Case Design

Modeling external events in the three-level analysis of multiple-baseline across-participants designs: A simulation study

Mariola Moeyaert & Maaike Ugille & John M. Ferron & S. Natasha Beretvas & Wim Van den Noortgate

Published online: 12 December 2012 # Psychonomic Society, Inc. 2012

Abstract In this study, we focus on a three-level meta- analysis for combining data from studies using multiple- baseline across-participants designs. A complicating factor in such designs is that results might be biased if the depen- dent variable is affected by not explicitly modeled external events, such as the illness of a teacher, an exciting class activity, or the presence of a foreign observer. In multiple- baseline designs, external effects can become apparent if they simultaneously have an effect on the outcome score (s) of the participants within a study. This study presents a method for adjusting the three-level model to external events and evaluates the appropriateness of the modified model. Therefore, we use a simulation study, and we illus- trate the new approach with real data sets. The results indicate that ignoring an external event effect results in biased estimates of the treatment effects, especially when

there is only a small number of studies and measurement occasions involved. The mean squared error, as well as the standard error and coverage proportion of the effect esti- mates, is improved with the modified model. Moreover, the adjusted model results in less biased variance estimates. If there is no external event effect, we find no differences in results between the modified and unmodified models.

Keywords Multiple baseline across participants . Three-level meta-analysis . Effect sizes . External event effect

A multiple-baseline design (MBD) is one of the variants of single-subject experimental designs (SSEDs). SSED researchers observe and measure a participant or case re- peatedly over time. Observations are obtained during at least one baseline phase (when no intervention is present) and at least one treatment phase (when an intervention is present). By comparing scores from both kinds of phases, SSED researchers can assess whether the outcome scores on the dependent variable changed, for instance, in level or in slope when the treatment was present (Onghena & Edgington, 2005).

In an MBD, an AB phase design (with one baseline phase, A, and one treatment phase, B) is implemented simultaneously to different participants, behaviors, or settings (Barlow & Hersen, 1984; Ferron & Scott, 2005; Onghena, 2005; Onghena & Edgington, 2005). MBDs are popular among SSED researchers (Shadish & Sullivan, 2011) because the interven- tion is introduced sequentially over the participants (or settings and behaviors), which entails the advantage that the researchers can more easily disentangle effects of the intervention and effects of some external events, such as the illness of a teacher, an exciting class activity, the presence of a foreign observer, or

M. Moeyaert : M. Ugille University of Leuven, Leuven, Belgium

J. M. Ferron University of South Florida, Tampa, FL, USA

S. N. Beretvas University of Texas, Austin, TX, USA

W. Van den Noortgate University of Leuven, Leuven, Belgium

M. Moeyaert (*) Faculty of Psychology and Educational Sciences, University of Leuven, Andreas Vesaliusstraat 2, Box 3762, 3000 Leuven, Belgium e-mail: [email protected]

Behav Res (2013) 45:547–559 DOI 10.3758/s13428-012-0274-1

a teacher intern (Baer, Wolf, & Risley, 1968; Barlow & Hersen, 1984; Kinugasa, Cerin, & Hooper, 2004; Koehler & Levin, 2000). This is because, if an external event occurs at certain points in time, the outcome scores for all participants in that study might be simultaneously influenced. Figure 1 gives a graphical presentation of possible consequences for the occur- rence of an external event in a multiple-baseline across- participants design. In Fig. 1a, the external event has a constant effect on the dependent variable on subsequent measurements—for instance, the teacher is ill during subse- quent days, or there is a foreign observer during some measure- ment occasions. Figure 1b illustrates a gradually fading away external event effect. For instance, the influence of a teacher intern on the behavior of the students may be reduced over time.

Van den Noortgate and Onghena (2003) proposed the use of multilevel models to synthesize data from multiple SSED studies, allowing investigation of the generalizability of the results and exploration of potential moderating effects. In previous research evaluating this multilevel meta-analysis of MBD data (Ferron, Bell, Hess, Rendina-Gobioff, & Hibbard, 2009; Ferron, Farmer, & Owens, 2010; Moeyaert, Ugille, Ferron, Beretvas, & Van den Noorgate, 2012a, 2012b; Owens & Ferron, 2012), the data were typically simulated with a treatment effect and random noise only. Potential

confounding events that could have a simultaneous ef- fect on all participants within a study were not taken into account. In this study, we evaluate the performance of the basic three-level model when there are effects of external events, as well as that of an extension of the model that tries to account for potential event effects. In the following, we first present the basic model and a possible extension to account for external events. Next, we evaluate the performance of both models, by means of a simulation study and an analysis of real data.

Three-level meta-analysis

A meta-analysis combines the results of several studies addressing the same research question (Cooper, 2010; Glass, 1976). Study results are typically first converted to a com- mon standardized effect size before meta-analyzing them. The effect sizes may be reported in the primary studies or can be calculated afterward, using reported summary and/or test statistics.

One possible way to calculate effect sizes when SSEDs are used is to analyze the data using regression models and to use the regression coefficients as effect sizes. A

Fig. 1 Graphical display of a constant external event effect (a) and a gradually fading away external event effect (b) affecting the score on four subsequent moments (day 17, day 19, day 21, and day 23) for a

multiple-baseline design across 3 participants, with the treatment start- ing on day 6, day 16, and day 24, respectively

548 Behav Res (2013) 45:547–559

regression model of interest here is the one proposed by Center, Skiba, and Casey (1985–1986):

Yi ¼ b0 þ b1Ti þ b2Di þ b3T 0iDi þ ei: ð1Þ

The score of the dependent variable on measurement occasion i (Yi ) depends on a dummy coded variable (Di) indicating whether the measurement occasion i belongs to the baseline phase (Di 0 0) or the treatment phase (Di 0 1); a time-related variable Ti that equals 1 on the first measure- ment occasion of the baseline phase; and an interaction term between the centered time indicator and the dummy vari- able, T 0iDi, where T

0 i is centered such that T

0 i equals 0 on the

first measurement occasion of the treatment phase. b0 indi- cates the expected baseline level b1 is the linear trend during the baseline, b2 refers to the immediate treatment effect, and b3 refers to the effect of the treatment on the time trend.

Van den Noortgate and Onghena (2003) proposed using the ordinary least squares estimates for b2 and b3 from Eq. 1 as effect sizes in the three-level meta-analysis. At the first level, the estimated effect sizes of the immediate treatment effect, b2jk, and the treatment effect on the time trend, b3jk, for participant j from study k are equal to the unknown population effect sizes, β2jk and β3jk, respectively, plus ran- dom deviation s,r2jk and r3jk, that are assumed to be normal- ly distributed with a mean of zero:

b2jk ¼ b2jk þ r2jk with r2jk � N 0; σ2r2jk � �

b3jk ¼ b3jk þ r3jk with r2jk � N 0; σ2r3jk � �

: ð2Þ

The sampling variances of the observed effects, σ2r2jk and

σ2r3jk ; are the squared standard errors that are typically

reported by default when a regression analysis is performed. These variances depend to a large extent on the number of observations and the variance of these observations and, therefore, can be participant and study specific. At the second level, the population effect sizes b2jk and b3jk from Eq. 2 can be modeled as varying over participants around the study-specific mean effect, θ20k and θ30k (Eq. 3):

b2jk ¼ θ20k þ u2jk with u2jk � N 0; σ2u2jk � �

b3jk ¼ θ30k þ u3jk with u3jk � N 0; σ2u3jk � �

: ð3Þ

The population effects for studies can vary between studies (third level, Eq. 4):

θ20k ¼ g200 þ v20k with v20k � N 0; σ2v20k � �

θ30k ¼ g300 þ v30k with v30k � N 0; σ2v30k � �

: ð4Þ

The model parameters that we are typically interested in when using a multilevel model are the fixed effects regression coefficients (i.e., g200 , referring to the average immediate treatment effect over participants and studies, and g300, refer- ring to the average treatment effect on the linear trend over participants and studies in Eq. 4) and the variances (i.e., σ2v20k ,

referring to the between-study variance for the estimated im- mediate treatment effect; σ2v30k , indicating the between-study

variance for the estimated treatment effect on the time trend; σ2u2jk , the between-case variance for the estimated immediate

treatment effect; and σ2u3jk , referring to the between-case vari-

ance of the estimated treatment effect on the time trend).

Correcting effect sizes for external events

External events in a multiple-baseline across-participants de- sign can have an effect on the outcome score(s) of all partici- pants within a study. These external event effects are common in SSEDs, because practitioners often implement these designs in their everyday setting (for example, in the home, school, etc.), where they cannot control for outside experimental factors (Christ, 2007; Kratochwill et al., 2010; Shadish, Cook, & Campbell, 2002). If we do not model these external events, the results might be biased. For instance, suppose that a re- searcher is interested in a change in challenging behavior and staggers the beginning of the treatment across 3 participants. The 3 participants receive the treatment at day 6, day 16, and day 24, respectively (see Fig. 1) and are observed every 2 days. On days 17, 19, 21, and 23, the teacher is ill, and as a conse- quence, a substitute teacher takes his or her place, and the participants exhibit more challenging behavior. In this situation, the estimated treatment effect for participants 1 and 2 will be smaller, and the estimated treatment effect for participant 3 will be larger, and therefore differences between participants in the treatment effects are also likely to be overestimated, unless we correct the effect sizes for possible external events.

A possible way to calculate effect sizes corrected for an external event in an SSED is by estimating effect sizes for participants per study, by performing a regression analysis with a model including possible event effects, and by as- suming that external events simultaneously affect all partic- ipants in a study. Thereafter, the corrected effect sizes can be combined over studies in the three-level meta-analysis.

For the first step, we propose to use an extension of the Center et al. (1985–1986) model, including dummy varia- bles for measurement occasions:

Yij ¼ b0j þ b1jTij þ b2jDij þ b3jDijT 0 ij

þ XI�1 m¼2

b mþ2ð ÞMmi þ eij: ð5Þ

Behav Res (2013) 45:547–559 549

The score on the dependent variable Y on measurement occasion i (0 1, 2, . . . , I) from participant j (0 1, 2, . . . , J) is modeled as a linear function of the dummy-coded variable (Dij) indicating whether the measurement occasion i from participant j belongs to the baseline phase (Dij 0 0) or the treatment phase (Dij 0 1); a time-related variable Tij, which equals 1 at the start of the baseline phase; an interaction term between the dummy variable indicating the phase and the time indicator centered around its value at the start of the

treatment phase, DijT 0 ij; and finally, dummy-coded variables

indicating the moment (Mmi 0 1 if m 0 i, zero otherwise). By including the effects of individual moments, coefficients β2j and β3j can be interpreted as the treatment effects, corrected for possible external events.

We do not include a dummy variable for one mea- surement moment in the baseline phase and one mea- surement moment in the treatment phase. This is to ensure that the model is identified; if we included these parameters as well, an increase in the effects for each moment in the baseline phase could be compensated for by a decrease of the intercept, illustrating that without constraining these parameters, there would be an infinite number of equivalent solutions. For our study, we select the first and last moments as the times at which to set the moment effects to zero, but different moments can be chosen if we suspect a moment effect during one of these times.

While the baseline level and slope (β0j and β1j) and both treatment effects (β2j and β3j) are participant specific, the moment effects are assumed to be the same for all partic- ipants from the same study and, therefore, have to be esti- mated for each study, using all data from that study. To this end, we propose to extend Eq. 5 by including a set of dummy participant indicators. For 2 participants, using dummy participant indicators P1 and P2, respectively, this results in Eq. 6:

Yij ¼ b01P1j þ b02P2j þ b11Ti1P1j þ b12Ti2P2j þ b21Di1P1j þ b22Di2P2j þ b31Di1T

0 i1P1j

þ b32Di2T 0 i2P2j þ

XI�1 m¼2

b mþ2ð ÞMmi þ eij: ð6Þ

After using Eq. 6 for each study to estimate the corrected effect sizes (β2j and β3j) for each participant, we can use the three-level meta-analysis (see Eqs. 2–4) to combine the corrected effect size estimates from multiple participants. In principle, we could also use a two-level model per study to estimate the participant- specific effects, but given the typically very small num- ber of participants per study, using a multilevel model might not be recommended.

A simulation study

Simulating three-level data

To evaluate the performance of the basic model and its extension, we performed a simulation study. We simulated raw data using a three-level model. At level 1, we used the following model:

Yijk ¼ b0jk þ b1jkTijk þ b2jkDijk þ b3jkTijkDijk þ eijk with eijk � N 0; σ2e

� � ;

ð7Þ

with measurement occasions nested within participants, which form the units at level two:

b0jk ¼ θ00k þ u0jk b1jk ¼ θ10k þ u1jk b2jk ¼ θ20k þ u2jk b3jk ¼ θ30k þ u3jk

8>>< >>:

with

u0jk u1jk u2jk u3jk

2 664

3 775 � N 0; @uð Þ: ð8Þ

The participants are, in turn, clustered within studies at the third level:

θ00k ¼ g000 þ v00k θ10k ¼ g100 þ v10k θ20k ¼ g200 þ v20k θ30k ¼ g300 þ v30k

8>>< >>:

with

v00k v10k v20k v30k

2 664

3 775 � N 0; @vð Þ: ð9Þ

Varying parameters

On the basis of a thorough overview of 809 SSED studies, Shadish and Sullivan (2011) enumerated some parameters that characterize SSEDs. On the basis of their results and our reanalyses of meta-analyses of SSEDs (Alen, Grietens, & Van den Noortgate, 2009; Denis, Van den Noortgate, & Maes, 2011; Ferron et al., 2010; Kokina & Kern, 2010; Shadish & Sullivan, 2011; Shogren, Faggella-Luby, Bae, & Wehmeyer, 2004; Wang, Cui, & Parrila, 2011), we decided to vary the following parameters that can have a significant influence on the quality of model estimation:

& g200 , represents the immediate treatment effect on the outcome and had values 0 (no effect) or 2.

& The treatment effect on the time trend, defined by g300, was varied to have values 0 (no effect) or 0.2.

& The regression coefficients of the baseline, g000 and g100 , did not vary and were set at 0, because the interest is in the overall treatment effects (e.g., the immediate treatment effect and the treatment effect on the time trend).

& The number of simulated participants, J, equaled 4 or 7. & The number of measurements within a participant, I, was

15 or 30. We chose to keep I constant for all participants within the same study.

550 Behav Res (2013) 45:547–559

& The number of studies, K, was 10 or 30. & The between-case covariance matrix: Covariances between

pairs of regression coefficients were set to zero. Therefore,u

is a diagonal matrix: P

u ¼ diag σ2u0; σ2u1; σ2u2; σ2u3 � �

¼ diag 2; 0:2; 2; 0:2ð Þ or Pu ¼ diag σ2u0; σ2u1; σ2u2; σ2u3

� � ¼

diag 0:5; 0:05; 0:5; 0:05ð Þ. & The between-study covariance matrix: Covariances between

pairs of regression coefficients were set to zero. Therefore, v

is a diagonal matrix: P

v ¼ diag σ2v0; σ2v1; σ2v; σ2v3 � �

¼ diag 2; 0:2; 2; 0:2ð Þ or Pv ¼ diag σ2v0; σ2v1; σ2v2; σ2v3

� � ¼

diag 0:5; 0:05; 0:5; 0:05ð Þ. & The moment of introducing a treatment effect was stag-

gered across participants within a study (see Table 1), depending on the number of measurements.

In a first scenario, a constant external event was added to influence four subsequent scores of all the participants with- in a study (as in Fig. 1a). The moment was randomly generated from a uniform distribution for each study sepa- rately. Because we did not include a moment effect for the first and the last moments to make the model identified, the external event effect did not occur on these moments. The external event effect was 0 or 2, representing a null and a large external event effect, respectively.

In a second scenario, the effect of the external event effect was added, which fades away gradually (see Fig. 1b) for all the participants within a study. The effect across four time points was 3.5, 2.5, 1.5, 0.5, and 0, respectively, so that, on average, the overall effect was the same as in the first scenario. The start of the event effect was generated completely at random from a uniform distribution for each study separately, so that the external event effect did not occur on the first or last measurement occasion. Data were generated using SAS.

Analysis

We had a total of 29 (0 512) experimental conditions. We simulated 400 replications of each condition, resulting in 204,800 data sets to analyze. We analyzed the data twice and compared the results. First, we combined the uncorrected effect sizes in the three-level meta-analysis. Next, we analyzed the

three-level data by estimating the corrected effect sizes, β2j and β3j, using the regression analysis per study (see Eq. 5) before combining them in the three-level meta-analysis (see Eqs. 2–4).

In the two approaches, we used the SAS proc MIXED (Littell, Milliken, Stroup, Wolfinger, & Schabenberger, 2006) procedure to estimate the participant-specific effect sizes, β2jk and β3jk. In the first approach, the effect sizes were uncorrected for the external event effect, whereas the effect sizes in the second approach were corrected.

SAS proc MIXED was also used for the three-level meta- analysis. The Satterthwaite approach to estimating the degrees of freedom method was applied because this meth- od provides more accurate confidence intervals for estimates of the average treatment effect for two-level analyses of multiple-baseline data (Ferron et al., 2009).

In order to evaluate the appropriateness of both models, uncorrected and corrected for external events, we calculated the deviations of the estimated immediate treatment effect, bg200, from its population value, g200, and the deviations of the estimated treatment effect on the time trend, bg300 , from its population value,g300. The mean deviation gives us an idea of the bias. Next, we calculated the mean squared deviation (the mean squared error [MSE]), which gives information about the variance of both estimated treatment effects (bg200 andbg300) around the corresponding population effect (g200 and g300). Furthermore, we discuss the standard error and the 95 % confidence interval coverage proportion (CP) of the estimated immediate treatment effect and the treatment effect on the time trend. We also evaluate the bias of the point estimates of the between-study and between-case variances.

We used ANOVAs to evaluate whether there were signifi- cant effects (α 0 .01) of each model type (e.g., model using effect sizes corrected vs. uncorrected for external event effects) and of the simulation design parameters (g200, g300, K, I, J, σ

2 u2 ,

σ2v2) on the bias, the MSE, the standard error, and the CP.

Results of the simulation study

We present the results in two sections. In the first section, we discuss the constant external effect over four subsequent measurement occasions. The second section considers the case where the external effect gradually fades away over

Table 1 Time of introducing the treatment

Start of intervention

I Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 Participant 6 Participant 7

15 5 6 7 8 9 11 13

30 5 8 11 14 17 20 23

Behav Res (2013) 45:547–559 551

four subsequent measurements. Each section presents the results of the three-level analysis of uncorrected and cor- rected effect sizes.

When there is no external event effect, the results of the three-level meta-analysis (i.e., bias in the fixed effects, MSE of the fixed effects, estimated standard errors of the fixed effects, CP for the fixed effects, and bias in the variance components) were found to be independent of the model type (corrected or not corrected for external events).

We found no significant bias for bg200 and bg300 when using the corrected or uncorrected model. Therefore, we discuss the results of the analyses of the data including only external event effects conditions.

Constant external event over four subsequent measurement occasions

Overall treatment effect

Bias When we estimate g200 and the effect sizes are uncor- rected, the estimated treatment effect is, on average, signifi- cantly larger than the population value (g200 0 0 or 2). Over all conditions, the bias equals 0.032, t(51199) 0 17.32, p < .0001, whereas there is no significant bias for the corrected effect sizes (−0.0015), t(51199) 0 −0.96, p 0 .34. Table 2 presents the bias estimates for bg200, when g200 0 2 and g300 0 0.2.

Similar results are obtained for bg300. The bias is significantly negative for the uncorrected effect sizes and equals −0.20, t(51199) 0 −255.27, p < .0001, whereas the bias is not signif- icant for the corrected effect sizes, t(51199) 0 −0.00020, p 0 .79. Moreover, an analysis of variance on the deviations reveals a significant difference between the two different models, for both bg200 and bg300 [F(1, 102398) 0 192.06, p < .0001 for bg200, and F (1, 102398) 0 33,695.1, p < .0001 for bg300�. The differences are largest when there is a small number of measurement occasions (I 0 15) and studies (K 0 10). In the following condition, the

largest difference was identified: g200 0 2, g300 0 0, K 0 10, I 0 15, J 0 4, σ2u2 0 0.5, and σ

2 v2 0 2 (with a difference of 0.23).

MSE Similar to the bias, the MSE of the estimated treatment effect depends significantly on the model type; using an analysis of variance on the squared deviations, F(1, 102398) 0 882.77, p < .0001 for bg200 and F(1, 102398) 0 7,076.91, p < .0001 for bg300 . When using the corrected model, the MSE for bg200 and bg300 equals 0.12 and 0.028, respectively, whereas it is 0.18 and 0.070, respectively, for the uncorrected effect sizes. Differences between both mod- els are larger if the number of observations and the number of studies are small (see Table 3 for bg200; similar results are obtained for bg300 ). So especially in these conditions, the modified model is recommended.

Estimates of the standard errors In order to evaluate infer- ences regarding the treatment effects, we constructed confidence intervals around the estimated treatment effects, bg200 and bg300 Therefore, we needed to estimate the standard errors of the estimated treatment effects. Because we obtained 400 estimates of the effects in each condition, the standard deviations of the effect estimates can be regarded as a relatively good estimate of the standard deviation of the sampling distribution and can, therefore, be used as a criterion to evaluate the standard error. We looked at the relative standard error biases, which are the differences between the median standard error estimates and the standard deviation of the estimates of the effect divided by the standard deviation of the estimates ofbg200 andbg300. The relative differences are negative for bg200, which means that the median standard error estimates are smaller than expected. For bg300 , these differences are positive, referring to median standard error estimates larger than expected. The relative standard error biases for both bg200 and bg300 are, on average, larger across the conditions for the uncorrected effect sizes, in comparison with

Table 2 The bias of bg200 in the g200 0 2, and g300 0 0.2 conditions for the constant external event effect over four subsequent measurement occasions Corrected Unorrected

I 0 15 I 0 30 I 0 15 I 0 30

K J σ2u2 σ 2 v2 0 0.5 σ2v2 0 2 σ

2 v2 0 0.5 σ2v2 0 2 σ

2 v2 0 0.5 σ2v2 0 2 σ

2 v2 0 0.5 σ2v2 0 2

10 4 0.5 −0.003 0.007 0.025 −0.036 0.213 0.208 −0.027 0.027

2 0.015 0.002 −0.017 0.014 0.129 0.196 0.012 0.035

7 0.5 −0.026 −0.057 0.024 0.005 −0.093 −0.058 −0.019 −0.074

2 −0.028 −0.015 −0.011 −0.003 −0.099 −0.060 −0.016 −0.026

30 4 0.5 0.009 0.028 0.004 −0.005 0.219 0.185 −0.008 0.013

2 0.018 0.021 0.004 −0.011 0.210 0.222 0.008 0.035

7 0.5 0.023 0.005 0.002 −0.009 −0.075 −0.105 −0.004 −0.016

2 0.001 0.026 −0.006 −0.012 −0.077 −0.088 −0.003 0.006

“Corrected” and “uncorrected” refer, respectively, to corrected effect size and uncorrected effect size for external event effects

552 Behav Res (2013) 45:547–559

the corrected effect sizes. For bg200, the average relative standard error biases equal −1.8 % and −2.0 % for the corrected and uncorrected models, respectively. The average relative standard error biases difference forbg300 for the uncorrected model is 2 %, whereas it is substantial (more than 10 %; Hoogland & Boomsma; 1998) for the uncorrected model (25.7 %). So the difference between the model types becomes more apparent when g300 is estimated, F(1, 254) 0 38.9, p < .0001. The conditions with the largest relative standard error bias when the uncorrected model for bg300 was used tended to coincidence with the conditions where 30 studies, an immediate treatment effect of 2, and a treatment effect on the time trend of 0.2 were involved, with the bias mounting to 107 % in the condition where g200 0 2, g300 0 0.2, K 0 30, J 0 7, I 0 30, σ

2 v2 0 0.5, and

σ2u2 0 0.5.

Coverage proportion We estimated the CP of the 95 % confi- dence intervals, which allowed us to evaluate the interval esti- mates of bg200 and bg300. The confidence intervals were estimated by using the standard errors and the Satterthwaite estimated degrees of freedom. The CP of these confidence intervals was estimated for each of the combinations. A positive significant difference between the corrected model and the uncorrected model in the CP is found for bg200, F(1, 254) 0 27.56, p < .0001 (see Table 4). Also, for bg300, the mean CP depends significantly on the model type, F(1, 254) 0 20.96, p < .0001 (see Table 4). The conditions with a CP less than .93 all have 15 measurements in common and occur when the effect sizes are uncorrected, for both bg200 and bg300. Moreover, for bg300, the CP is not only too small when I 0 15 and K 0 30, but also too

Table 3 The MSE of bg200 in the g200 0 2, and g300 0 0.2 conditions for the constant external event effect over four subsequent measurement occasions Corrected Unorrected

I 0 15 I 0 30 I 0 15 I 0 30

K J σ2u2 σ 2 v2 0 0.5 σ2v2 0 2 σ

2 v2 0 0.5 σ2v2 0 2 σ

2 v2 0 0.5 σ2v2 0 2 σ

2 v2 0 0.5 σ2v2 0 2

10 4 0.5 0.17 0.28 0.11 0.26 0.32 0.43 0.14 0.25

2 0.20 0.32 0.14 0.28 0.31 0.49 0.16 0.36

7 0.5 0.09 0.24 0.07 0.23 0.18 0.31 0.09 0.22

2 0.11 0.26 0.09 0.24 0.20 0.31 0.09 0.28

30 4 0.5 0.06 0.10 0.04 0.09 0.14 0.19 0.04 0.10

2 0.06 0.11 0.04 0.09 0.15 0.20 0.05 0.12

7 0.5 0.03 0.07 0.03 0.08 0.06 0.10 0.03 0.09

2 0.04 0.08 0.03 0.08 0.07 0.10 0.04 0.08

“Corrected” and “uncorrected” refer, respectively, to corrected effect size and uncorrected effect size for external event effects

Table 4 The coverage proportion of bg200 and bg300 in the g200 0 2, g300 0 0.2, and σ2u2 0 2 conditions for the constant external event effect over four subsequent measurement occasions

bg200 bg300 Corrected Uncorrected Corrected Uncorrected

K J σ2v2 I 0 15 I 0 30 I 0 15 I 0 30 I 0 15 I 0 30 I 0 15 I 0 30

10 4 0.5 .96 .96 .96 .96 .94 1.00 .97 1.00

2 .95 .95 .92 .95 .96 .98 .93 1.00

7 0.5 .96 .95 .94 .97 .99 1.00 .97 1.00

2 .97 .96 .95 .95 .96 .97 .84 .99

30 4 0.5 .97 .96 .89 .97 .97 1.00 .90 1.00

2 .97 .96 .91 .94 .96 .98 .49 1.00

7 0.5 .94 .94 .92 .96 .98 1.00 .93 1.00

2 .96 .95 .96 .96 .96 .97 .26 .96

Values smaller than .93 and larger than .97 appear in bold. “Corrected” and “uncorrected” refer, respectively, to corrected effect size and uncorrected effect size for external event effects

Behav Res (2013) 45:547–559 553

large when I 0 30 (values for the CP range from .99 to 1.00). When the effect sizes are uncorrected, the CP is well estimated when I 0 30 for bg200 and I 0 15 and K 0 10 for bg300. The difference in CP for bg200 is largest when there is only a small number of measurements (I 0 15) and a large number of studies (I 0 30).

Variance components

In the three-level analyses, the between-study and between- case variances were estimated for both the immediate treat- ment effect and the treatment effect on the trend. Because variance estimates are expected to be positively skewed, due to truncation of negative estimates to zero, we calculated the median (relative) deviation of the estimates from the popu- lation value, rather than the mean (relative) deviation, to evaluate the (relative) bias in the estimates. We discuss only the between-case variance and the between-study variance of the immediate treatment effect (σ2u2 and σ

2 v2 ), because

similar results are obtained for the treatment effect on the time trend (σ2u3 and σ

2 v3 ). The bias of the estimated between-

study variance and the estimated between-case variance of the immediate effect is larger when there are only 10 studies and 15 measurement occasions involved. The conditions with the largest relative bias all had 15 measurements, 4 participants, and a small between-study variance (σ2v2 0 0.5)

in common. If the effect sizes are corrected and we estimate the between-study variance of the immediate treatment ef- fect, we find relative parameter bias values across conditions ranging from 17 % to 55 %, while the relative bias goes up to a value of 313 % when the effect sizes are uncorrected. Similar results are found for bσ2u2, where the relative bias in a condition is maximum 119 % for the corrected effect sizes and 326 % for the uncorrected effect sizes (see Table 5). Overall, the adjusted model results in less biased variance estimates.

External event fades away gradually over four subsequent measurement occasions

Overall treatment effect

Bias The bias of bg200 for uncorrected and corrected effect sizes is, respectively, −0.0073, t(51199) 0 −418, p < .001, and 0.00057, t(51199) 0 38, p 0 .74. This means that there is a significant negative bias for the uncorrected effect sizes, whereas this is not the case for the corrected effect sizes, and the models differ significantly, F(1, 102398) 0 0.009, p 0 .77. The bias for bg300, depends largely on the model type, F (1, 102398) 0 30,476.1, p < .0001. The bias for the uncor- rected effect sizes is significant (−0.19), t(51199) 0 −246.23, p < .0001, whereas this is not the case for the corrected T

a b le 5

M ed ia n o f re la ti v e d ev ia ti o n o f th e v ar ia n ce

es ti m at es

o f g 2 0 0 in

th e g 2 0 0 0 2 , an d g 3 0 0 0 0 .2

co n d it io n s fo r th e co n st an t ex te rn al ev en t ef fe ct o v er

fo u r su b se q u en t m ea su re m en t o cc as io n s

bσ2 v 2 bσ2 u 2

C o rr ec te d

U n co rr ec te d

C o rr ec te d

U n co rr ec te d

K 0 1 0

K 0 3 0

K 0 1 0

K 0 3 0

K 0 1 0

K 0 3 0

K 0 1 0

K 0 3 0

I J

σ 2 v 2

σ 2 u 2 0 0 .5

σ 2 u 2 0 2

σ 2 u 2 0 0 .5

σ 2 u 2 0 2

σ 2 u 2 0 0 .5

σ 2 u 2 0 2

σ 2 u 2 0 0 .5

σ 2 u 2 0 2

σ 2 u 2 0 0 .5

σ 2 u 2 0 2

σ 2 u 2 0 0 .5

σ 2 u 2 0 2

σ 2 u 2 0 0 .5

σ 2 u 2 0 2

σ 2 u 2 0 0 .5

σ 2 u 2 0 2

1 5

4 0 .5

0 .4 7

0 .6 1

0 .5 5

0 .5 8

2 .7

3 .0 9

3 .1 1

2 .9 1

1 .2 4

0 .2 4

1 .1 9

0 .2 7

3 .0 1

0 .7 2

3 .1 3

0 .7 9

2 0 .0 5

− 0 .0 3

0 .1 3

0 .1 2

0 .7 1

0 .5 7

0 .7 4

0 .6 5

1 .1 6

0 .2 0

1 .1 9

0 .3 1

3 .1 1

0 .7 5

3 .2 6

0 .8 0

7 0 .5

− 0 .0 7

− 0 .1 1

0 .0 9

0 .0 6

0 .9 9

0 .9 9

1 .1 0

0 .9 8

1 .1 5

0 .2 5

1 .2 0

0 .2 9

2 .8 7

0 .7 1

2 .8 5

0 .7 0

2 − 0 .0 2

− 0 .0 8

0 .0 1

− 0 .0 2

0 .1 5

0 .1 6

0 .2 9

0 .2 8

1 .1 8

0 .2 7

1 .2 0

0 .2 8

2 .7 9

0 .6 3

2 .9 3

0 .7 3

3 0

4 0 .5

− 0 .1 2

− 0 .0 9

− 0 .0 2

0 .0 5

0 .3 3

0 .2 6

0 .4 5

0 .4 8

0 .8 3

0 .2 1

0 .8 0

0 .1 7

0 .6 1

0 .1 3

0 .8 1

0 .2 0

2 − 0 .0 8

− 0 .0 6

− 0 .0 1

− 0 .0 4

0 .1 4

0 .1 0

0 .0 8

0 .1 0

0 .8 1

0 .1 6

0 .8 5

0 .2 1

0 .5 9

0 .1 0

0 .8 0

0 .2 0

7 0 .5

− 0 .1 7

− 0 .1 2

0 .0 2

− 0 .0 3

− 0 .0 8

− 0 .0 7

0 .0 4

0 .0 4

− 0 .1 2

− 0 .0 6

− 0 .1 0

− 0 .0 3

0 .9 4

0 .1 9

0 .8 9

0 .2 2

2 − 0 .1 1

− 0 .1 2

− 0 .0 1

− 0 .0 4

− 0 .0 7

− 0 .0 2

0 .0 0

0 .0 0

− 0 .1 1

− 0 .0 5

− 0 .1 2

− 0 .0 1

0 .9 1

0 .2 3

0 .9 2

0 .2 2

“C o rr ec te d ” an d “u n co rr ec te d ” re fe r, re sp ec ti v el y, to

co rr ec te d ef fe ct

si ze

an d u n co rr ec te d ef fe ct

si ze

fo r ex te rn al

ev en t ef fe ct s

554 Behav Res (2013) 45:547–559

effect sizes (0.000179), t(51199) 0 0.24, p 0 .81. For both bg200 and bg300, the difference is largest when there are a small number of measurements (I 0 15) involved.

MSE For both estimated treatment effects, the MSEs are larger for the uncorrected effect sizes, in comparison with the corrected effect sizes (see Table 6). For both bg200 and bg300, the model type has a significant influence on the MSE, F(1, 102398) 0 724.69, p ≤ .0001 forbg200, and for bg300, F(1, 102398) 0 5,431.15, p < .0001. For both estimated treatment effects, the MSE is large when the studies are heterogeneous (σ2v2 0 2) and a small number of measure-

ment occasions (I 0 15) and studies (K 0 10) are used. The difference between the models is largest when a small number of measurements is used.

Estimates of the standard errors The difference between the average relative bias in the standard errors of the uncorrect- ed effect sizes equals 0.02 for both uncorrected and cor- rected effect sizes when g200 is estimated.

Similar to the constant external event effect results, the difference between the average relative bias in the standard errors of the uncorrected effect sizes (M 0 39.3) and corrected effect sizes (0.06) for bg300 is larger and statistically significant, F(1, 254) 0 129.66, p 0 .0001 (see Table 7). The difference in results due to the model type is more obvious if there are a small number of studies involved (K 0 10).

Coverage proportion Similar to the CP for the constant external event effect, the mean CP for the uncorrected and corrected effect sizes for the estimate of the immediate treatment effect differ significantly at the 5 % significance level for both bg200, F(1, 254) 0 3.92, p 0 .05, and bg300, F(1,

254) 0 3.25, p 0 .007. The CP with values smaller than .93 all have 15 measurement occasions, have a large between- study variance (σ2v3 0 2.0), and occur when the effect sizes

are uncorrected (for both bg200 and bg300 ). Similar to the constant external event effect, the CP is overestimated for bg300 and when the effect sizes are uncorrected in the condi- tion where 30 measurement occasions are included. In the condition where I 0 15 and σ2v3 0 2.0, the difference between

corrected and uncorrected effect sizes is largest.

Variance components

The results are similar to the results of the constant external event effect, and results are less biased using the adjusted

Table 6 MSE of bg200 and bg300 in the g200 0 2, g300 0 0.2, and σ2u2 0 0.5 conditions for the external event effect fading away gradually over four subsequent measurement occasions

bg200 bg300 Corrected Uncorrected Corrected Uncorrected

K J σ2v2 I 0 15 I 0 30 I 0 15 I 0 30 I 0 15 I 0 30 I 0 15 I 0 30

10 4 0.5 0.14 0.11 0.34 0.12 0.09 0.01 0.12 0.01

2 0.31 0.24 0.47 0.27 0.13 0.03 0.14 0.03

7 0.5 0.09 0.06 0.18 0.09 0.01 0.01 0.10 0.01

2 0.22 0.23 0.32 0.22 0.03 0.02 0.12 0.02

30 4 0.5 0.05 0.03 0.11 0.04 0.04 0.004 0.11 0.01

2 0.11 0.09 0.17 0.09 0.04 0.01 0.12 0.01

7 0.5 0.03 0.02 0.06 0.02 0.004 0.002 0.09 0.01

2 0.08 0.08 0.10 0.07 0.01 0.01 0.1 0.01

“Corrected” and “uncorrected” refer, respectively, to corrected effect size and uncorrected effect size for external event effects

Table 7 Difference between the median of the standard error estimates and the standard deviation of bg300 in the g200 0 2, g300 0 0.2, and σ2u3 0 0.05 conditions for the external event effect fading away gradually over four subsequent measurement occasions

Corrected Uncorrected

K J σ2v3 I 0 15 I 0 30 I 0 15 I 0 30

10 4 0.05 0.010 0.037 0.076 0.103

0.2 −0.031 −0.002 0.029 0.050

7 0.05 0.004 0.035 0.069 0.068

0.2 −0.001 −0.007 0.010 0.012

30 4 0.05 −0.018 0.022 0.039 0.061

0.2 −0.009 0.0003 0.022 0.024

7 0.05 0.002 0.021 0.038 0.040

0.2 −0.002 0.001 0.004 0.003

“Corrected” and “uncorrected” refer, respectively, to corrected effect size and uncorrected effect size for external event effects

Behav Res (2013) 45:547–559 555

model. We only discuss the estimated variances for the immediate treatment effect, because the results are similar for the estimated treatment effect on the trend. When we estimate the between-study variance and the effect sizes are uncorrected, the bias ranges from −0.002 to 3.41, while it ranges from 0.002 to 0.73 for the corrected effect sizes. So the estimated variances depend on the model type, F(1, 102398) 0 1,631, p < .0001. Similar results are obtained for the estimate of the between-case variance. The maximum bias for the corrected effect sizes is 1.60, while it is 3.21 for the uncorrected effect sizes, and these estimates depend on the model type, F(1, 102398) 0 5,628.62, p < .0001.

Empirical illustration

In this section, we give empirical illustrations of the com- parison of the modified three-level model in which external events are taken into account with the uncorrected model. Therefore, we used a part of the meta-analytic data set of Heyvaert, Saenen, Maes, and Onghena (2012) in which restraint interventions for challenging behavior among per- sons with intellectual disabilities was investigated. We give two empirical illustrations of the consequences of ignoring the external event effect in a multiple-baseline across- participants design. We illustrate first the consequences of ignoring external events in a single study, and next the consequences of ignoring external events in a three-level meta-analysis.

Ignoring external events in a single study

To illustrate the regression analysis of a multiple-baseline across-3-participants design, we use the study of Thompson, Iwata, Conners, and Roscoe (1999), which was included in the meta-analysis of Heyvaert et al. (2012). In their study, the effects of benign punishment on the self-injurious behavior of individuals who had been diagnosed with mental retardation was investigated. The 3 participants were measured repeatedly over time on 22 measurement occasions, and the intervention started on sessions 11, 13, and 20, respectively (see Fig. 2). From this figure, we might expect that there is an immediate reduction in challenging behavior when the treatment is intro- duced and that the effect of the treatment on the challenging behavior decreases over time (so there is a positive effect on the time trend during the treatment). We also see that the 3 participants’ scores on measurement occasions 4 and 10 are possibly influenced by an external event.

Results

If we ignore possible external events in the regression analysis before combining the effect sizes in the two-level meta-

analyses, the average immediate treatment effect over cases for that study equals −25.58, and the average treatment effect on the time trend over cases from that study equals −2.58. If we take the external event into account by correcting the effect sizes before combining them, the immediate treatment effect equals −23.23, and the treatment effect on the time trend is 1.24. This means that bg200 is 9.19 % smaller when the effect sizes are corrected, in comparison with the uncorrected effect sizes. Moreover bg300 is positive for the corrected effect sizes, whereas it is negative for the uncorrected, which means that the effect of the treatment over time decreases for the corrected effect sizes, whereas it increases for the uncorrected.

Ignoring external events in a three-level meta-analysis

The three-level analysis of SSED data includes summarizing the immediate treatment effect and the treatment effect on the time trend over participants and over studies.

We estimate the immediate treatment effect and the treat- ment effect on the time trend across seven studies. Again, we use the meta-analysis of Heyvaert et al. (2012) to ran- domly select multiple-baseline across-participants studies. We combined the multiple-baseline across-participants

Fig. 2 Graphical display of a multiple-baseline design across-3- participants designs using data from the study of Thompson, Iwata, Conners, and Roscoe (1999)

556 Behav Res (2013) 45:547–559

study of Lindberg, Iwata, and Kahng (1999), Chung and Cannella-Malone (2010), Zhou, Goff, and Iwata (2000), Thompson et al., (1999), Hanley, Iwata, Thompson, and Lindberg (2000), Rolider, Williams, Cummings, and Van Houten (1991), and Roscoe, Iwata, and Goh (1998). In all these studies, the same dependent variable was measured— namely, the reduction in self-injurious behavior. Again, we compare the three-level meta-analysis of uncorrected and corrected effect sizes.

Results

With the uncorrected effect sizes in the three-level meta- analysis, the overall immediate treatment effect equals −33.14, t(6.39) 0 −3.44, p 0 .012, and the overall treatment effect on the time trend equals −4.42, t(3.95) 0 −1.52, p 0 .19. When the effect sizes are corrected before estimating the effects over participants, the immediate treatment effect equals −21.07, t(6.88) 0 −1.13, p 0 .30, and the treatment effect on the time trend equals −0.43, t(1) 0 −0.28, p 0 .83. This means that the immediate treatment effect of the corrected effect sizes is 36.42 % smaller, as compared with the uncorrected effect size, and the treat- ment effect on the time trend for the corrected effect size during the treatment is 90.27 % smaller.

This is consistent with the results of the simulation study, where we found that the estimated treatment effects are biased when the effect sizes are uncorrected before combin- ing them in the three-level meta-analysis.

Discussion

External event effects are common in SSEDs because single-case researchers often implement these kinds of designs in everyday scenarios where they cannot control for outside factors (Christ, 2007; Kratochwill et al., 2010; Shadish et al., 2002). External events are not always antic- ipated by researchers, and thus, they may not be measured during the conduct of the study. Furthermore, the size of an event effect may be small, and researchers may be unaware of it even after the study has been completed. Whether researchers recognize an external event or not, the failure to account for the event in a meta-analysis can bias the estimate of the treatment effect. Thus, we searched for a method with which to model external events that could be applied even when the events had not been previously identified. Because we used a multiple-baseline across- participants design, there was a need to take into account the interdependence of the participants. Therefore, an exter- nal event that influenced the scores of 1 participant was assumed to influence the scores of the other participants in the same study.

We discussed two possible scenarios. In one scenario, the external event effect remains constant and influences the scores of all participants within a study on four subsequent moments. This occurs, for example, when a teacher is ill and a substitute teacher takes over the classroom or when a foreign observer is present on subsequent measurement occasions. In the second scenario, the external event’s effect would likely gradually fade away over four subsequent moments. For instance, the influence of a teacher intern on the behavior of students reduces over time. Moreover, the model adjusted for external event effects takes into account that measurement occasions closer in time are more related than measurement occasions further in time.

We evaluated this approach using a large simulation study and gave some empirical examples. If there is an external event effect of zero, both models (the one that corrects for moment effects and the one that does not) are appropriate. If the external event influences subse- quent scores for all the participants within a study, the three-level approach for uncorrected effect sizes is not recommended, because the estimates of both treatment effects (i.e., immediate effect on level and effect on time trend) are substantially biased. The MSE, standard error, and CP are better estimated when the modified model, which includes moment effects, is used. The difference between the corrected and uncorrected effect sizes is largest when there are a small number of studies and measurement occasions, so in this context, we advise using the adjusted model. Moreover, the adjusted model results in less biased variance estimates.

But, of course, we should be aware of some limita- tions. We assumed that all the participants within a study are influenced the same way by the external event effect. It is possible that different participants from the same study are at separate locations and, therefore, are not all influenced by the external event. Modeling event effects that are not common to all participants in a study is an important avenue for future research. We chose to keep the number of measurements within a study constant for all participants within the same study. Of course, it is possible that different participants of the same study have different series lengths. Furthermore, we cannot generalize these results to other conditions not involved in this simulation study, but we partially addressed this by simulating a large number of condi- tions and choosing realistic values for the parameters.

Another limitation is that we assumed linear trajecto- ries in the treatment phase, which might not be true in some real situations. To simplify the simulation model, we further did not account for a possible dependence between regression coefficients, which can be accounted for in a multilevel analysis by estimating the covariance at the various levels.

Behav Res (2013) 45:547–559 557

In addition, subjects in MBDs are repeatedly measured, and succeeding measurements may be more related to each other than measurements further away in time. We did not account for this possible autocorrelation and suggest that this as a useful extension to the present study.

Kazdin (2010) argued that there needs to be a minimum of three measurement occasions between the participants in an MBD in order to show an experimental effect. We did not take this into account in the condition where the number of measurement occasions was 15, because it was not possible to do this and provide each of 7 participants a unique baseline. We could alter the intervention schedule to intro- duce the treatment for some participants (e.g., randomly selected pairs) at the same moment. Examining this strategy specifically and alternative intervention schedules more generally would allow further research to extend results to a wider range of multiple-baseline applications.

It can be difficult to attribute simultaneously unusual outcome scores for all participants within a study to an external event effect. If there is no external event effect, we can still use the corrected model, because both the corrected and uncorrected effect sizes will be unbiased and, thus, there is no need to identify before the analyses whether an external event effect occurred or not. We advise single-case researchers to first use both models in the sen- sitivity analysis and then decide which model to use. If researchers are interested in the occurrence of external event effects, we recommend that they keep a log in order to identify potential outside factors that may influence the scores at certain measurement occasions and include dum- my indicator variables at least for these moments.

The extension of the three-level model for multiple- baseline across-participants designs to include modeling of potential external effects makes it even more appropriate and useful for the analysis of realistic SSED data sets. This study has indicated that the three-level model corrected for external event effects provides better results than does the uncorrected model for combining results from multiple- baseline across-participants data, especially if there is only a small number of observations (I 0 15) and a small number of studies (K 0 10) in the synthesis. As was found here, even when an external event effect is small, a failure to correct for it can lead to biased effect sizes. Thus, applied SSED researchers are encouraged to consider use of the three- level model that corrects for external event effects when synthesizing results of MBD data.

Author Note Mariola Moeyaert, Faculty of Psychology and Educa- tional Sciences, University of Leuven, Belgium; Maaike Ugille, Faculty of Psychology and Educational Sciences, University of Leuven, Belgium; John M. Ferron, Department of Educational Measurement and Research, University of South Florida. S. Natasha Beretvas, Department of Educa- tional Psychology, University of Texas; Wim Van den Noortgate, Faculty

of Psychology and Educational Sciences, ITEC-IBBT Kortrijk, University of Leuven, Belgium.

This research is funded by the Institute of Education Sciences, U.S. Department of Education, Grant R305D110024. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

For the simulations, we used the infrastructure of the Flemish Supercomputer Center, financed by the Department of Economy, Science and Innovation–Flemish Government and the Hercules Foundation.

References

Alen, E., Grietens, H., & Van den Noortgate, W. (2009). Meta-analysis of single-case studies: An illustration for the treatment of anxiety disorders. Unpublished manuscript.

Baer, D. M., Wolf, M. M., & Risley, T. R. (1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1, 91–97.

Barlow, D. H., & Hersen, M. (1984). Single-case experimental designs: Strategies for studying behavior change (2nd ed.). New York: Pergamon Press.

Center, B. A., Skiba, R. J., & Casey, A. (1985–1986). A methodology for the quantitative synthesis of intra-subject design research. Journal of Special Education, 19, 387–400.

Christ, T. J. (2007). Experimental control and threats to internal valid- ity of concurrent and nonconcurrent multiple-baseline designs. Psychology in the Schools, 44, 451–459.

Chung, Y.-C., & Cannella-Malone, H. I. (2010). The effects of pre- session manipulations on automatically maintained challenging behavior and task responding. Behavior Modification, 34, 479– 502. doi:10.1177/0145445510378380

Cooper, H. M. (2010). Research synthesis and meta-analysis: a step- by-step approach. London: Sage.

Denis, J., Van den Noortgate, W., & Maes, B. (2011). Self-injurious behavior in people with profound intellectual disabilities: A meta-analysis of single-case studies. Research in Developmental Disabilities, 32, 911–923.

Ferron, J. M., Bell, B. A., Hess, M. F., Rendina-Gobioff, G., & Hibbard, S. T. (2009). Making treatment effect inferences from multiple-baseline data: The utility of multilevel mod- eling approaches. Behavior Research Methods, 41, 372– 384.

Ferron, J. M., Farmer, J. L., & Owens, C. M. (2010). Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilvel-modeling approaches. Behavior Research Methods, 42, 930–943.

Ferron, J., & Scott, H. (2005). Multiple baseline designs. In B. Everitt & D. Howell (Eds.), Encyclopedia of Behavioral Statistics (Vol. 3, pp. 1306–1309). West Sussex, UK: Wiley & Sons Ltd.

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.

Hanley, G. P., Iwata, B. A., Thompson, R. H., & Lindberg, J. S. (2000). A component analysis of "stereotypy as reinforcement" for alternative behavior. Journal of Applied Behavior Analysis, 33, 285–297.

Heyvaert, M., Saenen, L., Maes, B., & Onghena, P. (2012) Systematic review of restraint interventions for challenging behaviour among persons with intellectual disabilities: Experiences and effective- ness. Manuscript submitted for publication.

Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling. An overview and a meta- analysis. Sociological Methods & Research, 26, 329–367.

558 Behav Res (2013) 45:547–559

Kazdin, A. E. (2010). Single-case Research Designs: Methods for Clinical and Applied Settings (2nd ed.). New York: Oxford University Press.

Kinugasa, T., Cerin, E., & Hooper, S. (2004). Single-Subject Research Designs and Data Analyses for Assessing Elite Athletes’ Condi- tioning. Sports Medicine, 34, 1035–1050.

Koehler, M. J., & Levin, J. R. (2000). RegRand: Statistical software for the multiple-baseline design. Behavior Research Methods, Instru- ments, & Computers, 32, 367–371.

Kokina, A., & Kern, L. (2010). Social story interventions for students with autism spectrum disorders: A meta-analysis. Journal of Autism and Developmental Disorders, 40, 812–826.

Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M. & Shadish, W. R. (2010). Single-case designs technical documentation. Retrieved from What Works Clearing- house website: http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf

Lindberg, J. S., Iwata, B. A., & Kahng, S. W. (1999). On the relation between object manipulation and stereotypic self-injurious behavior. Journal of Applied Behavior Analysis, 32, 51–62.

Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., & Schabenberger, O. (2006). SAS© system for mixed models (2nd ed.). Cary, NC: SAS Institute Inc.

Moeyaert, M., Ugille, M., Ferron, J., Beretvas, N., & Van den Noortgate, W. (2012a). Three-level analysis of single-case experimental data: Empirical validation. Journal of Experimental Education.

Moeyaert, M., Ugille, M., Ferron, J., Beretvas, N., & Van den Noortgate, W. (2012b) The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Manuscript submitted for publication.

Onghena, P. (2005). Single-case designs. In B. Everitt & D. Howell (Eds.), Encyclopedia of statistics in behavioral science (Vol. 4, pp. 1850–1854). Chichester: Wiley.

Onghena, P., & Edgington, E. S. (2005). Customization of pain treatments: Single-case design and analysis. The Clinical Journal of Pain, 21, 56–68.

Owens, C. M., & Ferron, J. M. (2012). Synthesizing single-case studies: A Monte Carlo examination of a three-level meta- analytic model. Behavior Research Methods.

Rolider, A., Williams, L., Cummings, A., & Van Houten, R. (1991). The use of a brief movement restriction procedure to eliminate severe inappropriate behavior. Journal of Behavior Therapy and Experimen- tal Psychiatry, 22(1), 23–30. doi:10.1016/0005-7916(91)90029-5

Roscoe, E. M., Iwata, B. A., & Goh, H.-L. (1998). A comparison of noncontingent reinforcement and sensory extinction as treatments for self-injurious behavior. Journal of Applied Behavior Analysis, 31, 635–646.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton-Mifflin.

Shadish, W. R., & Sullivan, K. J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 971–980.

Shogren, K. A., Faggella-Luby, M. N., Bae, S. J., & Wehmeyer, M. L. (2004). The effect of choice-making as an intervention for problem behavior: A meta-analysis. Journal of Positive Behavior Interventions, 4, 228–237.

Thompson, R. H., Iwata, B. A., Conners, J., & Roscoe, E. M. (1999). Effects of reinforcement for alternative behavior during punishment of self-injury. Journal of Applied Behavior Analysis, 32, 317–328.

Van den Noortgate, W., & Onghena, P. (2003). Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly, 18, 325–346.

Wang, S., Cui, Y., & Parrila, R. (2011). Examinging the effectiveness of peer –mediated and video-modeling social skills interventions for children with autism spectrum disorders: A meta-analysis in single-case research unsing HLM. Research in Autism Spectrum Disorder, 5, 562–569.

Zhou, L. M., Goff, G. A., & Iwata, B. A. (2000). Effects of increased response effort on self-injury and object manipulation as competing responses. Journal of Applied Behavior Analysis, 33(1), 29–40.

Behav Res (2013) 45:547–559 559