Introducing
Your new presentation assistant.
Refine, enhance, and tailor your content, source relevant images, and edit visuals quicker than ever before.
Trending searches
21
SIMULATION MODELING META-ANALYSIS of EFFICACY, DOSAGE, & SEQUENCING of
BRAVE START MODULES
SAMUEL PEER, PhD &
STEVE MAZZA, PhD
SINGLE SUBJECT DESIGNS (SSDs)
Participants serve as their own controls in comparisons of phases.
(Vannest, Peltier, & Haas, 2018)
*PS: No disclosures.
Topic 4
SSD ANALYSIS
ANALYSIS
Visual Analysis
- Reliability (Ninci, Vannest, Willson, & Zhang, 2015)
- Type I errors, especially with small to medium effects (Franklin, Gorman, Beasley, & Allison, 1996; Tincani et al., 2006; Todman & Dugard, 2001)
- Calls to supplement visual analyses with statistical analyses (Brossart, Parker, Olson, & Mahadevan, 2006; Brossart, Vannest, Davis & Patience, 2014; Franklin, Allison & Gorman, 1996; Manolov, 2017; WWC, 2017)
SSD EFFECT SIZES (ESs)
SSD ESs
- Standardized mean differences (SMD; Shadish et al., 2014)
- Non-overlap of all pairs (NAP; Parker & Vannest, 2009)
- Improvement rate difference (IRD; Parker, Vannest, & Brown, 2009)
- Percent of data exceeding phase A median (PEM-T; Wolery et al., 2010)
- Tau, Tau-U, and Tau-C (Parker et al., 2011; Tarlow, 2017)
- Bayesian (Rindskopf, 2013)
- Multilevel analysis (Baek et al., 2014)
- Randomization (Heyvaert & Onghena, 2014)
- Regression (Swaminathan et al., 2014)
- Generalized least squares (GLS; Swaminathan et al., 2014)
- Hierarchical linear modeling (HLM; Gage et al., 2012)
SSD META-ANALYSIS
META-SSDs
Relatively lacking in literature (Byiers et al., 2012; Jamshidi et al., 2017)
Publication biases (Denis et al., 2011; Gage et al., 2017; Goldman & Meghan, 2016; Gough et al., 2013; Talbott et al., 2017)
Insufficient replications (King, Lemmon, & Davidson, 2016)
Visual analysis even more problematic (Ninci et al., 2015; Smith, 2012)
Insufficient ES use (~50%; Meline & Wang, 2004; Peng et al., 2013)
Autoregression (Borckardt & Nash, 2014; Vannest et al., 2018)
AUTOREGRESSION (AR)
AUTOREGRESSION
- Data series whose values partially or wholly determine a subsequent value (Borckardt et al., 2008; Gottman & Ringland, 1981; Wolery et al., 2010)
- Ubiquitous: Ms = .20–.80 (Busk & Marascuilo, 1998; Sharpley & Alavosius, 1988)
- Creates serious inferential bias, regardless of AR’s significance (Byiers et al., 2012; Franklin, Allison, & Gorman, 1996; Jones et al., 1978; Matyas & Greenwood, 1990; Robey et al., 1999; Suen, 1987; Wolery et al., 2010)
AR & VISUAL ANALYSIS
VISUAL
Expert visual analysis unreliable and prone to Type I errors: 16–84% (Borckardt et al., 2004; 2008; DeProspero & Cohen, 1979; Furlong & Wampold, 1982; Jones et al., 1978; Matyas & Greenwood, 1990; Ottenbacher, 1993; Robey, Schultz, Crawford, & Sinner, 1999)
AR & STATS
STATISTICAL
Conventional parametric/nonparametric statistics (Byiers et al., 2012; Borckardt et al., 2008; Franklin, Allison, & Gorman, 1996; Robey et al., 1999)
Specialized multivariate software programs (Borckardt et al., 2008; Price & Jones, 1998)
- Examples:
- Autoregressive integrated moving average models (ARIMA)
- Hierarchical linear modeling (HLM)
- Interrupted time-series analysis correlational analysis (ITSACORR)
- Problems:
- Costly
- Oft-prohibitive requirement of data points: i.e., >30–50, while time-series studies typically have 10-20 data points (Borckardt et al., 2008; Center et al., 1985; 1986; Kratochwill et al., 2010; Sharpley, 1987)
- Partial out AR (Borkardt et al., 2008)
SIMULATION MODELING ANALYSIS (SMA)
- Bootstrapping technique similar to Monte Carlo method
- Designed to analyze data streams with serial dependence (ARs = 0–.9)
- Maximizes power (4–30 data points/phase)
- Minimizes Type I error rates
- Calculates commonly used statistics (r & p)
- Performs effect-of-phase and process-change
- Free
- Accessible: MacOSX, Windows 95+; SMA-Version 9.9.28 (Borckardt et al., 2008; ClinicalResearcher, 2008)
Topic 3
4
2
3
1
5
STEP 1
Calculate the correlation between DV values and IV vector phase (r-observed).
STEP 2
Calculate AR, with n = number of data points and k = lag.
STEP 3
Generate simulated data sequences derived randomly from a known null distribution of data sequences with phase n and AR parameters matching observed values, via the following formula:
yi= a +εi
where εi = pεi-1 + normal random error n(0, 1); p = programmed autocorrelation; a = programmed intercept change = 0; and i = 1 to phase n. This step is repeated s times (e.g., r1 to r5,000).
STEP 4
Calculate correlation between each null autocorrelated data sequence and the original phase vector (r-s; where s = 1 to the number of simulation data streams to be generated).
STEP 5
Compare the absolute value of each r-s with r-original. If r-s < r-original equals “miss”; whereas, r-s > or = r-original equals “hit”. Compute p-value using number of “hits”/s.
BRAVE START
EFFICACY
Topic 1
- Overall treatment effect
- Phase, symptom, and sequence effects
- Process change
TREATMENT EFFICACY
Anxiety: d = 1.30
p
d
Conduct: d = 1.39
p
d
Anxiety: r = .59
CDI
Conduct: r = .54
PDI vs BDI
Anxiety:
- CDI+BDI: r = .64
- CDI+PDI: r = .43
- t(5) = 1.51, p = .10, g = 1.18
PDI & BDI
Conduct:
- CDI+BDI: r = .38
- CDI+PDI: r = .67
- t(5) = 1.87, p = .06, g = 1.46
PDI & BDI
Anxiety:
- BDI+PDI: r = -.08
- PDI+BDI: r = -.15
PDI & BDI
Conduct:
- BDI+PDI: r = -.20
- PDI+BDI: r = .12
PROCESS CHANGE
F(2) = 6.44, p = .03, partial eta squared = .52
PROCESS
r
p
CONCLUSIONS
Results:
- Contextualize Brave START's efficacy
- Inform module sequencing and dosage decisions for comorbid anxiety and conduct problems
- Support the utility of SMA for SSD meta-analyses