Study Design and Data Analysis Resources

Even the most sophisticated research teams seek professional development on methodological issues as technical work in design and analysis are continually advancing. ARC is available to consult with REESE project teams about their methodological and statistical strategies and challenges. A number of online resources on study design, (including experimental and quasi-experimental design, power, sampling, synthesis & meta-analysis, and general topics) and data analysis (including Bayesian statistical methods, categorical data analysis, general test theory, hierarchical learning modeling, item response theory (IRT), longitudinal analysis, multilevel analysis, multivariate analysis, regression models, statistical software, and other topics) are also provided below. Please contact us to schedule a consultation with ARC investigators or to recommend additional resources you would find it helpful to have added to this portion of the REESE website.

STUDY DESIGN

EXPERIMENTAL AND QUASI-EXPERIMENTAL DESIGN
Cook, T. D., and Shadish, W. R. (1994). “Social experiments: Some developments over the past fifteen years.” Annual Review of Psychology, 45, 545-580.

Cook. T. D., and Wong, V. C. (In Press). “Empirical tests of the validity of the regression discontinuity design.” Annales d'Economie et de Statistique.

Dean, A. and Voss, D. (1999). Design and Analysis of Experiments. New York: Springer.

Designing Cluster-Randomized Trials. The National Center for Education Research (NCER) hosts an annual Summer Research Training Institute on Cluster-Randomized Trials to increase the national capacity of researchers to develop and conduct rigorous evaluations of the effectiveness of education interventions.

Grissmer, D. W., Subotnik, R. F., & Orland, M. (2008). A guide to the use of Randomized Controlled Trials (RCTs) in assessing intervention effects: The promise of multiple methods. Retrieved December 1, 2008, from http://www.apa.org/ed/schools/cpse/randomized-control-guide.pdf.

Kirk, R. E. (1995). Experimental design: Procedures for the behavioral sciences. Pacific Grove, CA: Brooks/Cole.

The Late Pretest Problem in Randomized Control Trials of Education Interventions. This NCEE Technical Methods report addresses pretest-posttest experimental designs that are often used in randomized control trials (RCTs) in the education field to improve the precision of the estimated treatment effects.

Manski, C. (1995). Identification problems in the social sciences. Cambridge, MA: Harvard University Press.

Precision Gains from Publically Available School Proficiency Measures Compared to Study-Collected Test Scores in Education Cluster-Randomized Trials. This NCEE Technical Methods Report (2010-4003) evaluates the use of pre-test scores versus school proficiency data for adjusting impact estimates in cluster randomized trials.

Replicating Experimental Impact Estimates Using a Regression Discontinuity Approach. This NCEE Technical Methods Paper compares the estimated impacts of an educational intervention using experimental and regression discontinuity (RD) study designs using data from two large-scale randomized controlled trials.

Shadish, W.R. (2002). “Revisiting field experimentation: Field notes for the future.” Psychological Methods, 7 (1), 3-18.

Shadish, W.R., Clark, M.H., & P. Steiner, M. (2008). “Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments.” Journal of the American Statistical Association, 103 (484), 1334-1356.

Shadish, W., Cook, T., and Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.

Shadish, W.R., and Cook, D. T. (2009). “The renaissance of field experimentation in evaluating interventions.” Annual Review of Psychology, 60, 607-629.

Shadish, W.R., Hu, X., Glaser, R.R., Kownacki, R., & Wong, S. (1998). “A method for exploring the effects of attrition in randomized experiments with dichotomous outcomes.” Psychological Methods, 3 (1), 3-22.

Shadish, W. R., Rindskopf, D. M. and Hedges, L. V. (2008). “The state of the science in the meta-analysis of single-case experimental designs.” Evidence-Based Communication Assessment and Intervention, 2 (3), 188-196.

Tabachnick, B.G., and Fidell, L.S. (2001). Computer-assisted Research Design and Analysis. Allyn and Bacon.

Using an Experimental Evaluation of Charter Schools to Test Whether Nonexperimental Comparison Group Methods Can Replicate Experimental Impact Estimates, an NCEE Technical Methods Paper that compares experimental estimates of the impacts of the offer of charter school enrollment with estimates from four different approaches to creating non-experimental comparison groups.

Whether and How to Use State Tests to Measure Student Achievement in a Multi-State Randomized Experiment: An Empirical Assessment Based on Four Recent Evaluations, an NCEE Reference Report, measures how sensitive result findings are to the type of assessment used (state test or a study-administered test) and using state tests in samples with multiple grades and states.

WWC Standards for Single-Case Design Research. In this paper, a panel of national experts provides an overview of SCDs, specifies the types of questions that SCDs are designed to answer, and discusses the internal validity of SCDs.

POWER
Building Capacity to Evaluate Group-Level Interventions: Optimal Design Software - Software for conducting power analyses and planning designs for a variety of multilevel designs: single level trials, cluster randomized trials, multi-site randomized trials, multi-site-cluster randomized trials, cluster randomized trials with treatment at level three, trials with repeated measures, and cluster randomized trials with repeated measures.

Do Typical RCTs of Education Interventions Have Sufficient Statistical Power for Linking Impacts on Teacher Practice and Student Achievement Outcomes? This NCEE Technical Methods report develops statistical power formulas for exploratory analyses under clustered school-based RCTs using ordinary least squares (OLS) and instrumental variable (IV) estimators, and uses these formulas to conduct a simulated power analysis.

G*Power 3 - A free software package that calculates power for many different statistical tests.

Introduction to Power - Website accompanies the article, Anderson-Cook, C. M., and Dorai-Raj, S. (2003), "Making the Concepts of Power and Sample Size Relevant and Accessible to Students in Introductory Statistics Courses using Applets," Journal of Statistics Education [Online], 11(3). Includes an applet that visually represents the idea.

Konstantopoulos, S. (2008). “Computing power of tests for the variability of treatment effects in designs with two levels of nesting.” Multivariate Behavioral Research, 43: 327-352

Konstantopoulos, S. (2008). “The power of the test for treatment effects in three-level cluster randomized designs.” Journal for Research on Educational Effectiveness, 1: 265-288.

Statistical Power Analysis in Education Research. This paper from IES/NCSER by L. Hedges and C. Rhoads provides a guide to calculating statistical power for the complex multilevel designs that are used in most field studies in education research.

Statistical Power for Regression Discontinuity Designs in Education Evaluations. This NCEE Technical Methods report examines theoretical and empirical issues related to the statistical power of impact estimates under clustered regression discontinuity (RD) designs.

SAMPLING
Hedges, L.V., & Hedberg, E.C. (2007). ”Intraclass correlation values for planning group-randomized trials in education.” Educational Evaluation and Policy Analysis, 29(1): 60-87.

Kish, L. (1965). Survey sampling. New York: Wiley.

Lohr, S.L. (1999). Sampling: Design and Analysis. Pacific Grove, CA: Duxbury Press.

Michael, R. T., and O’Muircheartaigh, C. A. (2008). “Design Priorities and Disciplinary Perspectives: The Case of the US National Children’s Study.” Journal of the Royal Statistical Society 171 (2): 465-480.

Sheaffer, R. L., Medenhall, W., & Ott, L. (2006). Elementary Survey Sampling (Sixth Edition). Belmont, CA: Thomson Brooks/Cole.

Spencer, B. D., and Foran W. (1991). "Sampling Probabilities for Aggregations, with Applications to NELS:88 and Other Educational Longitudinal Surveys." Journal of Educational Statistics 16: 21-34.

What to Do When Data Are Missing in Group Randomized Controlled Trials. This NCEE Technical Methods report examines how to address the problem of missing data in the analysis of data in Randomized Controlled Trials (RCTs) of educational interventions, with a particular focus on the common educational situation in which groups of students such as entire classrooms or schools are randomized.

Williams, B. (1978). A Sampler on Sampling. New York: John Wiley & Sons.

SYNTHESIS AND META-ANALYSIS
Cooper, H. M., Hedges, L. V., & Valentine, J. (Eds.) (2009). The Handbook of Research Synthesis and meta-analysis (2nd Edition). New York: The Russell Sage Foundation.

Hedges, L. V. 2007. Meta-analysis. Pages 919-953 in The Handbook of Statistics, ed. C. R. Rao. Amsterdam: Elsevier.

Konstantopoulos, S., & Hedges, L. V. (2004). Meta-Analysis. In D. Kaplan (Ed.), Handbook of Quantitative Methodology for the Social Sciences (pp. 281–297). New York: Sage.

Shadish, W.R., & Haddock, C.K. (2009). Combining estimates of effect size. In H. Cooper, Hedges, L. V., & Valentine, J. (Eds.), The Handbook of Research Synthesis (2nd ed.) (pp. 258-276). New York: Russel Sage Foundation.

Shadish, W. R., Rindskopf, D. M. and Hedges, L. V. (2008). “The state of the science in the meta-analysis of single-case experimental designs.” Evidence-Based Communication Assessment and Intervention, 2 (3): 188-196.

GENERAL
Basic and Applied Research Methods - Website for Lee A. Becker’s course on Basic and Applied Research Methods; includes general information on design, ethical, and measurement issues, among others, and suggested readings on these topics.

Estimating the Impacts of Educational Interventions Using State Tests or Study-Administered Tests, an NCEE Reference Report, identifies and describes the factors that could affect the precision of impact estimates when evaluations use state assessments instead of study-administered tests.

Kaplan, D. (Ed.). (2005). The SAGE Handbook of Quantitative Methodology for the Social Sciences. Thousand Oaks, CA: SAGE Publications.

NCES State Education Reforms website. This site, which draws primarily on data collected by organizations other than NCES, compiles and disseminates data on state-level elementary and secondary education reform efforts in accountability, assessments and standards, staff qualifications and development, state support for school choice, and student readiness and progress through school.

Resources for Researchers. A compilation of various methodological resources compiled by the Institute for Educational Sciences (IES) at the U.S. Department of Education.

DATA ANALYSIS

BAYESIAN STATISTICAL METHODS
Bayesian Methods, a Reading List - A compilation of readings that introduce tools of Bayesian statistics and machine learning, with a focus on applications to cognitive psychology.

A First Course in Bayesian Statistical Methods - An on-line book by P. D. Hoff; introduces Bayesian data analysis.

The BUGS Project - The BUGS (Bayesian inference Using Gibbs Sampling) project is concerned with flexible software for the Bayesian analysis of complex statistical models using Markov chain Monte Carlo (MCMC) methods.

CATEGORICAL DATA ANALYSIS
Applied Categorical & Nonnormal Data Analysis - Lecture notes for Phil Ender’s course; includes materials on categorical data analysis.

Categorical Data Analysis – Lecture notes for Richard Williams’s course; includes an introduction to models for binomial, ordinal and multinomial outcomes, and a discussion of categorical data analysis with complicated survey designs.

Categorical Data Analysis with Graphics - Intro
Categorical Data Analysis with Graphics – html version of notes
Lecture notes for Michael Friendly’s course, offered through the Statistical Consulting Service at York University; includes an introduction to categorical data analysis, tests of association for two-way tables and n-way tables, logistic regression, and loglinear models.

Modeling Categorical Data: Loglinear models and logistic regression – Lecture notes for Brendan Halpin’s course (in html format); includes an introduction to categorical data analysis.

GENERAL TEST THEORY
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Harcourt Brace Jovanovich.

Linn, R. L. (Ed.). (1989). Educational Measurement (3rd ed.). New York: Macmillan Publishing.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric Theory (3rd ed.). New York: McGraw-Hill.

Variability in Pretest-Posttest Correlation Coefficients by Student Achievement Level, an NCEE Reference Report, compares how states' pretests and posttests are related and how the correlations vary among students of different achievement levels.

HIERARCHICAL LINEAR MODELING (HLM) (see also resources on MULTILEVEL ANALYSIS)
HLM - Webpage for HLM software; includes FAQs, examples and free downloadable software (student version of HLM).

Multilevel Analysis/Hierarchical Linear Modeling - Lecture notes for C. J. Anderson’s course.

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.

ITEM RESPONSE THEORY (IRT)
IRT Modeling lab - Tutorial on IRT; includes notes on various introductory and intermediate IRT topics.

Hambleton, R. K., and Swaminathan, H. (1985). Item response theory: Principles and applications. Boston, MA: Kluwer-Nijhoff.

Wright, B. D., and Masters, G. N. (1982). Rating scale analysis: Rasch measurement. Chicago, IL: MESA.

LONGITUDINAL DATA ANALYSIS
Applied Longitudinal Data Analysis - Companion website for Singer, J. and Willett, J. (2003). Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. Oxford Press. Provides data, computer programs, and output for book.

Applied Longitudinal Data Analysis - Lecture notes and course materials for Marie Davidian’s course; includes examples.

Longitudinal Data Analysis - Lecture notes and course materials for Don Hedeker’s course; provides introduction to longitudinal data analysis and relates traditional approaches to repeated measures to the more flexible models that handle unbalanced data.

Repeated Measures/Longitudinal Data - Lecture notes from Alan Hubbard’s course.

Suggested Readings in Longitudinal Data Analysis - Patrick Curran’s suggested reading list for longitudinal data analysis. Provides references for traditional approaches to longitudinal data analysis and those for the Latent Growth Curve (LGC) approach to modeling longitudinal data.

MULTILEVEL ANALYSIS [see also resources on HIERARCHICAL LINEAR MODELING]
Advanced Multilevel Models - Lecture notes from Lesa Hoffman’s course; includes information on intermediate topics such as repeated measures, multivariate models, and generalized models.

The Estimation of Average Treatment Effects for Clustered RCTs of Education Interventions. This NCEE Technical Methods report examines the estimation of two-stage clustered RCT designs in education research using the Neyman causal inference framework that underlies experiments.

Kreft, I. and de Leeuw, J. (1998). Introducing Multilevel Modeling. Thousand Oaks, CA: Sage.

Useful Books- List of books on Centre for Multilevel Modeling website on multilevel modeling, including: General books aimed at a social science audience; books on longitudinal data analysis emphasizing (multilevel) random-coefficient models; more specialized volumes (e.g., spatial models, technical accounts of mixed models); books linked to, or that use, particular software; and books that discuss Markov chain Monte Carlo (MCMC) analysis.

MULTIVARIATE ANALYSIS
Companion website for Using Multivariate Statistics - Website for Tabachnick, B.G., and Fidell, L.S. (2007). Using Multivariate Statistics (5th Edition); provides an introduction to multivariate statistical analysis, and includes computer syntax and interpretations of computer output.

Hair, J.F., Black, B., Babin, B., Anderson, R.E., and Tatham, R.L. (2005). Multivariate Data Analysis (6th Ed.). Prentice Hall.

Tabachnick, B.G., and Fidell, L.S. (2007). Using Multivariate Statistics (5th Ed.). Boston: Pearson/Allyn and Bacon.

StatNotes: Topics in Multivariate Analysis – An online rolling publication by David Garson; includes notes on various statistical topics and models, with examples on data analysis and interpretation of the models.

REGRESSION MODELS
Cohen, J., Cohen, P., Aiken, L. S., & West, S. G. (2002). Applied multiple regression - correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.

Draper, N. R., & Smith, H. (1998). Applied regression analysis (3rd ed.). New York: Wiley.

Multilevel Regression - Lecture notes from Jason Newsom’s course.

STATISTICAL SOFTWARE
G*Power 3 - A free software package that calculates power for many different statistical tests.

HLM - Webpage for HLM software; includes FAQs, examples and free downloadable software (student version of HLM).

Research and Statistical Support (November 2002) - Issue of journal which lists Statistical Resources on the internet, including statistical software, articles, tutorials, interactive demos, software downloads, links for listserv archives, course notes and lectures. Topics include general statistics, more advanced methods, and information about specific instruments such as the Children’s Depression Scale (CDS) and the Satisfaction with Life Survey.

Software Packages – Statistics - A list of free software packages for use in the behavioral sciences.

SPSS Course Content - Website for Lee A. Becker’s course on SPSS; covers measures of association, t-tests, and ANOVA.

Stata Tutorial - Tutorial for learning Stata.

Statistical Computing - Website on statistical computing; provides links to tutorials on running SAS, SPSS, and Stata; examples from textbooks with computer code and interpretations of the output; and links to over 60 on-line seminars/classes on various statistical techniques and statistical packages.

Tutorials - The tutorials presented here provide a general introduction to various software packages; including AMOS, HLM, LISREL, MPlus, SAS, and SPSS, with sample syntax and interpretation of the output.

West, B. T., Welch, K. B., and Galcki, A. T. (2006). Linear Mixed Models: A Practical guide Using Statistical Software. Boca Raton: Chapman and Hall/CRC.

Statistics Concepts and Controversies
Statistical applets designed to help students master concepts covered in David S. Moore’s Statistics: Concepts and Controversies, 5e, including probability, the Central Limit Theorem, and p-value of a statistical test.

StatPower - Lecture notes for James Steiger’s course; includes slides covering introductory statistical topics.

VassarStats: Website for Statistical Computation - Tool for performing statistical computation. Provides descriptions and calculators for topics such as proportions, ordinal data, correlation & regression, t-tests, ANOVA, and ANCOVA.

OTHER
Alho, J. M., and Spencer, B. D. (2005). Statistical Demography And Forecasting. New York: Springer.

Applications of Latent Trait and Latent Class Models in the Social Sciences - An on-line book edited by J. Rost and R. Langeheine; introduces latent analysis methods with applications.

Confirmatory Factor Analysis for Applied Research - Companion webpage for Tim Brown’s book (2006); provides an introduction to confirmatory factor analysis.

Downloadable Books - List of books on various statistical topics for download at UCLA Academic Technology Services website.

Effect Size - Website that provides an introduction to effect size; includes links to calculators for effect sizes.

Effect Sizes in Research on Children and Families. This special issue of Child Development Perspectives (December 2008, Volume 2 Issue 3) contains a number of articles on the application of effect sizes in research on children and families.

FedStats - Website offering publicly available statistics from more than 100 agencies.

Fundamental Statistics for the Behavioral Sciences, 7th Edition – includes a glossary of general statistical terms and methods.

Inter-University Consortium for Political and Social Research (ICPSR) - is the world's largest archive of digital social science data.

Practical Assessment, Research & Evaluation - A peer-reviewed electronic journal with articles about various topics in research design, analysis and assessment.

Randomized Trials to Inform Education Policy - Website for Geoffrey Borman’s course; provides an introduction to implementation and analysis of randomized trials in the social sciences.

Research and Statistical Support (November 2002) - Issue of journal which lists Statistical Resources on the Internet, e.g. statistical software, articles, tutorials, interactive demos, software downloads, links for listserv archives, course notes and lectures. Topics include general statistics, more advanced methods, and information about specific instruments such as the Children’s Depression Scale (CDS) and the Satisfaction with Life Survey.

Rice Virtual Lab in Statistics - Reference website for statistics including an online volume with links to other online resources; includes an online statistics book with simulations and video, Java applets that demonstrate various statistical concepts, examples of real data with analyses and interpretation, and some basic statistical analysis tools. Topics covered in this general resource include estimation of confidence intervals, description of the hypothesis testing framework, ANOVA, chi-square tests, effect size, power, and nonparametric statistics.

Schochet, P. Z. (2009). Technical Methods Report: The Estimation of Average Treatment Effects for Clustered RCTs of Education Interventions (NCEE 2008-4026). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Scoring Office – A website providing an overview of writing test items.

Spencer, Bruce D. (1983). "On Interpreting Test Scores as Social Indicators: Statistical Considerations,” Journal of Educational Measurement 20, 317-334.

Statistical Notes - Notes on topics such as Excel for researchers, factor analysis, psychological research methods, and scale construction; all content is in PDF format.

Statistics Glossary - Glossary of statistical terms.

Translating the Statistical Representation of the Effects of Education Interventions into More Readily Interpretable Forms - This IES report is meant to help researchers translate effect size statistics into more interpretable forms that are helpful to practitioners, policymakers, and researchers.

Web Center for Social Research Methods - Website on applied social research and evaluation including online hypertext textbook on applied social research methods, The Knowledge Base, An online statistical advisor, volume of manual (i.e., dice-rolling) and computer simulation exercises of common research designs, The Simulation Book, a resource guide for learning about structured conceptual mapping.

Top