NSF asks REESE projects to develop an evaluation plan that includes measures that the project team intends to use in assessing its success and meeting its objectives. As described in the current REESE solicitation (Solicitation 12-552):
"Project evaluations should be sufficiently distant from the project to be objective but should be designed to be of most help to the project team pursuant to its responsibilities to the field. All projects must have a substantive external expert review mechanism (e.g., advisory committee) that provides regular feedback on the project's research methods and progress, analysis procedures, interpretation of data into findings, and dissemination activities. In some cases, this may be all that is necessary. In other cases, additional external evaluation might be more appropriate, given the nature of the project. Finally, proposals should describe how evaluation input will be used to shape the project."
Following are some resources that may be helpful to REESE investigators. Please contact us to recommend additional evaluation resources.
Select Readings on Evaluation Research and Design
Bickman, L., & Rog, D. J. (2009). The SAGE handbook of applied social research methods. Los Angeles: SAGE.
Boruch, R.F., & Cordray, D.S. (Eds.). (1980). An Appraisal of Educational Program Evaluations: Federal, State, and Local Agencies. Washington, D.C.: U.S. Department of Education.
Chelimsky, E. (1987). The politics of program evaluation. Society, 25(1): 24-32.
Clewell, B. & Fortenberry, N. (Eds.). (2009). Framework for Evaluating Impacts of Broadening Participation Projects. Washington DC: National Science Foundation
Cook, T. D. 2000. The false choice between theory-based evaluation and experimentation. New Directions in Evaluation: Challenges and Opportunities in Program Theory Evaluation 87:27–34.
Cook, T. D. (2008). "Waiting for life to arrive": A history of the regression-discontinuity design in psychology, statistics and economics. Journal of Econometrics, 142(2), 636-654.
Cook, T. D., M. Scriven, C. L. S. Coryn, and S. D. H. Evergreen. (2009). Contemporary thinking about causation in evaluation: A dialogue with Tom Cook and Michael Scriven. American Journal of Evaluation. Published online on December 4, 2009 as doi:10.1177/1098214009354918.
Cook, T. D., & Steiner, P. M. (2010). Case matching and the reduction of selection bias in quasi-experiments: the relative importance of covariate choice, unreliable measurement and mode of data analysis. Psychological Methods, 15(1), 56–68.
Cook, T. D., Steiner, P. M., & Pohl, S. (2009). Assessing how bias reduction is influenced by covariate choice, unreliability and data analytic mode: an analysis of different kinds of within-study comparisons in different substantive domains. Multivariate Behavioral Research, 44, 828–847.
Cook, T. D., Shadish, W. J., & Wong, V. C. (2008). Three conditions under which observational studies produce the same results as experiments. Journal of Policy Analysis and Management, 27(4), 724-750.
Cook, T. D., & Wong, V. C. (2008). Empirical tests of the validity of the regression discontinuity design. Annales d’Economie et de Statistique, 91-92, 127-150.
Cronbach, L.J. (1982). Designing Evaluations of Educational and Social Programs. San Francisco, CA: Jossey-Bass.
Cronbach, L.J., Ambron, S.R., Dornbusch, S.M., Hess, R.D., Hornik, R.C., Phillips, D.C., et al. (1980). Toward Reform of Program Evaluation: Aims, Methods, and Institutional Arrangements. San Francisco: Jossey-Bass.
Davis, D.F. (1990). Do you want a performance audit or a program evaluation? Public Administration Review, 50, 35-41.
Diamond, S. S., Bowman, L., Wong, M., & Patton, M. M. (2010). Efficiency and cost: The impact of videoconferenced hearings on bail decisions. Journal of Criminal Law and Criminology, 100(3), 869-902.
Fitzpatrick, J.L., Mark, M. M., & Christie, C. A. (2008). Evaluation in Action: Interviews with expert evaluators. Thousand Oaks, CA: Sage.
Fitzpatrick, J.L., Sanders, J.R., & Worthen, B.R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th Ed.). New York: Allyn & Bacon.
Frechtling, J.A. (2007). Logic Modeling Methods in Program Evaluation. San Francisco, CA: John Wiley & Sons, Inc.
Frechtling, J. (2002). The 2002 User Friendly Handbook for Project Evaluation. Rockville, MD: National Science Foundation
Friedman, A. (2008). Framework for Evaluating Impacts of Informal Science Education Projects. Washington DC: National Science Foundation
Grinnell, R. M., & Unrau, Y. A. (2011). Social work research and evaluation: Foundations of evidence-based practice. New York: Oxford University Press.
Hahn, T., & Van der Klaauw. (2001). Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica, 69(1): 201-209.
Ham, J.C., & LaLonde, R.J. (1996). The effect of sample selection and initial conditions in duration models: Evidence from experimental data on training. Econometrica, 64(1): 175-205.
Heckman, J.J., Smith, J.A., & Clemens, N. (1997). Making the most out of programme evaluations and social experiments: Accounting for heterogeneity in programme impacts. Review of Economic Studies, 6(4), 487-535.
House, E.R. (1980). Evaluating with Validity. Beverly Hills, CA: Sage.
Johnson, K. A., Greenseid, L., Toal, S.A., King, J. A., Lawrenz, F. & Volkov, B. (2009). Research on Evaluation Use: A Review of the Empirical Literature From 1986 to 2005. American Journal of Evaluation, 30(3), 377-410.
Julnes, G., & Rog, D.J. (Eds.). (2007). Informing Federal Policies on Evaluation Methodology: Building the Evidence Base for Method Choice in Government Sponsored Evaluation. New Directions for Evaluation, no. 113. A publication of Jossey-Bass and the American Evaluation Association.
Kellaghan, T., Stufflebeam, D.L., & Wingate, L.A. (2003). International Handbook of Educational Evaluation. Dordrecht, The Netherlands: Kluwer Academic Publishers.
King, J.A., & Lawrenz, F. (Eds.). (2011). Multisite Evaluation Practice: Lessons and Reflections from Four Cases. New Directions for Evaluation, no. 129. A publication of Jossey-Bass and the American Evaluation Association.
Ladd, H. F., & Fiske, E. B. (2008). Handbook of research in education finance and policy. New York: Routledge.
LaLonde, R.J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76(4): 604-620.
Lipsey, M.W., & Cordray, D.S. (2000). Evaluation methods for social intervention. Annual Review of Psychology, 51, 345-375.
Mertens, D.M. & Wilson, A.T. (in press). Program evaluation: Theory to practice. NY: Guilford
Mertens, D. M. (2010) Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods, 3rd ed. Thousand Oaks, CA: Sage
Murnane, R.J., & Willett, J.B. (2010). Methods Matter: Improving Causal Inference in Educational and Social Science Research. Oxford University Press.
Ottoson, J.M., & Hawe, P. (Eds.). (2009). Knowledge Utilization, Diffusion, Implementation, Transfer, and Translation: Implications for Evaluation. New Directions for Evaluation, no. 124. A publication of Jossey-Bass and the American Evaluation Association.
Owen, John M. (2007). Program Evaluation: Forms and Approaches, 3rd edition. New York, NY: The Guilford Press.
Patton, M.Q. (1990). Qualitative evaluation and research methods, 2nd edition. Thousand Oaks, CA: Sage Publications, Inc.
Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009). Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis, 31(4), 463–479.
Ragin, C. C., Nagel, J., White, P., & National Science Foundation (U.S.). (2004). Workshop on Scientific Foundations of Qualitative Research. Arlington, VA: National Science Foundation
Rogers, P.J., Hacsi, T.A., Petrosino, A., & Huebner, T.A. (Eds.). (2000). Program Theory in Evaluation: Challenges and Opportunities. New Directions for Evaluation, no. 87. A publication of the American Evaluation Association.
Rossi, P.H., Lipsey, M.W., & Freeman, H.W. (2004). Evaluation: A Systematic Approach, 7th edition. Thousand Oaks, CA: Sage Publications.
Royse, D., Thyer, B., & Padgett, D. (2010). Program Evaluation: An Introduction, 5th edition. Belmont, CA: Wadsworth.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin Company.
Shadish, W.R., Jr., Cook, T.D., & Leviton, L.C. (1991) Foundations of Program Evaluation: Theories of Practice. Newbury Park, CA: Sage Publications, Inc.
Shadish, W.R., Galindo, R., Wong, V.C., Steiner, P.M., Cook, T.D. (2011). A randomized experiment comparing random to cutoff-based assignment. Psychological Methods, 16(2), 179-191
Steiner, P. M., Cook, T. D., & Shadish, W. R. (2011). On the importance of reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics, 36(2), 213.
Steiner, P. M., Cook; T. D., Shadish, W. R., & Clark M. H. (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15(3), 250-67.
Stufflebeam, D.L. (2001). Evaluation Models. New Directions for Evaluation, no. 89. A publication of the American Evaluation Association.
Stufflebeam, D.L., & Shinkfield, A.J. (2007). Evaluation Theory, Models, and Applications. San Francisco, CA: Jossey-Bass.
Toal, S. A. (2009). The validation of the evaluation involvement scale for use in multisite settings. American Journal of Evaluation, 30(3), 349-362.
Toal, S. A., King, J. A., Johnson, K., & Lawrenz, F. (2009). The unique character of involvement in multi-site evaluation settings. Evaluation and Program Planning, 32, 91-98
Weiss, C.H. (1972). Evaluation Research: Methods for Assessing Program Effectiveness. Englewood Cliffs, NJ: Prentice Hall.
Weiss, C.H. (1998). Improving the use of evaluations: Whose job is it anyway? Advances in Educational Productivity, 7, 263-276.
Guidelines, Toolkits and Other Resources for Evaluation Research
American Evaluation Association (AEA)
"The American Evaluation Association is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness" (see AEA online at http://www.eval.org/). Numerous AEA resources may be of interest to STEM education researchers including:
The American Evaluation Association’s principles of evaluation are available online. AEA has developed training resources for introducing and understanding the Guiding Principles, including guides, articles, and power-points, which are available at http://www.eval.org/gptraining/gptrainingoverview.asp.
Additional online evaluation resources, including information on instruments, consultants, forums, and handbooks are provided in the Resources section of the AEA website.
The Guide to the Systems Evaluation Protocol (2012)
“This Guide to the Systems Evaluation Protocol is intended to serve several purposes: it is first and foremost designed to be a step-by-step guide for both program and evaluation professionals who wish to integrate a systems evaluation perspective into their evaluation work in order to enhance the quality and appropriateness of program evaluations.” (pp. vi)
This website is an online community for informal learning projects, research, and evaluation. In particular, it contains many useful links for evaluation purposes, including evaluation publications, organizations, guides, and how to’s.
Joint Committee on Standards for Educational Evaluation
"The Joint Committee is a coalition of major professional associations concerned with the quality of evaluation. The Joint Committee has published three sets of standards for evaluations that are now widely recognized" (see: http://www.jcsee.org/about). The program evaluation standards are available on the committee's website at: www.jcsee.org/program-evaluation-standards/program-evaluation-standards-statements and in a volume designed "to support evaluation quality through appropriate application of these standards" (p. xxx) at: www.sagepub.com/booksProdDesc.nav?prodId=Book230597.
Online Evaluation Resource Library
"The purpose of the Online Evaluation Resource Library is to collect and make available evaluation plans, instruments, and reports for NSF projects that can be used as examples by Principal Investigators, project evaluators, and others outside the NSF community as they design proposals and projects." (see: http://oerl.sri.com/)
Western Michigan University: The Evaluation Center
"The Evaluation Center is committed to advancing the theory, practice, and utilization of evaluation through research, education, service, and leadership" (see: http://www.wmich.edu/evalctr/about/). Resources include:
“This site provides refereed checklists for designing, budgeting, contracting, staffing, managing, and assessing evaluations of programs, personnel, students, and other evaluands; collecting, analyzing, and reporting evaluation information; and determining merit, worth, and significance. Each checklist is a distillation of valuable lessons learned from practice”
This site provides current projects with evaluation reports that can be referenced as examples.
“In keeping with The Evaluation Center’s mission to advance the theory, practice, and utilization of evaluation, our staff are engaged in publishing and presenting their research and evaluation work, as well as leading training sessions for national and international audiences. Listed below are the publications by Center staff, many of which also feature Interdisciplinary Evaluation doctoral students. This page shows EC staff publications since 2007.”
WWC Registry of Evaluation Researchers
A searchable list of external evaluators is available via the What Works Clearinghouse. General information about the registry is available online at http://ies.ed.gov/ncee/wwc/references/registries/index.aspx
NSF's PRIME Program
Promoting Research and Innovation in Methodologies of Evaluation (PRIME) As described in the current PRIME solicitation (Solicitation 12-508):
"The Promoting Research and Innovation in Methodologies for Evaluation (PRIME) program seeks to support research on evaluation with special emphasis on exploring innovative approaches for determining the impacts and usefulness of STEM education projects and programs; building on and expanding the theoretical foundations for evaluating STEM education and workforce development initiatives, including translating and adapting approaches from other fields; and growing the capacity and infrastructure of the evaluation field." (p. 1-2)
To learn more about the work PRIME supports and view abstracts of research being conducted in this program click here.
Please note that the websites and publications included in these directories are not affiliated with or endorsed by ARC. ARC does not guarantee their timeliness and is not responsible for their content.