A Phase II Trial of the Systems Evaluation Protocol for Assessing and Improving STEM Education Evaluation

Principal Investigator: 
Project Overview
Background & Purpose: 

This study is meant to assess whether the Systems Evaluation Protocol (SEP) (which was developed with funding from a previous NSF grant) when implemented in several different STEM contexts is associated with changes in short and intermediate outcomes related to evaluation, the evaluation capacity of participating organizations, and the quality of evaluation plans and products. A self-guided, virtual version of the SEP will also be developed and compared with the original facilitated SEP. In addition, a cyberinfrastructure for Evaluation Capacity Building will be significantly revised to better support both the facilitated and virtual SEP.

Setting: 

Study participants include a large cross-section of the 61 Cornell Cooperative Extension county offices that offer Science, Engineering and Technology programs across New York State, as well as at least 14 MRSEC sites across the United States.

Research Design: 

This study will be conducted using a longitudinal-sequential (or cohort-sequential) design with four cohorts over five years. The first cohort is comprised of eight pilot sites (seven CCE, one MRSEC) already engaged in the SEP process. Data from the initiation stage of this process (pilot years 1 and 2) have already been collected. The second cohort 20 CCE and three MRSEC sites who are working through the facilitated SEP. The Third cohort will consist of 10 sites who will be used to develop a virtual version of the SEP, and the fourth cohort will use only the virtual SEP. Over the next five years we will follow these sites through their implementation and utilization stages in an effort to determine the degree to which any changes in evaluation capacity are sustained over time. In addition, we will monitor the evaluation activity of five CCE sites and five MRSEC sites who have not participated in the SEP process in REESE year 2 as a control group. The longitudinal-sequential design will allow us to make a number of different comparisons both between and within the four cohorts. Primarily, we will use change score analysis to determine a) if there are any changes over time that are associated with exposure to the SEP process, b) if these changes differ between groups, and c) if these changes are sustained over time.

Findings: 

See Publications & Presentations for more information on this project.

Publications & Presentations: 

HTTP://WWW.NEWS.CORNELL.EDU/STORIES/DEC08/TROCHIM.NSF.SH.HTML

HTTP://PAD.HUMAN.CORNELL.EDU/CHE/NEWS_EVENTS.CFM?ID=113339&PAGETYPE=NEWS-STORY&RELATED

HTTP://WWW.CCMR.CORNELL.EDU/NEWS/NEWS.HTML?ID=211

HTTP://WWW.SRI.CORNELL.EDU/SRI/PROJECTS.PROJECT.CFM?PROJID=123542
(POST INTERVIEWS FROM FIRST STUDY, AND WILL BE REPEATED.)

Other Products: 

This project expects to generate an improved SEP for evaluation planning; SEP steps for evaluation implementation and utilization; a Virtual SEP Cyberinfrastructure; and tools for measuring evaluation capacity.