Building STEM Education and Evaluation Capacity through Research on Logic Model Use

Principal Investigator: 
Project Overview
Background & Purpose: 

The overall goal of this training, research, and dissemination project is to build STEM education and evaluation capacity, and the capacity of the evaluation profession at-large. To achieve this outcome the project has four objectives: 1) Increase the pool of evaluators of color with skills/experience in the evaluation of STEM education programs, qualitative methods, and culturally-responsive evaluation; 2) Understand the nature, extent, and quality of logic model use in the design, implementation, and evaluation of STEM education programs; 3) Understand the nature and extent of logic model use for evaluators at-large; 4) Produce and disseminate refinements in the theory and practice of logic model use generally, and for STEM program design, implementation, and evaluation.

Setting: 

NSF-funded Math and Science Partnerships

Research Design: 

The research design for this project is cross-sectional, and is designed to generate descriptive evidence through case study, and observation. This project collects original data using personal observation, survey research [online self-completion questionnaire, structured interviewer-administered questionnaire (computer assisted telephone interview, CATI), and semi-structured/informal interviews (face-to-face, and telephone)]. Qualitative data collection instruments were structured around gathering information about use of logic models as pertains to MSP being study, the benefits and challenges of using logic models, and the extent of prior experiences with logic models.

Qualitative data analysis software (Atlas-ti) was used to assign codes to interview segments and produce code specific output across interviews. Code specific segments were synthesized and summarized to develop a picture of logic model use among these MSPs.

Findings: 

There is variation in the development, content and use of models being used by MSPs that could be described as a logic model or related schemes, some of which do not adhere to logic models as described in the literature. While the models emerge and evolved at different points in each case study site’s development, similarities in the function of the models exist. For example, the models tend to be used as a guiding framework and as a communication tool. These two key features of the models further translate into a tool for orienting others to the project, developing shared understanding /shared language for describing the project and its processes among key stakeholders, which with more regular or sustained interaction with and/or reference to the model can enhance staff and stakeholders’ internalized understanding of the work of the project. This functionality serves day-to-day and as well as larger decision-making by various individuals and project leadership. Factors that have been seen to influence the use and benefit of logic models at case study sites tend to relate to four main categories: 1) The organization/project (scale, complexity, leadership, org culture); 2) The model itself, 3) The role of the evaluation team, and 4) The role of the funder. Case studies reports are currently being completed and further synthesis and cross-case analysis are continuing to further articulate findings and implications for practice.

Publications & Presentations: 

Torres, R. T., Hopson, R. K., & Casey, J. (2008). Case Study Findings on Logic Model use: Building STEM Education and Evaluation Capacity through Research on Logic Model Use. Presented at The Annual Meeting of the American Evaluation Association. Denver, Co.

Other Products: 

A book highlighting factors that influence the extent of use and benefit of logic models among large-scale multi-dimensional initiatives with implications for those seeking to use/make better use of logic models and related schemes.