Culture, Cognition, and Evaluation of STEM Higher Education Reform: A Mixed-Methods Longitudinal Study

Principal Investigator: 
Project Overview
Background & Purpose: 

The purpose of this study is to identify the mental models that instructors have for teaching introductory STEM courses in their fields, and their relationship to cultural norms regarding teaching and other organizational characteristics. We address the unit of analysis problem by adopting a situated view of cognition and focus on the mental representations and heuristics of individual faculty members as being embedded within the social, cultural, and technical aspects of their organizations. The study is a mixed-methods design that draws on the strengths of both quantitative and qualitative approaches to studying cognition and culture, and is intended as a pilot approach to program evaluation that improves the assessment of program impacts.


This study will take place at the following research universities: the University of Wisconsin-Madison, the University of Washington – Seattle, and the University of Colorado-Boulder. 

Research Design: 

The research design for this project is longitudinal, involves synthesis research, and is also designed to generate evidence that is descriptive and associative/correlational through ethnography and observation. This project will use a web-based survey instrument, a semi-structured interview protocol, and an observation protocol. The web-based survey will elicit information about instructor beliefs about teaching and learning, pedagogical content knowledge, pedagogical strategies, and level of participation in reform activities. The semi-structured interview protocol will include a retrospective think-aloud procedure and questions focused on the subjects’ perceptions of how their organizational environment influences their instructional practices. Finally, the observation protocol will be used to observe instructors teaching in large lecture and small discussion environments.

The survey data will be analyzed using the cultural consensus model (principal-components analysis) to identify groups of subjects who exhibit consensus on certain survey items, and individual cultural knowledge in that area. ANOVA and hierarchical linear modeling will be used to test relationships among different variables from the survey. Protocol analysis will be used to analyze the think-aloud data, and grounded theory will be used to analyze interview data. Findings from each type of analysis will be triangulated with others when they assess similar measures or phenomenon.

Other Products: 

We expect to develop survey and interview instruments that can be used by program evaluators that would be an improvement over current methods. These new instruments would be based on current theories of faculty cognition and their relationship to teaching practices, and would capture the relationship between cognition, organizational context, and actual teaching practices.