EAGER Proposal for Research in Measurement and Modeling: Dynamic STEM Assessment through Epistemic Network Analysis

Principal Investigator: 
Co-Investigator: 
Project Overview
Background & Purpose: 

In this project, we lay the ground work for developing a potentially transformational approach to STEM assessment in the 21st century. We propose the development of a new method of STEM assessment—called epistemic network analysis (ENA)—that focuses not on whether students master specific scientific facts, math skills, or engineering concepts, but on whether and how students link the skills, knowledge, identity, values, and epistemology of a STEM practice into a coherent way of thinking about complex STEM problems.

Setting: 

We are using sample data from an after-school STEM learning game.

Research Design: 

The research design for this project is comparative and is designed to generate data which is causal (statistical modeling). The epistemic frame hypothesis suggests that learning to solve complex STEM problems comes from being a part of a community of practice—that is, a group of people who share similar ways of solving problems. Within a STEM discipline, there are not only specific facts and skills to be mastered, but also a set of intellectual values: particular ways of justifying decisions and developing solution. Any community of practice, in short, has a culture and that culture has a grammar: a structure composed of skills (the things that people within the community do); knowledge (the understandings that people in the community share); values (the beliefs that members of the community hold); identity (the way that members of the community see themselves); and epistemology (the warrants that justify actions or claims as legitimate within the community). This collection of skills, knowledge, values, identity, and epistemology forms the epistemic frame of the community.

In formal terms, the epistemic frame of a given community of practice, P, has elements f1…n, where each fi is some skill, knowledge, value, aspect of identity, or epistemology that is part of the profession P. We analyze learning by observing a series of activities about which we collect data, Dt, which represents information about the players’ epistemic frames during the activities at each time t. For any learner, we can code each Dt for evidence that frame elements f1…n were used at time t. From this coding, we construct an adjacency matrix showing whether each pair of frame elements (i,j) was used by the learner at time t. We can quantify the characteristics of the developing epistemic network for learner by summing, for each pair of frame elements, the number of times they are recorded in the same Dt. That is, we can construct a cumulative adjacency matrix which shows the strength of the association between each pair of frame elements (i,j) for a given player in the data set. Once an epistemic frame is represented as a series of cumulative adjacency matrices, we can quantify the characteristics of the network using concepts from SNA, such as network density and centrality of individual nodes [21]. That is, we can quantify the changes in a learner’s developing STEM frame over time and compare it to the trajectory of expert performance.

We plan to use regression models to compare ENA analyses to traditional measures of STEM learning. We will use jackknife techniques to analyze the reliability of ENA models.

Findings: 

Project findings will be added as they become available.

Other Products: 

We aim to develop a new technique for assessing STEM learning.