Foundations of 21st Century Assessments

Principal Investigator: 
Project Overview
Background & Purpose: 

The Foundations project aims to investigate and compare the construct validity of traditional and technology-based assessment tasks designed to measure complex learning. Further, the project will identify design principles of technology-enhanced assessments that elicit understanding of science systems and inquiry abilities.

Setting: 

Studies will be carried out in middle school classrooms in California.

Research Design: 

The research design for this project is comparative, and is designed to generate associative/correlational evidence using quasi-experimental methods. The project includes an intervention, which is a series of assessments of scientific inquiry skills that are either static, dynamic or interactive. The study is a within-subjects design so students will be compared with their own performance across conditions. This project collects original data using assessments of learning/achievement tests, and Cognitive Lab/Think aloud data. Static, dynamic and interactive items will be developed for the empirical study and will be carefully designed to measure scientific knowledge and inquiry in the two science domains (force and motion or ecosystems). These items will be reviewed by the AAAS-led experts and the Advisory Panel.

We will establish the psychometric properties of the test items by using classical test statistics such as such as p-values and point biserials, as well as fitting an IRT partial credit model to examine item fit statistics. We will also use Cronbach’s alpha to ensure test reliability for each test session. Items that have poor psychometric characteristics such as out of range point biserials or item misfit in IRT will be excluded from the subsequent analyses. Finally, we will fit a series of partial credit IRT models to the student response data from each test session to investigate how each fits as we start from a unidimensional IRT model through successive models that add dimensions for each science practice. 

Publications & Presentations: 

Quellmalz, E.S., Davenport, J., Timms, M., & Buckley, B., WestEd, (2009). Quality Science Simulations for Formative and Summative Assessment. Presented at the NCME Annual Meeting 2009 in San Diego, CA. 

Other Products: 

This project will generate new assessments that test scientific reasoning and inquiry in physical and life sciences.