The first set of investigations focused on text analyses and statewide performance data showed that several features and composite factors affect student performance. These include: a) linguistic load (i.e., the number of semantic units in the item, in the answer choices, and/or the presence of negation in the stem), and b) extended science knowledge (i.e., items that present unfamiliar contexts and/or that require atypical perspective-taking). When these features are present, student performance tends to decrease. Preliminary analysis of interview data suggests that features not significantly correlated with performance in the first study (e.g., forced comparison, multiple frames) may in fact present students with considerable challenge. Importantly, the interviews suggest that these features may be sources of difficulty for students who know the targeted science as well as those whose scientific knowledge is questionable. The interviews also suggest that the features of forced comparison and atypical perspective may interact in surprising and unpredictable ways to create conditions in which students, sometimes creatively and often quite reasonably, generate multiple alternative interpretations for the same item. In combination, these features may present insurmountable challenges to students.
Test-makers generally worry about two kinds of error patterns. They worry about a) students who do not know the subject matter being tested but manage to get the right answer, and b) students who do know the subject matter being tested and, for whatever reason, get the wrong answer. When these patterns appear, it suggests that items may not be measuring what they purport to measure. Our results suggest that both kinds of error may be in operation for the 5th Grade Science MCAS. Perhaps not surprisingly, we found that students from communities historically placed at risk in school primarily fell into error pattern (b), that is, they knew the subject matter and selected the wrong answer. Interestingly, however, we also found instances in which middle class students who knew the subject matter fell into this error pattern as well.
Children and Science Tests Research Team (March 2008). Making sense of children’s performance on achievement tests: The case of the 5th grade science MCAS. Paper presented at the Annual Meeting of the American Education Research Association, New York City, New York.