Transforming STEM Assessment Methods: Research on Cyber-Enabled Measurement of Cognitive Models of Natural Selection

Principal Investigator: 
Co-Investigator: 
Project Overview
Background & Purpose: 

This project addresses a central problem in STEM education--assessing students’ cognitive models of natural selection--that must be addressed in order for substantial progress to be made in the teaching and learning of this extremely important but greatly misunderstood core idea in biology. The project harnesses the expertise from an unprecedented collaboration among science educators, evolutionary biologists, a psychometrician, statistician, cognitive psychologist, and Latent Semantic Analyst. The multidisciplinary team is working to transform the basic approaches used to perform biology assessment by constructing a cyber-enabled assessment cascade system for measuring learners’ knowledge of natural selection.

The website for this project is:

http://evolutionassessment.org

Setting: 

Science majors attending a research university in the Midwestern USA.

Research Design: 

The research design for this project is cross-sectional and comparative and is designed to generate evidence which is associative and/or correlational (quasi-experimental). This project collects original data using: school records (of students) or policy documents; assessments of learning and achievement tests; online self-completion questionnaire; face-to-face and computer assisted personal interviewer-administered questionnaires; face-to-face informal interviews; and focus groups.

Our research design includes testing the efficacy of four assessment methods relative to a “gold standard” (oral interviews about instances). These methods represent a spectrum of assessment strategies and instruments that vary in practical utility, ease of implementation, validity, and precision. We are using a statistical model to evaluate the degree of proximity that these different methods approach the gold standard. One of the methods is a novel computer-based assessment system.

Findings: 

Numerous psychometric constraints characterize extant multiple-choice Concept Inventories of natural selection and evolution (Nehm and Schonfeld 2008, 2010). Our study of computer assisted scoring of constructed response evolutionary explanations by biology undergraduates demonstrated that: (1) text analysis tools (i.e., SPSS Text Analysis 3.0) may be used to successfully diagnose fine-grained explanatory elements comprising students’ mental models of natural selection as represented in open-response text and (2) text analysis assessment scores are comparable to expert human-generated assessment scores in the vast majority of cases. Collectively, these findings affirm our view that text analysis may be a transformative method for STEM assessment in general and natural selection measurement in particular. Numerous disadvantages also characterize our approach, however, and should be weighed carefully relative to other text analytic strategies such as machine learning. Our future research will leverage the advances made in our term library expansion and rule generation to tackle other knowledge elements common to student evolutionary explanations—notably naıve ideas or misconceptions—and attempt to build more sophisticated and holistic representations and measures of students’ evolutionary
thinking using computational tools.

Publications & Presentations: 

Journal Articles:

(16) Beggrow, E. P., Ha, M., Nehm, R. H., Pearl, D., & Boone, W. J. (in press). Assessing scientific practices using machine-learning methods: How closely do they match clinical interview performance? Journal of Science Education and Technology. DOI: 10.1007/s10956-013-9461-9

(15) Ha, M., & Nehm, R. H. (in press). Darwin's difficulties and students' struggles with trait loss: Cognitive-historical parallelisms in evolutionary explanation. Science & Education. DOI
10.1007/s11191-013-9626-1.

(14) Rector, M., Nehm, R.H., & Pearl, D. (2013). Learning the language of evolution: Lexical ambiguity and word meaning in student explanations. Research in Science Education, 43, 3, 1107-1133.

(13) Opfer, J., Nehm, R.H., Ha, M. (2012). Cognitive Foundations for Science Assessment Design: Knowing What Students Know about Evolution. Journal of Research in Science Teaching, 49(6),744–777

(12) Rector, M., Nehm, R.H., Pearl, D. (2012). Learning the Language of Evolution: Lexical Ambiguity and Word Meaning in Student Explanations. Research in Science Education.

(11) Beggrow, E., Nehm, R.H. (2012). Students' Mental Models of Evolutionary Causation: Natural Selection and Genetic Drift. Evolution Education and Outreach. 5:429-444

(10) Nehm, R.H., Beggrow, E., Opfer, J., and Ha, M. (2012). Reasoning About Natural Selection: Diagnosing Contextual Competency Using the ACORNS Instrument. The American Biology Teacher. 74(2):92-98

(9) Ha, M., Nehm, R. Urban-Lurain, M., Merrill, J. (2011). Applying Computerized Scoring Models of Written Biological Explanations across Courses and Colleges: Prospects and Limitations. CBE-Life Sciences Education, 10(4) 379-393

(8) Nehm, R.H., Ridgway, J. (2011). What do experts and novices “see” in evolutionary problems? Evolution Education and Outreach. 4(4):666-679

(7) Nehm, R.H., Ha, M., Mayfield, E. (2012). Transforming Biology Assessment with Machine Learning: Automated Scoring of Written Evolutionary Explanations. Journal of Science Education and Technology. 21(1):183-196

(6) Nehm, R.H., Haertig, H. (2012). Human vs. Computer Diagnosis of Students’ Natural Selection Knowledge: Testing the Efficacy of Text Analytic Software. Journal of Science Education and Technology. 21(1):56-73

(5) Nehm, R.H., Ha, M. (2011). Item feature effects in evolution assessment. Journal of Research in Science Teaching. 48(3):237–256.

(4) Nehm, R.H., Rector, M., and Ha, M. (2010). “Force Talk” in Evolutionary Explanation: Metaphors and Misconceptions. Evolution Education and Outreach, (3) 506-613,

(3) Nehm, R.H. (2010). Understanding Undergraduates’ Problem Solving Processes. Journal of Biology and Microbiology Education, 1(2):119-122.

(2) Ha, M., Haury, D., Nehm, R.H. (2012). A feeling of certainty: Uncovering a missing link between evolutionary knowledge and acceptance. Journal of Research in Science Teaching. 49(1):95–121

(1*) Haudek, K., Kaplan, J. Knight, J., Long, T., Merrill, J., Munn, A., Nehm, R.H., Smith, M. Urban-Lurain, M. (2011). Harnessing Technology to Improve Formative Assessment of Student Conceptions in STEM: Forging a National Network. CBE-Life Sciences Education Vol. 10, 149–155

*collaboration with NSF CCLI AACR project

Technical Reports:

(3) Nehm, R.H., Ha, M., Rector, M., Opfer, J., Perrin, L., Ridgway, J., Mollohan, K. (2010). Scoring Guide for the Open Response Instrument (ORI) and Evolutionary Gain and Loss Test (EGALT). Technical Report of National Science Foundation REESE Project 0909999. 40 p.

(2) Baronda, S. (2011). Installation Guide for the Assessment Cascade System (ACS). Technical Report of the the National Science Foundation REESE Project 090999. 14p.

(1) Mollohan, K., Baronda, S. Nehm, R. (2011). User’s Guide for the Assessment Cascade System (ACS). Technical Report of the the National Science Foundation REESE Project 090999. 13p. 

Conference Papers:

(14) Ha, M., Moharreri, K., & Nehm, R.H. (2013). EvoGrader: A free, online assessment tool for evaluating undergraduates' written evolutionary explanations. Joint annual meeting of the Society for the Study of Evolution (SSE), the Society of Systematic Biologists (SSB), and the American Society of Naturalists (ASN), Snowbird, UT, June 21-25.

(13) Nehm, R.H., Ha, M., & Rector, M. A. (2013). Automated feedback in the assessment of students' written explanations of evolutionary change. Paper presented at the 2013 Annual Meeting - American Educational Research Association. San Francisco, CA, April 27 – May 1.
 

(12) Beggrow, E. P., Ha, M., Nehm, R. H., & Boone, W. J. (2013). Do computer-generated written explanation scores closely approximate oral interview scores? Evidence from Rasch modeling. Paper in proceedings of the National Association for Research in Science Teaching, Rio Grande, Puerto Rico. April 6 - April 9.

(11) Federer, M. R., Nehm, R. H. Beggrow, E. P., Ha, M., & Opfer, J. E. (2013). Evaluation of a new multiple-true-false concept inventory for diagnosing mental models of natural selection. Paper in proceedings of the National Association for Research in Science Teaching, Rio Grande, Puerto Rico. April 6 - April 9.

(10) Ha, M., & Nehm, R. H. (2013). Exploring the efficacy of machine learning and translation software in international comparison studies. Paper in proceedings of the National Association for Research in Science Teaching, Rio Grande, Puerto Rico. April 6 - April 9.

(9) Ha, M., Dennis, S., & Nehm, R. H. (2013). Optimizing machine-learning models for automated computer scoring of natural selection concepts. Paper in proceedings of the National Association for Research in Science Teaching, Rio Grande, Puerto Rico. April 6 - April 9.

(8) Nehm, R. H., Ha, M., Großschedl, J., Harms, U., & Roshayanti, F. (2013). American, German, Korean, and Indonesian pre-service teachers' evolutionary acceptance, knowledge, and reasoning patterns. Paper in proceedings of the National Association for Research in Science Teaching, Rio Grande, Puerto Rico. April 6 - April 9.

(7) Ha, M., Nehm, R.H. (2012). Using Machine-Learning Methods to Detect Key Concepts and Misconceptions of Evolution in Students’ Written Explanations. Proceedings of the National Association for Research in Science Teaching (NARST) annual conference, Indianapolis, IN, March 25-March 28.

(6) Rector, M.A., Nehm, R.H., and Pearl, D. (2012). Item Sequencing Effects on the Measurement of Students’ Biological Knowledge. Proceedings of the National Association of Research in Science Teaching conference, Indianapolis, IN, March 25-28.

(5) Beggrow, E.P., Nehm, R.H. (2012). Exploring the Role of Non-Adaptive Reasoning in Students’ Evolutionary Explanations. Proceedings of the National Association for Research in Science Teaching (NARST) annual conference, Indianapolis, IN, March 25-March 28.

(4) Rector, M.A., Nehm, R.H., and Pearl, D. (2011). Lexical ambiguity in evolutionary discourse: Implications for teaching, learning, and assessment. Paper presented at the National Association of Research in Science Teaching, Orlando, FL

(3) Ha, M., Nehm, R.H. (2011). Comparative Efficacy of Two Computer-assisted Scoring Tools for Evolution Assessment. Paper presented at the National Association of Research in Science Teaching, Orlando, FL.

(2) Opfer, J. E. et al. (2011). Applying Cognitive Science to Evolution Education Assessment. Paper presented at the National Association of Research in Science Teaching, Orlando, FL.

(1) Nehm, R.H., Haertig, H. & Ridgway, J. (2010). Human vs. Computer Diagnosis of Mental Models of Natural Selection: Testing the Efficacy of Lexical Analyses of Open Response Text. Paper presented at the National Association of Research in Science Teaching, Philadelphia, PA. Nehm Haertig Ridgway NARST 2010.pdf

Presentations:

Federer, M.R. & Nehm, R.H. (2013). Using Rasch analysis to explore gender bias in written scientific explanations. Presented at the Society for the Advancement of Biology Education and Research, Minneapolis, MN, July 9-11.

Nehm, R.H., Beggrow, E., Ha, M., Rector, M.A., Opfer, J. (2012). Development and Evaluation of a New Multiple-True-False Concept Inventory for Diagnosing Students’ Mental Models of Natural Selection. SABER (Society for the Advancement of Biology Education Research) National Meeting. Minneapolis, Minnesota. July 12-15.

Rector, M.A., Nehm, R.H., Pearl, D., Opfer, J. (2012). Explanations as Scientific Practice: Exploring Bias in Constructed-Response Biology Assessments. SABER (Society for the Advancement of Biology Education Research) National Meeting. Minneapolis, Minnesota. July 12-15.

Ha, M., Nehm, R.H. (2012). Using Machine-Learning Methods to Detect Diverse Evolutionary Reasoning Patterns in Undergraduates’ Written Explanations. SABER (Society for the Advancement of Biology Education Research) National Meeting. Minneapolis, Minnesota. July 12-15.

Beggrow, E.P., Nehm, R.H. (2012). Students’ Mental Models of Evolutionary Causation: Natural Selection and Genetic Drift. SABER (Society for the Advancement of Biology Education Research) National Meeting. Minneapolis, Minnesota. July 12-15.

Ha, M. and Nehm, R.H. (2012). Computer Tools For Scoring Written Test Questions: Examples From Introductory Biology At OSU. Innovate! 2012. Columbus, OH.

Beggrow, E.P., Nehm, R.H. (2012). Beyond natural selection: Exploring the role of nonadaptive reasoning in undergraduate students' evolutionary explanations. 81st Annual Meeting of the American Association of Physical Anthropologists, Portland, OR.

Rector, M., Nehm, R.H., and Pearl. D. (2011). Constructing evolutionary explanations: Differences in word use and meaning by college biology majors. SABER (Society for the Advancement of Biology Education Research) National Meeting. Minneapolis, Minnesota. July 29-30.

Ha, M., & Nehm, R.H. (2011). Transforming biology assessment with machine learning and text analysis software: A case study using SIDE and SPSSTA. SABER (Society for the Advancement of Biology Education Research) National Meeting. Minneapolis, Minnesota. July 29-30.

Beggrow, E., Nehm, R.H., Opfer, J. and Ha, M. (2011). Assessing student reasoning about evolutionary concepts: Introducing the ACORNS. SABER (Society for the Advancement of Biology Education Research) National Meeting. Minneapolis, Minnesota.July 29-30.

Ha, M., & Nehm, R.H. (2011). Transforming biology assessment with machine learning and text analysis software: A case study using SIDE and SPSSTA. Joint annual meeting of the Society for the Study of Evolution (SSE), the Society of Systematic Biologists (SSB), and the American Society of Naturalists (ASN), Norman, OK, June 17-21.

Nehm, R.H., Ha, M. (2010). Measuring students' evolutionary understanding: context, coherence, and competence. Joint annual meeting of the Society for the Study of Evolution (SSE), the Society of Systematic Biologists (SSB), and the American Society of Naturalists (ASN), Portland, OR, June 25-29.

Ha, M., Nehm, R.H., Haertig, H., Ridgway, J. (2010). Computerized scoring of students' evolutionary essays: testing the efficacy of Text Analytic software. Joint annual meeting of the Society for the Study of Evolution (SSE), the Society of Systematic Biologists (SSB), and the American Society of Naturalists (ASN), Portland, OR, June 25-29.

Rector, M. Ha, M., Nehm, R.H., (2010). Evolutionary 'pressures': Biologists' conceptions and students' misconceptions. Joint annual meeting of the Society for the Study of Evolution (SSE), the Society of Systematic Biologists (SSB), and the American Society of Naturalists (ASN), Portland, OR, June 25-29.

Nehm, R.H., Ridgway, J., Gee, M., Baronda, S. (2010). Revolutionizing science assessment at OSU: Development and evaluation of an online assessment cascade system. Presentation at: Innovate! eLearning in Action conference. Columbus, Ohio, May 20.

Nehm, R.H., Ridgway, J. Haertig, H., Gee, M., Baronda, S., Opfer, J. Pearl, D. (2010). Transforming STEM assessment methodologies: Research on cyber-enabled measurement of cognitive models of natural selection. National Science Foundation REESE Conference. Arlington, VA, USA, March 10-12.

Nehm, R.H., Haertig, H. & Ridgway, J. (2010). Human vs. Computer Diagnosis of Mental Models of Natural Selection: Testing the Efficacy of Lexical Analyses of Open Response Text. Paper presented at the National Association of Research in Science Teaching, Philadelphia, PA.

Other Products: 

Websites for dissemination:

(1) Visit www.evolutionassessment.org to download products and view videos.

(2) Machine-learning system: visit www.evograder.org