Making Math Tutors More Engaging and Effective through Interaction Design Patterns and Educational Data Mining

Principal Investigator: 
Project Overview
Background & Purpose: 

Recent years have seen substantial increases in usage of online software for mathematics education, yielding a wealth of data concerning student learning processes and effectiveness of different software elements (e.g. variations in interface and content). This study aims to use such data to identify design features associated with optimal learning and engagement, and develop Design Patterns (i.e. structured definitions of high-quality design solutions) that facilitate effective mathematics problem-solving online. It is hoped that resulting design patterns will serve as empirically supported prototypes upon which effective online mathematics problems can be designed, as well as contributing to existing knowledge on characteristics of effective online instruction.

Setting: 

Public schools in Massachusetts and Maine.

Research Design: 

The project uses a mixture of methods. In some cases, correlational evidence is generated from existing data, and in other cases experimental methods are used to obtain causal evidence. Original data are being collected on middle school students (grades 6-8) using assessments of learning, field observation, and log data.

The ASSISTment system is a web-based middle-school mathematics tutor incorporating diverse problems designed by educational designers and classroom teachers. This system is being used by over 50,000 students in 2013-2014. Learning is facilitated through formative assessment as problems across a broad range of topics are completed. Where errors are made, scaffolding questions, which break problems down into cognitive steps, are provided. This enables identification of individual steps with which students are struggling so that additional exercises can be selected for a particular student and assessment reports can be generated to aid teachers in providing in-class help. The ASSISTment system also generates logs that have been used in many prior analyses and enables running of automated experiments.

Design patterns are structured descriptions of high quality design solutions to recurring problems in a particular domain. Existing mathematics problems will be examined and design features (e.g. degree of difficulty, feedback and usability) from different problems will be enumerated. To identify effective design patterns we will use educational data mining to validate which features or combinations of features are associated with superior learning and engagement. Design features (e.g. those relevant to interface design, domain content and pedagogical strategies) present in existing mathematics problems will first be hand-labeled for a subset of problems before educational data mining is used to replicate these labels on a massive scale. We will then apply automated detectors of student learning, engagement and affect to assess the effectiveness and engagingness of over 20 000 mathematics problems. The design feature combinations that best cultivate learning, engagement, and affect will be identified using association rule mining on collected data.

Mathematics problems will then be improved based upon the design features, and the improved problems will be tested in randomized controlled trials deployed to students automatically. Improved problems will be evaluated in terms of their impacts on student learning and engagement.

Findings: 

Findings will be posted as they become available.

Other Products: 

Design patterns for the creation of engaging and effective online mathematics problems will be developed and made publicly available.