Effectiveness of Pedagogical Agents in Regulating Students' Understanding of Science

Principal Investigator: 
Project Overview
Background & Purpose: 

This award focuses on examining the effectiveness of using animated pedagogical agents (APAs) as external regulatory agents designed to foster middle school and college students' understanding of complex and challenging science topics (e.g., the circulatory system). The focus of our grant is on conducting interdisciplinary research examining: (1) the role of embedded animated pedagogical agents in collecting data of the complex interactions between cognitive and metacognitive processes during learning about complex science topics with MetaTutor; (2) the effectiveness of animated pedagogical agents as external regulating agents used to detect, trace, model, and foster students' self-regulatory processes during learning about complex science topics with MetaTutor; and (3) the effectiveness of scaffolding methods delivered by animated pedagogical agents in facilitating middle school and college students' self-regulated learning about complex science topics with MetaTutor.

Setting: 

Laboratory setting with non-science majors and high school students (10th-12th grade) in the Mid-South (Memphis, TN).

Research Design: 

The research design for this project is comparative, and is designed to generate evidence that is descriptive (observational and concurrent think-aloud and log-file trace data) and causal (experimental). This project includes the training of cognitive and metacognitive self-regulatory strategies for learning about complex science topics as an intervention, with a comparison condition consisting of no training of cognitive and metacognitive self-regulatory strategies for learning about complex science topics (i.e., reading about some other topic to account for time on training regiment). This project collects original data using assessments of learning/achievement tests; observation (videography and web logs); and online and paper and pencil survey research. The following data analytic approaches and methods are used: (1) inferential statistics (e.g., ANCOVAs) to examine pretest-posttest differences in performance measures; (2) non-inferential statistics (e.g., chi-squares) to examine some of the process data from the concurrent think-aloud protocols; (3) different regression analyses to determine if certain predictor variables are associated with certain learning outcomes; (4) think-aloud protocol analyses; (5) cluster analysis to examine navigational paths to ascertain learner profiles and their relation to certain outcomes measures; (6) log-file trace data analysis from the MetaTutor, an AI-based system that collects hundreds of data points during learning; and, (7) several methods and approaches from computational linguistics and machine learning algorithms to determine students’ understanding of the science topics, manage the dialogue between students and embedded agents, grade mental model essays, and to determine the micro- and macro-adaptive tutoring strategies to be deployed by the animated agents during learning.

Findings: 

Product Data—Learning Outcomes
This is a very brief summary of the learning outcomes data from the college students. In relation to research question one, the data show that students in the Training condition scored significantly higher on the SRL quiz after training on the SRL processes and were also able to maintain their knowledge of the SRL processes ([t (32) = - 5.02, p < .05]). As for the second research question, preliminary results also indicate that there was no statistically significant difference between conditions for the matching task of the circulatory system [t (32) = .82, p < .05], however, there were significant differences between groups on the labeling task ([t (32) = 2.02, p < .05]) and multiple choice of the circulatory system [t (32) = 1.85, p < .05]. Participants in the training condition outperformed those in the control condition. We found the same pattern of statistically significant results on the three parts of the nervous system (p < .05).

Process Data—Concurrent Think-Aloud Processes
We have also transcribed, coded, re-coded the think-aloud data for several days of the experiment. Here is a brief summary of the process data. We calculated several independent t-tests on the means of the coded SRL processes used by the learners in each of the two conditions. Results indicate that learners in the MetaTutor SRL Training condition deployed significantly the training condition engaged in more activation of prior knowledge activation, recycled goals in the working memory (WM), monitored their emerging understanding by using (positive) judgments of learning, monitored their progress towards goals during the learning session, and used knowledge elaboration as a learning strategy.

In keeping with our goal of deriving instructional implications, it is important to highlight a few key observations. First, the variability reported in some key SRL process such as creating sub-goals, feeling of knowing, coordinating informational sources, summarizing, and taking notes is surprising since these were four of the thirteen SRL processes used in the SRL training regimen. The five processes are quite difficult to acquire and use immediately following the training regimen. As such, the adaptive version of MetaTutor should be able to detect, model, and provide extensive scaffolding regarding the use of the processes.

Second, the low raw frequencies observed is some SRL processes related to planning (planning, time and effort planning), monitoring processes (content evaluation, monitoring use of strategies, self-questioning), use of effective strategies (reading notes), and handling task difficulties and demands (help-seeking) has implications for the design of the adaptive version of MetaTutor. These processes have repeatedly shown to be associated with learning gains in several hypermedia learning studies [e.g., Azevedo, 2007, 2008, 2009]. For example, content evaluation is a key monitoring process that is used by learners (when using non-linear, multi-representational hypermedia learning environments) to compare current external representation of information (i.e., text, diagram, animation) with their current learning goal. This can lead to two situations—1) there is no discrepancy (content evaluation positive; CE+) and therefore the representation is ideal for the current goal, or 2) there is a discrepancy (CE-) and therefore there is a need to select or search the environment for an appropriate external representation. The issue of valence leads to a complex issue regarding adaptivity in MetaTutor since the system must be able to “map” each representation to the current goal and determine when and what type of scaffold to administer (e.g., do you think this diagram of the heart fits with your current goal), in which case the scaffold prompt is generic and is designed to raise a learner’s metacognitive awareness. The system can then wait for the learner’s response and instruct him/her to indicate (see the SRL palette in Figure 1; right-hand side of the screen) their content evaluation. This would then lead to various dialogue moves and corresponding feedback cycles necessary to search for and settle on an ideal representation [see 18]. A related issue, related to socio-cognitive models of SRL [Zimmerman, 2006] is the need for MetaTutor to continuously model these key (yet difficult processes to acquire and internalize) during key phases of learning with MetaTutor. For example, Mary the Monitor can model CE when the system recognized that the current content is complex (e.g., what is the difference between systemic circulation and pulmonary circulation) and the leaner needs to select several diagrams and sequence them in order to foster mental model development.

Navigation Paths
MetaTutor also traces learners’ navigational paths during each learning session (see Witherspoon et al., under review for a complete analysis). The work here it emphasizing the need to examine how various types of navigational paths are indicative (or not) of strategic behavior expected from self-regulating learners (Winne, 2005).

Figures 2a-2d illustrate the navigational paths of two learners from our dataset while they use MetaTutor to learn about the circulatory system. In figure 2a and 2b, the x-axis represents move x and the y-axis represents x+1. Figure 2a shows the path of a low-performer (i.e., small pretest-posttest learning gains) while Figure 2b illustrates the path of a high-performer. These figures highlight the qualitative differences between a low- and high-performer in terms of the linear vs. complex navigational paths and reading times. For example, the low-performer tended to progress linearly through the content until they got to a key page (e.g., page 16 on blood vessels) and decided to return to a previous page. In contrast, the high-performer’s path is more complicated and is more consistent with a strategic, self-regulated learner by the complexity shown in Figure 2b. This learner progresses linearly and at times makes strategic choices about returning to previously visited pages, and deployed twice as many SRL processes as the low-performing learner (i.e., 212 moves vs. 102 moves, respectively). This is symbolically illustrated in the Figure by the difference between the “space” between the dots—i.e., more space between dots = longer reading times.

The SRL processes deployed by these were hand-written on their navigational paths and presented in Figures 2c and 2d. In Figures 2c and 2d, the x-axis represents time (in minutes) within the learning session and the y-axis represents pages of content (and their corresponding titles). There are several key observations to highlight in terms of keeping with our goal of extracting information for the design of the adaptive MetaTutor. First, there is more complexity in the navigational paths and deployment of SRL processes as seen in the number of processes hand-written in the figures. Second, 70% of the low-performer’s SRL moves were coded as taking notes while the high-performer only used 39% of his processes for taking notes. Third, one can infer (from the “space” between moves) that the low-performer spent more time acquiring knowledge from the environment while the high-performer spent less time reading throughout the session. Fourth, the high-performer used a wider variety of SRL processes compared to the low-performer. A related issue is the non-strategic move by the low-performer to create a new sub-goal near the end of the learning session. However, the high-performer is more strategic in his self-regulatory behavior throughout the learning session. For example, he engages in what can best be characterized as “time-dependent SRL cycles”. These cycles involve creating sub-goals, previewing the content, acquiring knowledge from the multiple representations, taking notes, reading notes, evaluating content, activation prior knowledge, and periodically monitoring their understand of the topic.

Publications & Presentations: 

Azevedo, R. (in prep). Computers as MetaCognitive tools for enhancing learning: International perspectives. Cambridge University Press.

Azevedo, R., & Witherspoon, A. (under review). Regulating the learner? Advances in the science of using external regulatory agents to facilitate complex learning with advanced technologies. In M. Khine & I. Saleh (Eds.), New science of learning: Computers, cognition, and collaboration in education. Amsterdam, The Netherlands: Springer.

Azevedo, R., & Witherspoon, A.M. (in press). Self-regulated learning with hypermedia. In A. Graesser, J. Dunlosky, D. Hacker (Eds.), Handbook of metacognition in education. Mahwah, NJ: Erlbaum.

Azevedo, R. (in press). Theoretical, methodological, and analytical challenges in the research on metacognition and self-regulation: A commentary. Metacognition and Learning.

Azevedo, R. (2008). The role of self-regulation in learning about science with hypermedia. In D. Robinson & G. Schraw (Eds.), Recent innovations in educational technology that facilitate student learning (pp. 127-156). Charlotte, NC: Information Age Publishing.

Azevedo, R., & Jacobson, M. (2008). Advances in scaffolding learning with hypertext and hypermedia: A summary and critical analysis. Educational Technology Research & Development, 56(1), 93-100.

Witherspoon, A.M., & Azevedo, R. (under review). The dynamic nature of self-regulatory processes during self- and externally-regulated learning episodes. Cognition and Instruction.

Witherspoon, A.M., & Azevedo, R. (under review). The role of multiple representations in learning with hypermedia. Learning and Instruction.

Azevedo, R., Witherspoon, A. (in prep). The temporal dynamics of self-regulatory processes during learning with hypermedia: A micro-analysis, Journal of the Learning Sciences.

Jeon, M., & Azevedo, R. (in prep). Using CohMetrix to analyze the complex nature of human tutorial dialogues during hypermedia learning. Discourse Processes.

Azevedo, R., & Witherspoon, A. (2008). Detecting, tracking, and modeling self-regulatory processes during complex learning with hypermedia. In A. Samsonovich (Ed.), Proceedings of the AAAI Fall Symposium on Biologically Inspired Cognitive Architectures (p. 16-26). Menlo Park, CA: Association for the Advancement of Artificial Intelligence (AAAI) Press.

Azevedo, R., Witherspoon, A., Lewis, G., & Siler, E. (2008). The role of prior knowledge and system structure on self-regulated learning with hypermedia [Abstract]. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (p. 2351). Austin, TX: Cognitive Science Society.

Jeon, M., & Azevedo. (2008). Automatic analyses of cohesion and coherence in human tutorial dialogues during hypermedia learning: A comparison among mental model jumpers. In B. Woolf, E. Aimeur, R. Nkambou, & S Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems: Lecture Notes in Computer Science (LNCS 5091)(pp. 690-692). Berlin: Springer.

Witherspoon, A., Azevedo, R., & D’Mello, S. (2008). The dynamics of self-regulatory processes within self- and externally-regulated learning episodes. In B. Woolf, E. Aimeur, R. Nkambou, & S Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems: Lecture Notes in Computer Science (LNCS 5091)(pp. 260-269). Berlin: Springer.

 

Conference Presentations

Azevedo, R. (August, 2009). Measuring and modeling metacognitive processes using on-line methods: Theoretical and methodological issues. Paper to be presented at an invited symposium at the biennial meeting of the European Association for Research on Learning and Instruction, Amsterdam, The Netherlands.

Azevedo, R., & Witherspoon, A.M. (August, 2009). A critical analysis of multi-method approaches. Paper to be presented at an invited symposium at the biennial meeting of the European Association for Research on Learning and Instruction, Amsterdam, The Netherlands.

Azevedo, R., Witherspoon, A.M., Siler, E., Cox, M., Chauncey, A., Graesser, A., McNamara, D., Lintean, M., Cai, Z., & Rus, V. (August, 2009). The effectiveness of MetaTutor in training college students to deploy key self-regulatory processes during learning. Paper to be presented at the biennial meeting of the European Association for Research on Learning and Instruction, Amsterdam, The Netherlands.

Azevedo, R., & Witherspoon, A. (April, 2009). Capturing, identifying, and classifying the deployment of self-regulatory processes during learning with MetaCognitive tools. Paper to be presented at annual meeting of the American Educational Research Association, San Diego, CA.

Azevedo, R., & Witherspoon, A. (April, 2009). The effectiveness of pedagogical agents in orienting learners to deploy key cognitive and metacognitive processes during hypermedia learning. Paper to be presented at annual meeting of the American Educational Research Association, San Diego, CA.

Azevedo, R., Witherspoon, A., Graesser, A., McNamara, D., Rus, V., Cai, Z., & Lintean, M. (November, 2008). MetaTutor: An adaptive hypermedia system for training and fostering self-regulated learning about complex science topics. Paper presented at annual meeting of the Society for Computers in Psychology, Chicago, IL.

Azevedo, R., & Witherspoon, A. M. (November, 2008). Detecting, tracking, and modeling self-regulatory processes during complex learning with hypermedia. Paper presented at the annual meeting of the American Association for Artificial Intelligence Symposium on Biologically-Inspired Cognitive Architectures, Washington, D.C.

Azevedo, R. (March, 2008). Intelligent multi-layered regulatory learning environments for fostering complex learning. Paper presented at the annual meeting of the American Educational Research Association, New York, NY.

Other Products: 

New AI-based hypermedia science learning environment with embedded animated pedagogical agents designed to trace, model, and foster students’ cognitive and metacognitive self-regulatory processes.