The research design for this project is longitudinal, and is designed to generate evidence that is descriptive through observation, and associative/correlational through diaries, problem-solving activities, and concept mapping. This project collects original data using diaries/journals/records kept by study subjects, school records/policy documents, assessments of learning/achievement tests, and face-to-face structured interviewer-administered questionnaires.
The following section briefly describes the scoring and coding for each measure described: CMs, problem-sets, and diaries. For each measure, all scoring and coding will be conducted by one experimenter, and at least 40% of all measures will be scored and coded by an additional experimenter to assess inter-rater reliability. Our previous studies have demonstrated good inter-rater reliability with each measure described.
Concept Maps (CMs). CMs will be scored in a similar manner to previous investigations from our labs. CMs for both students and instructors will be scored for their propositions (number, accuracy, and crosslinks) and number of hierarchy levels. Thus, each CM will be given scores on the basis of these four measures and an overall score. Accuracy of propositions will be assessed using proposition inventories for each learning unit. These inventories will contain all possible relations between pairs of concepts for a given learning unit. For each relation, propositions are scored on a 5-point scale, from 0 for inaccurate to 4 for excellent. Inventories will be created using required textbooks and verified by our organic chemistry consultants.
Problem-Sets. Participants will be requested to think-aloud while solving problems from each problem-set. Response time and accuracy data (based on WE-LEARN problem-set data) will be collected for each problem. Think-aloud protocols will be coded according to previous studies conducted in our labs. Such studies coded protocols according to processing categories, such as ‘superficial’, ‘integrating’, and ‘connecting’. For example, superficial processing could include merely reading the question over, or taking for granted that a mechanism is correct without verifying. Integrative processing could include emphasizing relations between concepts in a problem. Connective processing could include recognizing relevant prior knowledge assessed by the problem. Separate category scores (or strategy scores) will be assigned for each problem and overall for each participant.
Daily Diaries. Diaries will be coded according to previous studies conducted by our labs. Studying activities will be coded according to SRL strategies, such as ‘reviewing materials’, ‘organizing’ and ‘seeking information’. For example, ‘seeking information’ could include clarifying uncertainties using non-assigned materials, such as online resources or library books. For each week, strategy presence, frequency and duration (in hours) will be calculated for each strategy.
Design, Analyses and Expected Results. This study will examine its four specific aims in a 3x2x2x4 [prior achievement level (low, medium, high) x ethnicity (minority/non-minority)x gender (female/male) x occasion (four occasions/units)] split-plot design with repeated measure on the last factor. SP-ANOVAs are conducted to examine main effects and interactions on, for example, knowledge structures (CM scores/ discrepancy scores), problem-solving, and studying activities. For those questions involving the identification of strategies for problem solving and studying, principle-component analyses will be conducted to identify interpretable clusters for further analyses. Moreover qualitative analyses will be done to characterize, for example, misconceptions apparent in CMs. A final analysis will regress end-of-course grades on prior achievement, CM discrepancy score, and problem-solving and studying composites to estimate their relative contributions to “course success.”