«A Dissertation SUBMITTED TO THE FACULTY OF UNIVERSITY OF MINNESOTA BY Matthew Donald Beckman IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE ...»
ASSESSMENT OF COGNITIVE TRANSFER OUTCOMES FOR
STUDENTS OF INTRODUCTORY STATISTICS
SUBMITTED TO THE FACULTY OF
UNIVERSITY OF MINNESOTA
Matthew Donald Beckman
IN PARTIAL FULFILLMENT OF THE REQUIREMENTS
FOR THE DEGREE OF
DOCTOR OF PHILOSOPHYJoan Garfield, Advisor Robert delMas, Co-Advisor October 2015 © Matthew Donald Beckman, 2015 Acknowledgements Everything that matters in my life can be traced to God’s amazing grace, my sweet wife Sarah, and our children Eden and Jack. I am deeply grateful to Sarah for her constant support, encouragement, and inspiration on so many days and nights when I am away working or studying. Although only one of us will be named on the diploma, her hard work, sacrifice, and perseverance have contributed every bit as much to this accomplishment as mine. I can see no possible way that I would have or could have completed this challenge without her.
I am thankful to my dad, mom, sister, brother, and grandparents for setting such fantastic examples of lives lived well, and for teaching me from a young age to set big goals and work hard toward them. I also thank my employers, Medtronic, for many years of tuition support and professional development. Especially Tom Keenan, Pat Zimmerman, Michael Soma, Jared Hanson, Nancy Figueroa, Michelle Nelson, and Daryle Peterson who have shown endless patience to help me balance my work schedule with the demands of classes, teaching, and research. I am extremely grateful to students and faculty in the Statistics Education cohort at the University of Minnesota and to Laura Le, Sandy Weisberg, Roxy Peck, Sashank Varma, Tim Jacobbe, Beth Chance and Marsha Lovett for thoughtful feedback supporting my dissertation research. I thank Don Richards and Michelle Everson for taking an interest in a student they didn’t know and completely changing the trajectory of his life. Lastly, I thank my advisors, Joan Garfield and Bob delMas, for much guidance and encouragement, and most of all for taking a chance on a part-time student who just loves teaching statistics.
i Abstract This study chronicles the creation of an assessment tool that quantifies cognitive transfer outcomes for introductory statistics students. Literature suggested that outcomes associated with cognitive transfer are closely aligned with statistical thinking and are indicative of students’ ability to apply learning to novel scenarios beyond the classroom.
No assessment tool had been developed and published for the purpose of measuring cognitive transfer outcomes among statistics students. The results of this study suggest that the Introductory Statistics Understanding and Discernment Outcomes (I-STUDIO) assessment tool may effectively serve this purpose.
The assessment tool was developed according to a rigorous protocol of expert feedback and iterative piloting. Data were collected and analyzed from a nationwide sample of nearly 2,000 students attending a wide variety of post-secondary institutions, and the I-STUDIO instrument was found to measure both forward-reaching and backward-reaching high road transfer outcomes with good psychometric properties.
Data analysis indicated high reliability and diverse validity evidence. This evidence included confirmatory factor analysis models with compelling alignment to the theoretical model and analysis of qualitative themes among expert feedback. Analysis of scoring consistency also showed strong inter-rater agreement. Although the sample size of the scored responses is somewhat small by convention for item response theory, a graded response model generally showed good item functioning. Furthermore, the data suggested that the I-STUDIO assessment estimated student ability with consistent precision across a wide range of above-average and below-average students.
curricula. Additionally, the I-STUDIO instrument can be used to measure the effect of curriculum changes designed to improve transfer outcomes. Furthermore, the instrument and scoring rubric were designed to accommodate diverse curricula for the purpose of refining course outcomes.
List of Tables
List of Figures
1.1 Rationale for the Study
1.2 Problem Statement
1.3 I-STUDIO Assessment Tool
1.4 Structure of the Dissertation
2 Literature Review
2.1 Introduction to Literature Review
2.1.1 Consensus of traditional approach based on Normal distribution theory. 7 2.1.2 Summary of efforts to retool the introductory curriculum.
2.2 Cognitive Transfer Literature
2.2.2 Foundations of cognitive transfer research.
2.2.3 Development of schema and cognitive elements.
2.2.5 Cognitive load.
2.2.6 Strategies for the assessment of cognitive transfer.
2.3.1 Summary and critique.
2.3.2 Implications for teaching.
2.3.3 Implications for research.
2.3.4 Problem statement
3.1 Research Question
3.2 Study Overview
3.3 Instrument Development Cycle
3.3.1 Defining the construct for measurement.
3.3.2 Test blueprint
3.3.3 Item writing.
3.3.4 Iterative piloting process.
3.4 Data Analysis
3.4.1 Contribution of the instrument.
3.4.2 Rubric consistency.
3.4.3 Descriptive statistics
3.4.4 Reliability of the instrument
3.4.5 Validity of the instrument.
3.4.6 Item analysis
3.5 Chapter Summary
v 4 Results
4.2 Expert Reviewer Feedback
4.2.1 Contribution of the instrument.
4.2.2 Test blueprint
4.2.3 Draft I-STUDIO assessment tool.
4.3 Student Cognitive Interviews
4.3.1 Summary of feedback
4.3.2 Summary of changes to the instrument.
4.4 Field Test Data Analysis
4.4.1 Scoring rubric.
4.4.2 Descriptive statistics
4.4.4 Confirmatory factor analysis.
4.4.5 Item analysis
5.1 Study Summary
5.2 Synthesis of Results
5.2.1 General comments from expert feedback
5.2.2 Evidence of quality of the I-STUDIO assessment tool.
5.4 Implications for Teaching
5.5 Implications for Future Research
Appendix A: Test Blueprint Prior to Expert Feedback
Appendix B: Expert Feedback Questionnaire Accompanying Test Blueprint............... 162 Appendix C: Final Test Blueprint
Appendix D: Draft I-STUDIO Version Prior to Expert Feedback
Appendix E: Expert Feedback Questionnaire Accompanying Draft I-STUDIO Assessment Tool
Appendix F: I-STUDIO Version for Cognitive Interviews
Appendix G: I-STUDIO Version for Field Test
Appendix H: I-STUDIO Draft Scoring Rubric
Appendix I: I-STUDIO Final Scoring Rubric for Field Test
Appendix J: I-STUDIO Scoring Rubric Use Instructions
Table 1 Experimental Treatment Groups in Text Editor Experiment
Table 2 I-STUDIO Development Timeline
Table 3 Example of items classified by assessment goals
Table 4 Distribution of usable responses by institution
Table 5 Homogeneous item groups for split-half reliability estimation
Table 6 Distribution of expert feedback for questions 1 and 2 (blueprint questionnaire) 74 Table 7 Distribution of expert feedback for questions 3-7 (blueprint questionnaire)...... 77 Table 8 Distribution of expert feedback for questions 8-13 (blueprint questionnaire).... 80 Table 9 Distribution of expert feedback for questions 2-10 (draft assessment questionnaire)
Table 10 Distribution of expert feedback for questions 9 and 10 (draft assessment questionnaire)
Table 11 Inter-rater agreement by scoring element
Table 12 I-STUDIO summary statistics by item and testlet.
Table 13 Summary statistics of I-STUDIO scores
Table 14 Confirmatory factor analysis (CFA) fit diagnostics for independent item models
Table 15 Parameter estimates for 2LV-Transfer model fit
Table 16 Confirmatory factor analysis (CFA) fit diagnostics for correlated item models
Table 17 Parameter estimates for 2LV-Corr model fit
Table 19 Confirmatory factor analysis (CFA) fit diagnostics for testlet models............ 115 Table 20 Parameter estimates for 2LV-Testlet model fit
Table 21 Item fit diagnostics associated with MIRT graded response model................ 118 Table 22 Factor loadings associated with MIRT graded response model
Table 23 I-STUDIO graded response model coefficient estimates
Table 24 Flawed example responses (verbatim) to several I-STUDIO items................ 125 Table 25 Probability of Difference Reversal with Repeated Testing for Classes of 25 Students
Figure 1. Conceptual model of I-STUDIO outcomes.
Figure 2. Histogram of I-STUDIO total scores.
Figure 3. Mean scores with 95% confidence intervals by course ID.
Figure 4. 500,000 simulated Spearman-Brown split-half reliability estimates with 95% confidence interval based on 0.
025 and 0.975 quantiles.
Figure 5. Confirmatory factor analysis model CFA-2
Figure 6. Confirmatory factor analysis model for testlet data on two dimensions.
........ 117 Figure 7. I-STUDIO Test information curves for Forward-Reaching and BackwardReaching transfer dimensions.
Figure 8. I-STUDIO item information curves.
Figure 9. I-STUDIO option response functions for Backward-Reaching transfer testlets.
Figure 10. I-STUDIO option response functions for Forward-Reaching transfer testlets.
1.1 Rationale for the Study Statistical thinking has been described in part to concern comprehension of “how, when, and why” a statistical framework can inform some inquiry (Ben-Zvi & Garfield, 2005). In learning and cognition research, an important mechanism by which students accomplish this sort of comprehension is sometimes referred to as cognitive transfer—or simply transfer. Singley and Anderson (1989) defined transfer to concern “how knowledge acquired in one situation applies (or fails to apply) in other situations.” Similarly, Perkins and Salomon (1988) described transfer as “knowledge or skill associated with one context reach[ing] out to enhance another.” Additionally, researchers noted a number of specific types of transfer including vertical transfer, near transfer, far transfer, and negative transfer (Bransford, Brown, & Cocking, 2000; Perkins & Salomon, 1988; Singley & Anderson, 1989).