Evidence Of Impact

This individual random assignment study tested the impact of immediate vs. delayed feedback (an “active ingredient” of ASSISTments) on student learning among middle grades mathematics students (Grades 7 and 8). This study randomly assigned 63 students in Massachusetts to a traditional homework condition (TH) or a web-based homework (WBH) condition (Kelly, Heffernan, Heffernan, Goldman, Pellegrino, & Goldstein, 2013). All students entered their homework into the ASSISTments system and were assigned the same problem set. However, all feedback features were disabled for students assigned to the TH condition. In other words, WBH students were given correctness-only feedback at the end of each problem; TH students were simply told that their answer was recorded but were not provided any information about the correctness of their response. Students completed a brief five-item pre-test prior to random assignment, a post-test on the day following the lesson and a second post-test one day later. WBH students did not differ significantly from TH students on the pretest but significantly outperformed TH students on Post-tests 1 and 2 (effect sizes 0.52 and 0.56).

Kelly K., Heffernan N., Heffernan C., Goldman S., Pellegrino J., Soffer Goldstein D. (2013) Estimating the Effect of Web-Based Homework. In: Lane H.C., Yacef K., Mostow J., Pavlik P. (eds) Artificial Intelligence in Education. AIED 2013. Lecture Notes in Computer Science, vol 7926. Springer, Berlin, Heidelberg.
‍

This small-scale study conducted a test using a randomized, controlled crossover within-subjects design (Kehrer, Kelly & Heffernan, 2013). A total of 65 seventh-grade mathematics students in a suburban middle school in Massachusetts were blocked on prior knowledge and randomly assigned to a delayed or immediate feedback condition. Students in the immediate feedback condition entered their homework answers into the ASSISTments system and were provided immediate feedback about the correctness of their answers; their delayed-feedback counterparts entered answers from their worksheet into ASSISTments the following day. Similar to the first study, students completed a pre- and post-test. The following week, students switched conditions for a new lesson (students who previously received immediate feedback now entered their responses on a worksheet and received delayed feedback; students who previously received delayed feedback now received immediate feedback). The study found that students learned 12% more when given immediate feedback than when given delayed feedback (effect size of 0.37)
‍
Kehrer, P., Kelly, K. & Heffernan, N. (2013). Does immediate feedback while doing homework improve learning. In Boonthum-Denecke, Youngblood (Eds.), Proceedings of the Twenty-Sixth International Florida Artificial Intelligence Research Society Conference (pp. 542-545). AAAI Press
‍

When ASSISTments was given a $500,000 grant from the Gates Foundation, they used SRI as an outside evaluator to study the effectiveness of the 17 different products the Gates Foundation funded. For an assessment tool, SRI chose to use the NWEA's computer-adaptive test called Measures of Academic Progresses (MAPs). The schools that adopted ASSISTments saw a reliable improvement in their growth scores of .03 standards deviations. To translate that into more meaningful terms, we note the average growth on MAPs at seventh grade is about .3 standard deviations, so that is about a 10% improvement in learning.
‍

This article presents a quasi-experiment that evaluates whether ASSISTments use has an effect on improving middle school students’ year-end test scores. The data was collected from 1240 seventh graders in three treatment schools and one comparison school. Post-test (7th grade year-end test) results indicate, after adjusting for the pre-test (6th grade year-end test), that students in the treatment schools significantly outperformed students in the comparison school and the difference was especially present for special education students. A usage analysis reveals that greater student use of ASSISTments is associated with greater learning consistent with the hypothesis that it is useful as a tutoring system. We also found evidence consistent with the hypothesis that teachers adapt their whole class instruction based on overa student performance in ASSISTments. Namely, increased teacher use (i.e., having more students use the system more often) is associated with greater learning among students with little or no use, suggesting that those students may have benefited from teachers adapting their whole-class instruction based on what they learned from ASSISTments use reports. These results indicate potential for using technology to provide students instruction during assessment and to give teachers fast and continuous feedback on student progress.
‍
Koedinger, K., McLaughlin, E. & Heffernan, N. (2010). A Quasi-Experimental Evaluation of an On-line Formative Assessment and Tutoring System. Journal of Educational Computing Research. Baywood Publishing. 4, 489-510
‍

This study compared learning for fifth grade students in two math homework conditions. The paper-and-pencil condition represented traditional homework, with review of problems in class the following day. The Web-based homework condition provided immediate feedback in the form of hints on demand and step-by-step scaffolding. We analyzed the results for students who completed both the paper-and-pencil and the Web-based conditions. In this group of 28 students, students learned significantly more when given computer feedback than when doing traditional paper-and-pencil homework, with an effect size of .61. The implications of this study are that, given the large effect size, it may be worth the cost and effort to give Web-based homework when students have access to the needed equipment, such as in schools that have implemented one-to-one computing programs.
‍
Mendicino, M., Razzaq, L., & Heffernan, N. T. (2009). Improving learning from homework using intelligent tutoring systems. Journal of Research on Technology in Education, 41(3), 331-346. 
‍