Poverty Action Lab Calls ASSISTments One of Two Most Promising Interventions

‍A study conducted at MIT's Poverty Action Lab was recently featured in U.S. News & World Report that hailed ASSISTments as one of two promising educational technologies in the US. The MIT investigation conducted a review of 100+ educational technology studies, and concluded that ASSISTments was one of the two most promising interventions in the United States due to the fact that it increased student learning and does not require that schools adjust their curriculum or textbooks, and is available free of charge.

The U.S. News & World Report article by Jill Barshay said “ASSISTments improved seventh-graders’ math scores in Maine when students spent only 10 minutes a night on the software, three or four times a week, as homework. Teachers didn’t have to change their existing lesson plans or textbooks to incorporate it."
‍
The reason for this is that WPI put the answers for the 20 math textbooks used by the Maine schools into ASSISTments in a way that did not violate the textbook published copyright. This made it very easy for the teachers to adopt this technology as they did not have to radically change their curriculum and teaching.

Quote from the MIT review:

"ASSISTments represents an especially promising example. ASSISTments is a math homework platform released by the Worcester Polytechnic Institute that does not require that schools adjust their curriculum or textbooks, and is available free of charge. The program is designed to carry out “formative assessments,” i.e., to use “data from students’ independent work to give them helpful feedback and guidance while enabling the teacher to use the data to adjust instruction to meet students’ learning needs.” As students work through individual problems, the computer informs them about whether their answer is correct and offers guidance if necessary. Students are expected to benefit from the customized practice, the rapid feedback of responses, as well as the data supplied to teachers (in addition to, in some cases, supplementary professional development to train the teachers on optimizing use of ASSISTments). Two small-scale proof-of-concept studies found promising effects, but these studies had samples numbering only in the dozens of students and implementation time numbering only in the days.

More recently, however, a full-scale impact evaluation of an ASSISTments intervention was conducted with a sample of 2,850 seventh-graders across 43 schools in Maine. The authors found that the program improved math scores for treatment students by 0.18 standard deviations. This impact is particularly noteworthy given that treatment students used the program on average for less than ten minutes per night, three to four nights per week. It is worth noting that the program depends on students’ ability to access a laptop or tablet.

This is part of the reason that this evaluation was conducted in Maine, given its state policy of lending laptops to students. While this hurdle may raise some external validity concerns with regard to this particular study, a variety of possibilities exist for enabling access in other states, especially given that software and licensing are free so costs are otherwise low. Also noteworthy is that impact was significantly stronger for students at or below median than for those above, with an effect size of 0.29 standard deviations."
‍
Heffernan said, "The SRI evaluation found that the size of the effect of ASSISTments doubled the amount of learning, compared to how much learning normally occurring on the standardized test. I would have been thrilled to simply show we improved student learning by 10%, but SRI found that we improved students by 100%. However, the thing that makes my wife, and co-creator, and I most proud of is that we built a tool that helps the lower performing students catch up while also helping the higher knowledge students. Every policy maker that is thinking about solving the digital divide in their community by making sure students from lower socio-economic backgrounds can have access to technology for homework, needs to know that this study showed that. By combining something like ASSISTments with their technology they can expect large learning gains. Thousands of suburban districts are already assigning ASSISTments-supported homework, but more financially vulnerable schools can't do this unless policy makers decide that solving the digital divide is important."

The MIT review concluded that, "Two interventions in the United States stand out as being particularly promising—a fairly low-intensity online program that provides students with immediate feedback on math homework ... In light of the promising evidence, more research is needed to understand the mechanisms behind computer-assisted learning, specifically how software interacts with teachers and current curriculum."
‍
Professor Heffernan says "Companies are selling ed tech products to teach kids, they are not selling products built to help teachers learn if their own lessons were effective. We want teachers to use our tool to review homework differently. Teachers need to see what was hard on last night's homework. They need data that allows them to discuss common wrong answers students may have had; this video shows such a conversation."

Barshay cited Heffernan's approach "is unusual in the education technology industry. He financed the development of his software with federal funds and gives it away free to schools. The federal government, through the U.S. Department of Education’s Institute for Education Sciences, also foots the bill for the studies to see if his software improves math education. Heffernan now has grants to test whether the Maine results can be replicated in North Carolina, and whether teachers can be trained online to use the system across the country, especially in more urban settings."

Neil Heffernan said, "My purpose in life is to continue to make this a 100% free university-based platform and to make it even better over time by building the technology to learn how to share what teachers are doing in the platform with each other. Some teachers are writing Youtube feedback messages for their textbook's questions and we want to learn which feedback works best for which students. We want to work with teachers to create the best feedback possible and to make it free of charge."