E-TRIALS
Ed Tech Research Infrastructure to Advance Learning Science Research
Originally conceived as a research initiative at Worcester Polytechnic Institute, ASSISTments has since transformed into a highly regarded tool employed by thousands of educators. Because our primary goal revolves around enhancing mathematics education and fostering a more research-backed learning environment for students, we place significant emphasis on supporting researchers.
.svg)
Worcester Polytechnic Institute
Large Scale RCTs
Examples of large scale studies that use ASSISTments.
.svg)
A Systematic Replication Study of Interleaved Mathematics Practice
The purpose of this project is to conduct a systematic replication study of a highly promising mathematics learning intervention, interleaved practice, in 7th grade classrooms. Though psychologists have long known that interleaving and spacing improves long-term learning, the practice problems in most mathematics curricula are arranged so that the majority of problems relating to the same skill or concept are blocked together. With the interleaved practice intervention, some of the practice problems are rearranged so that 1) problems of different kinds are mixed together, which improves learning, and 2) problems of the same kind are distributed across multiple assignments, which improves retention. Numerous studies in the laboratory and classroom have demonstrated that merely rearranging practice problems so that the students receive a higher dose of interleaved practice can dramatically boost scores on researcher-developed measures of learning. The systematic replication study will determine whether this promising intervention can improve scores on externally-developed outcome measures and whether these intervention can scale to a widely-used online intervention that currently reaches tens of thousands of students in diverse settings.
From Here to There! A dynamic algebraic notation system improves understanding of equivalence in middle-school students.
Understanding equivalence is fundamental to STEM disciplines, yet misunderstandings and misconceptions inhibit students from fully appreciating or leveraging the concept. Using the game-based algebraic notation system, From Here to There! (FH2T), students explore ideas of equivalence by dynamically transforming expressions or equations among mathematically equivalent states. In the fall of 2019, 475 middle-school students participated in a randomized control trial where they worked in either FH2T or online problem sets with hints and feedback in ASSISTments over four 30-min sessions during their math class. We found that (a) students in both conditions improved their understanding of mathematical equivalence from pretest to posttest, (b) students in the FH2T condition performed better on posttest compared to students in the problem set condition, and (c) the condition effect was comparable between students with high versus low prior knowledge. Together, the findings suggest that FH2T is an effective intervention for improving middle-school students’ understanding of mathematical equivalence. The implications for research and practice on the usefulness of digital environments in mathematics education are discussed.
Chan, J. Y.-C., Lee, J.-E., Mason, C. A., Sawrey, K., & Ottmar, E. (2022). From Here to There! A dynamic algebraic notation system improves understanding of equivalence in middle-school students. Journal of Educational Psychology, 114(1), 56–71.
Personalizing Mathematics to Maximize Relevance and Skill for Tomorrow's STEM Workforce
We developed ASSISTments assignments that personalize algebra instruction to 8 career and 4 popular culture interest areas. A YouTube channel hosts 386 videos of STEM practitioners or popular culture interest enthusiasts telling stories of algebra in daily life and 12 more videos of students and teachers working with our materials in classrooms. These videos are organized by math and interest topics on our Stories of Algebra website, which also contains 225 templates students can use to consider interest areas and algebraic equations while building math problems.
Find our Stories of Algebra website here: https://sites.google.com/view/stories-of-algebra
Learn about the project here: https://stemforall2021.videohall.com/presentations/1917?display_media=video
The effect of language modification of mathematics story problems on problem-solving in online homework
Students’ grasp of the non-mathematical language in a mathematics story problem—such as vocabulary and syntax—may have an important effect on their problem-solving, and this may be particularly true for students with weaker language skills. However, little experimental research has examined which individual language features influence students’ performance while solving problems—much research has been correlational or has combined language features together. In the present study, we manipulated six different language features of algebra story problems—1) number of sentences, 2) pronouns, 3) word concreteness, 4) word hypernymy, 5) consistency of sentences, and 6) problem topic—and examined how systematically varying readability demands impacts student performance. We examined both accuracy and response time measures, using an assignment for learning linear functions in the ASSISTments online problem-solving environment. We found little evidence that individual language features have a considerable effect on mathematics word problem solving performance for a general population of students. However, sentence consistency reduced response time and problems about motion or travel had shorter response times than problems about business or work. In addition, it appears students may benefit or be harmed by language modifications depending on their familiarity with ASSISTments. Implications for the role of language in math word problems are discussed.
Walkington, C., Clinton, V., & Sparks, A. (2019). The effect of language modification of mathematics story problems on problem-solving in online homework. Instructional Science. 47, 499-529.
Small-Scale RCTs
Examples of small scale studies that use the E-TRIALS Platform
Rebalancing Fraction Arithmetic Practice
This study utilized a pretest-intervention-posttest design to empirically test the benefits of providing either a complement of the typical textbook distribution of problems (hyperbalanced practice) or an equal distribution of problems (balanced practice) compared to the benefits of providing a practice set that followed the typical distribution found in math textbooks. A MANCOVA and follow-up ANCOVAs revealed significant differences between students in the textbook condition and students in the balanced and hyperbalanced conditions. For items involving adding fractions with unequal denominators, students who received typical textbook practice showed greater improvement and made fewer strategy errors than students who received hyperbalanced practice. For items involving multiplying two fractions with equal denominators, the opposite was true. Students who received hyperbalanced practice showed greater improvement and made fewer strategy errors than students who received typical textbook practice on items involving multiplying two fractions with equal denominators. Finally, students who received fully balanced practice showed greater improvement and made fewer strategy errors than students who received typical textbook practice on problems involving multiplying one whole number and one fraction. This last finding was of particular interest since none of the practice conditions included practice with that item type.
Oppenzato, C. O. (2024). Rebalancing fraction arithmetic practice. (Publication No. 30990188). [Doctoral dissertation, Columbia University]. ProQuest Dissertations & Theses Global. https://doi.org/10.7916/8hm7-0h26 [Columbia Univ.]
Spacing Out! Manipulating Spatial Features in Mathematical Expressions Affects Performance
The study explores the effects of physical spacing within mathematical expressions on student performance. A total of 2,152 students in 5th-12th grade were randomly assigned to one of four conditions within an online problem set in ASSISTments, with terms in algebraic expressions spaced 1) neutrally, with no spaces in the expression, 2) congruent with the order of precedence through grouping terms, 3) incongruent with the order of precedence, or 4) mixed, a combination of the previous conditions. Results show that students who viewed incongruent problems made more errors and had to solve more problems to complete the assignment than those who viewed congruent or neutrally spaced problems. Additionally, students who viewed problems with mixed spacing had to solve more problems to complete the assignment than students who viewed congruent problems. These findings suggest that viewing expressions with spacing that is incongruent with the order of precedence, presents challenges for students. Overall, these results replicate prior research in perceptual learning in a natural homework environment and support the claim that physical spacing between terms does influence student performance on order of precedence problems.
Harrison, A., Smith, H., Hulse, T., & Ottmar, E. (2020). Spacing out!: Manipulating Spatial Features in Mathematical Expressions Affects Performance. Journal of Numerical Cognition, 6(2), 186-203. DOI: 10.5964/jnc.v6i2.243.
Do College Students Learn Math the Same Way as Middle School Students? On the Transferability of Findings on Within-Problem Supports in Intelligent Tutoring Systems
A widely used minimum criterion for intelligent tutoring system (ITS) status is that the system provides within-problem support. The ITS literature abounds with comparisons of supports, but investigations on the transferability of findings, particularly between education levels, are scant. There are, however, significant impetuses to investigate the transferability of these findings. First, ITSs are used throughout K-16, and findings from past studies naturally serve as guideposts for subsequent ITS developments and research. If findings do not transfer, these guideposts may be illusory. Furthermore, learners change over time, and the efficacy of ITS supports that do not adapt to these changes may vary. This study conceptually replicates investigations conducted at the middle school level by assigning 330 college students in introductory mathematics courses homework that have the same within-problem support formats. The support formats are either text hints, text explanations, or video explanations. We compare the efficacy of these support formats within the college student sample and between college student and middle school student samples. Our findings at the college level indicate that on-demand within-problem explanations displayed as text rather than videos lead to higher next problem correctness and that both outperform text hints. Our results differ from those in the literature using middle school student samples and therefore buttress the assertion that studies investigating the transferability of findings between education levels are necessary.
Smalenberger, M., & Smalenberger, K. (2022). Do College Students Learn Math the Same Way as Middle School Students? On the Transferability of Findings on Within-Problem Supports in Intelligent Tutoring Systems. In A. Mitrovic and N. Bosch, editors, Proceedings of the 15th International Conference on Educational Data Mining, pages 748– 752, Durham, United Kingdom, July 2022. International Educational Data Mining Society.
Providing feedback on computer-based algebra homework in middle-school classrooms
Homework is transforming at a rapid rate with continuous advances in educational technology. Computer-based homework, in particular, is gaining popularity across a range of schools, with little empirical evidence on how to optimize student learning. The current aim was to test the effects of different types of feedback on computer-based homework. In the study, middle school students completed a computer-based pretest, homework assignment, and posttest containing challenging algebraic problems. On the homework assignment, students were assigned to different feedback conditions. In Experiment 1 (N = 103), students received no feedback or correct-answer feedback after each problem. In Experiment 2 (N = 143), students received (1) no feedback, (2) correct-answer feedback, (3) try-again feedback, or (4) explanation feedback after each problem. For students with low prior knowledge, feedback resulted in better posttest performance than no feedback. However, students with high prior knowledge learned just as much whether they received feedback or not. Results suggest the provision of basic feedback on computer-based homework can benefit novice students’ mathematics learning.
Fyfe, E. (2016). Providing feedback on computer-based algebra homework in middle-school classrooms. Computers in Human Behavior 63, 568-574.
Blocking vs. Interleaving: Examining Single-Session Effects within Middle School Math Homework
The benefit of interleaving cognitive content has gained attention in recent years, specifically in mathematics education. The present study serves as a conceptual replication of previous work, documenting the interleaving effect within a middle school sample through brief homework assignments completed within ASSISTments, an adaptive tutoring platform. The results of a randomized controlled trial are presented, examining a practice session featuring interleaved or blocked content spanning three skills: Complementary and Supplementary Angles, Surface Area of a Pyramid, and Compound Probability without Replacement. A second homework session served as a delayed posttest. Tutor log files are analyzed to track student performance and to establish a metric of global mathematics skills for each student. Findings suggest that interleaving is beneficial in the context of adaptive tutoring systems when considering learning gains and average hint usage at posttest. These observations were especially relevant for low skill students.
Ostrow, K., Heffernan, N., Heffernan, C., & Peterson, Z. (2015, June). Blocking vs. interleaving: Examining single-session effects within middle school math homework. In International conference on artificial intelligence in education (pp. 338-347). Springer, Cham.
.svg)