Why Partnerships Between Tutoring Providers and Math Learning Platforms Make Sense: Lessons Learned from the ASSISTments + Cignition Partnership

The COVID-19 pandemic has massively disrupted the education of K–12 students in the U.S., impacting math the hardest. On average, students lost 5 months of learning, with schools serving majority-black students falling 6 months behind and students in low-income schools falling 7 months behind (Dorn et al., 2021). Even before the pandemic, students of color and low-income students were already facing what was termed the "opportunity gap" (Glossary of Education Reform, 2013). Instead of being presented with rigorous, grade-level materials, multiple studies (see analyses by the Education Trust, and TNTP) found these students are often given work below their grade-level. As such, even using the term "achievement gap" for these students is misleading since they were often not presented with the opportunity to challenge themselves (Carter & Welner, 2013).

Due to these compounding factors, teachers confront a very real challenge with how to make- up for lost learning time while also ensuring students get the opportunity to master the grade-level standards needed to stay on track. With this challenge in mind, Cignition and The ASSISTments Foundation partnered to develop a tutoring program that is highly responsive and aligned to student needs as relates to what is being taught during core instruction. We believe it’s important for the field to have evidence-based solutions that address students' unfinished learning just-in-time, so that it connects to and accelerates their mastery of grade-level content. We think giving tutors access to data from core instruction is a crucial first step to achieving this goal.

Tutoring has emerged as one of the most promising and research-based interventions schools can adopt. Multiple meta-analyses of 100 well-done randomized controlled experiments on tutoring estimate that tutoring interventions tend to have large and reliable effects (Dietrichson et al., 2017; Fryer, 2017; Nickow et al., 2020b). Cignition’s program already has many of the ingredients for effective virtual tutoring. Cignition recruits, vets and trains tutors with an average of 16 years of teaching experience. Tutors receive 3 hours of training to ensure they implement a consistent structure for each session, where students do the heavy lifting and engage in productive struggle while solving problems and  building conceptual understanding. Cignition’s infrastructure for online tutoring allows students and tutors to interact virtually using a carefully designed portal that includes an interactive workspace.

An independent evaluation of Cignition’s one-to-one tutoring model by Digital Promise showed significant gains in math achievement with elementary students. From January - December, 2021, Digital Promise partnered with Cignition to conduct a research study on how to make group virtual tutoring sessions most effective. Cignition then took the learnings from the group research study and completed a Randomized Controlled Trial (RCT) in Spring 2022 to evaluate their group tutoring solution (4:1 student tutor ratio).

‍

‍Problem: Limited Tutor Insight into What Students Learn During Core Instruction

Why bring a teacher-facing math learning platform (ASSISTments) together with a virtual tutoring provider (Cignition)? The reason is to solve a common problem tutoring providers face: limited insight into what students are learning during core instruction, including a lack of real-time data on student progress on those concepts. Without this insight, tutors may struggle to connect the content of tutoring sessions to students’ just-in-time needs and risk tutoring on isolated standards disconnected from the grade-level learning happening in class.

‍

‍A Promising Solution

Connecting tutors to real-time data from class through an integration with a math learning platform is a promising solution to this challenge, with ASSISTments being one such solution. ASSISTments provides a seamless process for teachers who are using Eureka or Illustrative Mathematics, two high quality math instructional materials, to assign the problems from their curriculum online. When completing assignments, students get multiple attempts to answer the problem and feedback on the correctness of their answer. Teachers then have the benefit of rich data reports that allow them to target their instruction. A rigorous study conducted by SRI showed a .22 effect size on a standardized math assessment for 7th graders. Students with lower priority achievement saw the greatest gains (.28).

With ASSISTments data, we hypothesized both teachers and Cignition tutors could better target instruction to real-time student learning needs, as well as increase the degree to which students will find tutoring useful and engaging. The below image is the underlying framework for this model.

‍

In January 2021, with support from the Bill & Melinda Gates Foundation, Cignition and ASSISTments were given the opportunity to develop and pilot this concept. Executing on this joint model required four key workstreams.

1. Building a tutor-facing dashboard, visualizing ASSISTments data from core instruction for Cignition tutors
2. Developing and delivering training for Cignition tutors on how to leverage and incorporate ASSISTments data into their tutor session planning and delivery
3. Developing and delivering training for teachers on how to use ASSISTments in their classroom to support both their instruction and the Cignition tutors
4. Creating processes to support this joint model, including an integration between the ASSISTments and Cignition platforms, and a shared system for enrolling students across systems

As there are multiple math tutoring providers and multiple at-scale math learning platforms serving schools, we wanted to share lessons learned from our partnership that could guide others in developing similar partnerships.

Lesson 1: Always start with a pilot

We identified two district partners excited to implement our joint model. For both, we began with a pilot, working with a 20-student Spring 2021 cohort in the first district, and a 10-student summer pilot cohort in the other. During this time, we engaged in robust formative learning, examining feedback from student, teacher and tutor surveys, usage data from the ASSISTments platform, and student progress data in tutoring sessions. We debriefed learning with district administrators after, to identify ways we can improve in partnership. From these pilots we learned invaluable lessons that set us up for greater success with implementation during the 2021-2022 school year, both in terms of program design and the processes that enable seamless collaboration between our systems. A few lessons learned from the pilot that would be applicable to similar partnerships:

• We ultimately doubled the amount of training and support we offered teachers, based on lessons learned from the pilot, and saw a payoff in terms of increased teacher usage of the platform (and thus, more data for tutors). We also learned it’s important for teachers to have lead time with getting comfortable with the platform - at least 3 instructional weeks to use the platform before tutoring begins.
• Given the entire class is using the learning platform, this model is particularly well suited with a universal model, in which all students have the opportunity to receive tutoring, based on their real-time learning needs. While it is certainly still beneficial in cases where a static group of students representing a subset of the class receive tutoring, it’s exciting these kinds of partnerships can support a more flexible, scalable approach to tutoring.
• It is essential to establish clear processes when working across two organizations with two distinct platforms. Getting to a place where we could seamlessly enroll students within our two platforms, and ensure appropriate linkages to tutors and data, required trial and error, as well as detailed process mapping, paired with the use of airtable to facilitate a clear workflow. In short, do not short change this part of the work.

‍

Lesson 2: Tutors need focused simple data that allows them to quickly hone in on how to maximize their time

We initially envisioned building elaborate data views for tutors of ASSISTments data, including providing curriculum-specific guidance and lesson information as additional context for student data. However, as we engaged Cignition tutors in a user-centered design process, we learned that tutors--given limited time to prepare for sessions--desired a high-level, easy-to-digest interface that answered a simple question: Which standards should I prioritize during the session? The below screenshot is the landing page of the tool we built. Tutors are able to select up to 4 students at a time, and view a snapshot of their progress by standard and by assignment. They can click into any one assignment to view data broken down further by problem and standard. Ultimately, we were able to develop a 10-minute session preparation sequence that allowed tutors to quickly use this data to determine the best use of their tutoring time.

‍

Lesson 3: Building a bridge between core instruction and tutoring requires a commitment from the teacher, and this has pros and cons

A common challenge with tutoring is that teachers may feel out of the loop about what is happening during tutoring sessions. Our model solves the problem by increasing alignment between tutor session content and the teacher’s instructional goals. However, during the pandemic--a time when school was unpredictably fluctuating between in person-and remote learning, and teachers were overwhelmed--implementation fidelity with ASSISTments was a challenge. When students did not complete work in ASSISTments during a given week, tutors had to default to a static scope and sequence from the teacher to determine what to focus on with students. While the pandemic is a hopefully once-in-a-lifetime situation, there will always be competing demands on teachers. Given this, ASSISTments is now exploring how to allow both teachers and tutors to assign content within our learning platform. This increases the consistency of data available to tutors, without losing the valuable bridge shared data built between tutors and teachers.

‍

Lesson 4: Models combining two distinct interventions do not always have a ready market

Even with the added value of our joint model, identifying a ready market for scale beyond our initial partners has been a challenge. Adding a teacher-facing learning platform into the mix feels like “one more thing” to a district seeking a tutoring provider; and on the ASSISTments side, district partners looking for a great formative assessment solution are not typically in the market for tutoring. Both of our organizations continue to be committed to identifying potential partners, as well as continuing to explore how ASSISTments can be further integrated into Cignition’s model.

Lesson 5: Ultimately, tutors found access to real-time data useful, and students felt the benefit of this as well.

After our initial pilot, we launched an improved version of our model with our two district partners, collectively serving 235 6th graders. After some trial and error around dosage and structure in the pilot, we implemented a 12-week model in which tutoring occurred twice a week, in 40 minute sessions (35 minutes for tutoring and 5 minutes buffer for post-tutoring tasks such as exit ticket completion). Student survey data throughout the program indicated that students were able to establish strong relationships with their tutors, and felt a sense of belonging (reporting on average a 4.2/5 for relationship, and a 3.9/5 for belonging). While attendance was not as strong as we hoped (68.4% on average across both districts), implementation occurred during a period during which students were experiencing sudden quarantines and shifts to distance learning. During weeks when class was in-person without disruption we saw strong numbers (for ex 75% in our last month), and one district partner reported that the program’s average attendance exceeded that of any other tutoring provider operating during the same time period.

Perhaps the strongest indicator of the value of the real-time data ASSISTments provided came directly from tutors. Below is a sampling of responses we received via our feedback surveys. This feedback was further validated in a series of user interviews we conducted with tutors following implementation.

‍

The ASSISTments data helped me choose which lessons and activities to choose and what kind of background knowledge the students had going into the lesson.

‍

ASSISTments data was very useful for planning my tutoring sessions, so that my tutoring goes synchronously with the school schedule and students get immediate help on the topics which they are currently working on. In most of my sessions students used to tell me "Yes! This is what we were working on at school today" and reached out for assistance, if they struggled on that topic. Sometimes they reached out to me saying that they needed more practice as the topic was so tricky. This gives me the feeling that students are getting the right assistance "JUST - IN - TIME".

‍

Overall I really liked having the ASSISTments data, and I especially appreciated seeing the students' individual answers to homework questions. This helped me feel more confident in sessions knowing in advance where students are already finding success or struggling, so I knew the session would be helpful.

Applying Lessons Learned

This invaluable year of learning has paved the way for both organizations to explore crucial questions about our work. For ASSISTments, we are heartened by the impact our data can have on tutoring, and in addition to continuing to explore next steps with Cignition, are diving into another round of deep user-centered design and development work on our tutoring product. We seek to expand the functionality and features that support tutors, and develop a version that can integrate with other tutoring providers, or be leveraged as a standalone system for school districts seeking to implement their own homegrown models. For Cignition, our goal in the next fiscal year is to set the stage for tutors to continue to nurture productive struggle and perseverance on the part of students enrolled in our tutoring programs. To that end, we are expanding the functionality of our platform to enable tutors and Cignition managers to collaborate more effectively with school staff members. We are also enhancing our assessment and reporting systems in order to facilitate the ability of all stakeholders to use data to inform instruction.      

‍

We are incredibly grateful for the investment from the Bill & Melinda Gates Foundation for setting us on this path of R&D and innovation, and look forward to continuing to share what we learn with the field.

‍

‍