2024 Summer Program Results

Our mission at the Math Agency is to close educational gaps in public schools. Shameless plug: if you are interested in helping support our partner schools as an academic coach, we’d love to hear from you!

This summer we were excited to run the largest in-person summer program we’ve ever done at the Math Agency. Our data indicates that summer programing is a key component to fully closing educational gaps, but we haven’t experimented with it nearly as much as we have with our school-year work.

The takeaway: we see that an intensive (five days/week) summer program can make material progress in closing educational gaps. We saw immediate gains for long-term Math Agency students of roughly 0.12 grade levels (corresponding to 4% of a grade level/week) consistent with our previous in-person summer program. Our data also highlights several factors for increasing the effectiveness of summer programming. If you are interested in the details, keep reading!

Attendance and Dosage

Our 2024 summer program was just one component of Bellevue School District’s summer program, and all results should be taken within that context. The full program was a half-day schedule that ran five days a week for four weeks. A subset of rising 3rd, 4th, and 5th graders received 45–60 minutes of Math Agency programming each day. Below we give a summary of the enrollment and attendance of our program broken out by grade. We also separate out new students (who we first began coaching during the summer) versus returning students who we coached during the 2023–24 academic year.

Figure 1. Enrollment and attendance of our summer program by grade, along with a breakout of new and returning students.

Our attendance percentage is calculated using days of “expected attendance”: a large number of students had regular field trips on Fridays and we didn’t count them absent on those days. Additionally, the first week of programming had a large number enrollment changes, which meant many students weren’t enrolled for the full program period. With these factors in mind, we find it helpful to track “average days attended”. On average, students in our summer program attended 13 days. This corresponds to 65% of the total program days, and suggests one area of improvement: optimizing the structure for students to get a larger percentage of available sessions.

Given our goal of closing educational gaps for all of the at-risk students at our partner schools, we’re also interested in how many of the students in our 2023–24 academic year program participated during summer. We find that 31% of our academic-year students joined the summer program; the results are broken out for our two 2023–24 Bellevue partner schools below.

Figure 2. The number of students enrolled in our 2023–24 academic year program and summer program (and relative percentage), broken out by school. All of the 2023–24 enrolled students were invited to the summer program.

The enrollment data points to another future focus area: improving our outreach to enroll a higher percentage of eligible students.

Practice and Growth

Once students are enrolled and attending, our goal is to get them practicing as much and as efficiently as possible. We used two digital learning platforms (Khan Academy and IXL), and tracked active practice time and progress on each. The results are broken out by grade in Figure 3. The practice time measurements only count digital practice, and thus leave out our “unplugged” activities, such as counting games, geometry puzzles, etc.

Figure 3. Practice time, mastery progress on Khan Academy, and growth measured via IXL broken out by student grade level.

Comparing New and Returning Students

At this point we’ll note an important consideration: new students tend to have higher growth rates that returning students. This follows from our program strategy, because 1) we start our program by teaching students how to learn our software tools, and we don’t want them distracted by content that is too hard, and 2) it is important to build confidence early, and so we don’t want them to begin with material beyond their current skill level. In other words, we intentionally err slightly on the side of easy content to begin, and then gradually ramp up the difficulty. We can see this in Figure 4 below, where new students had larger growth rates in both Khan Academy and IXL compared to returning students.

Figure 4. Khan Academy progress and IXL growth broken out by grade and for newly enrolled students vs. returning students.

The data shows that long-term (returning) students had 0.11–0.13 grade levels of growth over the course of the program. This is likely a more accurate estimate of actual learning than the average across all students. We can compare these practice and growth numbers to our previous in-person summer program, where we saw ~60 minutes/week of practice and IXL growth of 0.26 grade levels over 8 weeks. The results match up reasonably well, especially if we assume growth is linear with dosage, which we’ve observed with previous data. This suggests that a very natural way to increase our per-student impact is to extend the length of the program.

Finally, to put the growth from this summer in perspective, we’ll note that all of the students in our program were chosen because they were academically behind. During the 2023–24 school year, students entered our Bellevue programs having historically learned ~0.6 grade levels of math/year. With that in mind, picking up an extra 0.11–0.13 grade levels of math skills seems like a material benefit.

To summarize: students in our summer program (run in partnership with Bellevue School District) learned an average of 0.11–0.13 grade levels worth of math over the course of the 4-week program. Our data also highlights several areas of future improvement, including adjusting enrollment outreach, making sure students can attend all available sessions, and extending the length of the program.