By Bogdan Yamkovenko, efficacy research manager; Kodi Weatherholtz, principal efficacy and research scientist; and Phil Grimaldi, senior efficacy and research scientist at Khan Academy.
When it comes to educational technology, the million-dollar question is: does it actually help students learn? We all know that there are countless apps and platforms out there, but how many can prove they make a difference? At Khan Academy, we’re committed to not just asking this question but answering it with solid evidence. Read about how Khan Academy leads to learning gains.
The challenge of proving efficacy
Typically, ed tech companies, including Khan Academy, compare students who use their platform to those who don’t. While this approach gives us some insights, it’s not foolproof. After all, there could be many other factors at play. Maybe the students using the platform also have extra tutoring or perhaps they have more engaged teachers.
Conducting a randomized controlled trial (RCT) would be the gold standard for demonstrating efficacy, but it’s not always feasible. Imagine trying to find a school willing to randomly assign students to different teachers and teaching methods for an entire year!
A Different Approach: You vs. You
So, we tried something different. What if we compared students to their most similar counterparts—themselves? We tracked the same group of students over two school years to see how their performance changed from year to year depending on how their use of Khan Academy changed from year to year.
The results speak for themselves
The findings were encouraging. Kodi Weatherholtz, principal research scientist at Khan Academy, found that Students who amped up their Khan Academy usage from one year to the next saw notable improvements in their gains on the MAP Growth test. It was a consistent trend across different grades and demographic groups. Because of the study design (within-subjects comparison), these gains can’t be easily attributed to other student characteristics like motivation, making a strong case for the effectiveness of Khan Academy.
Digging deeper
For those who like the nitty-gritty details, we used a measure called the Conditional Growth Index (CGI) to track student growth. This measure takes into account various factors like starting scores and grade levels, making it a reliable indicator of genuine learning progress. This measure also allows us to compare student growth across grades on a standard deviation scale. The data showed that even a small increase in Khan Academy usage could lead to a meaningful bump in CGI scores. Read full research brief (pdf).
For example, a student who spent five hours on Khan in 2021 (less than 10 minutes per week) but then spent 10 hours in 2022 (an increase of five hours over the year), saw a one-point increase, on average, in MAP Growth scores (equivalent to ~0.1 increase in CGI). Again, because we are comparing a student to themself, this growth is not a special student effect. In fact, any student characteristic that is stable over time, like motivation or socio-economic-status, is controlled in this design because each student is essentially their own control group.
What about teachers?
It’s true that a student in the 2022 school year may have had a better teacher than they had in the 2021 school year. Could the improvement be due to that? This is possible, but there are two reasons why we are confident that’s not the case here. First, in public schools the students rarely choose their teachers. So, if someone got a better or worse teacher in one year compared to the teacher they got the other year, that’s largely due to chance. And given that we are looking at 100K students, about half of whom increased their usage year to year, it’s unlikely that all those students got better teachers in year two. Second, to rule out the teacher effect in our analysis, we account for the average time students of any given teacher spend on Khan as well as the average scores the students of any given teacher see on their tests. That means we are looking at the effect of change in usage on change in MAP gains for all types of teachers, and we average the effect. And after all that, the effect still holds.
What does this mean for you?
Simply put, the more you use Khan Academy, the more you’re likely to learn. We often say spend at least 30 minutes per week on Khan. But what this study means is that if you spend five minutes per week now, spending just a bit more time—say 10 minutes per week—will benefit you. Whether you’re a student, parent, or educator, this is great news. It means that the time invested in Khan Academy is time well spent, leading to real and measurable learning gains.
Conclusion
Our latest study builds on previous findings, showing that Khan Academy is a reliable tool for enhancing learning outcomes even in the challenging context of a pandemic. While no study is perfect, the evidence strongly suggests that Khan Academy can play a significant role in boosting academic achievement.
Bogdan Yamkovenko
Source link
