Khan Academy Improves State Test Scores: Results from New 3-Year Efficacy Study

By Bogdan Yamkovenko

At Khan Academy, we believe that every student deserves access to a free, high-quality education. That’s why we don’t just build learning tools and hope they work. We study them. Carefully. Over time. With the same kind of rigor you’d expect from academic institutions. Our latest research from Newark, New Jersey—where Khan Academy is widely used by ~8000 students (~5200 students observed over multiple years) across 13 North Ward schools— is a powerful example of that commitment.

A three-year panel study — not just correlation

Too often in education, programs are judged by surface-level results: comparing high-usage students, who are typically high-performing students, to low-usage students, who are typically lower-performing students, and claim success. But that’s not good science—and it doesn’t tell you what really works. Instead, our research team ran a three-year longitudinal study in which we isolated the effect of Khan Academy usage on student outcomes by controlling for stable and unobserved student and teacher characteristics (e.g. student motivation, teacher style). This approach is powerful because it eliminates the influence of factors that are difficult or impossible to measure, such as student motivation and teacher style. Once this confounding is removed, what remains is the effect of the Khan Academy usage that varies from year to year. As with all studies, there is one caveat to this. There could be other things that change over time that we do not account for in the analysis. But that’s a weakness of the study we are willing to accept, and we are developing methods to address this.

What we found

The results were compelling:

Every 10 new skills a student learns on Khan Academy leads to about a one-point gain on New Jersey’s state test (PARCC). Of course, 10 skills over the course of the school year is not a lot. Most of our courses cover between 120 and 140 skills. That means a student who learned 60 more skills in a given year than in a prior year could expect a six-to-eight-point gain on NJSLA—a meaningful increase on a 650–850 point scale.

On average, from year to year, students don’t change their NJSLA scores by a lot. They tend to stay within a couple of points of the prior year’s score (a two-point gain, on average, for NJ and a four-point gain for the district). So, gaining six additional points is close to triple the average state gain and 1.5 times the district gain. Results were consistent across demographic groups.

Figure 1. Estimated impact of change in skills on test scores

Figure 1 above shows a nuance in this finding. For increasing gains, what matters is increasing practice, or the number of skills you learn. For example, suppose a student spends very little time on practice in year 1. Then, in year 2, they increase their practice amount (e.g. learn 30 more skills). We’d expect their score to increase by about three points. In year 3, they practice the same amount as they did in year 2. We’d expect them to score about the same as in year 2. So, better than in year 1, but not much better than in year 2. Plus, reducing practice from one year to the next could lead to lower test scores.

Why it matters

This kind of research is rare, but essential, in the edtech world. When we say Khan Academy works, we want to be able to prove it—not with anecdotes but with data, and not just for some students but for many students. We included all students in our study regardless of how much they used Khan Academy.

We’re proud of the progress students in Newark made. Newark’s implementation of Khan Academy is extremely strong and has been increasing over the past two years. Over 50% of students master more than 60 skills in our courses. We’re even prouder to hold ourselves to a standard that puts their success first.

The post Khan Academy Improves State Test Scores: Results from New 3-Year Efficacy Study appeared first on Khan Academy Blog.