Kristen’s Corner Spring 2025

By Aviv Weiss, Khan Academy Districts

Each quarter, we interview Kristen DiCerbo, Chief Learning Officer of Khan Academy, to learn from her reflections as a leading researcher and product developer in the AI for education world. In this interview, first published in the April 2025 Sal’s Quarterly Newsletter for District Leaders, Kristen offered guidance to district leaders who are thinking deeply about implementing AI for learning.

Aviv Weiss: Given it is early days of AI for learning, do you have any guidance for district leaders on how they should be thinking about measuring success or efficacy? Can you speak to any of the research we are doing- what are you looking at, what’s important to you?

Kristen DiCerbo: In the early days of new technology, the first thing to understand is implementation. I would suggest district leaders understand if and how the tool is being used in classrooms. You can think of this as a chain of logic, if the technology isn’t being used, it definitely will not impact learning outcomes. So, the first thing to understand is the extent teachers are using it and, in cases where it isn’t being used, the barriers to usage. One common barrier is that teachers and students have never had time to explore what AI can do and therefore aren’t using many things it can do. So, a solution might be to find time for teachers to have 45 minutes or an hour to explore the available tools, modeling how they can also do this for their students. The key is understanding what the barriers are, so that solutions can be found.

We use a similar chain of logic in our research work. We are looking at the “tutoring move” that Khanmigo makes. Is it prompting students, offering procedural correction, summarizing, etc.? These are all moves that research on human tutors tells us are important. Similarly, we are looking at whether students are cognitively engaged in interactions with Khanmigo. This doesn’t mean clicking a lot but means asking meaningful questions and answering thoughtfully. We are doing work to analyze the tens of thousands of transcripts of student chats to understand how to improve meaningful engagement. If Khanmigo isn’t acting like a good tutor and students aren’t engaging, it isn’t likely to impact learning, so we are focusing on those two elements in our research.

Aviv: You’re constantly immersed in research—what’s one study, article, or piece of data that made you pause or rethink something this quarter? Any unexpected “aha!” moments from this quarter—whether from data, fieldwork, or product testing—that shifted your perspective?

Kristen: As a researcher, I almost never rely on one piece of information or data. As I lead our product, design, and content teams, I am almost always looking for results that have been replicated over and over with different populations of learners in different contexts. Those are the findings that, even when we are using cutting edge technology, we want to use to guide how we design the experience.

That said, the research I have been citing most frequently is a paper from researchers at Carnegie Mellon University called “An Astonishing Regularity in Student Learning Rate.” It essentially is a confirmation of the fundamental importance of practice. The authors combine 27 different datasets containing over 1.3 million events to show how each additional item of practice increases learning. It doesn’t matter if students are starting at a low or high level of achievement, each additional item of practice results in a similar learning gain. This means that if learners get the opportunity to engage in the practice they need, they will see gains in achievement that get them to our grade level standards. As we design our new AI tools, we keep in mind that even with this new technology, our goal should be providing students with the support and feedback they need to keep practicing.

Aviv: What’s one thing you wish more district leaders knew when they’re evaluating AI-powered learning tools?

Kristen: I wish more district leaders would ask developers of AI tools how they know if the output from the AI tools is good. For example, getting AI to generate a lesson plan is fairly easy. Getting it to generate a good lesson plan is much harder. As I write in this piece, we used a rubric for what makes a good lesson plan from a teacher preparation program as the standard our lesson plan had to meet before we released it. We have a variety of tools we use to evaluate the AI-powered experiences we create. It isn’t easy to evaluate these tools because every time you ask a generative AI-based tool to generate something, it does it differently. You can’t just ask it once and say it is good. In addition, once the tool is out in the world, we need to monitor the millions of interactions happening for changes in performance. When we change to a different underlying large language model, we need to ensure performance doesn’t change. Any provider should be able to talk to you about the measures they use before and after they release new tools and features to ensure you will consistently get quality output.

The post Kristen’s Corner Spring 2025 appeared first on Khan Academy Blog.