Today we’re pleased to announce that we’re releasing an anonymized dataset of math tutoring conversations for use in evaluating how AI models act like a tutor.
While many researchers and companies are exploring AI’s ability to do calculations, at Khan Academy we’re interested in the ability of AI to do calculations while acting like a tutor. As we explain in the paper accompanying the dataset, we think tutoring is an underexplored area of research that presents unique challenges—and one that holds great potential too.
About the anonymized dataset
The dataset we’re releasing today consists of 188 representative conversations covering elementary math through calculus. The representative conversations are based on conversations that took place between Khanmigo, our pilot tutor and teaching assistant, and students, and have been anonymized.
The dataset is a benchmark dataset, meaning it is a resource for researchers and companies to use to evaluate AI models.
Why a benchmark dataset about tutoring matters
There are many math datasets out there. We think today’s release of a tutoring dataset may be one of the first of its kind.
A tutoring dataset is important for our field because it captures how a conversation unfolds when Khanmigo tutors a student (while also preserving the student’s anonymity). The dataset shows interactions and two-way feedback, not just math problems.
This dataset focuses on one aspect of tutoring—the accurate evaluation of student work. We have found that AI models often struggle with this capability, either telling students they are right when they are wrong or vice versa. This struggle is partially due to calculation errors, but is also the result of the complex nature of doing those calculations in the context of a conversation with a student. Of course, tutoring involves much more than this, including what to offer in response to an error. But we believe this dataset will at least evaluate whether the model can correctly judge student work in a tutoring context. We think it will help our colleagues in the field evaluate AI’s ability to tutor in math so they can help improve AI in the future.
Our North Star is student learning
As a nonprofit organization, part of our goal is to contribute to the field of education by making learning accessible to all. By sharing this dataset, we hope to further advancements in AI in education to help students learn and succeed in their studies. We think the new dataset is an important step in the development of AI that not only gets math right but also acts as an effective tutor for students. Onward!
The post Introducing a New Dataset to Further the Field of AI Research appeared first on Khan Academy Blog.
Original Article Published at Khan Academy
________________________________________________________________________________________________________________________________
