In the fall of 2022, months before the public got a glimpse of ChatGPT, DiCerbo, a learning scientist and chief learning officer at Khan Academy, got access to a beta version of Open AI鈥檚 GPT-4 model. The startup needed Khan Academy鈥檚 help training it to pass the Advanced Placement biology exam, a requirement dreamed up by Microsoft co-founder Bill Gates, who wanted improved performance as a condition of handing Open AI more funding.

Khan Academy founder Sal Khan and DiCerbo negotiated a partnership with Open AI, and just five months later, their AI-powered Khanmigo tutoring bot debuted. Last summer, Khan Academy launched an .

Nearly two years in, DiCerbo remains bullish on the possibilities of AI tutoring, cheerfully engaging critics about the limitations of the technology, even as by all measures it evolves and improves. 

Much of the press for Khanmigo has been positive: late last year, produced an upbeat feature on Khan Academy鈥檚 efforts 鈥� host Anderson Cooper called Khanmigo鈥檚 potential 鈥渟taggering,鈥� but tempered the observation by adding, 鈥淚t鈥檚 still very much a work in progress.鈥�

Other media accounts have challenged Khan鈥檚 predictions that AI will anytime soon, with a Wall Street Journal reporter a year ago that Khanmigo didn鈥檛 consistently know how to round answers or calculate square roots and 鈥渢ypically didn鈥檛 correct mistakes when asked to double-check solutions.鈥�

Khan Academy has said improvements are ongoing, but that at least a few errors are likely to persist. The organization stresses that Khanmigo remains 鈥渋mperfect鈥� and 鈥渟till evolving.鈥� 

In March, DiCerbo will appear at , alongside Curriculum Associate鈥檚 Kristen Huff and Akisha Osei Sarfo of the Council of the Great City School to discuss how AI can improve school assessments. The panel will be moderated by 蜜桃影视鈥檚 Greg Toppo, who spoke recently with DiCerbo in a wide-ranging interview. 

They talked about Khanmigo, its critics and why she feels 鈥渃autiously optimistic鈥� about the role of AI in education.

The interview has been edited for length and clarity.

You’ve been with Khan Academy now for almost five years, and it’s been an eventful time. You’ve spent a lot of that time creating and improving Khanmigo. What are the latest developments? 

We have learned a lot in what is coming up on two years since Khanmigo launched. In terms of what students are doing, we definitely see some interesting things we didn’t necessarily expect. Students who are English language learners really like and use the supports in other languages. We probably shouldn’t have been surprised, but always need to be reminded that it’s important to just have instruction on how to use new technology and tools, and what that looks like. For students, how do you ask good questions? And for teachers, how do you integrate it? So both professional learning for teachers and supports for students have been important things that we’ve added over time.

The other thing is that we have found that Khanmigo as a tutor works best when it is paired with educational content we have already created. It is better integrated and has lower error rates when it’s using, and has reference to, the existing problems that were written and verified by people 鈥� and not just the problems, but the [step-by-step] hints and the answers that already exist in our system. When it can reference those, Khanmigo is better. And when students are just working on the practice that is part of Khan Academy generally, they are using Khanmigo as an assistant and as a help to get unstuck.

When we talked last year, you used that word “unstuck.” You guys have come in for some criticism from critics like and , who say Khanmigo gets math wrong, among other things. Meyer last year said he’s become a kind of pro bono consultant for you guys. [DiCerbo laughs.] You’re familiar with the criticisms, and I wonder: How have they landed? And have they had an effect on the product?

Dan has very good classroom experience and is extremely knowledgeable about teaching math. So many times, the things he says align to conversations internally that we’re already having. And the things he says are things that we end up changing and doing. We always appreciate criticism that helps us improve and move on. A lot of our work has been on things like working to better evaluate math accuracy, improve it, and get the balance right between how much Khanmigo gives help versus asks questions 鈥� all of the things we’re working to tune and get right in that sweet spot for what the student needs.

Dan actually just this week had . The thing he misinterprets about us is that he thinks we’re trying to replace teachers, and he thinks we don’t value teachers. That’s what that whole post was about. And that is just not how we see what we’re doing. We see Khanmigo as a tutor that’s also working in the same ecosystem, but the teacher is fundamental to this whole process.

I saw the 60 Minutes piece with Anderson Cooper a while back and I wonder how that landed.  

First of all, the writing piece they highlighted is something we’re pretty excited about. Very often in schools, kids do writing assignments and teachers end up with a huge stack of writing. As Sarah [Robertson], who’s our product manager, said in the piece, she had to limit herself to only 10 minutes per essay feedback, and still it would take her hours and hours as a secondary school teacher to grade all of these essays 鈥� and then the students get them back two weeks later. That’s not immediate feedback. So the idea that we can potentially provide more immediate feedback to students on their writing is pretty interesting to us.

There鈥檚 a lot of concern over cheating. 

We can say, “Hey, we’re going to flag this piece,” which Anderson did in his demo 鈥� he just cut and pasted in a whole bunch of content. We can say, “Hey, we don’t know where this content came from. We’re not going to make the judgment, but teacher, here’s a flag for you to check on.”. Not surprisingly, we got a lot of queries from school districts asking about getting access.

When I was writing the piece last year about IBM Watson and the effort to make it into a tutor, you expressed a cautious optimism that despite all the failures we’ve seen, this time was different. It’s been almost a year now. I wonder if your feelings have changed about AI tutoring generally and Khanmigo specifically?

I would still characterize how I feel as cautiously optimistic. I don’t think this is The Golden Ticket that’s going to save us all and be the sole reason that educational outcomes improve. I do think it still can be an important tool in the toolbox.

Does the change in presidential administrations have any bearing on your work, given that President Trump’s got an apparent interest in AI and support from big tech, specifically and ? 

There is a lot of noise about what may or may not happen. We are basically sticking to “What are our technology partners doing, and what are we able to then partner with them to build?” And we will see what actually comes to fruition and deal with it if and when anything actually happens. We’re not counting on anything either way.

My last question about this topic is the earthquake that happened with the Chinese AI startup . The interpretation that I’ve been hearing is that it has caused supreme havoc at places like Open AI. I wonder if any of this has redounded to you guys?

Not specifically the Deep Seek piece, but it’s just part of what we have thought is likely to be the future 鈥� it’s just a little bit sooner than perhaps we thought. The models themselves become a commodity. Even since we launched, the prices have come down so far that it’s significant. We’re able to offer what we do at significantly lower prices, and that’s just likely to continue. And it’s not going to be the models themselves that are the 鈥渕oat鈥� or the differentiator 鈥� it’s going to be what people build with them.  

Is it even in the realm of possibility that you would work with a company like Deep Seek?

Well, Deep Seek’s model is open-source, so you can install it on your own machine. And that’s part of the concern about security and privacy with the app that, of course, has ties to the Chinese government. Then there’s the question about the model itself, as an open-source model, How does it perform? I would not rule out us using open-source models from different sources, but they would have to be evaluated, like all our models are, for security and privacy and their performance. 

Let’s talk about South by Southwest. The session we’re doing is titled “How AI Makes Assessment More Actionable in Instruction,” which doesn’t exactly roll off the tongue. But it gets to an interesting idea, which is that AI can make assessments better: more invisible, more customizable, and help teachers adapt instruction. I wonder what you’re seeing in terms of the ways AI is moving into that field?

It’s interesting, because the assessment conversation has lagged a bit behind the learning conversation when it comes to AI. But it seems to be picking up speed this year, both at South by Southwest and at ASU+GSV.

Traditionally we’ve had multiple-choice tests. You and I know there’s the whole game-based, simulation-based movement. What does AI let us do? The idea of a conversational-based assessment is interesting. What if the assessment looks like what happens when a teacher sits down next to a student and says, “Explain your thinking. How did you get to that?” There’s a conversation there. And that could potentially be an interesting way of adding to assessments that we already have. Of course, there would be questions: Is that standardized? Because different kids might get different questions as they engage in this conversation. How do we deal with that when we’re talking about high-stakes assessment? 

The last thing I think is interesting is helping teachers and parents make sense of assessment data and get recommendations. Can AI help with that? Instead of getting this printout that says, “Your student got a 580 on this,” and you’re like, “What does that even mean? What should I do?” If you could have a conversation about that, that might be an interesting piece. We’ve been exploring that in something we have called and recommendations that allow teachers to talk about their students’ Khan Academy performance. What else might they assign? How might they group students based on those kinds of things? 

In the past couple of months I’ve been playing around with AI tools that summarize and analyze big chunks of text and YouTube videos and whatnot. It strikes me that we are going to become so used to having a tool like this break things down for us that if schools can’t help us break our students’ performance down, we’re going to be disappointed. Is my cart ahead of the horse?

I always try to figure out if I’m in a bubble or not, because I feel the same way. I know lots of people that similarly are really getting into the habit of whenever they get a large amount of information, put it into an AI tool and get the summarization. I’m not quite sure how broad-based that is when we think about all of the parents out there and all the schools, but that is what I’m seeing, and it might become an expectation in the near future. 

Is there something on the horizon that you are looking at that maybe others aren’t paying attention to 鈥� good, bad or other?

The video was out quite a while ago of Sal and his son [of Khanmigo]. We鈥檙e starting to get to a place where the AI is seeing what the student is working on, and is able to interact with that and move forward. I’m pretty excited about that.

To learn the geometric concept of transformations this year, Crystal Watson鈥檚 eighth-graders drew up blueprints of apartments. As they worked, she asked them to imagine designing affordable housing for Black and Hispanic families like theirs in Cincinnati who have been priced out of their neighborhoods.

But when she had them add a hallway down the middle of their floor plans, with apartments on either side, some struggled with the idea of reflection 鈥� flipping a figure to create a mirror image.

鈥淭here are still kids who mix up their x and y axis,鈥� said Watson, who teaches at Hartwell School.

At that point, she pulled students aside individually to explain the difference and offered tips for remembering. Her strategy 鈥� connecting math to socio-economic issues in the community and letting students proceed even if they haven鈥檛 mastered the skills 鈥� is captured in that gives teachers steps for 鈥渄ismantling racism鈥� in math instruction.

But the book鈥檚 claim that a focus on producing the right answer promotes 鈥渨hite supremacy culture鈥� alarmed some who question how inaccuracy in math could benefit students. And, partly in response to the controversy, California state board members recently recommended against incorporating the resource into a redesign of the state鈥檚 math program.

While history and literature seem like obvious battlegrounds for schools to address the effects of racial discrimination, some might question whether math 鈥� where achievement depends on precise calculations 鈥� is the appropriate venue for such fights. Those devoted to greater equity say the middle grades are a period when many Black and Hispanic students begin to turn off of math, only to continue struggling through high school. But the suggestion that answers to math problems are subjective became easy fodder for culture war conservatives.

鈥淢ath enjoyed this notion that it was somehow above the influence of the cultural and political issues of our time,鈥� said Rachel Ruffalo, the director of educator engagement at The Education Trust-West, the Oakland-based advocacy group that created the workbook.

Now, that is changing. The workbook is part of the organization鈥檚 larger 鈥� one that seeks to address persistent racial disparities in achievement. The Bill and Melinda Gates Foundation funded the $1 million initiative last spring as part of focused on making algebra more accessible to students of color, partly in response to learning disruptions caused by the pandemic. Districts in Georgia, Ohio and California are among those using the workbook in teacher training.

Conservative Fox News lampooned, sometimes out of context, a handful of the book鈥檚 ideas 鈥� for example, the notion that key teaching practices such as requiring students to show their work and complete assignments individually are based in racism. appeared in mid-February after the Oregon Department of Education teachers to a training featuring the book. The Fox piece sparked in other outlets and from columnists, most of whom neglected to mention that the authors later say, 鈥淥f course, most math problems have correct answers.鈥�

Erec Smith, a professor of rhetoric and composition at York College of Pennsylvania and co-founder of , is among those who accuse the book鈥檚 authors of their own form of bigotry.

鈥淭he workbook’s ultimate message is clear: Black kids are bad at math, so why don’t we just excuse them from really learning it,鈥� said Smith, who is Black.

Even leaders of the National Council of Teachers of Mathematics have reservations about the guide 鈥� though their reasons differ.

鈥淎re we building bridges or throwing grenades?鈥� asked David Barnes, associate executive director of the council. 鈥淲hen you get to page two and what鈥檚 bolded is 鈥榙ismantling white supremacy,鈥� there are some people that cannot read past that.鈥�

Other groups came to Oregon鈥檚 defense, offering positive reactions to the book and its broader effort to make math more culturally relevant for students of color.

鈥淵ou and I were taught that everything happened in Greece,鈥� said Kristopher Childs, director of Student Achievement Partners, a nonprofit focusing on academic success for historically underserved students. 鈥淓very culture and civilization contributed to mathematics. Students need to know that.鈥�

The authors, for example, prompt teachers to have students explore the Egyptian and Babylonian roots of the Pythagorean Theorem, before Pythagoras identified it in Greece during the 6th century B.C.

The guide draws inspiration from another document titled which includes what some scholars on race argue are characteristics of 鈥渨hite supremacy culture鈥� 鈥� ideals such as perfectionism, individualism and a sense of urgency they say allowed early American colonists to dominate over African slaves and Native Americans.

鈥楻oom for creativity鈥�

Samuel Rhodes, an assistant professor of elementary math education at Georgia Southern University, said focusing only on the right answer can at times be counterproductive. In a course last year for future K-8 teachers, he called on a student who gave a wrong answer to a problem.

He said he could have done what he鈥檚 observed in countless public school classrooms 鈥� go on to the next student until someone answered correctly, or repeat the steps.

But with that strategy, a 鈥渟tudent … just immediately shuts down,鈥� Rhodes said. 鈥淣ow they have to ignore everything they were thinking with the goal of trying to understand how the other students did it.鈥�

Instead, he asked the student how she arrived at that answer and learned she had a 鈥渃reative, brilliant process鈥� for finding the solution, but got derailed by a small computation mistake.

鈥淭here鈥檚 no room for creativity when there is a fixation on the answer,鈥� he said.

But knowing whether a student is 鈥渨ildly wrong鈥� or 鈥渙ff by just a hair鈥� takes deep expertise in math 鈥� something teachers, especially those at the elementary level 鈥� don鈥檛 always have, said Jay Wamsted, a longtime Atlanta math teacher working in high schools serving predominantly Black and HIspanic students.

He added, 鈥淚t’s not obvious to the layperson why 鈥榯he right answer鈥� isn’t always preferable and the workbook needs to be clear about why that is.鈥�

While the workbook discourages teachers from asking students to 鈥�’show their work鈥� in … prescribed ways,鈥� it does recommend that students have multiple options for demonstrating what they understand.

That鈥檚 a shift Lisa Owens, another Cincinnati math teacher, is still trying to make.

鈥淔or me, that was letting go of control. For a lot of teachers, that is where the issue is,鈥� said Owens. But she said she鈥檚 learned to spot shallow attempts at cultural relevance. 鈥淵ou can鈥檛 just put an ethnic name into a word problem.鈥�

Beginning her career in a Chicago suburb, she said she 鈥渨as raised that you don鈥檛 see color.鈥� But now Owens, who is white, teaches at Roberts Academy, which serves a predominantly Black and Hispanic population. She helped start a school equity coalition and opposed the school鈥檚 former practice of tracking fourth graders into low and high classes based on math scores.

She recognizes the hurdles involved in meeting the guide鈥檚 definition of an 鈥渁ntiracist math educator.鈥� Allowing students to arrive at mistakes on their own can take up valuable class time, and a lot of teachers, she said, still take a 鈥渢ough love鈥� approach and question whether such methods would improve test scores. According to state data, less than 10 percent of the eighth graders in the school score proficient in math.

鈥楾he role of education鈥�

Teaching practices like those in the workbook have been part of the San Francisco Unified School District鈥檚 shift in math instruction since 2014. That鈥檚 when the district stopped separating students into basic or Algebra I classes in middle school 鈥� a controversial policy that California is now considering statewide. The state will continue to collect public comments over the summer, and the state board will make a final decision in November. Advocates for gifted students are the proposed changes.

So far, administrators using the workbook have had a receptive audience of educators committed to 鈥渟ocial justice math.” But when they try to spread those ideas among colleagues at their schools, they often face resistance.

鈥淐hallenging the status quo is not easy for a lot of teachers,鈥� said Bernadette Andres-Salgarino, math coordinator for the Santa Clara County Office of Education in California.

The guide caused enough of a storm in California that members of the state board advised its Instructional Quality Commission, which is drafting the to remove references to it.

While Barnes, with the National Council of Teachers of Mathematics, thought the guide used 鈥渨ords that immediately divide,鈥� he appreciates the intent behind it 鈥� an effort to make math more accessible for students of color and give them a strong foundation when they enter high school.

It complements, he said, the push in California and to de-track math by keeping students in the same courses at least through the eighth grade. Virginia is considering changes.

less than 20 percent of Black students take Algebra 1 by eighth grade, compared to 67 percent of Asian students and 45 percent of white students. And even if they take higher-level math in middle school, Black students are less likely than white and Asian students to stay on an accelerated track in high school.

The pandemic has set students of color even further behind. from testing provider Renaissance showed that while all students performed below pre-pandemic levels in math, the decline was greatest among Black and Hispanic students. And on a national scale, the between Black and white eighth-graders hasn鈥檛 budged in years.

Williamson Evers, a former U.S. Department of Education official during the second Bush administration and a senior fellow at the conservative Independent Institute, suggested the social justice approach to math will put U.S. students further behind those in other countries.

鈥淥ur kids are going to be competing in a world with kids that have this in their heads. They鈥檙e doing better. They have the material under their belt,鈥� he said during a . His in the Wall Street Journal ran before California state board members turned their back on the workbook.

Josie McSpadden, a spokeswoman at the Gates Foundation, defended the project.

鈥淎t times, research has shown that racial bias and student mindsets can affect student academic achievement,鈥� she said, adding the workbook, 鈥渉ighlights a critical discussion 鈥� how students arrive at answers and demonstrate their understanding and conceptual grasp of important math concepts.鈥�

This fall in Cincinnati, math teachers throughout the district will walk through the practices recommended in the guide. Watson 鈥� who plays clean versions of rap songs in her class when students finish an assessment 鈥� said math is usually 鈥渟o cut and dried.鈥� The resource gives teachers ways to incorporate students鈥� opinions and family stories into her lessons.

鈥淚 don鈥檛 have to be an anti-racism and anti-bias guru,鈥� she said, 鈥渢o pick this up and do what鈥檚 good for kids.鈥�

Disclosure: The Bill & Melinda Gates Foundation provides financial support to and 蜜桃影视.

Lead Art: A math teacher works with a student at Willie L. Brown Jr. Middle School in San Francisco. The district is among those working to address racial disparities in math achievement.  (Lea Suzuki / The San Francisco Chronicle / Getty Images)