The open-source tool, which researchers say other educators can recreate and integrate into their tutoring systems, made the human tutors slightly more effective. And the weakest tutors became nearly as effective as their more highly-rated peers, according to a study . 

The tool, dubbed Tutor CoPilot, prompts tutors to think more deeply about their interactions with students, offering different ways to explain concepts to those who get a problem wrong. It also suggests hints or different questions to ask.

The new study offers a middle ground in what’s become a polarized debate between supporters and detractors of AI tutoring. It’s also the first randomized controlled trial — the gold standard in research — to examine a human-AI system in live tutoring. In all, about 1,000 students got help from about 900 tutors, and students who worked with AI-assisted tutors were four percentage points more likely to master the topic after a given session than those in a control group whose tutors didn’t work with AI.

Students working with lower-rated tutors saw their performance jump more than twice as much, by nine percentage points. In all, their pass rate went from 56% to 65%, nearly matching the 66% pass rate for students with higher-rated tutors.

The cost to run it: Just $20 per student per year — an estimate of what it costs Stanford to maintain accounts on Open AI’s GPT-4 large language model.

The study didn’t probe students’ overall math skills or directly tie the tutoring results to standardized test scores, but Rose E. Wang, the project’s lead researcher, said higher pass rates on the post-tutoring “mini tests” correlate strongly with better results on end-of-year tests like state math assessments. 

Wang said the study’s key insight was looking at reasoning patterns that good teachers engage in and translating them into “under the hood” instructions that tutors can use to help students think more deeply and solve problems themselves. 

“If you prompt ChatGPT, ‘Hey, help me solve this problem,’ it will typically just give away the answer, which is not at all what we had seen teachers do when we were showing them real examples of struggling students,” she said.

Essentially, the researchers prompted GPT-4 to behave like an experienced teacher and generate hints, explanations and questions for tutors to try out on students. By querying the AI, Wang said, tutors have “real-time” access to helpful strategies that move students forward.

”At any time when I’m struggling as a tutor, I can request help,” Wang said.

She said the system as tested is “not perfect” and doesn’t yet emulate the work of experienced teachers. While tutors generally found it helpful — particularly its ability to provide “well-phrased explanations,” clarify difficult topics and break down complex concepts on the spot — in a few cases, tutors said the tool’s suggestions didn’t align with students’ grade levels. 

A common complaint among tutors was that Tutor CoPilot’s responses were sometimes “too smart,” requiring them to simplify and adapt for clarity.

“But it is much better than what would have otherwise been there,” Wang said, “which was nothing.”

Researchers analyzed more than half a million messages generated during sessions, finding that tutors who had access to the AI tool were more likely to ask helpful questions and less eager to simply give students answers, two practices aligned with high-quality teaching.

Amanda Bickerstaff, co-founder and CEO of , said she was pleased to see a well-designed study on the topic focused on economically disadvantaged students, minority students, and English language learners.  

She also noted the benefits to low-rated tutors, saying other industries like consulting are already using generative AI to close skills gaps. As the technology advances, Bickerstaff said, most of its benefit will be in tasks like problem solving and explanations. 

Susanna Loeb, executive director of Stanford’s National Student Support Accelerator and one of the report’s authors, said the idea of using AI to augment tutors’ talents, not replace them, seems a smart use of the technology for the time being. “Who knows? Maybe AI will get better,” she said. “We just don’t think it’s quite there yet.”

At the moment, there are lots of essential jobs in fields like tutoring, health care and the like where practitioners “haven’t had years of education — and they don’t go to regular professional development,” she said. This approach, which offers a simple interface and immediate feedback, could be useful in those situations. 

“The big dream,” said Wang, “is to be able to enhance the human.”

Benjamin Riley, a frequent AI-in-education skeptic who leads the AI-focused think tank and writes a on the topic, applauded the study’s rigorous design, an approach he said prompts “effortful thinking on the part of the student.”

“If you are an inexperienced or less-effective tutor, having something that reminds you of these practices — and then you actually employ those actions with your students — that’s good,” he said. “If this holds up in other use cases, then I think you’ve got some real potential here.”

Riley sounded a note of caution about the tool’s actual cost. It may cost Stanford just $20 per student to run the AI, but he noted that tutors received up to three weeks of training to use it. “I don’t think you can exclude those costs from the analysis. And from what I can tell, this was based on a pretty thoughtful approach to the training.”

He also said students’ modest overall math gains raises the question, beyond the efficacy of the AI, of whether a large tutoring intervention like this has “meaningful impacts” on student learning. 

Similarly, Dan Meyer, who writes a on education and technology and co-hosts a on teaching math, noted that the gains “don’t seem massive, but they’re positive and at fairly low cost.”

He said the Stanford developers “seem to understand the ways tutors work and the demands on their time and attention.” The new tool, he said, seems to save them from spending a lot of effort to get useful feedback and suggestions for students.

Stanford’s Loeb said the AI’s best use is determining what a student knows and needs to know. But people are better at caring, motivating and engaging — and celebrating successes. “All people who have been tutors know that that is a key part about what makes tutoring effective. And this kind of approach allows both to happen.”

They’ve been busy.

“Business is booming,” Kori Smith, an assistant special agent in charge, told district officials at a conference earlier this year on oversight of federal programs. 

He offered hypothetical examples of fraud and abuse of relief funds meant for pandemic recovery — for example, the purchase of 700 Chromebooks when a district only needed 500, or stockpiling masks and other protective gear at staff members’ homes. 

One actual investigation, turned over to the Department of Justice, led to of two Louisiana Christian University students who stole the identities of nine students and used their names to obtain in emergency aid intended for housing, tuition and food. In a more outlandish example unconnected to education, a Texas man is serving prison time for bilking that helped businesses pay their employees during lockdown out of almost $25 million and using some of it to buy a Bentley convertible.  

“We think it’s going to get worse,” Smith said. “It’s a lot of money that was unexpected.” 

Indictments for defrauding the government make headlines. But they’re also just one lever in the complicated machinery of oversight districts face as they spend an unprecedented $122 billion from the American Rescue Plan. Districts face increased scrutiny from federal and state officials as they move from planning how to use the relief funds to signing contracts. But fear of audits has had an unintended consequence, some experts say: Districts are proceeding cautiously to spend funds meant to fix urgent problems.

“As compelling as it is to ask ‘Will this help our kids?’ the next question is ‘What will the auditors say about it?’” said Sheara Krvaric, co-founder of the Federal Education Group, a law firm specializing in federal K-12 programs.

That means Education Secretary Miguel Cardona’s frequent calls for districts to use the funds as soon as possible to address student learning loss and other needs sometimes fall on deaf ears.

For example, some districts have been reluctant to spend relief funds on non-academic programs, like sports and physical education “even though there is pretty convincing evidence it helps with learning loss,” Krvaric said.

And despite severe staff shortages and turnover, some districts have opted not to use relief funds on retention bonuses or other incentives to keep teachers “because their states have signaled that would be unallowable,” she said. “This is despite clear guidance from [the Education Department] that it is allowable.”

Those contradictions give district and state officials extra reason to be on their guard. “There’s a ton of confusion still about what you can spend the money on,” she said.

Meanwhile, auditors with the independent Office of Inspector General aren’t necessarily digging for subtlety. Just , they cited Oklahoma Gov. Kevin Stitt’s office for failing to keep more than $650,000 in relief funds from being spent on arcade games, Christmas trees and 131 sets of cookware. The state set up an $8 million program to offer $1,500 grants to low-income families and contracted with ClassWallet, an online payment platform for educators, to run the program. 

Auditors said the purchases didn’t meet the standard for “emergency educational services” and called on the state to return the funds spent on the “unallowable” items. In a response, the state blamed ClassWallet for the “deficiencies” and said it has improved oversight. 

‘A moving target’

A year ago, parents, teachers and community members were invited to advise districts on how they should spend the considerable federal windfall. But as auditors dig into the details, many parents accuse districts of stashing the money away and continue to clamor for increased and other opportunities for their children to catch up. 

Keri Rodrigues, president of the National Parents Union, argues that many parents haven’t seen these funds benefit their children. She meets every two weeks with Cardona or his staff, where the use of relief funds comes up frequently.

“He wants districts to be acting with urgency,” she said, ‘“but they are saying, ‘We don’t know what to do.’ ”

By law, districts can’t just hold onto the money. They have to obligate it by the end of September, 2024. While the department has said it will consider some extensions, their approval is not guaranteed.

At the same time, the rules, which require districts receiving at least $750,000 in federal funds to undergo an audit, are shifting rapidly. 

“I’m talking total reversal. It literally is a moving target,” said Bonnie Graham, a partner with Brustein & Manasevit, a law firm specializing in federal education policy. “School districts are in a tough spot. You can’t afford to make a mistake.”

In 2020, the Biden administration’s Office of Management and Budget districts were not required to track employees’ hours charged to Elementary and Secondary School Emergency Relief — known as ESSER. The 2021 and 2022 versions of the document said the . 

Districts are also allowed to use relief funds to pay themselves back for money they spent at the start of the pandemic, but their documentation often doesn’t go back that far, said Cathy Harlow, manager of an accounting firm in Pennsylvania that conducts district audits.

A former superintendent for the Tyrone Area School District — situated midway between Pittsburgh and Philadelphia — she sympathizes with districts.

“Our firm leans on the side of leniency,” she said. “We’re holding districts accountable, but understanding that the landscape has been changing rapidly as they’re going through it.”

‘New territory’ 

Sometimes audits don’t tell the whole story.

The inspector general’s office also reviewed how the Missouri Department of Elementary and Secondary Education used to connect more students to the internet. 

Due to a clerical error, the state didn’t reach all eligible districts, showed. What it didn’t show was that officials used ESSER funds to cover the rest. 

“We just had to make peace with it,” said Chris Neale, the department’s assistant commissioner. “We know there will be a ton of auditing that has never gone on before. It’s new territory for everyone.”

That’s particularly true for smaller districts and charter networks, which normally don’t spend enough federal funds to trigger an audit. The Colorado Charter School Institute has some charter management organizations facing the process for the first time.

“The question that I get from CMOs is, ‘Do we really have to do this?’” said Andi Denton, director of finance and operations. Most, she added, just don’t want to spend the $10,000 or so to pay an accounting firm to complete it. She reminds them they’ve gotten “a lot of money.”

As districts and charters apply for relief funds, some state officials are kicking those requests back for more information before approving them. 

In Georgia, for example, the state audit department initially rejected districts’ requests to use relief funds to cover salaries. They interpreted the law to mean those funds could only be used to make up for a drop in state revenue, said Matt Cardoza, a spokesman at the Georgia Department of Education.

To clear up the confusion, federal officials sent a letter explaining that using relief funds to pay staff is “not dependent on a shortfall in state and local funding.”

Marguerite Roza, director of the Edunomics Lab at Georgetown University, said some officials might not have kept up with the “twists and turns” in messages from the education department about “allowable” expenses. 

Relief funds for education, the law says, are meant to “prevent, prepare for and respond to” COVID. The most recent from the department says districts should use the funds to “emerge stronger post-pandemic” and address needs exacerbated by COVID. 

Districts are now submitting reimbursement requests to their state education departments, which typically turn around approvals quickly — except for a few questionable items.

“A couple districts asked for pressure washers to clean sidewalks. It’s very hard to connect that to COVID,” Cardoza said. “We don’t try to be so over the top [that] they can’t spend their money, but we’re trying to keep them from being audited.”

Now that effort has drawn the attention of one of the world’s wealthiest people, who happens to be giving her money away at a rapid clip. 

In March, just over five years after the group’s founding, MacKenzie Scott, ex-wife of Amazon founder Jeff Bezos, an unrestricted gift of $3 million. The donation is its biggest gift to date and nearly doubles the group’s revenue, according to recent .

The money, said founder Lakisha Young, will allow the group to “take our work to the next level” and plan for the long-term, which includes pushing to bring more parents and community members into schools in support roles.

The plan for the group is to map out a three-year growth strategy, expanding trainings that allow community members to become literacy tutors and work alongside teachers in the Oakland Unified School District.

“Our work needs to go where most of the students are,” Young said in an interview. 

California regulations restrict who can work as tutors, but she said the group is poised to lobby to tweak those regulations. 

Scott’s gift will also give the group a measure of stability as it pushes to bring in more funders. Having reliable funding, Young said, “allows the work to move just that much faster.” 

The donation was a surprise. Young said she got a call last October letting her know that an anonymous donor wanted to find out more about the organization. Then in March came the email with Scott’s announcement. Even now, Young hasn’t met or talked with Scott, who is known for in multi-billion-dollar flurries with little fanfare and posting notices to her , where she’s known simply as “Mom, writer, advocate.”

The $3 million donation represents a watershed moment for The Oakland REACH, which Young created in 2016, after an eye-opening experience trying to get her eldest daughter into a good public kindergarten. 

Many neighborhood schools at the time were in “school improvement” under No Child Left Behind, which meant they might well close within a few years. Young didn’t want that sort of disruption, so she placed her daughter’s name in a charter school lottery that offered just 11 slots. The lottery drew 93 applicants. 

Luckily, her daughter’s name popped up seventh, but Young said the victory “really sparked something in me” — a realization of how deeply unfair the system was to families of color. If thousands of families are forced to place their children’s futures in a hat and hope for the best, she recalled, “What do you think that kind of message is sending to them about what they have access to?”

Young created The Oakland REACH in December 2016 as a self-described network of “passionate and fearless parents” pushing to improve education in a city where more than 90 percent of students are non-white and nearly 60 percent receive free or reduced-price meals.

The group formed with 50 “unapologetic” parents across about 30 district and charter schools, “moms and grandmamas and daddies and uncles who were raised in Oakland, went to Oakland schools, were served poorly by those schools,” Young said in an interview last fall. As parents and grandparents, now they’re “basically saying, ‘This won’t be my child’s story. This won’t be my grandchild’s story.’”

She recalled how she chose her first members: “I don’t want the PTA parent. I want the parent that, when they come in, pencils move because they’re just coming in totally focused on, ‘What’s happening with my baby, what’s going on?’ They’re just solely focused on their kids.”

Since then, the group has pushed to shine a bright light on achievement in the city: Young last March that just 8.7 percent of Oakland eighth-graders scored proficient in math in fall 2021 — this in a city, according to the group’s December survey, where parents “showed a HUGE demand for math skills.” She noted that 81 percent of parents want “more high-dosage math tutoring. Parents want their kids to read and do math 
 well!”

‘Our families were already losing’

The group’s first big victory came in March 2019, when the city school board unanimously approved a policy change that gave families impacted by school closures and consolidations priority admission to schools they wanted their children to attend.

When COVID-19 hit exactly one year later, Young, with the help of the (CRPE), quickly built the digital Hub. 

Remembering back to the summer of 2020, Young said the group “didn’t have much to lose” by trying something new for online learning. “Our families were already losing. This was, actually, for us, an opportunity because kids were at home with their parents. It was an opportunity to do something really different and move from a ‘struggle’ model to a model of more privilege and abundance.”

In the program’s first five weeks, Young noted, assessments showed that 60 percent of K-2 students rose two or more levels on the district’s reading assessment and 30 percent rose three or more levels. 

“We did that,” she said triumphantly. “And we did that by bringing 
 teachers and a group of folks to the table that were doggedly focused on serving families. No politics, no adult politics, no drama. 
I’m telling you: If we were doing that all the time, our kids would be able to read. Our kids would be good at math.”

Families on their own

In addition to training literacy tutors, the Scott donation will also jumpstart a planned math fellowship this fall that will help caregivers become tutors.

Family members taking control of learning is key in a district awash in news of , and . One need the group hopes to help with: Oakland’s insatiable demand for substitute teachers. Young envisions training “community educators” who don’t simply show up to a new school each day, but who have “an investment in that community” and remain there through multiple assignments.

Nationwide, districts will also soon be figuring out what to do when federal COVID relief in 2024. 

Through it all, Young said, most parents must continue trusting their children to a public education system that’s full of uncertainty. 

“Superintendents leave,” she said. “Principals leave. Teachers leave. Families don’t leave. So they have to have what they need.”

In essence, families must create the solutions the district needs. “We can scream and holler as much as we want about what the system needs to do,” Young said, but “we need to be creating the talent.”

Christina Barnes, a mother of two who works as a family liaison for the organization, recalled helping a grandfather who was taking care of a child but didn’t know how to use email or send text messages. As a result, he missed alerts about school closures and other important information. “He would call and say, ‘I didn’t know school was closed today.’” 

Barnes helped arrange a tech fellowship for the grandfather that gave him the skills he needed to stay informed and to help his grandchild keep up with school.

Guadalupe Canchola, a mother of three young children and a so-called “parent liberator,” works with many Spanish-speaking families who feel unsafe speaking up, mostly because of language barriers or fears about their immigration status. “I love to bridge these gaps between families and the school system, just so they know that no matter their situation, they have rights. Their kids have rights.”

In one recent case, a parent asked Canchola to sit in on her child’s IEP meeting for special education services. But the parent handled it well.

“Honestly, the way I saw her advocate for herself and her son was the biggest surprise and takeaway for me,” she said. In so many cases like these, parents get intimidated “or cornered into a decision that’s not ours.” The more parents know about their rights, she said, the better. “They have to listen to you. They have to meet your needs. That’s just very powerful.” 

Bree Dusseault, a principal with CRPE, said Young’s work to elevate the voices of parents is “getting very clear results” in student achievement, with literacy rates climbing “at a pretty significant pace” for the kids they serve. “She has this very, very deep belief that parents need to be in the driver’s seat — and deserve to be in the driver’s seat — and need to be a part of the larger narrative of what their children are achieving.”

A lot of Young’s success and impact, Dusseault said, is the product of years of work in Oakland — much of that “well underway before the pandemic.”

Giving away money ‘quickly and without much hoopla’

Scott, 52, is one of the richest people in the world — as of May 16, the placed her at No. 35, just above Blackstone Group CEO Stephen Schwarzman. Her net worth stands at $32.9 billion, though Scott has vowed to give away half the fortune in Amazon stock she got in a divorce settlement with Bezos. 

Since 2020, the one-time novelist has given away at least $8 billion, with a heavy emphasis on education, public health, climate change, gender and racial equality, food security, and LGBTQ rights 

In early 2021, she married Dan Jewett, a Seattle science teacher.

The New York Times that Scott hands out money “quickly and without much hoopla,” moving the focus away from herself and onto the beneficiary organizations, which have included historically Black colleges and universities, Habitat for Humanity chapters, community-based education foundations, and many others. Many of these organizations “fly beneath the radar of major foundations,” The Times noted.

Young said the donation will help strengthen families and move people’s focus away from the “the inputs of drama and the inputs of chaos” roiling the Oakland district. “It’s a lot, but the question is: What can we still be doing in this moment to make sure our kids can read and do math? And we believe it’s what we’ve created. Our kids do not miss a beat, regardless of the political hoopla. But how many other kids did miss a beat because of it?”