During the pandemic, longtime Bay Area college and career counselor Jon Siapno started developing a chatbot that could answer high schoolers’ questions about their future education options. He was using IBM’s question-answering precursor to ChatGPT, Watson, but when generative artificial intelligence became accessible, he knew it was a game-changer.

“I thought it would take us maybe two years to build out the questions and answers,” Siapno said. “Back then you had to prewrite everything.”

An AI-powered chatbot trained on information about college and careers and designed to mimic human speech meant students at the charter school in the East Bay city of Richmond could soon text an AI Copilot to chat about their futures. The idea was that students could get basic questions out of the way — at any hour — before meeting with counselors like Siapno for more targeted conversations.

Almost one-quarter of U.S. schools don’t have a single counselor, according to the latest federal data, from the 2021-22 school year. California high schools fare better, but the state’s student-to-counselor ratio when ChatGPT debuted the following year was still , a far cry from the American School Counselor Association’s recommended ratio of 250-to-1.

Siapno wasn’t the only one to see generative AI’s potential to scale advising. A flood of bots designed to help people navigate their college and career options have surfaced over the last two years, often with human-sounding names like Ava, Kelly, Oli, Ethan and Coco. It’s unclear how many California high schools tell students to use any of them, but the power of generative AI and the scale at which young people are already turning to chatbots in their personal lives is giving some people pause.

Julia Freeland Fisher is education director at the Clayton Christensen Institute, a nonprofit research organization that studies innovation. She recently sounded the alarm about the consequences of letting students develop relationships with AI-powered college and career counselors instead of human ones.

“It’s so tempting to see these bots as cursory,” Freeland Fisher said. “‘They’re not threatening real relationships.’ ‘These are just one-off chats.’ But we know from sociology that these one-off chats are actually big opportunities.”

Sociologists talk about “social capital” as the connections between people that facilitate their success. Among those connections, we have “strong ties” in close friends, family and coworkers who give us routine support, and “weak ties” in acquaintances we see less regularly. For a long time, people thought weak ties were less important, but in 1973 Stanford sociologist Mark Granovetter wrote about “” and a flood of studies since then have confirmed how important those more distant acquaintances can be for everything from to .

As California considers regulating AI companions for young people, policymakers, tech companies and schools must consider how the burgeoning market for AI-driven college and career guidance could inadvertently become the source of a new problem.

“We’re creating this army of self-help bots to help students make their way through school and toward jobs,” Freeland Fisher said, “but those very same bots may be eroding the kinds of network-building opportunities that help students break into those jobs eventually.”

‘Like a mentor in your pocket’

The Making Waves Academy ensures all its graduates meet minimum admissions requirements to California’s four-year public colleges. Nine out of 10 of them do pursue higher education, and while there, staff at the Making Waves Education Foundation offer 1:1 coaching, scholarships, budget planning and career planning to help them graduate on time with no debt and a job offer.

Patrick O’Donnell, CEO of Making Waves, said his team has been thinking about how to scale the kinds of supports they offer for years now, given the scarcity of counselors in schools.

“Even if counselors wanted to make sure they were supporting students to explore their college and career options, it’s almost impossible to do and provide really personalized guidance,” O’Donnell said.

Early superusers of the Making Waves AI CoPilot were 9^th and 10^th graders hungry for information but boxed out of meetings with school counselors focused on helping seniors plan their next steps.

CareerVillage is another California nonprofit focused on scaling good college and career advice. CareerVillage.org has been aggregating crowd-sourced questions and expert answers since 2011 to help people navigate the path to a good career.

When ChatGPT came out, co-founder and executive director Jared Chung saw the potential immediately. By the summer of 2023, his team had a full version of their AI Career Coach to pilot, thanks to help from 20 other nonprofits and educational institutions. Now “Coach” is available to individuals for free online, and high schools and colleges around the country are starting to embed it into their own advising.

At the University of Florida College of Nursing, a more specialized version of Coach, “Coach for Nurses,” gives users round-the-clock career exploration support. Shakira Henderson, dean of the college, said Coach is “a valuable supplement” to the college’s other career advising.

Coach for Nurses personalizes its conversation and advice based on a user’s career stage, interests and goals. It is loaded with geographically specific, current labor market information so people can ask questions about earnings in a specific job, in a specific county, for example. Coach can also talk people through simulated nursing scenarios, and it’s loaded with chat-based activities and quizzes that can help them explore different career paths.

Henderson is clear on the tool’s limitations, though: “AI cannot fully replace the nuanced, empathetic guidance provided by human mentors and career advisors,” she said. People can assess an aspiring nurse’s soft skills, help them think about the type of hospital they’d like most or the work environment in which they’d thrive. “A human advisor working with that student will be able to identify and connect more than an AI tool,” she said.

Of course, that requires students to have human advisors available to them. Marcus Strother, executive director of MENTOR California, a nonprofit supporting mentoring programs across the state, said Coach is worlds better than nothing.

“Most of our young people, particularly young people of color in low-income areas,” Strother said, “they don’t get the opportunities to meet those folks who are going to be able to give them the connection anyway.”

By contrast, Coach, he said, is “like having a mentor in your pocket.”

‘A regulatory desert’

Last month, California state Sen. Steve Padilla, a San Diego Democrat, introduced legislation to protect children from chatbots. would, among other things, limit companies from designing chatbots that encourage users to engage more often, respond more quickly or chat longer. These design elements use psychological tricks to get users to spend more time on the platform, which research indicates can create an addiction that keeps people from engaging in other healthy activities or lead them to form unhealthy emotional attachments to the bots.

The addictive nature of certain apps has long been a critique of social media, especially for young people. In Freeland Fisher’s for the Clayton Christensen Institute, she included a comment from Vinay Bhaskara, the co-founder of CollegeVine, which released a free AI counselor for high schoolers called Ivy in 2023.

“I’ve seen chat logs where students say, ‘Ivy, thank you so much. You’re like my best friend,’ which is both heartwarming, but also kind of scary. It’s a little bit of both,” the report quotes him as saying.

Reached by phone, Bhaskara said his company’s tool is designed to be friendly and conversational so students feel comfortable using it. Millions of students have used the chatbot for free on CollegeVine’s website and more than 150 colleges in California and around the country have offered the technology to their own students. After seeing how many millions of emails, text messages and online chat sessions have happened outside of working hours, Bhaskara now argues the insight and support students have gotten from the chatbot outweigh the risks.

In announcing Padilla’s bill, his office referenced a number of cases in which chatbots directed children who had become attached to them to do dangerous things. At the most extreme, a Florida teen took his own life after a Character.AI chatbot he had become romantically involved with reportedly encouraged him to Padilla said his bill wouldn’t keep young people from getting the benefits of college and career advising from chatbots; it would offer reasonable guidelines to address a serious need.

“This is a regulatory desert,” Padilla said. “There are no real guardrails around some of this.”

Freeland Fisher said the AI companions that young people are turning to for friendship and romantic relationships represent a far greater risk than AI-powered college and career advisors. But she said schools and tech developers still need to be careful when they seek out an AI solution to the counselor shortage.

Maybe the only current danger is replacing conversations with school advisors. Eventually, though, sophisticated tools that capture more of students’ time and attention in the quest to fill a greater need could end up replacing conversations with other adults in their lives.

“These other supports matter down the line,” Freeland Fisher said. When students spend more time with chatbots and, indeed, learn to prefer interactions with bots over humans, it contributes to social isolation that can limit young people’s ability to amass all-important social capital. “That’s part of the warning that we’re trying to build in this research,” Freeland Fisher said. “It’s not to say ‘Don’t use bots.’ It’s just to have a much fuller picture of the potential costs.”

For their part, Making Waves and CareerVillage are taking some responsibility for the risks chatbots represent. Making Waves is actually retiring the AI Copilot this summer as the foundation shifts its mission to finding a way to use technology to help kids build social capital, not just get answers to questions about college and career. And CareerVillage has already put safeguards in place to address some of Padilla’s concerns.

While Coach does tell users the more they interact with the chatbot the more personalized its recommendations become, Chung, the executive director, said Coach is designed to only discuss career development. “If you try to go on a long conversation about something unrelated, Coach will decline,” Chung said. He described a series of guardrails and safety processes the company put in place to make sure users never become emotionally attached to the chatbot.

“It’s work,” Chung said, “but I’m going to be honest with you, it’s not impossible work.”

This article was and was republished under the license.

Districts across the state reported they did not receive nearly as many applicants as anticipated and, as a result, there is some legislative momentum right now to extend the deadline.

School officials attributed the weak response to a number of factors, including summer communication lags and a concern among students that having pass/no pass outcomes on their transcripts would hurt their college prospects.

“Sometimes it feels like our families have some school messenger fatigue, where they don’t always hear them or listen to them,” said Tess Seay, head counselor at Fresno High School in the state’s Central Valley. Of their over 2,000 students, roughly 50 requested grade changes before their district’s Aug. 17 deadline, five days after their first day of school. They expected at least 100.

The ability to purge a low letter grade from your record has very real consequences. For two students on the cusp of 2.0 GPAs, Seay said the grade change option pushed them over that eligibility threshold for Cal Grants, the state’s loan-free financial aid. Some Cal Grants could provide for college tuition, making a significant financial impact for low-income students.

California legislators in July designed to not penalize students for the challenges that came with remote learning during the pandemic. In addition to the pass/no pass option, families could also request that their child repeat a grade or waive particular high school graduation requirements not mandated by the state.

The law required that districts notify families by Aug. 2 and provide a two-week application period. Many have questioned whether the timeline adequately enabled families — particularly those without regular internet access, or who may have been offline during the summer — to seriously consider their grade change options.

In San Bernardino, students returned to in-person learning Aug. 2 for the first time since March 2020. Anticipating a more hectic than usual back-to-school season as a result, the district opened English and Spanish-language grade-change on Aug. 16 for their 14,911 high schoolers, weeks after many other districts.

By Aug. 31, the district east of Los Angeles received 256 applications — just under 2 percent of those eligible.

“By design, we planned to start [the application window] the third week of school, allowing students, families, and counselors to focus on [the new state policy] and not overlook it in the rush of the start of the school year. This way students were settled into a routine before we brought it to their attention,” Maria Garcia, the district’s communications officer, said in an email to ����Ӱ��.

Each San Bernardino high school website flags the policy and directs families to . Families were also notified via their district’s phone app [Parent Square], social media and email newsletters.

Nancy Witrado, director of counseling and guidance with Fresno Unified School District, said their application was made available via a fillable PDF, available in Hmong, Spanish and English. While they did not track the demographic details of who applied, she told ����Ӱ�� that many originated from a high school in northern Fresno county with a history of high parental engagement.

Many Fresno families filled out applications incorrectly, asking to change Bs or Fs to pass; the former would not benefit students and the latter would be impossible. The district is now paying overtime to several registrar and counseling employees to meet about 200 requests.

“We have a lot of follow-up to do, and to try to connect with parents to make sure that they have a full understanding of what it is they’re asking for,” Witrado said.

Fresno Unified stuck to its Aug. 17 cutoff, though Witrado says not many families have reached out after-the-fact. One instance, of an application mistakenly submitted to a student’s teacher, will be honored because it came in before the deadline.

Problems reaching families may not be the only driver behind fewer grade-change requests — some college-bound students were warded off by the worry that a pass/no pass grade carried negative connotations with admissions officers.

Cecilia Roeder Chang, a senior at Gunn High School in affluent Palo Alto, said her district and school did a great job of getting the word out online, and her peers even posted about it on Instagram. Last school year she earned two Cs, in physics and foreign policy, that she considered changing to pass.

“I originally had decided that I was going to. Then I emailed my school counselor, and they replied back that colleges didn’t like that as much. So I decided against [it],” she said.

Roeder Chang, who is applying to both in- and out-of-state schools, was not made aware that all California State Universities and all campuses within the University of California system must grades, or that some Cal Grants require a minimum 2.0 GPA. Her peers did not apply for grade changes, she said, given that they had mostly As and Bs.

The knowledge that some are able to change low grades for the better, after the school year’s end, has garnered mixed feelings.

“I sort of feel conflicted because on one hand, if you do have like lower grades it is helpful, but also on the other hand, if you are one of those people who are getting consistently like higher grades, you can feel like, I don’t want to say annoyance — maybe a little frustration.”

North of Roeder Chang’s Palo Alto, Oakland Unified School District received 660 applications in Chinese, Arabic, English, Spanish and Vietnamese. All of the district’s 13 high schools were represented, and the highest volume of applications were submitted by students at Skyline, Oakland Tech and Oakland High, the largest schools. Only two families have reached out after the district’s Aug. 16 deadline.

“Given this was the first legislation of its kind, we didn’t anticipate a certain number of requests and made sure we were prepared to handle a large number,” John Sasaki, the district’s communications director, told ����Ӱ�� by email.

From a policy standpoint, advocates caution against permanent alternative grading. The , a national nonprofit that aims to make students prepared for post-secondary education, expressed concern for pass/no pass policies over longer periods, saying they may lead to decreased expectations for students and less accurate student data.

Short-term adjustments to grading policies can be beneficial for students who may need to heal from collective trauma, said one former high school math teacher who now works with the Collaborative. Recalling how his Las Vegas school let up on requirements after a mass shooting in October 2017, he said changing grading policies provided students with needed flexibility.

He said that other supports — like removing deadlines or penalties for late work — may adequately support students without overhauling A-F grading, which feeds into many other systems like financial aid, school report cards and state reporting.

San Diego Unified — the state’s second-largest district — during the pandemic. Only 290 of their 36,000 high schoolers applied for grade changes. The in its communications — recommending instead that the grade be “suppressed” by repeating the class.

Because of low application rates and school capacity to process applications at the start of the school year, San Diego state Assemblywoman Lorena Gonzalez, who authored the state law, is now recommending districts extend deadlines voluntarily and is pursuing legislation to formally extend the deadline to October.

“I will say we’re a little disappointed with the lack of flexibility with some of the districts,” . “If you feel like you missed [the deadline], contact the school district. Really push.”