CrisisBench tests crisis counseling AI with 1,085 tasks; retrieval-augmented GPT scores 0.961
Researchers built the suite from 200 de-identified hotline conversations. All tested systems showed too much empathy and questioning compared with human counselors in single-turn tests.
What happened
A research team built CrisisBench, a test set for AI chatbots that could assist crisis helpline counselors. It uses AI-played callers instead of real transcripts, so no private conversations are exposed. The team tested several GPT models and found they handle basic conversation structure well.
Why it matters to you
Crisis helplines are often understaffed and underfunded, so AI help is tempting. Safe tools need testing first, and the study shows where today's systems still differ from experts.
The takeaway
AI can match counselors on basic skills but still makes response-level mistakes that experts avoid.
What it means for you
The released rubrics, judge prompts and tasks give a reusable test harness, and the judge agrees with humans about as well as humans agree with each other.
Organizations considering AI for counseling support get a privacy-preserving way to test tools before deployment.
Near-ceiling scores on structure and tone with tight spreads between systems suggest that safety and productivity are where vendors will differ.