At a Glance
- Alaska’s probate AI assistant, AVA, is now over a year in development instead of the planned three months.
- The chatbot has struggled with accuracy, empathy tone, and hallucinations that risk harming users.
- A streamlined 16-question test set will be used before its late-January launch.
- Why it matters: Families navigating probate rely on precise guidance; inaccurate AI could cause legal and financial harm.
Alaska’s court system has been building a generative-AI chatbot called the Alaska Virtual Assistant (AVA) to simplify probate paperwork, but the project has faced setbacks in accuracy and reliability.
From Three-Month Dream to Year-Long Reality
Consultant Aubrie Souza said the project was intended to take three months but is now over a year and three months, citing the due diligence needed to get it right.
- Original timeline: 3 months
- Current status: >13 months
- Reason: rigorous accuracy checks
Accuracy, Empathy, and Hallucinations
Court director Stacey Marz warned that users could suffer harm if the chatbot’s information is inaccurate or incomplete.
> “If people are going to take the information they get from their prompt and they’re going to act on it and it’s not accurate or not complete, they really could suffer harm. It could be incredibly damaging to that person, family or estate.”
The team also trimmed the chatbot’s overly empathetic tone after users complained of excessive condolences.
> “Through our user testing, everyone said, ‘I’m tired of everybody in my life telling me that they’re sorry for my loss.'”
Law professor Tom Martin explained the challenge of hallucinations.
> “We had trouble with hallucinations… For example, when we asked it, ‘Where do I get legal help?’ it would tell you, ‘There’s a law school in Alaska, and so look at the alumni network.’ But there is no law school in Alaska.”
Testing, Cost, and Ongoing Oversight
The team initially designed 91 probate questions, but found the test too time-consuming and narrowed it to 16 key questions.
| Question Set | Size | Purpose |
|---|---|---|
| Initial | 91 | Broad coverage |
| Refined | 16 | Practical and high-frequency |
A single query can cost as little as 11 cents for 20 queries.
> “Under one technical setup, 20 AVA queries would cost only about 11 cents.”
The system will require regular checks and updates as new AI models are released.
Key Takeaways
- AVA’s development has exceeded its original three-month goal, now over 13 months.
- Accuracy and hallucination issues risk harming users, prompting careful testing and tone adjustments.
- The chatbot’s low cost and planned launch in late January could improve access to probate information if reliability improves.
Despite the challenges, AVA’s launch in late January could give Alaskans easier access to probate information, provided the system’s accuracy improves.
