Let rules do the exact work. Let AI handle the leftovers.
What building a claims pre-check agent taught me about where an LLM belongs in a regulated workflow.
When people picture an AI agent for insurance claims, they often picture a model reading every claim and deciding what's wrong with it. That is the expensive, hard-to-explain version. While building my claims pre-check agent I landed on a simpler split, and I think it applies to most regulated workflows.
Most problems are exact
A large share of claim issues are deterministic. A required field is missing. A code isn't valid. A value falls outside an allowed range. These checks have one right answer, and a rule finds it every time, in microseconds, for free, with a reason anyone can read.
Handing those checks to a language model buys you nothing except cost, latency and a new way to be wrong. So the rules go first, and they stay in charge of everything they can decide.
Some problems are messy
Then there is the rest. A pharmacist's note says a prior authorization is still pending. A comment contradicts a coded field. Context lives in free text, and no reasonable set of rules covers every way a person writes it down. This is where a model earns its place: reading the leftovers the rules can't.
In the pre-check agent, the rules run first and the AI only reviews the notes. Its job is narrow and its output is structured, so a flag comes back with the risk and the text that triggered it.
Evals before prompts
The part I'd emphasize to any team starting out: build the labeled test set before tuning the prompt. I used synthetic claims with known answers, then compared rules alone against rules plus the agent. That comparison is the whole argument for the AI. On my eval set, the agent caught risks in free-text notes that the rules missed. Without the labels, I'd only have had a demo that felt good.
Why this matters in regulated work
- Explainability. Most decisions come from rules an auditor can read. The AI's contribution is small and visible.
- Cost and speed. The model only sees the slice of work that needs it.
- Safer failure. If the model has a bad day, the exact checks still run.
The same pattern shows up in payments, identity and reconciliation. Decide what has an exact answer, make that part boring and reliable, and point the AI at what's left.
The code and the synthetic data are on GitHub.