Could ClawMetry have caught IBM's prior-auth agents going wrong?

IBM Technology published a good 17-minute explainer on AgentOps, “Are Your AI Agents Flying Blind?”. Its example is one I keep thinking about. Two agents handle prior authorization for a specialty medication. A clinical documentation agent pulls diagnosis codes, lab results and failed prior treatments out of the hospital EHR. A payer authorization agent submits that package to the insurer’s portal, answers follow-up requests by calling back into the clinical agent, and tells the pharmacy and the doctor when a decision lands. A three-to-five-day process drops to under four hours.

Then the video asks the four questions that, in its words, keep the CISO up at night:

1. How do you know it’s not hallucinating diagnosis codes?
2. How do you know it’s not leaking patient data?
3. How do you know it’s not stuck in an infinite loop burning through your API budget?
4. And, later in the video, what happens when a handoff between the two agents fails?

I build ClawMetry, so I did the obvious thing: took each question literally and checked what ClawMetry, as it ships today, would actually have done. Not what the landing page implies. What the code on main does. The answer is one clear yes, two partials, and one no, plus a finding about the setup itself that I did not like.

First, how these agents would reach ClawMetry

This matters more than anything else in the post. ClawMetry reads most agents with zero instrumentation: it has adapters that tail the session files of Claude Code, Codex, Cursor, OpenClaw and the other supported runtimes. A hospital’s prior-auth pipeline is not one of those. It is a custom application, probably built on LangGraph, CrewAI or an agents SDK, running in the hospital’s own infrastructure.

For an app like that, the path in is OpenTelemetry. ClawMetry runs an OTLP receiver on /v1/traces, /v1/metrics and /v1/logs, turns span-only apps into sessions, and ships working examples for LangChain, Pydantic AI and Strands in examples/bring-your-own-agent/. Point the exporter at it and the traces show up.

Here is the finding I did not like. OTLP carries two kinds of signal, and ClawMetry treats them differently. Log records named tool_decision and tool_result (the shape Claude Code’s own telemetry uses) are turned into tool events, so the repeated-tool-failure detector fires on them. They carry no tool arguments, so the loop detector deliberately stays quiet rather than guess. Spans are different. The tracing instrumentation for frameworks like LangGraph reports tool calls as spans, and spans land in the trace store without feeding the event stream the detectors read. A custom agent instrumented that way, which is the likely setup for this hospital, gets traces but almost no detection. So every answer below comes in two versions: what happens for an agent on a supported runtime, and what happens for a custom agent that sends spans. Closing that gap is our top fix, and I come back to it at the end.

Question 3: stuck in a loop, burning the budget. Yes.

This is the question ClawMetry was built around, and it is the one clear yes.

• Detection. stuck_loop fires when the same tool call repeats; no_progress fires when a session keeps calling tools without writing anything. Thresholds are learned per runtime from your own sessions, clamped so a fleet that always loops cannot teach the detector to go quiet.
• Money, not severity. Every incident carries a spend at risk figure: the estimated cost of the stuck stretch, not the whole session. A warning that crosses your critical-dollar line is promoted to critical.
• Alerting. stuck_session and cost_velocity alert rules deliver to Telegram and the dashboard banner for free, and to Slack, Discord or a webhook on paid plans.
• Intervention. A Guard policy can pause the session, then kill it five minutes later if it is still stuck. It only acts when all three locks are open (the policy’s own action, CLAWMETRY_POLICY_ENFORCE=1 on the node, and an entitlement check that fails closed). New policies default to monitor-only.

For the hospital’s custom agent: you would see the loop in the trace view as a long run of identical spans, and the cost would show up in usage. The detector would not fire, because spans do not reach it. Guard cannot pause a process it does not control either: pause, stop and kill work on runtimes ClawMetry can signal, not on a worker pool inside someone else’s Kubernetes.

Question 4: a handoff between agents fails. Partly.

In the video, 1.3% of handoffs fail, almost always because the EHR is unavailable, and the fix is better retry logic. What would ClawMetry show?

• repeated_tool_failure fires when the same tool errors several times inside one session. An EHR outage that makes the clinical agent’s lookup fail again and again would trip it.
• Parent and child agents are linked. The sessions view shows the sub-agent tree, and the child’s activity rolls up into the parent’s timeline.

What it would not do is give you the number the video actually cares about. There is no handoff success rate, no handoff latency (the video’s 340 ms against a 500 ms target), and no alert when 1.3% creeps up to 3%. One failure inside one session is visible. A slow drift across a thousand sessions is not. That drift is the operational signal, and we don’t compute it.

Question 2: leaking patient data. Partly, and quietly.

ClawMetry has two layers here, and neither one is what a healthcare CISO means by “detecting a leak.”

Redaction on ingest. Before an event is written to the local DuckDB store, ClawMetry masks secrets (API keys, bearer tokens, private keys) and a personal-data tier: email addresses, phone numbers, card numbers, IBANs and national IDs such as US Social Security numbers. Each category is checksum-validated so a commit hash never matches. That is good hygiene for our own storage. It does not cover what a prior-auth agent actually handles: medical record numbers, patient names, dates of birth, ICD-10 diagnosis codes, NPI numbers. And redaction is masking, not alerting. When it fires, nothing tells you it fired.

Behavioural detectors. network_egress learns which hosts your agents normally talk to and flags a new one. If the payer agent started posting to a host that is not the insurer’s portal, that is exactly the signal you want. credential_access flags agents reading secret locations such as .env files and key stores. Both are real, and both run over the event stream, so the span caveat applies.

For the hospital’s custom agent there is a worse problem. Redaction runs at the event chokepoint. OTLP span payloads (inputs, outputs, attributes) are written to the span store without going through it. A span carrying a patient record would sit in plain text in the local database. It stays on the machine, and cloud sync is end-to-end encrypted, but “stored in plain text on the node” is not an answer you want to give an auditor. Closing it is on the list below.

Question 1: hallucinated diagnosis codes. No.

This is the honest no. The video reports 99.4% diagnosis-code accuracy and says “these are not guesses, we can validate against the source records.” That sentence is the whole requirement: an accuracy number needs ground truth, and ground truth lives in the EHR.

What ClawMetry has:

• A per-answer hallucination risk heuristic, free.
• A faithfulness evaluator (Pro) that checks whether each claim in the answer is supported by evidence the agent actually retrieved in the transcript.
• An eval_score_below alert rule over either score.
• A human review queue that samples sessions for someone to mark correct or wrong: the video’s pharmacist panel, roughly. It samples a fixed number per agent per day (10 by default), not a percentage, and nothing alerts when reviewer-marked accuracy drops.

Faithfulness would catch an agent inventing a code that appears nowhere in what it read. It would not catch an agent copying the wrong code off the record, or a code that is present but clinically wrong for this patient. For that you need a comparison against the source system, and ClawMetry has no hook for one. An outside observer cannot know what “correct” means in your domain unless you tell it.

The scorecard

What we’re building from this

Ranked by how much each one changes the answer for the prior-auth team:

1. Run the detectors on OTLP spans. Turn tool and LLM spans into the same events the detectors already read, so a LangGraph app gets stuck_loop, repeated_tool_failure and network_egress like any supported runtime.
2. Redact span payloads. Send OTLP inputs, outputs and attributes through the same redaction pass as events.
3. A pii_exposure incident. Report redaction hits as incidents so a policy or alert can act on them, and add healthcare patterns: MRN shapes, dates of birth next to names, ICD-10 and NPI formats.
4. Handoff success rate and latency. Roll parent-to-child handoffs up from spawn spans and child outcomes, with a handoff_failure_rate alert.
5. A ground-truth hook. An endpoint where your system posts the real outcome (the code was right, the payer approved first pass), so accuracy becomes a measured number instead of a judged one.

The honest part

The video frames AgentOps as three layers: observability, evaluation, optimization. ClawMetry is strongest where a runtime failure costs money: loops, stalls, repeated failures, spend. That is where it began, watching coding agents burn tokens overnight. It is weakest where the question is “was this output correct for this patient?”. No observer sitting outside your system can answer that without your ground truth.

If you are running agents in a regulated workflow today, use ClawMetry for the cost and runtime layer and be clear-eyed about the rest. If your custom agents report over OpenTelemetry traces, know that the detectors do not cover them yet. I’d rather you read that here than find it out during an incident.

The companion post scores all fifteen of the video’s metrics one by one: IBM’s AgentOps checklist, scored honestly against ClawMetry.