When a government agency reaches for a consumer chatbot to police billions in public money, two things are true at once: it is a sensible response to an impossible workload, and it is an accountability question waiting to happen. That tension sits at the centre of a new plan from the US Department of Health and Human Services, and it is a preview of a debate every finance ministry and audit body will soon have — India’s included.
What’s happening
The US Department of Health and Human Services (HHS) has launched an initiative it calls AERO (Audit Enforcement and Risk Oversight) that uses AI to scan five-plus years of state and grantee single-audit history across all 50 states. A nuance worth being precise about: HHS’s official announcement referred only to “AI” — the specific mention of ChatGPT and other LLMs came from Assistant Secretary Gustav Chiarello in comments to the Wall Street Journal, not the press release. The stated goal is broadly framed as payment integrity — surfacing audit noncompliance, chronic deficiencies and, ultimately, fraud, waste and abuse in federal health spending, with the AI flagging items for human reviewers rather than making enforcement calls — a category that runs into large, hard-to-track sums across Medicaid, Medicare and grant programmes.
The department has reportedly already alerted governors and treasurers in every state that their audit filings will be run through these tools. That detail matters: it signals this is not a quiet internal pilot but an explicit, ongoing oversight mechanism that states are being told to expect. In plain terms, HHS is proposing to point a general-purpose large language model at mountains of financial documentation and ask it to flag what looks wrong.
It is worth being precise about what this is and isn’t. HHS is not, on the available reporting, claiming that ChatGPT will make final determinations of wrongdoing. The framing is analysis and pattern-spotting at scale. But the line between “the AI flagged this” and “this state is under scrutiny” is thinner than it sounds, and that is where the promise and the peril both live.
The upside
Start with the honest case for doing this. State single-audit reports are long, dense, inconsistently formatted, and produced in volume. A human review team can read only so many pages, so fast, and audit backlogs are a chronic feature of public finance everywhere. An AI system that can ingest thousands of pages, normalise the language, and cluster anomalies is doing work that no realistic headcount could match on the same timeline.
Speed and scale are the obvious wins. A model can read all 50 states in the time it takes an analyst to get through one, and it can re-read them whenever a report is updated. That turns oversight from an annual, sampled exercise into something closer to continuous monitoring.
The subtler benefit is pattern-surfacing. Fraud rarely announces itself; it hides in the gap between what one document says and what another implies. Language models are unusually good at flagging inconsistencies — a vendor that recurs across unrelated programmes, spending that spikes without a matching justification, boilerplate that has been quietly copied between filings. These are the needles humans miss precisely because the haystack is too big to hold in one head.
And it is cost-effective, at least on paper. Off-the-shelf AI carries a low marginal cost per document compared with hiring and training specialist auditors. For an agency under pressure to show results without expanding staff, that arithmetic is seductive. Used well — as a triage layer that decides where trained humans should look — this is a genuinely defensible use of the technology.
The risks
Now the harder part. Using a general-purpose chatbot for high-stakes oversight raises accuracy, auditability and due-process questions that off-the-shelf tools were never designed to answer.
The first is accuracy. Large language models hallucinate — they generate plausible, confident, wrong statements. In a chatbot conversation that is an annoyance; in a fraud referral it is a false accusation. A false positive here does not just waste analyst time; it can put a state programme, and the people who run it, under a cloud they did nothing to earn. High-stakes contexts invert the usual tolerance for AI error: being roughly right most of the time is not good enough when the exceptions carry legal and reputational weight.
The second is auditability. If an AI flags a state’s spending, someone will eventually have to explain why — to that state, to Congress, potentially to a court. “The model said so” is not a chain of reasoning that survives contact with due process. General chatbots are notoriously poor at producing stable, reproducible, inspectable justifications; ask the same question twice and you may get two different answers. An oversight process that cannot show its working is not oversight, it is assertion.
The third is data governance and vendor dependence. Feeding sensitive federal audit data into third-party AI systems raises immediate questions about where that data goes, how it is retained, who can access it, and whether it trains future models. Building critical government oversight on a commercial vendor’s product also creates dependence — on that vendor’s pricing, uptime, model changes and terms of service. When a private company’s roadmap can reshape how a government detects fraud, the public has lost a measure of control it should be reluctant to give up.
None of this is an argument against the tool. It is an argument that the guardrails — human review of every consequential flag, transparency about how AI is used, and rigorous data governance — are not optional extras. They are the difference between a decision-support system and a due-process failure.
The India read
For an Indian audience, the temptation to copy this playbook is obvious and growing. India runs vast welfare and health schemes, from Ayushman Bharat to state-level programmes, generating exactly the kind of audit and utilisation data that overwhelms manual review. The CAG, state audit departments and scheme administrators all face the same problem HHS faces: too many documents, too few reviewers, and a public expectation that leakage will be caught.
Off-the-shelf AI is an understandable first reach — it is cheap, fast to deploy, and already familiar to staff. Used as a triage and pattern-detection layer, it could meaningfully sharpen public oversight. But the risks translate directly, and some sharpen. Feeding citizen-linked health and welfare data into foreign commercial AI systems collides with India’s own data-protection framework and its digital-sovereignty ambitions. A false flag against a beneficiary or a local official can do real harm in a system where the accused often lacks easy recourse.
The design principles worth borrowing are the boring ones. AI helps governance when it narrows where humans should look, keeps a human in the loop on every consequential decision, logs its reasoning in an inspectable form, and runs on infrastructure the government actually controls. AI endangers governance when it makes or effectively makes determinations, when its outputs cannot be audited, when sensitive data leaves sovereign control, and when “the algorithm flagged it” becomes a substitute for evidence.
The HHS move will be watched precisely because it tests that line in public, at scale, on money that matters. The efficient path and the accountable path are not the same path by default — they only converge if the guardrails are built in from the start. For any government eyeing the same shortcut, that is the whole lesson.
