The AI Pilot Era Is Over. What Comes Next Requires Governance, Not Enthusiasm.

The Quiet Takeover

Somewhere in a state benefits office right now, an AI agent is routing a case file. In a federal procurement shop, another is screening vendor bids. In a county HR department, one is answering employee questions about leave policy. None of this was announced with a press release. It happened the way most operational changes happen in government: gradually, then all at once.

This is not speculation. According to Nextgov/FCW reporting on official GAO findings, documented AI use cases across eleven federal agencies nearly doubled from 571 in 2023 to 1,110 in 2024, and generative AI usage specifically increased ninefold in the same period. That is not the trajectory of a technology in early evaluation. It is a technology in operational deployment.

The tone here is not alarm. Agencies moving this decisively on AI is, in many respects, a success story. But that momentum creates a tension that every public sector leader we work with feels acutely. Deployment has outpaced the frameworks meant to govern it, and in regulated environments, that gap is not merely an IT problem. It is a compliance problem, a mission-risk problem, and ultimately a public trust problem.

The question is no longer whether AI is running government operations. The question is whether anyone is accountable when it makes a mistake, and whether the systems in place can answer that question before it becomes a headline.

The Data Is Unambiguous

Abstract illustration of two diverging paths, one luminous and fast-moving, one lagging behind, representing the gap between AI deployment and governance readiness

The scale of current deployment is worth sitting with, because the instinct among many agency leaders is still to think of AI as something they are preparing for. The data suggests otherwise.

Appian, a process automation vendor with a commercial interest in public sector AI adoption narratives, has reported that a survey of U.S. public sector workers found 37% describe their organization's AI integration as advanced, with AI already embedded across HR, case management, procurement, grants, and cybersecurity. To put that in concrete terms: more than one in three public sector respondents say AI is not a side experiment but a core part of how work gets done today. That figure is vendor-commissioned and should be read as Appian's reported finding, but it is directionally consistent with the GAO's independent count of more than 1,100 documented federal AI use cases across just eleven agencies.

The counterweight, however, is stark. A survey by Smarsh and FTI Consulting, reported by Corporate Compliance Insights, found that 55% of enterprises are actively deploying AI while only 26% say their governance frameworks are fully aligned with that pace. To make that gap tangible: Consider a hospital where more than half the operating rooms are running new procedures, but fewer than one in four have updated their safety protocols to match. That is roughly the ratio at which enterprise AI is being deployed against the governance structures meant to oversee it.

McKinsey's 2025 global AI survey reinforces the pattern: 88% of organizations use AI in at least one business function, but only 33% have scaled beyond pilots, and just 6% report meaningful financial impact. The difference between deploying AI and realizing value from it, in most cases, comes down to governance: the structures that allow an organization to move from an interesting proof of concept to a trusted, auditable, operational system.

Governance frameworks are not keeping pace with deployment. In regulated environments, that lag has a name: liability.

Why Regulated Sectors Feel It Worst

A person typing on a laptop in a dimly lit office environment, a shadow cast across the screen suggesting unseen or unsanctioned activity

The governance gap is not evenly distributed. It is sharpest precisely where the compliance stakes are highest, and that is not a coincidence.

Nutanix, a cloud infrastructure vendor, found in its enterprise survey that public sector, healthcare, and financial services organizations face the greatest risks from shadow AI and data sovereignty failures. In regulated environments, the ordinary problems of ungoverned AI do not stay ordinary. They escalate into audit findings, FISMA violations, HIPAA breaches, or data sovereignty failures.

Shadow AI is the specific mechanism. When employees use AI tools without IT approval (and they are doing so in every organization we have worked with), they are potentially exposing sensitive constituent data to systems that have not been vetted for compliance. According to IBM's 2025 Cost of a Data Breach Report, as cited by SentinelOne, a cybersecurity vendor, incidents involving shadow AI carry a significant cost premium over other security breaches, and the vast majority of breached organizations lacked proper AI access controls. For agencies that adopted informal AI tools precisely because they seemed low-stakes, that finding reframes the risk calculus considerably.

Healthcare illustrates the problem with particular clarity. Research published in a peer-reviewed NIH repository, drawing on a systematic review of 35 governance frameworks, found that governance infrastructure consistently lags clinical AI deployment. Independent reporting in Healthcare Finance News found that the median 2026 budget share for AI governance and safety in hospitals is just 4.2%, and only 22% of hospitals could deliver a complete, auditable AI explanation to regulators within 30 days. For an industry operating under HIPAA, that is not an abstract risk posture. It is an audit finding waiting to happen.

The Cloud Security Alliance, a nonprofit industry consortium, notes that AI agent systems are categorically different from the tools that informed existing governance frameworks, and cites Gartner's projection that 40% of enterprise applications will embed task-specific agents by end of 2026. For regulated sectors, that compression of timeline is not an abstraction. It is an operational deadline.

The sectors with the most to gain from AI are also those with the most to lose when it goes ungoverned.

A Governance-First Playbook That Actually Fits Government

The diagnosis is clear enough. The harder question is what to do about it, practically, in organizations with legacy systems, constrained budgets, and staff already using AI tools informally.

The first move is to know what you actually have. AI use cases are proliferating faster than most agencies realize, and shadow AI is almost certainly part of the picture. A complete AI inventory, including unsanctioned tools, is the prerequisite for everything that follows. OMB Memorandum M-24-10 already requires federal agencies to maintain AI inventories and designate Chief AI Officers. Its successor, OMB M-25-21, issued in April 2025, directed agencies to develop AI strategies and invest in the technology with governance as a co-requirement rather than an afterthought. For many agencies, the inventory mandate is not a new idea. It is an unfulfilled one. Starting there is a strategic intelligence exercise that reveals what the organization is actually doing with AI, before an auditor does it for you. In our experience conducting these assessments, a thorough AI inventory typically surfaces two to three times more active tools than IT leadership initially estimates.

The second move is to establish human oversight and auditability requirements before scaling. Every AI system touching rights, benefits, or public safety decisions needs a defined human review mechanism and a documented audit trail. The UK Government's AI Playbook, developed with input from over 20 government departments and more than 50 experts, operationalizes this principle with practical guidance on team structure, ethical procurement, and oversight mechanisms. Singapore's Responsible AI Playbook takes the same structured approach, providing a practical starting point for guarding AI applications against basic risks at the deployment and monitoring stage. At least 30 U.S. states have now issued guidance on state agency AI use, and legislators considered over 150 bills on government AI in 2024 alone. The policy environment is moving. Agencies that build oversight mechanisms now will be ahead of the compliance curve rather than scrambling to catch up.

The third move is to align procurement with the NIST AI Risk Management Framework. Buying AI without governance criteria embedded in the contract is how shadow AI gets institutionalized at scale. The NIST AI RMF gives procurement teams a structured vocabulary for specifying auditability, explainability, and data handling requirements, connecting governance processes with technical security controls and operational monitoring. As FedScoop has noted, in the absence of a comprehensive federal AI framework, information governance has become the control layer agencies can act on today.

Governance is not a gate. It is a map.

California Showed It Can Be Done

On June 29, 2026, Governor Newsom announced that Claude would become the first AI productivity tool available to all California state agencies, as well as cities and counties, at a 50% discounted rate. The partnership is not just a procurement deal. It is a governance architecture. By centralizing access to a single vetted platform, the state eliminates the proliferation of ad hoc tool adoption across dozens of departments, precisely the dynamic that creates shadow AI exposure. StateScoop reporting confirms the practical scope: agencies can use the tools to streamline administrative work, analyze large volumes of information, draft documents, improve constituent services, and power Poppy, an AI-powered digital assistant for public employees.

This is what governed, centralized AI adoption looks like in practice. Not a cautious pilot in one department, but a structured rollout with pricing, training, and oversight built in from the start. The UK's AI Playbook, launched in February 2025 with contributions from over 20 government departments and 50 experts, demonstrates the same model is replicable across jurisdictions. What California built is not a California solution. It is a template others can follow.

Governance Is the Accelerator, Not the Brake

Abstract illustration of bright green light accelerating through a dark tunnel with motion blur, representing governance as an enabler of speed and scale rather than a barrier

There is a persistent instinct in many organizations to treat governance as a friction layer, as the thing compliance requires before you can do the interesting work. That instinct is wrong, and the data says so.

The underlying logic is corroborated by independent research. UC Berkeley's Center for Long-Term Cybersecurity documented 35 AI governance implementation efforts in practice, providing a concrete picture of how organizations move from aspirational principles to operational systems. In our reading of that body of evidence, the organizations that scale are the ones that treat governance as an operational requirement, not a policy aspiration.

Without governance, every new AI deployment requires a new round of stakeholder reassurance, legal review, and risk negotiation. That cycle keeps organizations permanently in pilot mode. McKinsey's finding that only 33% of organizations have scaled AI beyond pilots is not primarily a technology problem. It is a governance problem. To put that in concrete terms: if the same ratio held across the roughly 1,100 documented federal AI use cases, fewer than 400 would be operating at true production scale. Agencies that govern well can scale with confidence. Agencies that skip governance will eventually be forced to stop, audit, and rebuild at far greater cost, under conditions of external pressure rather than internal choice.

StateScoop's reporting on a UC Berkeley School of Information analysis found that states formalizing governance frameworks, including governance councils and clear rules around human oversight and data privacy, are expanding AI use with confidence rather than caution. The organizations that will get the most from AI over the next three years are those building governance infrastructure now, not those waiting for a perfect policy environment that may never arrive.

The point of governing AI well in government is not to satisfy auditors. It is to serve people better, at scale, with accountability. That is the work we do with our clients, and it is why governance is where every AI engagement we take on begins.