Too many organisations still approach AI governance as a legal workstream rather than an organisational capability. Programmes rarely struggle because teams misunderstand regulation. They struggle because governance, control capacity and operational ownership were never designed into the project in the first place.
The AI model is not the problem. The operating model is.
By the time governance conversations begin, the business case has usually been approved and the budget allocated — with no meaningful assessment of the operational impact on Risk, Compliance and the other control functions. Governance then arrives as an additional cost, rather than as part of the investment decision.
One sentence recurs in industry discussions when AI projects reach Compliance:
“We didn’t budget for that.”
As the regulatory landscape tightens — the EU AI Act’s requirements for high-risk systems, DORA, sector-specific rules, the expectations of national competent authorities — governance is no longer optional.
But the real challenge is not complying with one more regulation. It is building an organisation capable of operating AI responsibly across Product, Engineering and the control functions.
01Governance starts with investment decisions
AI governance does not begin when Compliance reviews the project. It begins when someone decides that the project is worth funding.
At that moment, the organisation is not only defining the technology investment. It is also deciding whether the project budget includes the operational impact on Product, the first line of defence, Risk, Compliance and, where relevant, Internal Audit.
Product, Engineering, Risk and Compliance should therefore shape the business case together before the budget is fixed. The project should assess not only development and delivery costs, but also the FTE capacity required to review, monitor, challenge and evidence the control framework.
Run focused workshops by theme rather than one oversized governance meeting. Separate the business use case, data, model risk, prompts, code, human oversight, vendor management, access, ownership, documentation and operational controls.
Approving an AI project also means approving the governance capacity required to operate it.
02Governance has an operating cost
This is one of the most underestimated aspects of AI governance.
Every new AI use case creates work for the control functions: Risk assessments, Compliance reviews, periodic testing, monitoring, incident management, documentation updates and, where relevant, Internal Audit assurance.
A widely reported pattern across the sector is AI projects being approved and launched before any additional FTE capacity has been provisioned for the control functions.
Product teams are staffed. Engineering teams are funded. But no equivalent capacity assessment is completed for Risk, Compliance or Internal Audit.
As a result, governance becomes constrained by available headcount rather than designed into the operating model from the outset.
None of these activities happen automatically. They require people.
A control that exists on paper but cannot be performed because no operational capacity was funded is not an effective control.
Launching an AI project without provisioning the FTE capacity required to operate its governance framework is no different from launching a product without funding the engineering team that will build it.
Controls without people to execute them are not controls. They are intentions.
03You are not governing a model. You are governing an AI system.
Governance discussions still focus too heavily on the model.
Modern AI systems also include prompts, orchestration workflows, agent configurations, APIs, application code, external integrations, permissions and autonomous actions.
Organisations should be able to answer:
- Who designs, tests and approves production prompts?
- Who can modify an agent’s instructions or behaviour?
- How are prompts and configurations version-controlled?
- Who reviews the code before deployment?
- Who authorises connections to external systems?
- Who can stop or roll back the agent?
Whether an agent is written in Rust, Python or another language, its code should follow the same governance standards as any critical application: source control, peer review, testing, traceability, controlled deployment and rollback capability.
Challenging a system in production is only possible if the operational tooling makes its behaviour visible and reversible. Concretely, this requires:
- Observability and logging — capturing inputs, outputs, tool calls and decisions so behaviour can be reconstructed and examined after the fact. Without it, there is nothing concrete to challenge.
- Continuous evaluation and monitoring — automated checks against defined quality, safety and drift metrics on live traffic, with alerting thresholds, rather than one-off pre-deployment testing.
- Adversarial testing (red-teaming) — structured attempts to break the system through prompt injection, jailbreaks, data exfiltration or tool misuse, before launch and periodically after it.
- Guardrails and enforced limits — input and output controls, scoped tool permissions, action and rate limits, and human-in-the-loop checkpoints for high-impact actions.
- Versioning and reproducibility — version control of prompts, models, configurations and datasets, so any production behaviour can be traced to a specific version and rolled back.
The practical question is simple: who in the organisation is actually able to challenge a production prompt, an agent’s behaviour, an integration or a code deployment?
Defining who reviews these components, what evidence they require and when technical escalation is mandatory is not an implementation detail. It is a governance decision.
AI agents are software systems. Their prompts, code, permissions and integrations all require defined ownership and review.
04Responsibility follows operational control
Governance does not belong only to Compliance.
It belongs to everyone who can influence how the AI behaves: the person writing prompts, the engineer modifying code, the Product Manager approving the use case, the administrator granting permissions and the operator connecting the agent to external systems.
Responsibility should follow real decision rights and operational power.
- What is the person authorised to change?
- How are their actions reviewed?
- Who approves material changes?
- Who accepts the residual risk?
- Who can suspend or retire the system?
The person prompting or operating the agent should have the required expertise, training, authority and supervision. The more operational power a user has, the clearer the accountability and escalation path should be.
Accountability without operational control is unclear. Operational control without accountability is dangerous.
05Buying AI does not buy governance
Using a third-party LLM, agent platform or AI solution does not transfer accountability. The provider supplies the technology; the deploying organisation remains responsible for how that technology behaves in its own context.
Vendor due diligence should therefore extend beyond cybersecurity questionnaires and contractual assurances. Three dimensions decide whether a vendor dependency is actually governed, rather than merely contracted.
- Effective audit rights. A contractual right to audit is not a control unless it can be exercised. That requires access to what matters — model and system documentation, evaluation results, change notifications and operational logs — and the internal technical capacity to interpret them. An audit clause nobody can execute is paperwork, not assurance.
- A real ability to suspend, roll back or replace. Continuity and exit are governance controls, not procurement footnotes. The organisation should be able to answer concretely: can this system be suspended or rolled back without halting the business, is there a fallback, and can the service be exited and the data recovered within an acceptable timeframe? Where switching or rollback is operationally impossible, the dependency itself becomes the risk — a concern sharpened by third-party ICT resilience and exit-strategy expectations such as those under DORA.
- Non-transferability of accountability. Under the EU AI Act, procuring a system does not remove the deploying organisation’s own obligations — using it within its intended purpose, ensuring human oversight and monitoring its behaviour. And under general civil liability principles, allocating risk in a contract governs the relationship between the parties; it does not extinguish exposure toward regulators or affected third parties. Contracts distribute risk. They do not distribute accountability.
Whether an AI use case is insurable — and on what terms and exclusions — is a useful external signal of how well the residual risk is understood and controlled. Insurance does not replace governance: a risk that cannot be insured, or only with broad exclusions, is usually a risk the organisation does not yet fully control.
Technology can be outsourced. Accountability cannot.
06Documentation is not governance
One reason AI governance is particularly difficult is that it sits at the intersection of Legal, Compliance, Risk, Software Engineering, Data, Cybersecurity and Product.
Unlike more traditional compliance programmes, AI governance requires organisations to understand technical concepts such as foundation models, prompts, orchestration, APIs, agent architectures, source code, permissions and deployment pipelines.
That complexity often leads organisations to compensate with documentation.
More policies. More procedures. More registers. More templates. More committees.
Documentation is necessary. But documentation cannot compensate for a lack of technical understanding. Nor can it replace clear ownership, operational capacity or effective controls.
Compliance, Risk and Product teams do not need to become software engineers. But they do need enough technical literacy to understand what they are governing, challenge the design and identify where risk enters the system.
AI governance may be the first governance discipline that cannot succeed without a shared understanding of software engineering concepts across business and control functions.
Technical literacy should therefore be translated into a practical competency framework for the first, second and third lines of defence — not left as a generic training objective.
Every document should still have a clear operational purpose.
Ask one simple question:
Who uses this document to make which decision or perform which control?
If nobody can answer, the document probably exists to satisfy a process rather than improve governance.
Governance programmes rarely fail for lack of documentation. They fail when nobody owns the controls, when nobody has the capacity to perform them, or when those responsible for governance do not understand the technical system well enough to challenge it.
Good governance enables informed decisions. Bad governance generates paperwork.
Final thoughtDo you have an organisation capable of governing AI?
Do we have the budget to build it?
Do we have the operational capacity to govern it?
Both questions deserve the same level of attention.
When organisations discuss AI governance, they often ask whether they have the required documentation.
The more important question is:
Do we have an organisation capable of governing AI?
Governance is not a policy. It is not a committee. It is not an AI Act checklist.
It is an operating model that connects investment decisions, budget, strategy, product, engineering, operational capacity, control functions and accountability.
And that operating model must exist long before the first model goes live.