Programmable Governance & Probabilistic Humans
How probabilistic humans and collective intelligence could reshape AI governance
It might sound counter-intuitive at first, but good AI governance needs to become both programmable and probabilistic if organisations are to make meaningful use of human judgement alongside machine intelligence.
The idea of ‘human-in-the-loop’ governance is simple and reassuring - AI systems may assist, recommend or automate, but somewhere in the process a human remains responsible for oversight and final judgement.
For early pilot deployments this model works reasonably well. Humans review outputs, approve sensitive actions or intervene when something appears wrong. But as AI systems become faster and more autonomous, we hit the limits of this approach quite quickly.
The scale and speed of modern systems will soon outstrip the cognitive bandwidth of manual oversight. A governance model built around humans inspecting individual outputs simply does not scale.
This does not make human judgement less important. If anything, the opposite is true. But the role humans play in governance needs to evolve.
In practice, most leaders already hold nuanced views about emerging risks. A security leader may suspect that monitoring systems could fail under certain conditions. A product leader may worry about reputational edge cases. Legal teams often sense regulatory ambiguity long before it becomes formal policy.
Yet governance structures rarely capture these insights clearly. They compress judgement into binary decisions: approved or rejected, compliant or non-compliant, acceptable or unacceptable, red or green.
An under-used superpower that organisations already possess, and which could help here, is the ability to harness distributed human judgement about uncertainty.
A probabilistic human expresses judgement differently. Instead of presenting certainty where none exists, they estimate likelihoods and levels of confidence. When those signals can be aggregated across many individuals, organisations gain a clearer picture of how risk is evolving across the system. This opens the door to a different kind of governance. Rather than inserting humans into isolated approval points, organisations can begin to treat collective judgement as a continuous signal about how uncertain the system really is.
Several mechanisms are beginning to emerge to support this shift. Some organisations experiment with prediction markets, allowing distributed expertise to converge into probabilistic forecasts about emerging risks, whilst others introduce structured dissent mechanisms, deliberately creating space for people to challenge prevailing assumptions and surface potential failure modes. And some leadership teams are beginning to convene probabilistic risk councils, where uncertainty is discussed explicitly and collective judgement informs governance decisions.
Taken together, these approaches allow organisations to move beyond episodic oversight toward something more adaptive: continuous calibration of uncertainty.
Let’s explore how these techniques work in practice and how leaders can use them to make human judgement legible inside increasingly complex AI systems.
Why human-in-the-loop governance begins to strain
For many leadership teams, the idea of human oversight in AI systems feels like a reassuring safeguard. If automated systems introduce uncertainty, the obvious response is to ensure that a person remains responsible for the final decision.
Yet organisations deploying AI at scale quickly encounter a different reality.
The challenge is not simply that systems are autonomous. It is that they operate within environments defined by speed, complexity and data volume that far exceed what traditional governance processes were designed to handle.
AI systems interact with constantly changing data, evolving models and interconnected workflows. Decisions that once occurred occasionally may now occur thousands of times per hour. Risk signals appear not as clear incidents but as patterns emerging across vast streams of activity.
Under these conditions, governance based on periodic review begins to struggle. Human judgement remains essential, but it cannot function effectively if it is only inserted at isolated approval points. This is where the idea of programmable governance becomes important.
Rather than relying entirely on manual oversight, programmable governance embeds certain rules, constraints and escalation paths directly into the systems themselves. Authority boundaries can be checked automatically before actions occur. Certain thresholds can trigger human review. Conflicts between objectives can halt execution and escalate to decision-makers.
In other words, governance becomes structural rather than procedural. Some forms of accountability are handled automatically within the system, while human judgement is reserved for the decisions that genuinely require interpretation, trade-offs and values.
When governance is structured this way, the human role changes. Instead of reviewing every decision, leaders focus on calibrating how the system interprets risk and uncertainty. And this is where probabilistic thinking becomes essential.
Most governance processes still ask leaders to express judgement in binary terms: approved or rejected, compliant or non-compliant, acceptable or unacceptable.
Yet the judgments leaders actually hold are rarely so definite.
A CISO may believe there is a moderate chance that monitoring systems would fail under certain conditions. A product leader may suspect that a new feature introduces reputational risk without being able to quantify it precisely. Legal teams may sense regulatory ambiguity long before it becomes formal policy.
These kinds of judgements contain valuable information. But governance structures compress them into simple approvals or objections.
The idea of the probabilistic human offers a different approach.
Instead of presenting certainty where none exists, probabilistic humans express judgement in terms of likelihood and confidence. When those signals can be aggregated across many individuals, organisations gain a clearer picture of how risk is evolving over time.
And once judgement can be expressed probabilistically, a new set of governance techniques becomes possible, and we enable a far richer audit trail of decision-making that can be used to train AI systems and improve future decisions.
Once judgement can be expressed this way, a new set of governance techniques becomes possible and we create a richer audit trail of how judgement is exercised, which can inform both future governance decisions and the training of AI systems themselves.
Where this tension shows up for leaders
The limitations of human-in-the-loop governance rarely appear as an explicit design flaw. Instead, they surface indirectly, as uncertainty that feels difficult to resolve through existing oversight structures.
Different leadership roles can encounter this tension in different ways, depending on where they sit in the organisation’s decision and accountability landscape.
CISO pain points: signals that arrive too late
For CISOs and security leaders, the strain often appears as a timing problem: risk reviews take place and systems are assessed against known threat models. Yet concerns about AI behaviour can emerge gradually, often through operational signals rather than formal governance channels.
A model may drift slowly outside expected parameters, monitoring alerts begin to cluster in unusual ways, small anomalies appear that do not yet justify escalation, but suggest that the system’s behaviour is shifting.
Traditional governance frameworks expect risks to be identified and addressed at defined checkpoints. But many of the signals that matter most in AI environments are probabilistic rather than definitive.
Security teams therefore find themselves working with a growing set of partial signals, indicators that something may be wrong, without a clear threshold that justifies intervention.
Legal and compliance pain points: decisions without certainty
Legal and compliance leaders tend to experience the tension differently. Governance processes often require them to classify a system in categorical terms: compliant or non-compliant, acceptable or unacceptable. Yet many AI deployments sit in ambiguous territory, particularly when regulations are evolving or when systems operate across jurisdictions.
Legal teams frequently recognise emerging risks early. They may sense that a deployment could attract scrutiny, or that regulatory expectations are shifting in ways that are difficult to formalise.
However, governance structures typically force those insights into binary decisions. A deployment is either approved or blocked, even when the underlying judgement is far more nuanced.
This can create uncomfortable dynamics. Legal teams appear cautious or obstructive when they are simply responding to uncertainty that has not yet stabilised.
Product leadership pain points: innovation slowed by rigid oversight
Product leaders encounter the same structural issue from another direction. AI-enabled features often evolve iteratively. Teams test new capabilities, refine workflows, and adjust behaviour based on real-world feedback. In this environment, risk rarely presents itself as a clear go-or-no-go moment.
Instead, risk appears as shifting probabilities.
A feature may be broadly safe, but introduce edge-case failure modes. A system may work well under typical conditions, but become fragile when interacting with other services.
When governance frameworks rely on discrete approvals, product teams can find themselves navigating a process that feels mismatched to the way the technology evolves. Reviews occur at specific milestones, while risk emerges gradually over time.
The result is often friction rather than clarity.
Executive leadership pain points: oversight that becomes symbolic
At the executive level, the tension appears as a widening gap between formal oversight and operational reality.
Leadership teams approve governance frameworks, establish policies, and review risk dashboards. Yet the speed and complexity of AI systems can make those structures feel increasingly abstract.
Executives may receive polished summaries of model performance or compliance posture, while sensing that the organisation’s true exposure is harder to quantify.
This does not usually reflect a failure of diligence. Rather, it reflects a mismatch between the episodic rhythm of traditional governance and the continuous evolution of AI-enabled systems.
Under these conditions, leadership oversight risks becoming symbolic: reassuring in principle, but too distant from the system to provide real-time calibration.
Across all of these perspectives, the underlying issue is the same - human judgement is present throughout the governance system, but it is expressed in ways that hide uncertainty rather than revealing it. Leaders are often forced to present confidence where what they actually possess is a probability.
This is where collective intelligence techniques become valuable. They allow organisations to capture and aggregate the probabilistic judgement already present across the system, turning it into signals that governance structures can actually use.
Read on to explore three techniques that can help organisations embed structured collective intelligence into AI governance:
prediction markets
structured dissent mechanisms / red team markets
probabilistic risk councils.
