A JSON schema Is not an AI governance layer

In the space of two weeks this September, the argument about who should govern AI produced four different answers, all of them from the people who build the systems.

Demis Hassabis floated an industry-funded oversight body, modeled loosely on FINRA, to test frontier models before deployment. Dario Amodei called on labs to slow frontier development and give independent evaluators genuine access, and days later Anthropic and Accenture committed at least $2 billion over five years to embedded evaluation. OpenAI started publishing reports on unexpected model behavior. And according to the Wall Street Journal, Mark Zuckerberg, Elon Musk and Jensen Huang each spoke with President Trump and persuaded him to drop the oversight body, arguing that a common regulator would protect the lead of the biggest labs, and that companies can secure their own systems without one.

Whichever of these positions you agree with, none of them talks about the code that decides whether the agent in your application is allowed to send an email, rotate a key, merge a branch or move money.

That decision is not made by a lab, and it is not written in a policy; it is made by a function in your own code, and that function returns true or false.

I want to show you a bug in that function, and it’s not a thought experiment: an independent stress test found it in PIC, an open protocol I maintain for checking agent actions before execution. The schema accepted an uppercase SHA-256 digest, the reference runtime quietly normalized it and carried on, and the specification had never said whether that representation was valid. No cryptography failed. The protocol simply had more than one defensible meaning, which is harder to spot than broken cryptography, and I suspect many agent deployments running today have the same problem.

Governance ends in a boolean

Stack Overflow’s April 2026 pulse survey of 1,100 developers and technologists found that 60% of them block agents from making unapproved system changes. That makes sense, but it leaves out the difficult part: what counts as approved, and will every component in the path agree on the answer?

Here is an agent proposing a payment. It supplies the tool it wants to call, the inputs that shaped the decision, and the evidence behind its claim that the payment is authorized.

{
  "action": {"tool": "payments.transfer", "args": {"amount": 500}},
  "impact": "money",
  "provenance": [
    {"id": "invoice", "trust": "trusted"},
    {"id": "invoice", "trust": "untrusted"}
  ],
  "claims": [
    {"text": "The invoice was approved", "evidence": ["invoice"]}
  ]
}

A JSON Schema will confirm that provenance is an array, that every entry carries an identifier and a permitted trust value, and that the required fields are present. Unless the protocol adds a rule of its own, this payload is valid. The identifiers are not unique, and the evidence reference now points at two different entries.

Suppose the policy is simple: a payment may run only if the evidence behind its claim comes from a trusted source. (For now, take the trust labels in the payload at face value; I come back to who gets to set them later.) To apply that policy, the verifier has to work out which entry “invoice” refers to, and two reasonable engineers resolve it differently.

# Implementation A, Python: stop at the first match
def resolve(provenance, evidence_id):
    for entry in provenance:
        if entry["id"] == evidence_id:
            return entry
    return None

// Implementation B, TypeScript: index by id, last write wins
const index = Object.fromEntries(provenance.map(e => [e.id, e]));
const resolve = (id) => index[id];

Neither lookup is a bug on its own; both are normal, idiomatic code, and the specification never said which entry should win. Given the payload above, A resolves the evidence to the trusted entry and the payment executes, while B resolves it to the untrusted entry and the payment is blocked. Both verifiers run the same protocol version, with the same policy and the same trusted keys; the only difference is how each one looks up an id. So the same bytes, checked against the same schema, give two different results, and an attacker only needs to know which service is on the other end of the call.

The schema did its job, but that job is limited. It checks the shape of the payload: that fields are present, have the right type, and hold allowed values. It can even express some rules between fields, such as “if this field is present, that one is required”. What it can’t express is a rule like the one this payload needs: the “invoice” cited as evidence in claims has to point to exactly one entry in provenance. Standard JSON Schema has no way to say that a value in one place must match exactly one value somewhere else in the document; and uniqueItems doesn’t help either, because the two entries have different trust values, so the schema sees them as two different items. Whether the payment goes through depends on that link, and the schema leaves it to each verifier to work out.

This is a parser differential, and it is much older than agents

Security people have seen this many times: two components read the same input in two different ways, and the difference between them is where the vulnerability is. HTTP parameter pollution is one example, where a proxy reads the first id and the application reads the last. XML signature wrapping is another: the verifier checks the signature on one element, while the parser reads a different one. Then there is JWT alg confusion, where the token itself says which algorithm should be used to check it. And with HTTP request smuggling, a proxy and the server behind it disagree about where one request ends and the next one begins.

This kind of bug has been around for about twenty years. What we have learned from it is fairly simple: when two implementations need to agree on what the same bytes mean, any detail the specification leaves open can be read in two ways, and sooner or later an attacker will take advantage of that.

Agent protocols are now running into the same problem, and PIC did too: the stress test I mentioned found these gaps, and we closed them by tightening the specification and adding each case to the conformance tests I describe below. Most of these protocols are only a year or two old, and they haven’t yet been tested the way token and signature formats have been over the years; so I think a governance layer written this year should be checked with the same care we would give a new token format.

Helpful normalization quietly rewrites the protocol

Developers normalize input constantly. We trim whitespace, lowercase identifiers, repair Base64 padding, accept alternate encodings, all because rejecting data that is obviously recoverable feels hostile. That instinct is right in a search box and dangerous at an authorization boundary.

Suppose a policy permits exactly one tool:

expected_tool = "payments.transfer"
proposed_tool = "payments.transfer "   # note the trailing space

Compare those strings exactly and the proposal fails. Call strip() on both first and it passes. Neither behavior is irrational on its own; the defect is that individual implementations get to choose.

With signatures, the problem is more serious than it looks. A SHA-256 digest is usually rendered as hexadecimal, and uppercase A through F denote the same number as lowercase a through f, so lowercasing the input looks harmless. But it’s not, because signatures cover bytes rather than numbers. If the producer signed a payload containing the uppercase form and the verifier lowercases before constructing its canonical payload, verification fails on an authentic proposal.

Flip it around and it gets worse: a verifier that checks the signature over the original string and then normalizes the value before applying the policy is no longer enforcing exactly what was signed. For a hex digest, lowercasing gives the same number, so nothing changes; but the same habit applied to other fields can change which policy rule matches (a trimmed or lowercased tool name, for example, can match an allow-list entry that the signed name would not), and at that point the policy is acting on a value the signer never signed.

Base64 has the same problem: some libraries tolerate embedded whitespace, some restore missing padding, some silently translate the URL-safe alphabet, while others reject all three. When a signature decides whether an action may execute, those choices made by your Base64 library become part of your authorization policy, even though nobody on your team wrote them or reviewed them.

It helps to separate two decisions here. The first is how the signed payload is turned into bytes. Standards such as RFC 8785 for JSON, XML c14n before it, and the deterministic encoding rules in COSE fix things like field order, number format, and escaping, so that every implementation signs exactly the same bytes; but they leave the contents of a string alone, so RFC 8785 keeps an uppercase digest uppercase. The second decision is which values the protocol accepts at all. Here you can either normalize them (for example, lowercase every digest before signing and before checking) or accept a single form and reject everything else; so uppercase hex fails, unpadded Base64 fails, and a tool name with a trailing space fails.

PIC does both. In its canonical attestation mode it signs a canonical serialization (it still keeps a legacy mode that verifies raw payload bytes), and for values it took the second route, for a simple reason: normalization is one more piece of code that has to be written again in every language that uses the protocol, and the history of c14n bugs suggests that different implementations of the same rules end up disagreeing, just as parsers do. With strict rejection, a payload in the wrong format is refused as soon as it arrives, with a clear error, instead of getting through and causing two verifiers to disagree later, when the problem is much harder to track down. If you are designing something similar, pick the approach you can describe completely in your specification; but do pick one, because otherwise the meaning of a signed proposal depends on whichever library happens to decode it.

Trust cannot come from the thing asking for permission

The ambiguity that mattered most was not about encoding at all, but about who gets to declare an input trustworthy.

{
  "id": "invoice",
  "source": "email_attachment",
  "trust": "trusted"
}

That field is easy to serialize, but on its own it tells the verifier almost nothing it can rely on. The proposal is produced upstream of the verifier, frequently by the very agent requesting permission to act. Honoring the field at face value lets the requester supply its own authority, which is the classic confused deputy problem, only written in JSON.

Trust has to be derived inside the enforcement boundary. The producer may identify a source and attach evidence, while the verifier decides whether that evidence leads back to something the operator actually trusts, such as a known signing key.

effective_trust = "untrusted"

if signature_is_valid(evidence, operator_keyring):
    effective_trust = "trusted"

if proposal.impact == "money" and effective_trust != "trusted":
    return BLOCK

The difference between integrity (the data wasn’t changed) and authority (someone allowed to approve it actually did) is what matters most here. A matching hash proves the verifier received the bytes it expected, but it says nothing about who produced them, who approved them, or whether they should authorize a payment. An attacker can supply a file together with the correct hash of that same file; the integrity check passes, because nothing was changed, but nobody with any authority has approved anything.

A signature carries authority only when the verifier looks up the signing key in a list of keys the operator trusts (a list the proposal has no way to add to), and only when the signature covers the fields that matter: the tool, the arguments, the impact class, the relevant provenance, and the expiry. Otherwise a genuine signature can approve one thing while the action itself is something else; for example, a signature for a transfer of 500 could end up attached to a request for 50,000.

In other words, it is the verifier that marks an input as trusted, only after checking the evidence, and it should never take that label from the agent.

Where this lands in the stack you are already running

None of this is specific to my protocol. Take the Model Context Protocol: tool definitions describe their inputs with JSON Schema; there are separate SDKs written in several languages; hosts add their own approval prompts and allow-lists on top; and before the arguments reach the tool, they have usually passed through a model, a client, a transport and a host policy check. That is a lot of separate code reading the same data, and each piece can read it a little differently. The same is true if you wrote your own approval step, meaning the code that checks a tool call from the model before your application actually runs it.

You don’t need to adopt anyone’s protocol to check this. Take your own approval path and ask three questions. If the same field appears twice, which one wins, and does every component agree? If a tool name arrives with a trailing space, what happens, and is it the same thing in your CLI as in your HTTP bridge? If the payload says it is trusted, who decided that?

Three layers, not one (A schema validates shape, not authority)

JSON Schema describes the structure of JSON data, and it is good at required properties, types, enumerations, patterns, and composition. Those controls still matter, because rejecting malformed input early leaves the semantic layer fewer cases to worry about.

But the questions that decide whether an agent should act usually involve more than one field, and sometimes more than one system. Does an evidence identifier resolve to exactly one provenance entry? Does the signature cover the same tool and arguments now being proposed? Was the signing key trusted at the moment of verification, not merely at generation? Which error wins when several checks fail at once?

If you force all of that into one schema, you get a document that looks precise but still doesn’t say how the runtime should behave. A stronger design separates three layers, each narrowing what the next is allowed to see:

1. Schema validation rejects malformed structure and non-conformant wire representations.
2. Semantic verification resolves references, validates bindings, and derives effective trust.
3. Policy evaluation decides whether this verified proposal may perform this action now.

Each of these layers needs its own code and tests, and a note in the documentation asking implementations to behave sensibly can’t replace any of them.

Conformance vectors are the executable specification

A written specification cannot cover every interpretation its authors failed to imagine. A second implementation finds those gaps within days, because another language, another library, and another developer make different reasonable choices.

The practical answer is a set of conformance tests you can run. Each vector holds the exact input, the expected verdict, and a stable error code where the proposal must fail. Every implementation runs the same vectors, and the suite deliberately covers the cases ordinary application tests never reach.

These vectors do more than catch regressions. They also define how the protocol behaves in the cases people could disagree about, and once an ambiguous case is settled, it becomes a permanent test rather than another paragraph that future implementers get to interpret for themselves.

The order in which errors are reported should be settled in the same way. One proposal can contain duplicate identifiers, the wrong tool name and an invalid signature all at once. If one implementation reports the duplicate identifier first and another reports the bad signature first, the operator gets different advice for the same input, depending only on which service they happened to call; and every adapter ends up with its own order. Deciding which check runs first, and writing that into the conformance tests, makes the behavior predictable and much easier to debug.

Secure defaults have to survive every adapter

The verifier itself can be strict and still be undone by the code around it. For example, the CLI turns on strict checking but the HTTP bridge doesn’t; a framework adapter keeps an old, looser default months after the main verifier changed; or a caller turns off evidence checks to save time and keeps accepting inputs that say they are trusted, without noticing that the two together are unsafe.

So every way into the verifier (the CLI, the HTTP bridge, each framework adapter) has to start with the same strict settings. When the protocol changes, you may still need to accept an old field or an older, looser behavior for a while, so that people who built on the previous version have time to migrate. That is reasonable, but the old behavior should only apply when someone turns it on deliberately, and the software should show a clear warning whenever it is used. If a migration flag can bring back the old behavior without telling anyone, then whether a deployment is secure depends on how it happened to be configured, and that is rarely written down.

Why the policy fight makes this worse rather than better

Earlier this year I argued that agents expose security checks organizations never actually wrote, because those checks had always lived in human judgment. Writing them down turns out to be only the first step: the code that replaces that judgment also has to mean the same thing everywhere it runs.

This is why one of those four positions matters more here than the others. The argument that each company should secure its own AI, without a shared body defining what secure means, is a description of many independent teams making many independently reasonable choices about normalization, trust derivation, and error precedence. A disagreement between companies is not the same thing as a disagreement between parsers; but in practice it leads to a similar situation, where different implementations give different answers to the same question.

The concern behind it is fair enough; a testing body set up by the biggest labs could well have helped those same labs stay ahead. But if nobody agrees on shared rules at the top, those rules have to be agreed lower down, by the people who write verifiers, in public and through conformance vectors that every implementation can run; and in practice, a good part of that work will fall to developers.

Governance becomes real in the verifier

Policies, schemas, signatures, and human approvals are all useful, but none of them actually stops an agent until some code turns them into a yes or no before the tool runs. That code doesn’t have to predict everything an advanced model might do; it has a much shorter list of jobs: reject ambiguous identifiers, compare security-sensitive values exactly as specified, keep integrity and authority distinct, decide trust only from evidence the verifier can check itself, and, for the same proposal and policy, return the same verdict everywhere the protocol is implemented.

If any of these is left to each implementation, one service can allow a proposal that another one, with the same policy, blocks.

The public argument is about how fast AI should advance and who referees it, but the question for your own code is a simpler one: whether the systems controlling today’s agents can agree on what they are reading. You can find out by running the same test inputs through each of them and comparing the answers.