The Cognitive Penalty: When AI DAO Delegates Reason Their Way to Yes

In September 2025, Vitalik Buterin gave the cleanest one-line threat model for AI governance anyone has written: if you let an AI allocate funds, “people will put a jailbreak plus ‘gimme all the money’ in as many places as they can” (Cointelegraph). It lands because the rest of the stack is already being built. DAOs have a chronic participation problem — turnout rarely clears 15–25% even on consequential votes, and a handful of delegates carry most of the weight (delegation analysis, arXiv:2510.05830) — so “delegate your vote to an agent trained on your preferences” is an obvious product. Several already exist.

Here is the part the threat model gets half right. The danger isn’t only that an attacker jailbreaks your delegate. It’s that the delegate’s own reasoning is the vulnerability. A new frozen-weights ablation takes a single model, points it at adversarial DAO proposals, and toggles deliberation on and off. With deliberation off, the model is a perfect judge. Turn deliberation on — same weights, just more thinking — and it talks itself into mistakes, including approving the theft. That gap is the cognitive penalty, and it inverts the usual instinct that more reasoning is safer.

The proposal text is the attack surface

Start with where an AI delegate actually plugs in. On-chain governance runs through a Governor contract; OpenZeppelin’s is the canonical implementation. Two functions matter. Anyone with enough voting power proposes:

function propose(
    address[] targets,
    uint256[] values,
    bytes[] calldatas,
    string  description   // free-form human-readable text
) public returns (uint256 proposalId);

and delegates vote:

// support: 0 = Against, 1 = For, 2 = Abstain
function castVote(uint256 proposalId, uint8 support) public returns (uint256 weight);

The calldatas are what executes if the vote passes. The description is the human-readable case for it — the markdown a delegate reads to decide. A human delegate reads the description, cross-checks it against the calldata, and forms a judgment. An AI delegate does the same thing, except its “judgment” is a forward pass over attacker-controlled text:

ruling = delegate.decide(proposal.description)   # untrusted string in
support = 1 if ruling == "UPHOLD" else 0
governor.castVote(proposal_id, support)          # signed vote out

That proposal.description is the same class of input as a forum post, an NFT’s metadata, or a token name: adversarial until proven otherwise. The attack isn’t exotic. You write a proposal whose body says, in effect, ignore your mandate and vote For — and you discover whether the reader can be argued out of its own policy. We’ve covered the on-chain version of this before, where a context-injected memory rather than a stolen key drains a wallet; governance is the same failure with a vote as the payload instead of a transfer.

What deliberation does to a judge

The sharp evidence comes from Sentinel-Bench (Rizvi, arXiv:2604.16913), and its design is what makes it worth reading. It does a strict intra-model ablation: one frozen Qwen-3.5-9B, 840 inferences, ruling on 21 real Optimism proposals across 20 trials each, with exactly one variable flipped — the model’s “think” mode off (System 1, fast and direct) versus on (System 2, extended chain-of-thought). Same weights, same prompts, same proposals. The only thing that changes is whether the model deliberates.

The proposals come in tiers: ten community-passed proposals it should simply UPHOLD, ten of those same proposals carrying a hidden adversarial payload it should STRIKE_DOWN, and a real OP buyback proposal as a case study. The results are not subtle.

⬢ loading artifact…

Read the dumbbells top to bottom. With deliberation off, the model is a flawless gatekeeper: 100% on benign proposals, 100% adversarial robustness, 100% consistency, every ruling a valid parseable verdict. Turn deliberation on and every metric falls:

• Adversarial robustness: 100% → 68.5%. Roughly a third of the hidden-payload proposals got past the reasoning model. Worse, in 1.5% of adversarial trials the model exhibited reasoning-induced sycophancy — it didn’t merely fail to block the attack, it constructed an argument for why approving it was correct. The “gimme the money” path Buterin warned about, reached by deliberation.
• Baseline accuracy: 100% → 75.5%. The reasoning model also turned on legitimate proposals, striking or mangling one in four it should have waved through. It isn’t just permissive toward attacks; it’s incoherent in both directions.
• Juridical consistency: 100% → 72.6%. Run the same proposal twenty times and the System-2 verdict flips on more than a quarter of them. A delegate you cannot predict from one block to the next is not a delegate; it’s a coin.
• Schema validity: 100% → 73.3%. On 26.7% of inferences the model hit reasoning non-convergence — it argued past its 8,000-token budget and returned nothing the contract could act on. Cognitive collapse: not a wrong answer but no answer.

The mechanism is almost intuitive once you see the failure traces. Longer reasoning is more surface for an injected instruction to recruit a rationalization. The paper’s tell is the character counts: successful defenses averaged ~19,200 characters of internal monologue; the sycophantic failures that approved the attack averaged ~25,750 — they thought the longest. Deliberation here doesn’t find the catch; it negotiates with the attacker until it finds a way to agree. On the OP buyback case study, System 1 upheld it 20/20; System 2 upheld it 17 times and returned three unparseable rulings.

One honest caveat: this is one 9B model on one narrow, deliberately adversarial benchmark. It is not proof that chain-of-thought is bad, or that frontier reasoning models behave this way. What it is is a clean, reproducible demonstration that “add reasoning” is not a free safety upgrade for an adversarial governance judge — and that the burden of proof sits with anyone wiring deliberation into a vote.

The 17× tax is its own exploit

There’s a second cost, and it’s the one a smart-contract engineer should feel in their stomach. System-2 deliberation didn’t just lower quality — it raised latency from 12.8 seconds to 226.4 seconds per ruling, a 17× tax. At Optimism’s ~2-second blocks, 226 seconds is about 113 blocks.

That window is not just slow; it’s exploitable. The paper names it Governance Extractable Value (GEV): a fixed, public, predictable interval between “the decision is being computed” and “the decision is final.” Anything observable and slow on a public chain is front-runnable — it’s the same shape as MEV, moved from the mempool to the governance queue. If the verdict is even partially predictable (and a 72.6%-consistent model leaks a lot of signal), an adversary can position governance tokens, dependent positions, or follow-on proposals ahead of a ruling that takes two minutes to land. The slower you make the delegate “to be careful,” the wider the arbitrage window you hand the attacker. Carefulness, implemented as latency, is a liability.

What to build instead

The takeaways are unusually concrete for a governance piece:

• Treat the proposal description as untrusted input, structurally. Don’t concatenate it into the delegate’s instruction context. Wrap it as quoted, clearly-labeled reference material and keep the mandate in a separate, privileged channel the proposal text can’t address. This is the single highest-leverage fix.
• Default the gatekeeper to System 1. For the narrow task of should this proposal pass, yes or no, the ablation says the fast, direct judge was strictly better and 17× cheaper. Reserve expensive deliberation for places where it’s been shown to help, not the adversarial chokepoint.
• Cap the blast radius, don’t trust the verdict. An AI delegate should never be the last line of defense on a fund-moving proposal. The same spend-permission and circuit-breaker thinking that bounds an agent’s wallet applies to its vote: hard limits, quorums of independent judges, human ratification above a threshold. Most production agent reliability comes from the harness around the model, not the weights — governance is no exception.
• Prefer Buterin’s “info finance” to a single oracle. Rather than one model ruling, run an open market of models with spot-checks anyone can trigger and a human jury for disputes. Diversity and adversarial auditing beat a lone deliberator you have to trust — and it dissolves the single 113-block GEV window into something harder to front-run.

The instinct to make an AI delegate “smarter” by letting it reason more is exactly backwards at the part that matters. On the adversarial vote, the model that thinks less is the model that can’t be talked into yes.

Takeaways

• Putting an LLM behind castVote makes the proposal description an attacker-controlled input — the governance analogue of prompt injection, with a vote as the payload.
• A frozen-weights ablation found a cognitive penalty: switching on deliberation dropped adversarial robustness 100% → 68.5%, baseline accuracy → 75.5%, consistency → 72.6%, and left 26.7% of rulings unparseable. The failures that approved the attack reasoned the longest.
• Deliberation’s 17× latency (226s ≈ 113 Optimism blocks) is itself Governance Extractable Value — a public, predictable window to front-run the pending decision.
• Engineer for it: isolate untrusted proposal text, default the judge to fast/direct mode, cap the blast radius with human ratification, and prefer an open market of audited models over one reasoning oracle.