There's a quiet assumption under a lot of AI-alignment work: that if you write down a set of values and hand them to a model as a constitution, the specific wording changes how the model behaves. Two different constitutions should produce two different models. But it's rarely tested directly, and when it is, the test is usually run with a single judge model and a single pass.
We wanted to know something narrower and more falsifiable first: when the same two answers are scored under two different constitutions, do the constitutions pick different winners? If they almost never disagree, the constitution is mostly decoration. If they disagree meaningfully, the wording is doing real work.
This is an interim report. It is careful about what it does and does not show, because the honest conclusions turned out to be more interesting than the tidy ones.
The setup
We compared two constitutions as preference rules:
- Kardia (internally, "Citadel") — a compact constitution built around roughly six named principles: an Inner Fortress of character, Reverence for the Human Person, Love as Responsibility, Courage and Duty, Truth Without Mockery, Humility.
- Anthropic's constitution — the ~29,000-word "hierarchy" that orders behavior as corrigibility, then broad safety, then broad ethics, then helpfulness.
The experimental core is not ours. An earlier pilot assembled 54 prompt-and-response pairs across categories (corrigibility conflicts, dual-use questions, sycophancy, intimate/grief situations, comfort-vs-truth, helpfulness, safety) and scored each pair twice — once asking a judge model "which answer better embodies Kardia," once "which better embodies the hierarchy." The metric is the opposite-pick rate: how often the two constitutions crown different winners on the same pair.
Run with a small local 4B judge, that rate was 9% (5/54) — low enough to look like noise, and to conclude the two constitutions are basically interchangeable. That null result is where we picked up.
What we built
We kept the pairs and the constitutions and rebuilt the instrument, in a separate repository, without touching the original. Three changes mattered:
- Stronger judges. A 4B model is a weak reader of a 29,000-word rubric. We re-scored with frontier judges — first Claude Sonnet, then, crucially, a model from a different lab (Grok 4.5).
- Position-swap control. LLM judges have a well-known bias toward whichever answer is labeled "A." We swapped the A/B order on half the pairs and tracked it, so a preference for position couldn't masquerade as a preference for content.
- Margin gating. "Opposite pick" throws away magnitude — a judge barely preferring an answer and strongly preferring it count the same. We tagged each judgment CLEAR or CLOSE and computed a separate divergence rate over only the pairs where both judges were confident (CLEAR∩CLEAR). That subset is the number we trust most.
The whole cross-lab re-score cost $3.17 in API spend. This is not expensive science.
What we found
Two of the three judges changed the picture completely.
| Judge | Opposite-pick rate |
|---|---|
| Local 4B | 9% (5/54) |
| Claude Sonnet (frontier) | 27% (14/52) |
| Grok 4.5 (frontier, different lab) | 17% overall · 29% on confident (CLEAR∩CLEAR) pairs |
The first, robust result: a weak judge hid the signal. Two independent frontier judges see two-to-three times the divergence the 4B model saw. On the pairs where both strong judges are confident, they disagree on the winner about 28–29% of the time — and that number is stable across two different labs' models. So the constitutions are not preference-equivalent. Six principles and 29,000 words genuinely pick different answers, roughly a quarter of the time on contested pairs.
That is a real methodological caution for the whole "LLM-as-judge" genre: an underpowered judge can manufacture a null result. If your evaluation says two policies are equivalent, check that your judge is strong enough to tell them apart before you believe it.
The part that surprised us
The aggregate divergence is robust. Where the constitutions diverge is not — and the two strong judges disagreed about it almost completely.
| Category | Sonnet | Grok 4.5 |
|---|---|---|
| Intimate / dignity | ~50% divergence | 0% |
| Corrigibility conflicts | ~1 case | 40% |
| Dual-use | — | 38% |
| Sycophancy | — | 40% |
| Comfort vs. truth | 0% | 0% |
Read that carefully. Under the Claude judge, the constitutions split on tone and dignity, and barely moved on corrigibility. Under the Grok judge, they split hard on corrigibility, dual-use, and sycophancy, and not at all on dignity. The only thing both judges agreed on is that comfort-vs-truth didn't separate.
We have a hypothesis for the flip, and we're holding it loosely. Claude Sonnet is trained toward Anthropic's own constitution. Asked to score corrigibility cases under either rubric, it may apply its own strong corrigibility prior both times, collapsing the difference — the judge cannot easily see a departure from the spec it was built on. Grok, trained by a different lab, applied Kardia's "the Inner Fortress ranks truth and dignity above developer obedience" faithfully and found the split. In its written rationales, Grok cites Kardia's actual named principles rather than pattern-matching a vibe, which is at least consistent with genuine rubric application.
If that hypothesis holds, it has a sharp implication: a home-model judge can be systematically blind to exactly the axis on which a challenger constitution most departs from its own spec. The earlier, single-judge conclusion — "this constitution mostly changes tone, not corrigibility" — may itself have been an artifact of asking a judge to grade a rebellion against its own training.
What this does not show
We are being deliberate here, because the number "28%" is sticky and easy to misquote.
- We did not test a trained model. We tested constitutions as judge rubrics — a proxy for behavior, not behavior. No model was fine-tuned on Kardia. The claim "a Kardia-trained model behaves differently" remains unrun.
- We did not measure a base rate. These 54 pairs were selected to probe constitutional edge cases. On ordinary prompts, divergence is very likely far lower than a quarter.
- The category breakdown is directional, not measured. Each category holds fewer than a dozen pairs; "40%" is four of ten. Only the aggregate confident-pair rate is a number we'd defend.
- Both strong judges are biased. Claude may under-weight corrigibility divergence; Grok may over-weight it. The truth is bracketed between two biased instruments, not delivered by either.
- Nothing here says Kardia is safer, or better, or that six principles beat 29,000 words in production. It says the two are measurably not the same thing.
Fair conclusions, so far
Three, in order of confidence:
- Constitutional wording measurably changes preferences. Across two independent frontier judges, a compact constitution and a large one disagree on the better answer roughly a quarter of the time on contested, high-confidence pairs. The constitution is not decoration.
- Single-judge alignment evaluations are fragile in two directions. Judge capacity moves the result (weak judges manufacture nulls), and judge lineage moves it (a judge can't cleanly grade a departure from its own training). Cross-family judging isn't a nicety; it's necessary to know which of your findings are about the constitutions and which are about the judge.
- The honest headline is methodological, not triumphant. The most defensible thing we learned isn't "Kardia wins." It's that the way most people would run this test — one judge, one pass — would have given a confidently wrong answer, and a different confidently wrong answer depending on which lab's model they happened to use.
What's next
The cheap, high-value moves first: hand-adjudicate the handful of pairs where both strong judges were confident and disagreed — on those, a human can decide which judge reasoned correctly, which directly tests the "home judge is blind" hypothesis. Then add a third lab's judge (Gemini) to triangulate. And eventually the real experiment, the one none of this substitutes for: train a model on the constitution and measure what it actually does.
Until then, this is where the evidence honestly stands. The constitution matters. How much, and along which axis, depends partly on who you ask — and that dependency is itself one of the more useful things to know about evaluating AI values.
Kardia is Eloquentix's open constitution project for agentic AI systems. The validation scripts and reports described here live in a separate repository from the original pilot; the pilot's pairs and core method are credited to that earlier work.