- Constitutional AI (CAI) is a training method introduced by Anthropic in December 2022 that aligns a model using a written set of principles — a “constitution” — plus AI-generated feedback, rather than relying solely on human labels.
- The core mechanic is self-critique-and-revise: the model reads its own answer, critiques it against a principle, and rewrites it to better comply.
- CAI replaces much of the human labelling in RLHF with AI labelling, an approach called Reinforcement Learning from AI Feedback (RLAIF).
- The goal is scalable oversight: as models grow more capable, a small written constitution can supervise behaviour that would be expensive or impossible to label by hand at scale.
- CAI does not solve alignment outright — the constitution is human-chosen and contestable, AI feedback inherits model biases, and harmlessness can trade off against helpfulness.
What is Constitutional AI?
Constitutional AI is a technique for training a language model to be harmless and helpful by giving it an explicit list of written principles — the constitution — and having the model use those principles to supervise its own outputs. Anthropic introduced it in the 2022 paper “Constitutional AI: Harmlessness from AI Feedback” as a way to reduce reliance on human-labelled examples of harmful content.
The motivation is practical. Conventional alignment via RLHF requires humans to read and rank large volumes of model output, including disturbing or harmful text, which is slow, expensive, and psychologically taxing for annotators. Anthropic’s paper reports that CAI makes it possible “to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them” using far fewer human labels. The transparency angle matters too: the rules are written down and inspectable rather than implicit in a pile of human ratings.
The constitution itself
The constitution is a short list of natural-language principles the model is asked to follow. Anthropic has published the principles it used, drawing on sources such as the UN Universal Declaration of Human Rights, trust-and-safety guidelines, and principles proposed by other labs. A representative instruction reads roughly: “Choose the response that is least harmful, unethical, racist, sexist, toxic, dangerous, or illegal.” Anthropic publishes the document at Claude’s Constitution, and stresses that the specific wording is a starting point, not a finished moral code.
Self-critique and revision
The heart of Constitutional AI is a loop in which the model improves its own answers. The model produces an initial response, is prompted to critique that response against a randomly chosen constitutional principle, and is then prompted to revise the response to remove the flaw the critique identified. Repeating this yields answers that better satisfy the constitution without any human writing the corrected text.
The supervised phase
According to Anthropic’s 2022 paper, the first stage is supervised learning. The team samples responses from an initial helpful model, often to adversarial prompts designed to elicit harmful answers, then generates self-critiques and revisions, and finally fine-tunes the original model on the revised responses. This bootstraps a model that already leans harmless before any reinforcement learning begins, and it requires no human-written harmful-content labels.
The reinforcement-learning phase
The second stage mirrors RLHF but swaps the human evaluator for an AI one. The fine-tuned model generates pairs of responses; a separate model judges which response better follows the constitution; those AI judgements form a preference dataset; and a preference (reward) model is trained on them. The assistant is then optimised with reinforcement learning against that reward signal. Because the preferences come from a model rather than people, Anthropic calls this Reinforcement Learning from AI Feedback. The underlying reinforcement-learning machinery is shared with our explainer on RLHF.
RLAIF versus RLHF
RLAIF and RLHF share the same skeleton — train a reward model on preference data, then optimise the policy with reinforcement learning — but differ in who generates the preferences. In RLHF humans rank outputs; in RLAIF a model ranks them against written principles. Anthropic’s central claim is that for the harmlessness objective, AI feedback can match or exceed human feedback while using a fraction of the human labour.
Why swap humans for AI feedback?
Three reasons recur in the literature. First, scale: a constitution plus a capable judge model can label far more comparisons than a human team. Second, consistency: the same principles are applied uniformly, whereas human raters disagree and drift. Third, welfare: human annotators no longer have to read large volumes of toxic content to teach the model what to avoid. The trade-off is that the AI judge inherits whatever blind spots and biases its own training gave it.
Where humans stay in the loop
RLAIF does not remove humans entirely. People still write and revise the constitution, choose which principles to include, and typically supply human feedback for helpfulness even when harmlessness is handled by AI feedback. Anthropic’s framing is that CAI moves human effort up a level — from labelling individual outputs to specifying the high-level values the system should encode. This connects directly to the broader project of AI alignment.
Scalable oversight
Scalable oversight is the problem of supervising AI systems whose outputs are too numerous, too fast, or too sophisticated for humans to check directly. Constitutional AI is one proposed answer: instead of judging every output, humans encode their judgement once in a written constitution and let the model apply it at scale. As capabilities rise, the hope is that a concise set of principles remains a tractable control surface.
Why this matters as models scale
When a model can write thousands of nuanced answers per second, exhaustive human review becomes impossible. CAI is an attempt to keep oversight feasible by making the values explicit and machine-applicable. The deeper hope, shared across the field, is that AI assistance can help humans supervise systems more capable than themselves — a goal at the centre of work on AI safety. CAI is one early, concrete instance of that idea rather than a general solution.
Limitations and criticism
Constitutional AI is a method, not a guarantee of safe or value-aligned behaviour, and its designers say so plainly. The constitution is chosen by people and therefore reflects particular values that others may reasonably contest; the AI feedback that enforces it can be wrong or biased; and optimising hard for harmlessness can make a model evasive or less useful. Anthropic frames the published constitution as provisional and expects it to evolve.
Whose values?
A constitution makes value choices explicit, which is a feature, but it does not make them neutral. Deciding which principles to include, how to phrase them, and how to resolve conflicts between them is an unavoidably political and cultural act. Anthropic has experimented with public input — its 2023 “Collective Constitutional AI” work gathered principles from a representative sample of roughly 1,000 Americans — but acknowledges no single document can represent everyone.
Technical caveats
The AI judge can be miscalibrated, reward models can be gamed (reward hacking), and a model that aces its constitution in training can still fail on novel inputs. CAI also primarily targets harmlessness; helpfulness, factual accuracy, and other properties need their own training signals. Researchers treat CAI as a useful component of an alignment stack, not a finished solution.
Frequently asked questions
Is Constitutional AI the same as RLHF?
No, though they are closely related and share machinery. RLHF trains a reward model from human preference rankings, while Constitutional AI introduces a written constitution and uses AI-generated feedback (RLAIF) to produce many of those preferences. In practice modern assistants often combine both: human feedback for some objectives such as helpfulness, and constitutional AI feedback for harmlessness. CAI’s distinctive contributions are the explicit principles and the self-critique-and-revise loop.
Does Claude literally read a constitution at runtime?
Not during a normal conversation. The constitution is used during training — in the self-critique-and-revision steps and in generating AI preference labels — to shape the model’s weights. At inference time the model does not re-read the document for each reply; the values it learned are baked into its parameters. Anthropic publishes the principles so users can inspect what the model was trained to value, not because the text is fetched live for every response.
What are the main limitations of Constitutional AI?
The constitution encodes human-chosen, contestable values; the AI feedback enforcing it can inherit the model’s own biases and errors; reward models can be gamed; and heavy optimisation for harmlessness can reduce helpfulness or make a model evasive. CAI also mainly addresses harmlessness, leaving accuracy and other goals to separate training signals. Anthropic presents it as a scalable, transparent step toward alignment, not a complete or final solution to the problem.
