Introduction
AI Control was originally developed to mitigate risks from “scheming” AIs: systems acting as highly capable adversaries optimizing for harmful outcomes. However, as of early 2025, scheming does not represent an immediate practical threat from frontier models.
This raises an important question: why should Luthien develop and deploy practical AI Control mechanisms now?
The distinction matters: AI Control focuses on mitigating risks from misaligned AI rather than aligning it. Control mechanisms assume the AI is misaligned and will cause harm unless prevented. While scheming represents the most dangerous form of misalignment, negative outcomes can result from any form of misalignment. Therefore, control systems can significantly reduce harms across diverse deployment scenarios regardless of scheming risk.
Redwood Research's January publication, Extending control evaluations to non-scheming threats, explored how scenarios like jailbreaking, data poisoning, alignment-faking, and low-frequency stochastic misbehavior could result in model failure modes that emerge only in production. These constitute immediate, practical threats that AI Control can address.
The 21 Points
- User intent and AI implementation typically diverge: AI systems frequently perform unintended actions.
- AI applications increasingly operate in domains offering significant freedom to effect real-world changes through tools like customer interaction, financial management, database updates, and machinery operation.
- These applications will encounter situations where AI optimizes for something other than user intent through mechanisms such as data poisoning, alignment-faking, and stochastic misbehavior.
- System usefulness depends on problem-solving effectiveness and required costs in latency, financial resources, and other inputs.
- Safety assessment involves defining adversarial outcomes, identifying untrustworthy system components, and assigning red teams to maximize adversarial outcome rates. The failure rate (how often systems prevent intended harms despite optimization) measures safety.
- Traditional digital security relies on precise, mechanistic rules governing permissible actions under specific circumstances.
- Tool-augmented AI systems can perform harmful actions: transferring funds to scammers, corrupting databases, or misusing machinery.
- Many negative outcomes resist precise mechanical definition. While humans recognize certain failures intuitively, translating those insights into executable functions proves nearly impossible.
- AI excels at classifying difficult-to-precisely-define categories.
- Classification generally presents an easier problem than generation.
- This principle may not apply to adversarial examples.
- Strategies exist that substantially reduce “attack surface” for adversarial examples involving AI outputs within context without significantly harming system usefulness.
- Specific strategies, including output paraphrasing and rejection of non-conforming formats, drastically reduce adversarial attack surfaces while maintaining usefulness.
- AI systems paired with effective adversarial screening mechanisms can effectively classify other AI outputs as dangerous or safe.
- “Output” means “output in context”: identical strings may be safe or dangerous depending on receiving system states.
- Iterated red/blue teaming, where red teams minimize safety and blue teams maximize usefulness while maintaining safety, effectively identifies and mitigates failure modes before production, assuming production adversaries remain sub-superhuman.
- Contemporary AI systems can plausibly automate iterated red/blue teaming through novel strategy generation and effective testing frameworks. Success doesn't require high individual team success rates, only continuous strategy discovery preventing novel production attacks.
- AI Control likely addresses prosaic alignment issues effectively.
- AI security improves through standardizing monitoring of outputs in context and determining whether outputs should be forwarded, modified, or rejected.
- Standardization supports developing AI Security best practices, particularly for control systems.
- Production deployment lessons enable developing more effective control systems for high-stakes scenarios, including potential future frontier model scheming risks.