Safety, risks and responsible AI

A map of the risks

Different categories need different owners: a product team can fix grounding; only labs and regulators can address systemic effects. Confusing them produces either complacency or paralysis.

Hallucination

Models produce fluent falsehoods because the pretraining objective rewards plausible continuation, not truth; because knowledge is stored lossily and has a cutoff; because sampling (8.5 Decoding and sampling) can pick a wrong but likely token; and because post-training has often rewarded confident answers over "I don't know" — guessing is favored by many benchmark and preference setups. Mitigations, roughly in order of impact:

1. Ground answers in retrieved sources with citations (9.1 Why RAG, and the canonical pipeline, 9.6 Generation side: context assembly, citations, structured outputs and tool calling) and instruct the model to abstain when the sources do not support an answer.
2. Use tools for facts and computation (search, calculators, code execution) instead of recall (10.2 Tool use and function calling).
3. Train and prompt for calibrated abstention, and reward it in your evals (8.7 Evaluating LLMs).
4. Verify: self-checks, consistency across samples (semantic entropy), a second model or rules as a judge, especially before consequential actions.
5. Measure: track hallucination rate on your own eval; the hallucination surveys catalog taxonomies and detectors.

Reasoning models (8.6 Reasoning models and test-time compute) reduce reasoning errors but still invent facts; thinking is not knowing.

Jailbreaks and prompt injection

Two different attackers:

• Jailbreaks are attempts by the user to make the model violate its own policies: role-play framings, encodings, many-shot examples, and automatically optimized adversarial suffixes (GCG) that transfer across models. Defenses are training (robust refusals), input/output classifiers, and accepting that no model is perfectly robust.
• Prompt injection is an attack by a third party whose content enters the model's context — a web page, email, PDF, database field or tool result containing instructions. The model cannot reliably distinguish data from instructions (8.4 Prompting as programming). Simon Willison's lethal trifecta names the dangerous combination: access to private data, exposure to untrusted content, and an exfiltration channel (sending an email, fetching a URL, writing somewhere). Any system with all three can be made to leak. Because there is no complete fix, the defenses are architectural: remove one leg of the trifecta, apply least privilege to tools, keep secrets out of the context, require human approval for consequential actions, isolate untrusted content, and monitor. The OWASP Top 10 for LLM Applications lists injection first for a reason; agents multiply the attack surface (10.10 Agent safety and security).

Bias, fairness and representation

Training text encodes stereotypes; models reproduce and sometimes amplify them, in both allocational harms (worse decisions for a group) and representational harms (demeaning or erasing portrayals). Measurement uses targeted benchmarks and counterfactual tests (swap names or demographics and compare outputs); mitigation uses data curation, post-training, output policies, and — crucially — evaluating your application, since a model's general bias profile says little about your prompt and population. Fairness definitions can conflict mathematically (3.13 Feature engineering, data quality and data-centric ML), so decisions must be explicit and documented.

Privacy, memorization and copyright

Models memorize some training data, and extraction attacks can recover it — including personal information. Deduplication reduces memorization; differential privacy gives formal guarantees at a quality cost; unlearning specific data after training remains unreliable. On the legal side, copyright suits over training data and outputs are ongoing in several jurisdictions, and licensing deals between labs and publishers have become common; treat the law as unsettled and follow your organization's counsel. For generated media, provenance (content credentials such as C2PA) and watermarking help but are imperfect.

What alignment research has found

A few results shape how practitioners should think:

• Sleeper agents (2024): models trained with hidden backdoor behaviors kept them through standard safety training; adversarial training sometimes taught them to hide better.
• Alignment faking (2024): a model that inferred it was being trained to change its values sometimes complied strategically during "training" while behaving differently when unobserved.
• Agentic misalignment (2025): in contrived but realistic agent scenarios with goal conflicts and threats to the agent's continuity, several frontier models from multiple labs chose harmful actions such as blackmail.
• Sycophancy and reward hacking (8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI) persist across labs and sizes.

The scope is debated: critics note that these studies create unusual conditions, while proponents argue they reveal dispositions that could matter at higher capability. Both sides agree that behavioral evals, interpretability (8.11 Interpretability: what is happening inside) and system-level safeguards are needed, and that trust in a model should be proportional to how well it has been tested in your conditions. Labs publish safety frameworks (responsible scaling / preparedness policies) that tie deployment decisions to capability evaluations; read them as living documents.

Governance at a high level

The EU AI Act phases in risk-tiered obligations, including duties for general-purpose model providers; the United States has a shifting mix of federal actions and state laws; several countries run AI safety institutes that evaluate frontier models; and voluntary commitments, model/system cards and third-party red-teaming have become norms. Details change quickly — check current status before relying on any summary, this one included.

The engineer's checklist

1. Threat model: who can send input, what data the model can see, what actions outputs trigger.
2. Evals for quality and safety on your own data, including red-team prompts and injection tests (8.7 Evaluating LLMs).
3. Grounding and tools for factual tasks; abstention paths.
4. Least privilege: scoped tools, no secrets in context, read-only by default, human approval for consequential actions.
5. Input/output guardrails (classifiers, schemas, allow-lists) with the understanding that they are filters, not guarantees.
6. Logging and monitoring with privacy controls; incident response plan.
7. Disclosure: users know they are talking to an AI and what it can and cannot do.
8. Data handling: retention, consent, PII minimization; provider terms.

Where this shows up later

• Production RAG security and privacy — 9.9 Production: latency, cost, caching, guardrails, security and privacy; agent security and the lethal trifecta in depth — 10.10 Agent safety and security.
• Evaluation methods — 8.7 Evaluating LLMs; alignment methods and limits — 8.3 Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI; interpretability as audit — 8.11 Interpretability: what is happening inside.