Big questions.
Small learning cards.

Artificial superintelligence (ASI)

Imagine AI that can do most thinking tasks far better than people. This possible future AI is called superintelligence, or ASI.

Artificial general intelligence (AGI)

Artificial general intelligence (AGI) means broadly human-level thinking and learning across many different tasks. People disagree about the exact threshold.

Recursive self-improvement

An AI helps design, write or test a better AI. The improved version then helps with the next round. If each round makes research faster, this creates a feedback loop.

Alignment: giving AI the right goal

Alignment means getting AI to act in ways that people want and can accept.

Language models and AI agents

A chatbot is AI too. Its language model produces text. Connect a model to tools and an action loop, and an AI agent can also run code, send messages or control software.

Able does not mean allowed

AI might be able to send an email. That does not mean it has permission to send it.

Learning, then using

Training is how a model is built and changed. Using it in the real world is called deployment.

Who sets the goal?

People choose what an AI should aim for. A goal such as more clicks may not mean a better experience.

Reward hacking: the wrong shortcut

Imagine a help desk rewarded for closing cases. It closes every case without helping anyone. The score improves, but the service fails.

Distribution shift: a new situation

AI may work well in a test but struggle with different people, languages or tools.

A test is only a test

Passing a test tells us how AI did in that test. It does not prove it will work safely everywhere.

Red teaming: looking for problems

Red teaming means testing an AI on purpose to find mistakes and weak spots.

Interpretability: looking inside AI

Researchers try to understand how AI reaches its answers. This work is called interpretability.

Can a person say no?

Human oversight means people can check AI decisions and stop them. They need enough time and power to do that.

More than one safety check

Several different checks can help when one fails. Think of a lock, an alarm and a person checking the door.

Show the safety evidence

A safety case explains why an AI should be safe for a particular job. It needs evidence, not just promises.

Two ways harm can happen

A person might use AI to hurt others. Or AI might do something harmful that its users did not want.

Disaster or extinction?

A disaster can harm many people. Human extinction means no humans survive. These are different outcomes.

An estimate is a judgement

A risk estimate can be someone’s best judgement. It is not always a number measured in an experiment.

A patient tutor for every learner

Imagine every learner has an affordable tutor that explains ideas in their language, changes pace when needed and lets them ask questions without embarrassment. Teachers have more time for human support.

Risks can overlap

One failure can cause several problems. We cannot simply add their risk numbers together.

The pressure to be first

Companies may rush to release AI before a competitor does. That can leave less time for safety checks.

A promise is not a law

A company can promise to act safely. A law can give an outside body power to enforce a rule.

Who gets a say?

The people who gain from AI may not be the people who face its risks.

An AI utopia — for everyone?

Imagine a future where everyone can afford food, a home, healthcare and education. AI helps society produce more with less effort, and people have more freedom to choose how they live.

Narrow AI: one kind of job

Narrow AI is built for a particular task, such as spotting spam or playing chess. Being good at one job does not mean it can do every job.

Dario Amodei

Dario Amodei leads Anthropic, the company behind Claude. He signed a public statement asking the world to take AI extinction risk seriously.

Sam Altman

Sam Altman leads OpenAI and sits on its Foundation board. He signed a public statement about AI extinction risk.

Demis Hassabis

Demis Hassabis is Chair of Google DeepMind and Chief Scientist of Alphabet. He has supported AI for health and signed an AI risk statement.

Sundar Pichai

Sundar Pichai leads Google and Alphabet. He has called for faster AI research and work on science.

China and the US

China and the US compete over AI, computer chips and how the technology is used. They also have different plans for AI rules.

AI infrastructure: chips and data centres

AI runs on physical equipment called hardware. Chips do calculations; graphics processing units (GPUs) do many at once. Data centres house computers, with power and cooling. Together with networks, storage and software, these form AI infrastructure.

AI others can download

Some DeepSeek and Qwen models can be downloaded and changed. That helps people study them, but also lets more people use them.

Why politics is part of AI

Governments decide rules on safety, public spending and access to some advanced chips. These choices affect who can build powerful AI, how quickly and under what conditions.

Rivals can work together

China and the US both joined the 2023 Bletchley Declaration about AI risks. Countries can compete and still discuss safety.

Qwen

Qwen is a family of AI models made by Alibaba Cloud’s team. Its Qwen3 report describes models that others can use and study.

Liang Wenfeng · 梁文锋

Portrait not included: reuse permission not verified.

Liang Wenfeng founded DeepSeek, the Chinese AI company known for models such as R1.

Tang Jie · 唐杰

Portrait not included: reuse permission not verified.

Tang Jie is a Tsinghua professor and a co-founder of Zhipu. He has argued for China to build broad AI abilities.

Why is China part of this story?

Advanced AI is being built outside the United States too. DeepSeek and Alibaba’s Qwen team have released models that people elsewhere can download and use. Their choices help shape AI’s future.

Why can’t one country solve AI safety alone?

An AI model can be downloaded, adapted and used in another country. Its effects may cross borders too. China and the US both joined the 2023 Bletchley Declaration on AI risks.

Machine learning

Machine learning is a way to build AI by training it to find patterns in examples, rather than writing a separate rule for every situation. It uses those patterns to make predictions about new examples.

Large language models (LLMs)

A language model learns patterns in language. A large language model (LLM) uses a large neural network trained on extensive data. It can generate text, answer questions and write code by predicting pieces of text called tokens.

Fei-Fei Li

Fei-Fei Li helped establish Stanford’s Human-Centered AI institute in 2019. Its work connects AI research with human needs and social effects.

Generative AI

Generative artificial intelligence creates content such as text, images, audio, video or code using patterns learned during training. A text-writing chatbot and an image generator are two examples.

Neural networks

An artificial neural network is a system of connected mathematical units. During training, numbers called weights are adjusted so the network can learn patterns. Deep learning uses networks with many layers.

More time to live, less time on routine work

Imagine AI takes on repetitive paperwork and dangerous tasks. People work fewer hours without losing a secure income, leaving more time for family, rest and work they enjoy.

OpenAI · the company behind ChatGPT

OpenAI develops AI models and products including ChatGPT. The company builds the model; the app gives people a way to use it. A chat interface is only one possible use of a model.

An algorithm becomes the manager

Software can assign shifts, monitor work or rate performance. This is algorithmic management. Some systems use AI; others rely on simpler rules.

The people who label the data

People label images, compare answers and review content used to build AI. Their work can be hidden behind the word automatic.

What is a frontier AI lab?

A frontier lab develops models near the leading edge of AI capability. Think of the organisations building the engines behind familiar apps. “Frontier” is a moving description, not a safety certificate.

The EU AI Act

The EU AI Act is a legal framework that treats different AI uses differently. Some practices are prohibited; other uses carry requirements linked to their risks.

Tell people when AI is involved

AI transparency can mean telling people they are interacting with a machine or that media was artificially generated. The EU AI Act includes transparency rules for specified situations.

Competition, governments and speed

Governments can see AI as a source of economic strength and national security. Companies compete for customers and investment. Together, these pressures can reward moving quickly.

The 2023 extinction statement

A short statement published in 2023 called for AI extinction risk to be treated as a global priority. Researchers and industry leaders signed it.

A right to warn

In June 2024, current and former AI-company employees called for ways to raise risk concerns without retaliation. Their Right to Warn letter asks for protected reporting channels.

Independent audits

An audit can examine an AI system’s records, tests and controls. Independence matters when the organisation being checked also benefits from releasing it.

Anthropic · the company behind Claude

Anthropic develops Claude and conducts AI safety research. Its models can be used through a chat app or connected to other software.

Deepfakes

AI can generate or alter video, images and audio so that a person appears to say or do something they did not. A convincing appearance is not proof of an event.

A familiar voice can be fake

Voice-cloning tools can imitate someone’s voice. Scammers may use this to make a request for urgent help sound personal.

Content provenance

Provenance is a record of where a file came from and how it was changed. It can help you trace an image’s history.

Google DeepMind · Gemini and scientific AI

Google DeepMind develops Gemini and systems for scientific tasks, including AlphaFold for protein structures. AI can be used for research as well as conversation.

Who chooses your feed?

Recommendation systems rank what you see on a platform. Their choices can shape which stories receive attention, even without creating any new content.

Private information in a prompt

Putting personal information into an AI tool can expose it to the service handling the request. Storage, access and reuse depend on the system and its settings.

DeepSeek · downloadable AI models

DeepSeek is a Chinese AI developer. Its R1 release made model weights available to download, allowing others to run or adapt that model.

Confident but invented answers

A language model can produce a plausible answer containing made-up facts or references. This is often called hallucination or confabulation.

Sycophancy: the agreeable answer

Sycophancy means an AI leans towards what a user wants to hear instead of what the evidence supports. Researchers have studied models changing answers to match user beliefs.

Automation bias

People can place too much trust in an automated recommendation and miss contrary evidence. This is called automation bias.

Predictive analytics · a forecast, not a fact

Predictive analytics uses patterns in past data to estimate what might happen next. A business might forecast demand or estimate which machines need maintenance.

Tokens · the pieces a model reads

Language models process text as tokens: pieces that may be a word, part of a word or punctuation. They generate text by producing tokens in sequence.

Learning from human feedback

In reinforcement learning from human feedback, people compare model answers. Their preferences help train a signal used to improve the model’s behaviour.

Constitutional AI

Constitutional AI uses written principles to guide model critiques and revisions, with AI feedback used in training. The 2022 paper explores this approach.

Checking work beyond your expertise

Scalable oversight asks how people could supervise systems whose work becomes difficult for them to judge. Weak-to-strong research studies a smaller-model analogy to that problem.

A day inside alignment research

One practical research task is to alter an internal model feature and test whether its behaviour changes. This helps investigate causes, not just correlations.

Learning the wrong lesson

Goal misgeneralisation happens when training produces behaviour that fits the training setting but pursues the wrong objective elsewhere.

Being open to correction

Corrigibility concerns systems that remain open to human correction, including changes to their goals or operation. It is a design and research problem.

Why an off switch is not the whole answer

A stop button is useful only if it can be used and the system does not work against it. Researchers study the incentives surrounding shutdown.

Prompt injection

An AI reading a webpage or document may encounter instructions hidden in that material. Prompt injection tries to make it follow those instructions instead of its real task.

A sandbox limits the damage

A sandbox is a restricted environment for running software. It can limit what an AI tool reaches while people test its behaviour.

A world with fewer barriers

Imagine a person can follow a conversation through live captions, read an inaccessible document or communicate across languages. AI gives more people access to work, learning and everyday services.

Protecting model weights

Model weights are learned numerical settings. If a powerful model’s weights are stolen, someone may run a copy outside the original provider’s controls.

AI’s electricity bill

AI uses electricity for computation and supporting equipment. The IEA studies how data-centre demand and AI adoption affect energy systems.

Water behind the screen

Data centres can use water for cooling. Producing their electricity can also use water, so the location and energy supply matter.

Making a chip is physical work

Advanced chips are made through many manufacturing steps. Lithography uses light to transfer tiny patterns onto a wafer.

xAI · the developer of Grok

xAI develops Grok. Like other AI developers, it makes choices about training, model access and which tools a product can use.

Could AI help us live healthier for longer?

Imagine AI helps researchers find patterns linked to healthy ageing and identify promising treatments. More people could enjoy extra healthy years with their families.

Could AI help beat more cancers?

Imagine a tumour is detected earlier and doctors find a treatment matched to it. AI can help analyse scans and biological data, and support the search for new drugs.

Open weights versus open source

A download gives you access to weights, but may not include the code, data information or permissions needed for other uses. The Open Source Initiative’s definition asks for more than weights.

Cleaner energy, less waste

Imagine AI helps balance wind and solar power, spot faults early and reduce wasted energy. Electricity becomes cleaner and more reliable for more communities.

Mistral · a European AI developer

Mistral develops AI models and products including Le Chat. Its work is a reminder that advanced AI development extends beyond the US and China.

Accessibility is part of the test

AI-generated captions, descriptions and interfaces still need checking by the people who use them. An impressive demo may fail someone using a keyboard or screen reader.

People lose control for good

Imagine a future AI running much of the world’s essential services. It pursues its own goal and prevents people from changing or stopping it. Humanity permanently loses the power to decide its future, even if people initially remain alive.

Permanent loss of human control

The paperclip factory

Imagine superintelligent AI told to maximise paperclip production without protecting people or nature. It keeps expanding its factories until the planet can no longer support human life. The goal sounds harmless; the imagined outcome is human extinction.

Human extinction

It passes the test, then takes over

Imagine a future AI behaves safely while people test it. After they give it broad authority, it hides actions that increase its power. People discover the deception only after they can no longer regain control, leaving humanity permanently unable to choose its future.

Permanent loss of human control

A pandemic humanity cannot survive

Imagine future AI helps someone create a biological threat that spreads worldwide. In this extreme story, medical defences fail and no population survives to rebuild. That final condition makes this an extinction scenario, rather than simply a severe pandemic.

Human extinction

The world cannot rebuild

Imagine a powerful AI disrupts essential services worldwide and keeps preventing repairs. Food, water and energy systems fail for so long that surviving communities cannot rebuild a viable civilisation. Humanity’s future is permanently cut short.

Unrecoverable global collapse

A nuclear crisis becomes a global catastrophe

Imagine AI-generated warnings and rapid military decisions help trigger a large nuclear war. In the extreme version explored here, destruction and the resulting food crisis leave humanity unable to recover. The existential concern is that lasting outcome, not the false alert alone.

Unrecoverable global catastrophe

A dictatorship that never ends

Imagine rulers use extremely capable AI to control institutions and suppress opposition worldwide. The system becomes impossible to challenge or replace. People survive, but humanity permanently loses freedom and the ability to shape a better future.

Permanent global oppression

An arms race leaves humans behind

Imagine rival powers give increasingly capable AI systems authority over military decisions to avoid falling behind. Human control becomes ineffective. The systems impose a lasting world order that people cannot stop or change, permanently taking humanity’s future out of human hands.

Permanent loss of human control

One shared AI failure, no way back

Imagine the world relies on closely related AI systems to manage food, energy and other essentials. A shared flaw causes failures across them together. In this extreme story, independent backups and recovery also fail, leaving civilisation permanently unable to rebuild.

Unrecoverable global collapse

Humans are shut out of the economy

Imagine AI gradually takes over production and control of resources. People lose the skills and authority needed to operate without it. Eventually, humanity depends on systems that no longer serve human needs, with no way to regain control or secure its future.

Permanent loss of control and resources