16 thinkers, books, and ideas connected to this field.
San Francisco AI safety company founded by former OpenAI leaders Dario Amodei, Daniela Amodei, and colleagues — creator of the Claude AI assistant series — notable for Constitutional AI and its focus on making AI systems helpful, harmless, and honest.
American AI safety researcher and forecaster — Research Associate at Oxford's Future of Humanity Institute — who is considered the most analytically rigorous thinker in the effective altruism and AI safety community — whose interview with Dwarkesh became one of the most discussed AI podcasts ever recorded.
American AI safety researcher — former OpenAI researcher — who left OpenAI in 2024 citing safety concerns and forfeited his vested equity — author of forecasting documents on AI timelines and transformation.
American AI safety researcher — co-founder of the Machine Intelligence Research Institute (MIRI) — creator of LessWrong — author of Harry Potter and the Methods of Rationality and the Sequences — who has been arguing since 2000 that AI poses an existential risk and has produced the foundational framework for AI safety thinking.
American philanthropist — co-founder of GiveWell (charity effectiveness evaluation) and Open Philanthropy — who has directed over $1B in philanthropic giving toward effective causes — and who has become increasingly focused on AI safety as a priority.
Stuart Russell's 2019 book arguing that the standard model of AI — maximizing a fixed objective — is fundamentally unsafe, and proposing a new approach in which AI systems remain uncertain about human preferences and defer to human correction.
The thesis that sufficiently capable agents with almost any terminal goal will converge on certain instrumental sub-goals — self-preservation, resource acquisition, goal preservation — making powerful AI potentially dangerous regardless of its specific objective.
American AI safety researcher and philosopher — Research Analyst at Open Philanthropy — who writes long-form philosophical analysis of AI risk, alignment, and related questions with unusual rigor.
American AI safety researcher and forecaster — former OpenAI researcher — who published Situational Awareness (2024), a detailed forecast of AGI development timelines arguing that AGI will arrive by 2027 and will trigger a geopolitical race with China — whose departure from OpenAI and publication became a major AI policy event.
Eliezer Yudkowsky and others' Berkeley AI safety organization — focused on foundational mathematical research for creating provably safe artificial general intelligence — one of the earliest dedicated AI safety organizations.
Executive Director of the Machine Intelligence Research Institute (MIRI) — a nonprofit focused on AI alignment research — known for essays on rationality, motivation, and the urgency of AI alignment, and for the 'replacing guilt' framework for productivity and motivation.
Swedish philosopher at Oxford — founder of the Future of Humanity Institute — author of Superintelligence — who created the intellectual framework for AI existential risk and the simulation argument — the most influential philosopher in AI safety discourse.
American AI safety researcher — former OpenAI safety team lead — founder of the Alignment Research Center (ARC) — who invented RLHF (Reinforcement Learning from Human Feedback) in its original conceptual form — the most technically sophisticated AI safety researcher.
Leopold Aschenbrenner's 2024 long-form essay arguing that AGI is coming by 2027–2030, that OpenAI and other labs lack adequate security measures against Chinese espionage, and that the AI race represents the defining geopolitical competition of the next decade.
British-American computer scientist at UC Berkeley, co-author of the standard AI textbook (Artificial Intelligence: A Modern Approach), and author of Human Compatible — which argues for a new paradigm of AI development based on uncertainty about human preferences.
AI safety researcher at Open Philanthropy whose 'biological anchors' framework for forecasting transformative AI timelines became the most widely discussed quantitative AI timeline model in the EA/AI safety community.