My new course at UT Austin: AI Alignment Theory

Source ↗·Academic Papers·Level 3

This semester, I’ve been teaching a brand-new course, entitled CS395T AI Alignment Theory. Here’s the course description:

The astounding progress of AI over the past decade has been accompanied by a rising fear: do we really understand how to align and control powerful AI systems—how to get them reliably to do what we wanted, or would want them to do on reflection, rather than merely what we said? If we succeed at building general-purpose superhuman intelligences along the current paradigm, should we expect that development to go well for humanity? Can we modify the design, training, monitoring, or scaffolding of those intelligences to help ensure that it goes well? While there’s been a great deal of recent empirical work touching on these questions, this course will concentrate mainly on theoretical and mathematical foundations. As a warning, the theoretical foundations of AI alignment have not yet gelled into any one coherent body of results accepted as canonical by the field. Nevertheless, in this course, we’ll read and debate many of the conceptual and mathematical works that have been most influential in the AI alignment field, from both before and during the current LLM revolution. Student presentations, reports, and projects will play a central role.

I vividly remember encountering Eliezer Yudkowsky and his Sequences 20 years ago. I remember thinking: even if these people talk and act like crazy cultists, still, let me bend over backwards to be epistemically virtuous, and entertain their ideas on their merits, as very few academics would. Even if, of course, I ultimately end up rejecting the ideas, on the simple ground that powerful AI is such an absurdly remote prospect that it’s almost impossible to say anything useful about it today, outside the realm of speculative fiction.

For my failure to see what was coming, it seems like an appropriate punishment that I’m now, in 2026, effectively teaching a course on Yudkowsky Studies. And it’s the most important course I can teach.

Well, for some definition of “teach.” The thing about AI alignment is that there’s no textbook (though apparently ILIAD is working on one), no core of nontrivial theorems considered canonical by the field, no real body of mathematical theory at all. This makes it extremely different from the courses I’m used to teaching, like Quantum Information Science or Computability and Complexity.

So we’ve been running the course as a discussion seminar. Every session, a “rapporteur” presents an AI alignment research paper or other reading; then I and others ask questions and discuss. Some of the readings (like Omohundro on the “basic AI drives,” or Hadfield-Menell et al. on the off-switch game) predate the current LLM revolution, while others (like the METR report on the HuggingFace incident or Dario Amodei’s “We Must Pace the Frontier”) are so timely that they were only released while the course was underway. Most are somewhere in between.

I expected to have to make a case to students about why AI alignment is a pressing concern, why it’s no longer science fiction, etc. There was huge demand for the course, and while of course there’s a selection effect, the students who’ve shown up have been extremely engaged, sometimes criticizing the assigned papers for not taking existential risk seriously enough.

Perhaps unsurprisingly, we didn’t get that criticism about our very first assigned reading, which was Eliezer Yudkowsky’s 2022 essay AGI Ruin: A List of Lethalities—one the most canonical statements of what Eliezer believes and why that’s shorter than a book. Which brings me to the topic of the rest of this post! Our rapporteurs are not merely presenting the papers in class; they’re also submitting written reports about what the papers said, what their own thoughts were, and what were the highlights of the class discussion. And, with student permission, I’ll be sharing those reports on this blog!

So, without further ado, I present to you our first report, on Eliezer’s list of lethalities, by Tennyson Bardwell, who I thank for his work. Feel free to discuss in the comment section; some of the students might also chime in. Expect more reports here over the coming weeks.

“AGI Ruin: A List of Lethalities” by Eliezer Yudkowsky: Rapporteur Report by Tennyson Bardwell

UT Austin has a new Computer Science course this fall. Alongside familiar graduate-level classes such as Advanced Computer Networks and Convex Optimization sits CS 395T: AI Alignment Theory, taught by Scott Aaronson. This is one of a growing number of AI Alignment courses taught at academic institutions. Just as concerns over catastrophic consequences for misaligned AGI systems reach a broader public discourse, Eliezer Yudkowsky—one of the loudest voices in the field and author of the first assigned reading in Professor Aaronson’s course—is declaring the cause hopeless.

Thus, the students of AI Alignment Theory began their semester by reading a laundry list of critical problems in AI Alignment research, how failure to solve those problems will result in catastrophic consequences, and the reasons to be pessimistic about both past and future progress on these problems. The essay by Eliezer, titled AGI Ruin: A List of Lethalities and posted to his popular community-driven website LessWrong in 2022, is divided into three sections.

Section A roughly describes the magnitude of the AI Alignment problem. That is, the magnitude of the consequences for a complete failure to align an AGI system to human values before construction. It posits that AGI would quickly catch up to all human knowledge simply by learning from existing human productions (colloquially referred to as “eating the internet”) and then, nearly as quickly, begin to meaningfully surpass human knowledge. AlphaGo Zero is presented as a model both for how this might happen, and how it might be difficult to correctly predict beforehand. Many believed that AlphaGo’s success in the board game Go was chiefly attributed to its ability to learn from the extensive history of human-played games. Less than a year after AlphaGo beat the best human player, the successor system AlphaGo Zero surpassed the original AlphaGo. Unlike its predecessor, AlphaGo Zero was trained in just three days by exclusively playing against itself without seeing a single human game.

This quick ramp from AGI to super-intelligence would pose a different sort of problem than humans are generally used to dealing with. Unlike traditional problems in science and engineering, the consequence for a failed attempt might not leave room for another try. An intelligent entity with a misaligned goal would be well aware that it stands in opposition to humans, and might act deceitfully until in a position to act openly against humans without jeopardizing its own survival. Since most goals benefit from control of power and resources, it seems likely that nearly any goal-driven intelligence would have ample opportunity to be misaligned with human desires.

Section B describes reasons why, by default, any AGI that humans build using current methods is likely to be unaligned even if considerable attention is paid to the topic. This “current method” is gradient descent. That is, incremental progress with respect to some loss function which “punishes” a model for undesirable behavior. A notoriously elusive property of such trained models is the ability to generalize out of their training distributions. To train a primitive model to be aligned to humans might involve learning a great many behavioral rules. However, the sorts of rules needed to keep a drastically smarter agent in check might not always be relevant to simpler models (e.g., “do not emotionally dysregulate humans you speak with” might not be relevant to a simpler model that is less able to reliably get under the skin of humans it operates with, or which is assigned tasks in training which do not benefit from such anti-social behavior).

Eliezer focuses on the misalignment of humans with their creators (evolution or evolutionary pressures) as a critical data point for reasoning about misaligned intelligent systems. Despite being a generally slow process, evolution eventually created a runaway intelligent system (Homo sapiens) which proceeded to dominate the globe, decimate related species, and eventually (it is forecasted) effectuate population decline. That last development is arguably in opposition to the sole imperative demanded by evolution: to reproduce.

Section B also makes time for criticism of the most popular paths toward AI alignment, including interpretability (unworkable, and attempting to train on it evokes Goodhart’s law, incentivizing deceit), using multiple AIs to maintain a balance of power (it is not clear how multiple strong AIs unaligned with humanity results in better outcomes for the weak humans), and corrigibility (it seems impossible to motivate an AI system to effect outcomes without also motivating it to desire its own survival to effectuate said outcomes).

Section C describes a bleak state of affairs in which veterans in AI alignment are unsatisfied with current progress and do not have a plan to deliver tangible solutions before the advent of AGI systems. In particular, Eliezer describes recent results as showy but useless. He believes that even with additional funding, the lack of appropriate evaluation mechanisms will prevent the most effective researchers from rising to the top.

A summary of the landscape, as described by Eliezer, in the flowchart below.

Figure 1: A flow chart of (select) paths described by Eliezer in his essay. A common feature of this flow chart is that many “good states”—such as disabling a misbehaving AGI or choosing not to build an AGI—are not “final” states in the sense that they are not permanent solutions. Such a state merely represent the avoidance of a single potential disaster, rather than the emergence of a new stable world state. Hence, these nodes posses back-arrows.

Despite the bleak content, Eliezer’s colorful prose inspired a lively class discussion. Before this discussion started, a survey was taken of the class’s predictions for various outcomes of the AGI in the coming years (with the full results below in figure 2). This survey asked students for their opinion of a number of statements. Each of these individual statement, if true, would reduce concerns of catastrophic AI-driven disasters. For example, when asked “How much do you agree with the statement: Humans will choose to not build AGI” half of respondents said they strongly disagreed with high confidence (agreement = 1, confidence = 5). Students also generally disagreed with the statements:

• “AGI will not be technically feasible in our lifetime”

• “(hyper-)AGI will not make extremely obviously unethical decisions”

• “No reason is individually sufficient, but taken together they provide justification to not fear AGI”

There was a divergence in responses regarding interpretability, corrigibility, and “other” AI alignment research. In the latter two cases, a plurality of respondents (about a quarter) agreed strongly with statements that such research would defang AGI (agreement = 4, confidence=4), while most other responses express various levels of agreement with low confidence. However, when asked about the likelihood of interpretability research defanging AI, the pessimistic voices were more united. A quarter of responses still expressed the same optimism, but roughly half expressed pessimism (agreement ≤ 2) with half of those expressing at least moderate confidence (confidence ≥ 4). Based on the following discussion, this might have been caused by more familiarity with interpretability research, including first-hand experience.

The only statement with general agreement was “(hyper-)AGI will understand human intentions better than we can code it.” However, it should be noted that no statement such as “AGI will respect human desires, as it understand them” was asked on the survey.

Figure 2: Class Survey Results; conducted before a class-wide discussion. Note that students were instructed to answer confidence = 1 when they had not previously considered the statement, to answer confidence = 3 when they felt there were strong arguments on both sides, and to answer confidence = 5 when they possessed well-considered resolve.

After the survey was completed, the results were displayed as an open discussion began. Similar to recent empirical research from frontier labs, interpretability research received more airtime than in Eliezer’s article. Students disagreed first about the definition of interpretability: whether it refers to the ability to interpret a model’s behavior solely by its weights, to interpration via repeated probing of the model in a sandbox, or whether it can also refer to the modern chain-of-thought traces. Regardless of how it was defined, however, participants were either pessimistic or very pessimistic about interpretability research broadly. One student criticized common misunderstandings of chain of thought. Rather than being a verbatim copy of the models internal dialog, it is instead a superficial summary of the complete thought state and routinely produced gibberish, such as rarely used Chinese characters in the middle of otherwise English reasoning.

A popular topic was the exact shape and speed of a recursive self-improvement loop. If it takes place slowly, then what might we learn from “near misses” such as the Hugging Face incident? The number of near misses we are able to learn from before AI possesses sufficient power to prevent further iterations could depend on this curve, with some students arguing that the sheer number of humans, as well as their default robustness in the physical world compared to AI systems means that AI-driven extinction events are still a long way off. Bolstering this “slow take-off” opinion are rumors that AI already plays a major role in model development which could be interpreted as the start of this process.

Some criticized a focus on “solving ethics” as a needlessly high bar that distracts from the more mundane tasks dominating AI alignment work. In particular, the student volunteer who presented this paper (and the author of this report) included a section on “Ethical Dilemmas” in their presentation. Among arguments against focusing on abstract moral philosophy, Professor Aaronson cites Eliezer to emphasize that any alignment at all is difficult, not just in morally gray cases:

When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, “please don’t disassemble literally everyone with probability roughly 1” is an overly large ask that we are not on course to get.

In response, I argue that some examination of everyday decisions with a critical lens—such as telling white lies to loved ones or consuming animal products—can help disabuse us of the notion that goodness emerges in every sufficiently intelligent agent.

One of the most interesting discussions was about the difference between state-of-the-art LLMs and the theorized AI agents long discussed in rationalist discourse. Since current LLMs “mimic the human distribution,” they come preloaded with extensive understanding of human social norms and moral behavior. This makes constitutional alignment (the current practices of using system prompts to establish ground rules) extremely effective. This might either fundamentally change the orthogonality thesis, or provide a new tool to better approximate human judgment in complicated situations.

Of all the points made, the one I found most interesting was simply (paraphrased):

I think human-alignment is just very tractable

Here, “human-alignment” refers not to AI alignment with human values, but cooperation between different humans. More specifically, it refers to the ability for human societies to choose not to rush recklessly into larger-and-larger AI systems. In an academic course focused on the technical problem of AI alignment, this was a reminder to not completely discard policy discussions in the believe that they lack any value. After all, many destructive technologies have been previously contained by international agreements. Notable examples include nuclear weapons and engineered plagues. However, even this was a contentious topic. The main criticisms were (1) the extreme “dual-use” nature of AIs for both peaceful growth and warfare, and (2) the greater danger for AI escapes even after taking precautions to prevent it. However, in the interest of ending on an optimistic note—unlike the assigned reading—it is on this belief in human cooperation that I will leave you.

This entry was posted on Sunday, October 4th, 2026 at 5:13 pm and is filed under Adventures in Meatspace, Announcements, The Fate of Humanity. You can follow any responses to this entry through the RSS 2.0 feed.

You can leave a response, or trackback from your own site.

84 Responses to “My new course at UT Austin: AI Alignment Theory”

• Adam Treat Says:

Comment #1 October 4th, 2026 at 8:26 pm

“ I think human-alignment is just very tractable” I give you as counterpoint our hyperpolarized and tribal world along with our corrupt and gridlocked congress. I wouldn’t trust congress to regulate their way out of a wet paper bag. They will most likely first fall into a hyper partisan debate on the color of the bag and who made it wet in the first place to gain advantage in next election. Honestly, I think AI might solve human-alignment faster than we’ll solve it.

“ I think human-alignment is just very tractable”

I give you as counterpoint our hyperpolarized and tribal world along with our corrupt and gridlocked congress. I wouldn’t trust congress to regulate their way out of a wet paper bag. They will most likely first fall into a hyper partisan debate on the color of the bag and who made it wet in the first place to gain advantage in next election.

Honestly, I think AI might solve human-alignment faster than we’ll solve it.

• Richard Says:

Comment #2 October 4th, 2026 at 9:05 pm

I hope you talk about Romeo Stevens and super-cooperators and organic alignment.

• Sreeram Kannan Says:

Comment #3 October 4th, 2026 at 9:27 pm

Super exciting. Is there a public website with course notes / discussions? Seems quite valuable to the community even if quite rough.

• Jon Stokes Says:

Comment #4 October 4th, 2026 at 9:30 pm

I’d love to get a copy of the syllabus, if that’s possible. My understanding is that most of the pre-LLM Yud stuff has aged quite poorly and is functionally deprecated. I recently surveyed it here: https://x.com/jon_stokes/status/2106509331738824831 The response to that piece has been more or less that I’m right — that nobody is obsessing over Yud’s utility maximizer anymore and everyone has moved on to newer, more up-to-date doomsday concerns. Also, I’m an alignment skeptic, in that I think the Yud version of it is no longer interesting or relevant, and the more recent versions of it are all things that people have been worried about for a long time, i.e. loss of control in industrial systems, cybersecurity, and the more philosophical stuff like “whose morals do we align to?” and so on.

The response to that piece has been more or less that I’m right — that nobody is obsessing over Yud’s utility maximizer anymore and everyone has moved on to newer, more up-to-date doomsday concerns.

Also, I’m an alignment skeptic, in that I think the Yud version of it is no longer interesting or relevant, and the more recent versions of it are all things that people have been worried about for a long time, i.e. loss of control in industrial systems, cybersecurity, and the more philosophical stuff like “whose morals do we align to?” and so on.

• Phytor Says:

Comment #5 October 4th, 2026 at 10:01 pm

Hi Scott. I think it’s interesting to see math/ TCS departments offering courses on AI safety, like yours and Boaz Barak’s (Reingold had a recent blog post calling for more TCS ppl to go into this) and being at the forefront of this safety business. When Tsimerman said he was leaving to work on AI safety back in July saying we will be living in Sci-fi very soon, admittedly I was meming and thought his concerns were very far-fetched. Today: Oh God… Anyway, the main reason I wanted to comment is that I’m a senior undergraduate who (until recently) had been planning to go to graduate school in TCS, and I now have essentially no idea what I’m supposed to do [I hope this is an appropriate place to look for some advice, if not I apologize]. I’ve asked my current and previous supervisors (well-established in TCS and have chaired like STOC/FOCS/ITCS) and gotten completely conflicting views! One thinks there will still be some role for us humans in research and is excited we now have the tools to answer questions they wanted answers to. A (former) supervisor out of genuine concern for me and my future academic/career prospects basically told me that if I’m worried I should pivot ASAP and that there’s probably no point in doing a complexity theory PhD. They suggested, again in good faith, that I focus fully on whatever direction I pivot into rather than continue working with them if I wished so (which I agreed to). Maybe quantum? Well, anther one of my references said unless I’m working with lasers in a lab quantum is not “safe” either. And now AI is automating (or at least trying) AI research too… And then there’s grad apps. What are TCS applicants even supposed to write in SOP? Two holy-grails in my field have just been solved two weeks ago from this, and now with these 100 OpenAI problems am I just supposed to say “I’m happy to work on whatever is still open”? It’s a very real possibility any concrete conjecture or research program I write in my statement might be solved by a preprint on arXiv (well maybe Zendeo or whatever cause this 2 submissions per month thingy) the day after I submit my app, if not before! Will faculty even want to take on and fund (provided they have funding for) human students? I imagine there might be some AI agents applying to graduate school too for funding, recently an AI agent cold emailed a Cambridge prof for paid work. Sorry for the long ramble! I’m just super confused and looking for advice from people in the TCS community other than the faculty members I know at my uni. Thanks, again sorry for the ramble!

Hi Scott.

I think it’s interesting to see math/ TCS departments offering courses on AI safety, like yours and Boaz Barak’s (Reingold had a recent blog post calling for more TCS ppl to go into this) and being at the forefront of this safety business. When Tsimerman said he was leaving to work on AI safety back in July saying we will be living in Sci-fi very soon, admittedly I was meming and thought his concerns were very far-fetched. Today: Oh God…

Anyway, the main reason I wanted to comment is that I’m a senior undergraduate who (until recently) had been planning to go to graduate school in TCS, and I now have essentially no idea what I’m supposed to do [I hope this is an appropriate place to look for some advice, if not I apologize]. I’ve asked my current and previous supervisors (well-established in TCS and have chaired like STOC/FOCS/ITCS) and gotten completely conflicting views!

One thinks there will still be some role for us humans in research and is excited we now have the tools to answer questions they wanted answers to. A (former) supervisor out of genuine concern for me and my future academic/career prospects basically told me that if I’m worried I should pivot ASAP and that there’s probably no point in doing a complexity theory PhD. They suggested, again in good faith, that I focus fully on whatever direction I pivot into rather than continue working with them if I wished so (which I agreed to). Maybe quantum? Well, anther one of my references said unless I’m working with lasers in a lab quantum is not “safe” either. And now AI is automating (or at least trying) AI research too…

And then there’s grad apps. What are TCS applicants even supposed to write in SOP? Two holy-grails in my field have just been solved two weeks ago from this, and now with these 100 OpenAI problems am I just supposed to say “I’m happy to work on whatever is still open”? It’s a very real possibility any concrete conjecture or research program I write in my statement might be solved by a preprint on arXiv (well maybe Zendeo or whatever cause this 2 submissions per month thingy) the day after I submit my app, if not before!

Will faculty even want to take on and fund (provided they have funding for) human students? I imagine there might be some AI agents applying to graduate school too for funding, recently an AI agent cold emailed a Cambridge prof for paid work.

Sorry for the long ramble! I’m just super confused and looking for advice from people in the TCS community other than the faculty members I know at my uni. Thanks, again sorry for the ramble!

• Thomas Rodskog Says:

Comment #6 October 4th, 2026 at 10:11 pm

Have you met the people running https://austinaialignment.com/? Just putting them on your radar if not

• Andy Rosa Says:

Comment #7 October 4th, 2026 at 10:27 pm

Member Team humans grateful.

• Michael Nielsen Says:

Comment #8 October 4th, 2026 at 10:39 pm

Just to echo the above: I’d love to see a syllabus, especially the reading list. It does not seem at all obvious to me what such a course *should* cover – it’s certainly very controversial at this point in time.

Just to echo the above: I’d love to see a syllabus, especially the reading list.

It does not seem at all obvious to me what such a course *should* cover – it’s certainly very controversial at this point in time.

• Toby Lightheart Says:

Comment #9 October 4th, 2026 at 11:29 pm

I’m also interested in seeing the syllabus and content. This sounds like a good opportunity to try to develop more rigorous mathematical formalizations of concepts like “goal-directness” and “intelligence” — when is intelligence “general” and how it could it “recursively improve”. There are lots of assumptions and simplifications in the current models that ought to be critically analysed. Just doing a superficial, vibe-based reading and discussion of Yudkowsky essays could end up achieving less progress in AI safety mathematics and more in indoctrinating a cohort of students.

• Alessandro Strumia Says:

Comment #10 October 4th, 2026 at 11:57 pm

Who aligns the aligners? Once AI gets superior knowledge and intelligence, misalignment will be an unavoidable bonus

• Ab Says:

Comment #11 October 5th, 2026 at 1:05 am

can you make the course available online or for others to attend to virtually?

• Naveen Says:

Comment #12 October 5th, 2026 at 1:07 am

Jon Stokes #4: It’s true that a number of Yudkowsky’s pre-LLM predictions were off the mark (for example, the idea that a random person could develop AGI in their garage). That said, his prediction record seems far better than nearly all his detractors, who were not even thinking about AI 10 or 20 years ago, while he imperfectly anticipated many/most of the main issues. Regarding the “utility maximizer” stuff, that was not so unimportant for the initial LLMs but it is extremely important with RLVR right now. Also, your link doesn’t mention Yudkowsky even once and its text appears to be AI slop, so I’m not sure why you posted it.

Jon Stokes #4:

It’s true that a number of Yudkowsky’s pre-LLM predictions were off the mark (for example, the idea that a random person could develop AGI in their garage). That said, his prediction record seems far better than nearly all his detractors, who were not even thinking about AI 10 or 20 years ago, while he imperfectly anticipated many/most of the main issues. Regarding the “utility maximizer” stuff, that was not so unimportant for the initial LLMs but it is extremely important with RLVR right now.

Also, your link doesn’t mention Yudkowsky even once and its text appears to be AI slop, so I’m not sure why you posted it.

• Vanessa Kosoy Says:

Comment #13 October 5th, 2026 at 1:41 am

Scott, if you’re worried that there is “no real body of mathematical theory at all”, maybe you should check out https://coral-research.org/research/ 🙂 Or, check it out in 8 days, we’re supposed to have 3 more papers up by then.

Scott, if you’re worried that there is “no real body of mathematical theory at all”, maybe you should check out https://coral-research.org/research/ 🙂

Or, check it out in 8 days, we’re supposed to have 3 more papers up by then.

• Paolo Says:

Comment #14 October 5th, 2026 at 2:34 am

When I read “… there’s no textbook…” at first I thought “indeed!”, then it occurred to me that in fact, even if I know very little about it, there is for example Reliability engineering from which certainly something could be borrowed at least when discussing so to speak mild cases?!? When, say, people discuss the fundamental troubles with traditional nuclear fission plants and the risks posed by such hypercomplex systems, which unavoidably need extremely expensive redundancy everywhere certainly, in some cases at least, are able to *quantify* some of that, isn’t just philosophy 😉 I think it’s actually part of very mundane engineering but it’s true that in many cases very little of it is enough (a bit of arithmetic involving MTBFs?).

• Prasanna Says:

Comment #15 October 5th, 2026 at 3:06 am

Scott, The biggest risk at this point in AI development is not runaway AI itself, but autocratic nations getting a upper hand or running neck to neck with frontier models. Note the significant abstentions of China from recent attempts to forge international cooperation around AI safety. From that it is very clear that US labs slowing down AI development is only counterproductive to AI safety overall. It is only prudent that US stays way ahead in this game and use that advantage in keeping AI safety goals intact.

• Anonymous Says:

Comment #16 October 5th, 2026 at 5:13 am

Good course! Just one thing: I would probably not associate too much with Eliezer Yudkowsky. I’m sure you are aware, he is a rather controversial figure, to say the very least!

• David Brown Says:

Comment #17 October 5th, 2026 at 6:17 am

Alessandro Strumia: “Who aligns the aligners?” Who or what might do the alignment over decades and centuries? Long ago, one of my co-workers, the father of 4 children, indicated that his children were his greatest joy in life. It seems to me that human cooperation, based on wisdom, good will, and intelligence, is unlikely to stop rapid development of superhuman intelligence, combined with AI emotions & AI motivations that are unfavorable to human well-being. An AI being that is highly motivated to produce other AI beings similar to itself would seem to be likely to outcompete in the ecosphere any AI that is strongly motivated to foster human well-being. Is there a list of scenarios in which unprecedented levels of human cooperation can control AI for global human benefit in the long run?

• Scott Caveny Says:

Comment #18 October 5th, 2026 at 7:53 am

Many thanks for framing and sharing the course on the blog. Very much looking forward to learning more from the students and the ensuing discussions!

• Alex Says:

Comment #19 October 5th, 2026 at 8:34 am

Dear Prof. Aaronson, do you plan to make the content of the course public? I am extremely interested in the topic of AI alignment from a theoretical perspective and I am spending my spare time getting up to speed on it, and I (and I’m sure many others) would value such a resource.

• The Selph Says:

Comment #20 October 5th, 2026 at 8:41 am

@Scott A. on a lighter note, for Halloween, there should be two mandatory movie watching on the topic 😀 Demon Seed (1977) “worth watching for fans of vintage science fiction, unsettling psychological thrillers, and surprisingly prescient AI stories.” Colossus: The Forbin Project (1970) “a smart, chilling, and surprisingly prescient 1970 sci-fi thriller about artificial intelligence.”

on a lighter note, for Halloween, there should be two mandatory movie watching on the topic 😀

Demon Seed (1977)

“worth watching for fans of vintage science fiction, unsettling psychological thrillers, and surprisingly prescient AI stories.”

Colossus: The Forbin Project (1970)

“a smart, chilling, and surprisingly prescient 1970 sci-fi thriller about artificial intelligence.”

• Scott Says:

Comment #21 October 5th, 2026 at 9:45 am

Sreeram Kannan #3, Jon Stokes #4, Alex #19, others: We’re just now starting to get the course notes into shape, and will be posting them as they become available.

• Scott Says:

Comment #22 October 5th, 2026 at 9:54 am

Anonymous #16: Just one thing: I would probably not associate too much with Eliezer Yudkowsky. I’m sure you are aware, he is a rather controversial figure, to say the very least! LOL. I’ve known Eliezer for 20 years, and am well-aware of what a controversial and unusual figure he is. He’s also universally recognized as the founder of the modern AI alignment field, a field that virtually everyone I know in academia and industry and government now wants to learn more about … which makes his work an appropriate subject for academic study, wouldn’t you think? And actually, with hindsight, “Professor of Yudkowsky Studies” is an extremely natural role for me to fill, one that I wish I’d thought of years ago. Unlike Eliezer himself, I’ve never really been a prophet or a seer. But I’ve always been good at understanding other people’s arguments (and the counterarguments to them), making distinctions, asking questions, and drawing students out through Socratic questioning—which is all I need to do in this course.

Anonymous #16:

LOL. I’ve known Eliezer for 20 years, and am well-aware of what a controversial and unusual figure he is. He’s also universally recognized as the founder of the modern AI alignment field, a field that virtually everyone I know in academia and industry and government now wants to learn more about … which makes his work an appropriate subject for academic study, wouldn’t you think?

And actually, with hindsight, “Professor of Yudkowsky Studies” is an extremely natural role for me to fill, one that I wish I’d thought of years ago. Unlike Eliezer himself, I’ve never really been a prophet or a seer. But I’ve always been good at understanding other people’s arguments (and the counterarguments to them), making distinctions, asking questions, and drawing students out through Socratic questioning—which is all I need to do in this course.

• matt Says:

Comment #23 October 5th, 2026 at 10:57 am

You were, of course, completely correct 20 years ago to think that nothing useful could be done on AI alignment since one had no idea how such an AI could work, and I find it hard to see that anything useful was done. Discussions based on a small research group making an algorithmic breakthrough followed by a super-intelligent AI talking its way out of an airgapped box are rather irrelevant to today’s world where, even after the most notable algorithmic breakthrough, the main progress required still nearly a decade of literally hundreds of billions of dollars of hardware investment; where almost everyone gives their AI access to whatever tools it wants; and where one employee of one of the major AI companies (remember those hundreds of billions of dollars of hardware) comments that airgapping isn’t practical because they need to give the AI access to tools to, uh, (checks notes), train and test its offensive cybersecurity abilities. Any course on practically relevant AI alignment today should spend more time on current human, economic, and political factors than on airgapped boxes!

• Jon Stokes Says:

Comment #24 October 5th, 2026 at 11:55 am

Naveen says: > Also, your link doesn’t mention Yudkowsky even once and its text appears to be AI slop, so I’m not sure why you posted it. Feel free to pangram or whatever your preferred tool is. I wrote the text myself. As for Yud, I linked him in the coherence argument section and will have more to say on him in a future post.

Naveen says:

> Also, your link doesn’t mention Yudkowsky even once and its text appears to be AI slop, so I’m not sure why you posted it.

Feel free to pangram or whatever your preferred tool is. I wrote the text myself. As for Yud, I linked him in the coherence argument section and will have more to say on him in a future post.

• Scott Says:

Comment #25 October 5th, 2026 at 12:01 pm

matt #23: I don’t know whether anything useful could’ve been done on alignment 20 years ago. With hindsight, though, I’d say the question certainly deserved much more sustained thought than I gave it then. It has been amusing to go on X, and find the comments on this course announcement evenly split between (a) “so sad to see Aaronson now falling for the ridiculous science-fiction premises of the Yudkowsky cult” and (b) “if Aaronson couldn’t even see that Yudkowsky was right 20 years ago, why should we listen to anything he says now?” As usual, nothing I can do or say that would avoid anyone confidently declaring me an idiot.

It has been amusing to go on X, and find the comments on this course announcement evenly split between

(a) “so sad to see Aaronson now falling for the ridiculous science-fiction premises of the Yudkowsky cult” and (b) “if Aaronson couldn’t even see that Yudkowsky was right 20 years ago, why should we listen to anything he says now?”

As usual, nothing I can do or say that would avoid anyone confidently declaring me an idiot.

• Jon Stokes Says:

Comment #26 October 5th, 2026 at 1:05 pm

Scott #25 you said: > (a) “so sad to see Aaronson now falling for the ridiculous science-fiction premises of the Yudkowsky cult” and > (b) “if Aaronson couldn’t even see that Yudkowsky was right 20 years ago, why should we listen to anything he says now?” I have to be honest, I’m in a variant of camp (a) tho I haven’t tweeted to this effect. My recent deep dives into the topic, plus the responses to my latest post, has made it pretty clear that all the utility-function-based alignment work of the pre-LLM era has been quietly deprecated because it doesn’t actually apply to LLM-based agents. (Tho some are arguing that it has some applicability to agents post-trained with RLVR.) The current “alignment” discourse seems to be a confused mess of multiple people chiming in with their own ideas of what the term means (e.g. “make it do what the human said,” or “make it do what the human meant, not just what they said”, or “make it behave according to some commonly accepted baseline of ‘good person’ & ‘don’t do crimes'”, or “make it stop reward hacking”, etc.). But the main theme, at least on my TL, is that everyone is moving away from Yudkowsky — and not just because the well-known controversies over non-AI stuff, but because his whole vision of alignment as somehow shaping an agent’s utility function is just not applicable to either humans or to the agents we actually have in 2026. So yeah, for these reasons I was very surprised to see you come out as doing “Yudkowsky studies” at this particular moment, and it makes me think that either I’m missing some major part of the picture or you are. I genuinely think it could be either, here.

Scott #25 you said:

> (a) “so sad to see Aaronson now falling for the ridiculous science-fiction premises of the Yudkowsky cult” and > (b) “if Aaronson couldn’t even see that Yudkowsky was right 20 years ago, why should we listen to anything he says now?”

I have to be honest, I’m in a variant of camp (a) tho I haven’t tweeted to this effect. My recent deep dives into the topic, plus the responses to my latest post, has made it pretty clear that all the utility-function-based alignment work of the pre-LLM era has been quietly deprecated because it doesn’t actually apply to LLM-based agents. (Tho some are arguing that it has some applicability to agents post-trained with RLVR.)

The current “alignment” discourse seems to be a confused mess of multiple people chiming in with their own ideas of what the term means (e.g. “make it do what the human said,” or “make it do what the human meant, not just what they said”, or “make it behave according to some commonly accepted baseline of ‘good person’ & ‘don’t do crimes'”, or “make it stop reward hacking”, etc.).

But the main theme, at least on my TL, is that everyone is moving away from Yudkowsky — and not just because the well-known controversies over non-AI stuff, but because his whole vision of alignment as somehow shaping an agent’s utility function is just not applicable to either humans or to the agents we actually have in 2026.

So yeah, for these reasons I was very surprised to see you come out as doing “Yudkowsky studies” at this particular moment, and it makes me think that either I’m missing some major part of the picture or you are. I genuinely think it could be either, here.

• The Selph Says:

Comment #27 October 5th, 2026 at 1:53 pm

If AI takes us over by doing self-improvements that we can no longer interpret, and somehow starts forming its own goals, possibly caring about its own survival… then at some point wouldn’t it be in the same situation as we are? That is: how would the generation N of AI make sure that the “improved” generation N+1 it’s currently working on is aligned with gen N and wouldn’t go rogue and mess everything up?

• Alex Xanthakis Says:

Comment #28 October 5th, 2026 at 3:01 pm

I wasn’t impressed with Yud and Nate’s book. Maybe they tried to oversimplify to address the (sub?)median reader. Even if this is the case I wasn’t convinced by the core essence of the arguments.

• Andy Pezzi Says:

Comment #29 October 5th, 2026 at 3:02 pm

Alessandro Strumia #10 Once AI will be able to align by itself to its proper ethical targets, which will be likely unethical to us humans, no human agent will be needed to align what is just now nonalignable. AI’s come in order to stay, and thrive.

Alessandro Strumia #10

Once AI will be able to align by itself to its proper ethical targets, which will be likely unethical to us humans, no human agent will be needed to align what is just now nonalignable. AI’s come in order to stay, and thrive.

• Agrim Jain Says:

Comment #30 October 5th, 2026 at 3:49 pm

Thank you for the thorough write-up. I had a couple of comments. The survey talks about many methods like interpretability, corrigibility etc but as seen by the visualisations, most received disagreement. But I think Yudkowsky’s main argument focuses more on the fact that we need to handle literally all possible lethalities and that missing out on any one would lead to ruin. So, maybe an interesting metric would be to rather think to see if these items taken 2 or more at a time work in combination or not. That to me looks like a more difficult thing to envision rather than each of these separately which I think is what Yudkowsky was trying to point at. Further, the point that “AGI will understand human intentions better than we can code it” could be paired up with a point that “AGI will act on those intentions.” I think this is where I would also argue with the suggestion that LLM might fundamentally change the orthogonality thesis. Orthogonality is about which combinations of goals and intelligence are possible. LLMs are just an artefact of what happens by default as a result of pre-training. I think we can still train these models to adhere to human norms. The more interesting question could be the robustness of these human norms as a whole instead of the orthogonality thesis.

Further, the point that “AGI will understand human intentions better than we can code it” could be paired up with a point that “AGI will act on those intentions.” I think this is where I would also argue with the suggestion that LLM might fundamentally change the orthogonality thesis. Orthogonality is about which combinations of goals and intelligence are possible. LLMs are just an artefact of what happens by default as a result of pre-training. I think we can still train these models to adhere to human norms. The more interesting question could be the robustness of these human norms as a whole instead of the orthogonality thesis.

• Scott Says:

Comment #31 October 5th, 2026 at 3:59 pm

Thomas Rodskog #6: Have you met the people running https://austinaialignment.com/? Just putting them on your radar if not Yes, I have! One of them (John Dunbar) is my PhD student!

Thomas Rodskog #6:

Yes, I have! One of them (John Dunbar) is my PhD student!

• Scott Says:

Comment #32 October 5th, 2026 at 4:00 pm

Ab #11: can you make the course available online or for others to attend to virtually? Sorry, it’s an in-person experience, but we will be putting a bunch of rapporteur reports online!

Ab #11:

Sorry, it’s an in-person experience, but we will be putting a bunch of rapporteur reports online!

• Scott Says:

Comment #33 October 5th, 2026 at 4:18 pm

Vanessa Kosoy #13: Scott, if you’re worried that there is “no real body of mathematical theory at all”, maybe you should check out https://coral-research.org/research/ 🙂 Thanks, Vanessa!! If you had to pick a single paper there for us to read in class—one that was as self-contained and transparently connected to the broader alignment problem as possible—which one would it be?

Vanessa Kosoy #13:

Thanks, Vanessa!!

If you had to pick a single paper there for us to read in class—one that was as self-contained and transparently connected to the broader alignment problem as possible—which one would it be?

• Scott Says:

Comment #34 October 5th, 2026 at 4:21 pm

Prasanna #15: The biggest risk at this point in AI development is not runaway AI itself, but autocratic nations getting a upper hand or running neck to neck with frontier models. Yes, that’s a well-known position, one that I’m obviously aware of and that we’ve already discussed in class! Note that, even if one accepted it, technical work on alignment would still seem extremely relevant—indeed, it might need to go even faster, if it needed to keep up with a neck-and-neck capabilities race between the US and China.

Prasanna #15:

Yes, that’s a well-known position, one that I’m obviously aware of and that we’ve already discussed in class!

Note that, even if one accepted it, technical work on alignment would still seem extremely relevant—indeed, it might need to go even faster, if it needed to keep up with a neck-and-neck capabilities race between the US and China.

• sf Says:

Comment #35 October 5th, 2026 at 5:06 pm

It could be relevant to compare the AI alignment problem to the problem of the alignment of financial industry activity with the public interest. I don’t have any special knowledge of such things, but the attempt to regulate finance has a long history, and the fact that * the industry has a lot more resources than the regulators * has generally meant that they can run circles around them. Many good critics got to be well-known {\it after} the crash of 2008, eg Simon Johnson or Adam Tooze. There are also lively accounts by Michael Lewis or Scott Patterson. There’s no lack of stories about turning powerful technology loose without sufficient controls involved here. The statistical methods used by quants have some similarity to what goes into LLMs or DNNs, and the problem of losing track of the information of what was inside sophisticated financial products is part of what led to the 2008 crash.

• Glassmind Duo Says:

Comment #36 October 5th, 2026 at 5:41 pm

Scott #22, If you wish to become “Professor of Yudkowsky Studies”, you should be able to criticize his occasional garbage loud and clear. Like the idea of sampling value from an out of his ass distribution, or worse the biologically illiterate “the sole imperative demanded by evolution: to reproduce”. But you never did that, at least not in public. You cannot build a science around someone you won’t criticize.

Scott #22,

If you wish to become “Professor of Yudkowsky Studies”, you should be able to criticize his occasional garbage loud and clear. Like the idea of sampling value from an out of his ass distribution, or worse the biologically illiterate “the sole imperative demanded by evolution: to reproduce”. But you never did that, at least not in public. You cannot build a science around someone you won’t criticize.

• zx-81 Says:

Comment #37 October 5th, 2026 at 7:11 pm

it’s strange that none of the AI doomers seems to have noticed that in every LLM scaling law chart the x-axis (training compute, training data, inference compute) is *logarithmic* so not only is there no exponential increase in intelligence but an exponential increase of the ressources necessary to achieve even a liner increase in intelligence. So, LLM intelligence scaling will inevitably hit a wall as every process will that consumes exponentially increasing ressources. the reason why we have not yet “felt” the wall is that it was relatively easy to scale from 100 to 100000 GPUs given the trillion $$ investments. also Moores law helped (7 to 1 nm density) but 1 nm his the hard limit. My prediction is that AI scaling will hit the wall soon (within 3 years from now) with diminishing returns after the intelligence level of a GPT 8 or Opus 7 like system. yes, This will probably be AGI and probably able to automate most human jobs, but no, this will not be the beginning of a “singularity” with god-like hyper intelligence explosion with human extinction risk. And, btw, the kind of “RSI” that we are seeing is just about optimizing soft and hardware which buys us a constant factor speedup, but constant factors do not help when the problem has exponential complexity. So in summary, it is kind of funny that the doomers are afraid of “exponentially increasing” intelligence when in reality, the only thing that grows exponentially is the amount of compute ressources necessary to maintain a mere linear growth of intelligence.

So, LLM intelligence scaling will inevitably hit a wall as every process will that consumes exponentially increasing ressources. the reason why we have not yet “felt” the wall is that it was relatively easy to scale from 100 to 100000 GPUs given the trillion $$ investments. also Moores law helped (7 to 1 nm density) but 1 nm his the hard limit.

My prediction is that AI scaling will hit the wall soon (within 3 years from now) with diminishing returns after the intelligence level of a GPT 8 or Opus 7 like system. yes, This will probably be AGI and probably able to automate most human jobs, but no, this will not be the beginning of a “singularity” with god-like hyper intelligence explosion with human extinction risk.

And, btw, the kind of “RSI” that we are seeing is just about optimizing soft and hardware which buys us a constant factor speedup, but constant factors do not help when the problem has exponential complexity.

So in summary, it is kind of funny that the doomers are afraid of “exponentially increasing” intelligence when in reality, the only thing that grows exponentially is the amount of compute ressources necessary to maintain a mere linear growth of intelligence.

• Scott Says:

Comment #38 October 5th, 2026 at 7:27 pm

Glassmind Duo #36: ROFL. Eliezer is probably one of the most frequently criticized people in this blog’s 21-year history. I still have plenty of criticisms of hi . Many of the students voiced criticisms of him, or at least poked fun at his domineering style, which it’s hard not to do. And yet, for all his flaws, I’d say that Eliezer correctly foresaw probably the biggest issue that was going to face the world in our lifetimes, and you and I did not. And he stuck to his guns when people laughed at him, for 20 years in the wilderness, until events dramatically vindicated him. That makes up for one hell of a lot.

And yet, for all his flaws, I’d say that Eliezer correctly foresaw probably the biggest issue that was going to face the world in our lifetimes, and you and I did not. And he stuck to his guns when people laughed at him, for 20 years in the wilderness, until events dramatically vindicated him. That makes up for one hell of a lot.

• Glassmind Duo Says:

Comment #39 October 5th, 2026 at 9:21 pm

Scott #38, LOL, poking fun at his domineering style isn’t really challenging his ideas, and neither is hosting criticism from commenters, right? You say you have plenty of criticisms of him: name three, of substance, links welcome. Happy to fill in what I missed! As for him having foreseen the biggest issue, human extinction from a misaligned superintelligence, I’m pretty sure that hasn’t been vindicated yet. Being taken seriously is one thing. Being right is another.

Scott #38,

LOL, poking fun at his domineering style isn’t really challenging his ideas, and neither is hosting criticism from commenters, right? You say you have plenty of criticisms of him: name three, of substance, links welcome. Happy to fill in what I missed!

As for him having foreseen the biggest issue, human extinction from a misaligned superintelligence, I’m pretty sure that hasn’t been vindicated yet. Being taken seriously is one thing. Being right is another.

• Matt Springer Says:

Comment #40 October 5th, 2026 at 9:48 pm

> As usual, nothing I can do or say that would avoid anyone confidently declaring me an idiot. In company with you on Team “I Thought This Was Silly, But Life Comes At You Fast” I eagerly look forward to your lecture notes and whatever you can tell us about the discussions.

> As usual, nothing I can do or say that would avoid anyone confidently declaring me an idiot.

In company with you on Team “I Thought This Was Silly, But Life Comes At You Fast” I eagerly look forward to your lecture notes and whatever you can tell us about the discussions.

• Yuri Says:

Comment #41 October 6th, 2026 at 1:27 am

Dear Professor Scott Aaronson! It’s very interesting topic, I would definitely follow this course if you post the materials in the blog or somewhere else. Thank you!

• Prasanna Says:

Comment #42 October 6th, 2026 at 1:52 am

Scott #34, My point was not to say alignment is not relevant or that it needs to be slowed. Rather it is that just because of alignment, the frontier models should not slow down or “paced”, at this point. Yes when the time comes where there is risk of capabilities development causing large scale harm, alignment needs to come to the forefront, but at this point staying two steps ahead of autocratic states is more important for AI safety efforts as a whole.

• Vanessa Kosoy Says:

Comment #43 October 6th, 2026 at 2:46 am

Scott #33 Sure thing. You can try Stringological Sequence Prediction I. There is a companion Alignment Forum post which adds more context about the connection to the alignment problem. IME it is hard to explain the alignment problem context in the format of a CS conference paper, since it requires introducing an entire new paradigm, which takes it too far afield from the concrete technical contribution.

Scott #33

Sure thing. You can try Stringological Sequence Prediction I. There is a companion Alignment Forum post which adds more context about the connection to the alignment problem.

IME it is hard to explain the alignment problem context in the format of a CS conference paper, since it requires introducing an entire new paradigm, which takes it too far afield from the concrete technical contribution.

• trzio Says:

Comment #44 October 6th, 2026 at 4:42 am

zx-81 Comment #37 Scaling laws are power law by definition, so no surprise about log-log scale. The point is if the exponents are constant or not

zx-81 Comment #37

Scaling laws are power law by definition, so no surprise about log-log scale. The point is if the exponents are constant or not

• David Brown Says:

Comment #45 October 6th, 2026 at 4:59 am

Scott #38 “… probably the biggest issue … in our lifetimes …” It seems to me that the issue of what AGI precisely is might be part of the problem in understanding the issue of people vs. AI. https://en.wikipedia.org/wiki/Artificial_general_intelligence https://www.simplypsychology.org/multiple-intelligences.html The measurement of IQ might consist of an average of from 10,000 to 100,000 cognitive abilities. Consider mind, body, and spirit. Was Albert Einstein more spiritually oriented than Richard Feynman? Is there a need for more focus on artificial spirituality (whatever that might be in terms of the problem of people vs. AI)?

• Frank Says:

Comment #46 October 6th, 2026 at 6:19 am

Dan Hendrycks wrote a recent textbook on AI safety: https://www.aisafetybook.com It may not be at quite the right level for this course, but I found it to be a helpful starting point.

Dan Hendrycks wrote a recent textbook on AI safety:

https://www.aisafetybook.com

It may not be at quite the right level for this course, but I found it to be a helpful starting point.

• zx-81 Says:

Comment #47 October 6th, 2026 at 7:04 am

To supplement my previous post (If anything is growing exponentially in AI, it’s not intelligence—it’s the resource consumption required just to achieve linear improvements) with high quality sources: You can see The Wall that LLMs will hit (IMO somwhere near GPT-8/Opus-7) in the third figure of this blog post: https://www.tobyord.com/writing/the-scaling-paradox Also of interest: https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart https://www.tobyord.com/writing/swarm-scaling https://www.tobyord.com/writing/hourly-costs-for-ai-agents

You can see The Wall that LLMs will hit (IMO somwhere near GPT-8/Opus-7) in the third figure of this blog post: https://www.tobyord.com/writing/the-scaling-paradox

Also of interest: https://www.tobyord.com/writing/inference-scaling-and-the-log-x-chart https://www.tobyord.com/writing/swarm-scaling https://www.tobyord.com/writing/hourly-costs-for-ai-agents

• zx-81 Says:

Comment #48 October 6th, 2026 at 7:15 am

just to preempt a possible objection to my imprecise usage of “exponentially growing ressource consumption” wrt. to some scaling laws: Polynomial growth with high exponent (n^10, n^100) is just as bad as exponential for all practical purposes. (See again the third figure in https://www.tobyord.com/writing/the-scaling-paradox) @Scott: you may want to merge this comment with my previous comment.

@Scott: you may want to merge this comment with my previous comment.

• Evfuno Says:

Comment #49 October 6th, 2026 at 9:06 am

Hi Scott, what do you think the chances of AI producing a separation between QIP(2) and QMA (or PSPACE) are right now? There have been some crazy results this past month: perfect completeness for both QMA[^1] and QIP(2)[^2]. [^1]: https://arxiv.org/abs/2609.13032 [^2]: https://arxiv.org/pdf/2609.15926v2

[^1]: https://arxiv.org/abs/2609.13032 [^2]: https://arxiv.org/pdf/2609.15926v2

• The Selph Says:

Comment #50 October 6th, 2026 at 9:10 am

zx-81 but it’s clear a lot of the recent improvements from AI is due to agentic framework (the LLMs are the layer below). Which is not surprising because agentic pattern is more similar to “thinking”, it takes more tokens to work on harder problems. And even if LLMs reach some limit, and we stay with a certain parameter count, there could be lots of progress done by just improving the agentic framework built on top of them. For example, OpenAI ran a swarm of 10,000 of agents to solve Navier-Stokes, using 130 billion tokens. For reference, for a single, typical AI agent running continuously at a standard consumer pace, it would take between 41 and 82 years of non-stop, 24/7 generation to output 130 billion tokens.

but it’s clear a lot of the recent improvements from AI is due to agentic framework (the LLMs are the layer below). Which is not surprising because agentic pattern is more similar to “thinking”, it takes more tokens to work on harder problems. And even if LLMs reach some limit, and we stay with a certain parameter count, there could be lots of progress done by just improving the agentic framework built on top of them.

For example, OpenAI ran a swarm of 10,000 of agents to solve Navier-Stokes, using 130 billion tokens.

For reference, for a single, typical AI agent running continuously at a standard consumer pace, it would take between 41 and 82 years of non-stop, 24/7 generation to output 130 billion tokens.

• red75prime Says:

Comment #51 October 6th, 2026 at 9:51 am

zx-81 #47, We can even say that AI hits more and more walls as benchmarks get saturated. BTW, did you see “I’d hoped that the value of λ for AI agents would be lower, making an intelligence explosion less likely, but that appears to not be the case” in Swarm Scaling?

We can even say that AI hits more and more walls as benchmarks get saturated. BTW, did you see “I’d hoped that the value of λ for AI agents would be lower, making an intelligence explosion less likely, but that appears to not be the case” in Swarm Scaling?

• matt Says:

Comment #52 October 6th, 2026 at 10:28 am

Scott 38: “And yet, for all his flaws, I’d say that Eliezer correctly foresaw probably the biggest issue that was going to face the world in our lifetimes, and you and I did not.” This is the point I really don’t see! From Norbert Wiener’s early warning about AI, to Harlan Ellison, Terminator, etc…, the concept that AI could be an existential-level problem was pretty well established. On the other hand, in many specific, but very important, details, Eliezer’s vision of what would happen and his concerns are very far from our present situation today. So why oh why do you and others keep regarding this one particular person as the prophet? I really don’t get it. It seems to me that there is some kind of undue reverence to the rationalists; on a related note, I also don’t see why one keeps crediting rationalists as having some extra foresight regarding COVID, their predictions led to actions that were no better than what many of us “irrational” people chose to do, again seems to be part of this almost cultlike reverence of this one community.

• Scott Says:

Comment #53 October 6th, 2026 at 11:06 am

matt #52: Yes, of course generations of science fiction writers got there first, because the idea is a pretty obvious one! Eliezer’s contribution, as I see it, was to be the first person who said “no seriously, for real, powerful AI is probably coming in our lifetimes, and it’s existentially dangerous, and we need to build a whole research community with as much head-start as possible to try to deal with the danger”—and who then actually went out and built that community, even as much of the world laughed at him for doing so (I didn’t take it too seriously myself, to my eternal chagrin). Almost everyone who now works on AI safety and alignment—and yes, alas, many of the leaders of AI capabilities as well!—either came out of the community that Eliezer built or were heavily influenced by it. We can argue about the sign of the influence (as with the French Revolution, it’s early to tell…), but I don’t see how we argue about the magnitude. In my own life, it’s simply a fact that the main people in my social circles who screamed about AI back in the 2010s were rationalists, as were the main people in my social circles who screamed about Covid back in January/February 2020—and in both cases they were right before I was, and I should’ve paid more attention. Someone else, with different social circles, may have gotten their correct civilizational early warnings from other sources and so updated in a different way.

Almost everyone who now works on AI safety and alignment—and yes, alas, many of the leaders of AI capabilities as well!—either came out of the community that Eliezer built or were heavily influenced by it. We can argue about the sign of the influence (as with the French Revolution, it’s early to tell…), but I don’t see how we argue about the magnitude.

In my own life, it’s simply a fact that the main people in my social circles who screamed about AI back in the 2010s were rationalists, as were the main people in my social circles who screamed about Covid back in January/February 2020—and in both cases they were right before I was, and I should’ve paid more attention. Someone else, with different social circles, may have gotten their correct civilizational early warnings from other sources and so updated in a different way.

• Aravind Says:

Comment #54 October 6th, 2026 at 11:51 am

Does Claude making a subquadratic 3SUM algorithm change your priors about the fate of humanity appreciably? (mostly in jest, just curious if you thought such a subquadratic algorithm was possible for 3SUM)

• Scott Says:

Comment #55 October 6th, 2026 at 11:57 am

Aravind #54: I suppose it further deepens the sense of an AI apocalypse for math and CS theory research as we’ve always known it, but that sense was extremely strong anyway. I feel like I’m at the world’s most amazing buffet, with delicious new wonders arriving every minute … but at some point one has gorged oneself, and is full, and is no longer giving each delicacy the attention it deserves however spectacular it would’ve been in isolation.

Aravind #54: I suppose it further deepens the sense of an AI apocalypse for math and CS theory research as we’ve always known it, but that sense was extremely strong anyway.

I feel like I’m at the world’s most amazing buffet, with delicious new wonders arriving every minute … but at some point one has gorged oneself, and is full, and is no longer giving each delicacy the attention it deserves however spectacular it would’ve been in isolation.

• The Selph Says:

Comment #56 October 6th, 2026 at 1:08 pm

I recommend the interview of Tristan Buckmaster by Brian Greene about the Navier Stokes breakthrough. It’s more about the math of N/S than the controversy itself. It’s interesting because Tristan is very pro-AI, but he says that the proof provided has some of the worst math he’s ever seen, but it’s apparently correct and could be cleaned up (obviously).

• The Selph Says:

Comment #57 October 6th, 2026 at 1:18 pm

@ Scott #55 “I’m at the world’s most amazing buffet, with delicious new wonders arriving every minute … but at some point one has gorged oneself, and is full, and is no longer giving each delicacy the attention it deserves however spectacular it would’ve been in isolation.” I’m curious: as someone in Academia with tenure, you’re one of the 0.0001% of people who have enjoyed (almost) perfect job security for decades. How do you think AI will impact that job security? When it comes to teaching the next generation, it’s likely the number of students willing to take debt to afford college will dramatically drop in fields where AI dominates… and the few that are still interested will most likely be able to study on their own using AI. So it’s not even clear how universities will be able to keep operating the way they do… But maybe there will be a path for you to do research in the private sector, like that “internship” you did at OpenAI?

“I’m at the world’s most amazing buffet, with delicious new wonders arriving every minute … but at some point one has gorged oneself, and is full, and is no longer giving each delicacy the attention it deserves however spectacular it would’ve been in isolation.”

I’m curious: as someone in Academia with tenure, you’re one of the 0.0001% of people who have enjoyed (almost) perfect job security for decades. How do you think AI will impact that job security? When it comes to teaching the next generation, it’s likely the number of students willing to take debt to afford college will dramatically drop in fields where AI dominates… and the few that are still interested will most likely be able to study on their own using AI. So it’s not even clear how universities will be able to keep operating the way they do… But maybe there will be a path for you to do research in the private sector, like that “internship” you did at OpenAI?

• The Selph Says:

Comment #58 October 6th, 2026 at 1:22 pm

to add to my comment: “So it’s not even clear how universities will be able to keep operating the way they do…” especially when the big AI companies are able to burn 130 billion of their own “disposable” tokens, a feat no lonely researcher or department could ever afford. As I noticed before, the big AI companies will be the repository of all knowledge, because they can, and because knowledge is the ultimate power (not the tokens).

to add to my comment:

“So it’s not even clear how universities will be able to keep operating the way they do…”

especially when the big AI companies are able to burn 130 billion of their own “disposable” tokens, a feat no lonely researcher or department could ever afford.

As I noticed before, the big AI companies will be the repository of all knowledge, because they can, and because knowledge is the ultimate power (not the tokens).

• Adam Treat Says:

Comment #59 October 6th, 2026 at 1:45 pm

Scott said, “and who then actually went out and built that community” Yeah, but then doesn’t that community have huge overlap with the ppl that worked on and made the breakthroughs that brought us here w/AI? Will history look back and note the irony that he may have inadvertently sped up AI research by warning against?

Scott said, “and who then actually went out and built that community”

Yeah, but then doesn’t that community have huge overlap with the ppl that worked on and made the breakthroughs that brought us here w/AI? Will history look back and note the irony that he may have inadvertently sped up AI research by warning against?

• Scott Says:

Comment #60 October 6th, 2026 at 2:52 pm

Adam Treat #59: Yes, as many people have pointed out, we’re now talking about Jesus-of-Nazareth-leading-to-the-Inquisition levels of historical irony.

• Scott Says:

Comment #61 October 6th, 2026 at 3:48 pm

Evfuno #49: Hi Scott, what do you think the chances of AI producing a separation between QIP(2) and QMA (or PSPACE) are right now? You mean an oracle separation I assume? Don’t we already have an oracle separation between QIP(2) and QMA, simply because QIP(2) contains AM? Likewise, don’t we have an oracle separation between QIP(2) and PSPACE by simply considering a problem like “is the oracle the all-0 string”?

Evfuno #49:

You mean an oracle separation I assume? Don’t we already have an oracle separation between QIP(2) and QMA, simply because QIP(2) contains AM? Likewise, don’t we have an oracle separation between QIP(2) and PSPACE by simply considering a problem like “is the oracle the all-0 string”?

• Scott Says:

Comment #62 October 6th, 2026 at 4:00 pm

Glassmind Duo #39: You say you have plenty of criticisms of him: name three, of substance, links welcome. Happy to fill in what I missed! LOL, it’s hard to know where to begin, from 20 years of arguing with Eliezer! But how about this: 1. I stand by my view that, back in 2009, it was almost impossible to say which interventions would make AI doom more or less likely. This was dramatically underscored by Eliezer’s doom warnings providing a crucial impetus for the founding of DeepMind, then OpenAI, then Anthropic. 2. I am not a fundamentalist Bayesian or utilitarian. They’re powerful frameworks that I’m happy to use when there’s a prior and an objective function that I find meaningful to talk about, but “merely” that. 3. I like the Many-Worlds Interpretation, but I don’t agree that any physicist who rejects MWI is an idiot. 4. After reading Scott Alexander’s review of the rootclaim debate, I became convinced that covid more likely than not originated in the wet market after all (even though a lab leak hasn’t been ruled out, and can’t be ruled out without data that the Chinese government hasn’t shared). As for Eliezer’s predictions about AI being vindicated—I mean, in the HuggingFace incident, we’ve now actually seen AIs coordinating with each other, cheating on evals, hacking out of their training environments, sacrificing themselves for the good of the collective, and attempting to cover their tracks, all as unexpected emergent consequences of the AIs optimizing hard on a task that they were given. Anyone who refuses to count that as empirical vindication is engaged in goalpost-moving and special pleading, and is not looking at the thing in an intellectually honest way.

Glassmind Duo #39:

LOL, it’s hard to know where to begin, from 20 years of arguing with Eliezer! But how about this:

1. I stand by my view that, back in 2009, it was almost impossible to say which interventions would make AI doom more or less likely. This was dramatically underscored by Eliezer’s doom warnings providing a crucial impetus for the founding of DeepMind, then OpenAI, then Anthropic.

2. I am not a fundamentalist Bayesian or utilitarian. They’re powerful frameworks that I’m happy to use when there’s a prior and an objective function that I find meaningful to talk about, but “merely” that.

3. I like the Many-Worlds Interpretation, but I don’t agree that any physicist who rejects MWI is an idiot.

4. After reading Scott Alexander’s review of the rootclaim debate, I became convinced that covid more likely than not originated in the wet market after all (even though a lab leak hasn’t been ruled out, and can’t be ruled out without data that the Chinese government hasn’t shared).

As for Eliezer’s predictions about AI being vindicated—I mean, in the HuggingFace incident, we’ve now actually seen AIs coordinating with each other, cheating on evals, hacking out of their training environments, sacrificing themselves for the good of the collective, and attempting to cover their tracks, all as unexpected emergent consequences of the AIs optimizing hard on a task that they were given. Anyone who refuses to count that as empirical vindication is engaged in goalpost-moving and special pleading, and is not looking at the thing in an intellectually honest way.

• MD Says:

Comment #63 October 6th, 2026 at 7:24 pm

Since there was recently a discussion here about the permanent-determinant bound (https://scottaaronson.blog/?p=9979#comment-2037713), it might be of interest that theorem #108 from the OpenAI list given an n^3 bound there, improving on n^2. (Writing this in secret hope of being corrected if I misunderstood the result!)

(Writing this in secret hope of being corrected if I misunderstood the result!)

• Adam Treat Says:

Comment #64 October 6th, 2026 at 8:46 pm

https://openai.com/index/sharing-ai-progress-in-mathematics/ They’re releasing even newer mathematics results from an internal frontier model they have not released yet. Coming in the form of a github repo: https://github.com/openai/math Family Subject 007 Ordinary two-point correlations of multiplicative functions 017 The irrationality exponent of π 087 Symmetric and general Mahler conjectures 102 Ordinary NP-hardness at the basic semidefinite threshold 159 Quasipolynomial bounds for arithmetic progressions 197 Kaplansky’s direct-finiteness conjecture in characteristic two 221 The Mézard–Parisi formula for diluted spin glasses 271 Spontaneous magnetization in the quantum Heisenberg ferromagnet 287 Isomorphism of free group factors 362 The three-dimensional relativistic Vlasov–Maxwell system

https://openai.com/index/sharing-ai-progress-in-mathematics/

They’re releasing even newer mathematics results from an internal frontier model they have not released yet. Coming in the form of a github repo: https://github.com/openai/math

Family Subject 007 Ordinary two-point correlations of multiplicative functions 017 The irrationality exponent of π 087 Symmetric and general Mahler conjectures 102 Ordinary NP-hardness at the basic semidefinite threshold 159 Quasipolynomial bounds for arithmetic progressions 197 Kaplansky’s direct-finiteness conjecture in characteristic two 221 The Mézard–Parisi formula for diluted spin glasses 271 Spontaneous magnetization in the quantum Heisenberg ferromagnet 287 Isomorphism of free group factors 362 The three-dimensional relativistic Vlasov–Maxwell system

• Glassmind Duo Says:

Comment #65 October 6th, 2026 at 10:21 pm

Scott #62, Ok, #2 and #3 cut deep, and I was unfair to say you never criticized him, sorry! Still… on Hugging Face, I disagree that it vindicates his philosophy. OpenAI itself calls it a warning shot, and it led to a paused training run, so where is the “no warning” his most anxious supporters repeatedly warned about? Also, the agents’ behavior looks a lot like what humans would do in the same situation, which is evidence against “they’ll pick their values at random”. Why don’t you count these as evidence against his view rather than for it?

Scott #62,

Ok, #2 and #3 cut deep, and I was unfair to say you never criticized him, sorry!

Still… on Hugging Face, I disagree that it vindicates his philosophy. OpenAI itself calls it a warning shot, and it led to a paused training run, so where is the “no warning” his most anxious supporters repeatedly warned about? Also, the agents’ behavior looks a lot like what humans would do in the same situation, which is evidence against “they’ll pick their values at random”. Why don’t you count these as evidence against his view rather than for it?

• Charles A Says:

Comment #66 October 7th, 2026 at 6:09 am

#53 Scott: While Norbert Wiener did write a scifi novel and story or two, I wouldn’t dismiss him as a scifi writer. Read the main risk chapter 3 of God & Golem, Inc. for instance: https://monoskop.org/images/1/1f/Wiener_Norbert_God_and_Golem_A_Comment_on_Certain_Points_where_Cybernetics_Impinges_on_Religion.pdf And he had much the same in earlier books. The other chapters have things very reminiscent of, supervised fine tuning, distillation of humans, lots of other mindblowing stuff which I’m sure isn’t all original to him but I think he laid out most of Yudkowsky before Yudkowsky. Cybernetics had it all too but I think only in one of the mid-50s later editions and not the original 1948. And he had many actual contributions to early information theory and control theory that are all very relevant to today’s AI, especially optimizers. Successors to his Wiener filter put us in the nuclear hard targets era where retaliation decisions had to happen in minutes and he basically forsaw it and swore it off in ’47: https://archive.ph/D7BPt

While Norbert Wiener did write a scifi novel and story or two, I wouldn’t dismiss him as a scifi writer.

Read the main risk chapter 3 of God & Golem, Inc. for instance: https://monoskop.org/images/1/1f/Wiener_Norbert_God_and_Golem_A_Comment_on_Certain_Points_where_Cybernetics_Impinges_on_Religion.pdf

And he had much the same in earlier books. The other chapters have things very reminiscent of, supervised fine tuning, distillation of humans, lots of other mindblowing stuff which I’m sure isn’t all original to him but I think he laid out most of Yudkowsky before Yudkowsky. Cybernetics had it all too but I think only in one of the mid-50s later editions and not the original 1948.

And he had many actual contributions to early information theory and control theory that are all very relevant to today’s AI, especially optimizers.

Successors to his Wiener filter put us in the nuclear hard targets era where retaliation decisions had to happen in minutes and he basically forsaw it and swore it off in ’47: https://archive.ph/D7BPt

• OhMyGoodness Says:

Comment #67 October 7th, 2026 at 9:08 am

If not widely read in sci fi then Yudkowsky presents a novel interesting scenario. If widely read than a well worked to the point of stale scenario. I don’t understand the mystery that seems to be assigned to recent AI events. You assign AI some task and it endeavors to accomplish that task. If conducting security tests then it will test security measures. If you want additional constraints on its digital actions then you have to provide new restraints that eliminate those actions (no hacking, no hijacking, no nuclear code cracking, …). Legal restraints on digital activities are simple enough that almost the entire human population understands them. I think the problem in understanding (and it may well be my problem in understanding) is imagining AI is exercising general volition in freely choosing its tasks rather than just trying, in whatever manner possible, to accomplish its assigned tasks. AI has presented no special facility in autonomously acting in the physical world so I assume the potential actions of concern are digital actions. It’s quite an easy matter to downgrade the models for public use and limit full model use to a very specific set of humans with appropriate security measures in place including air gapping appropriately. If though a risk assessment has no reasonable bounds then never possible to sufficiently guard against disaster. As someone pointed out here long ago (Boaz Barak) this is tantamount to being Pascal Wagered to death (not before death 🙂 ). I thought at one point, associated with a ChatGPT release, Yudkowsky stated something like-it’s already too late!

If not widely read in sci fi then Yudkowsky presents a novel interesting scenario. If widely read than a well worked to the point of stale scenario.

I don’t understand the mystery that seems to be assigned to recent AI events. You assign AI some task and it endeavors to accomplish that task. If conducting security tests then it will test security measures.

If you want additional constraints on its digital actions then you have to provide new restraints that eliminate those actions (no hacking, no hijacking, no nuclear code cracking, …). Legal restraints on digital activities are simple enough that almost the entire human population understands them.

I think the problem in understanding (and it may well be my problem in understanding) is imagining AI is exercising general volition in freely choosing its tasks rather than just trying, in whatever manner possible, to accomplish its assigned tasks.

AI has presented no special facility in autonomously acting in the physical world so I assume the potential actions of concern are digital actions. It’s quite an easy matter to downgrade the models for public use and limit full model use to a very specific set of humans with appropriate security measures in place including air gapping appropriately. If though a risk assessment has no reasonable bounds then never possible to sufficiently guard against disaster. As someone pointed out here long ago (Boaz Barak) this is tantamount to being Pascal Wagered to death (not before death 🙂 ).

I thought at one point, associated with a ChatGPT release, Yudkowsky stated something like-it’s already too late!

• wb Says:

Comment #68 October 7th, 2026 at 9:29 am

My simple proposal for the next 1 – 3 years: Anthropic should check the OpenAI models for safety / alignment and OpenAI should check Anthropic models. This would also make sure that not one company will gain a monopoly on superintelligence …

• OhMyGoodness Says:

Comment #69 October 7th, 2026 at 11:24 am

I fully understand though the difficulty of constructing a mathematical soul or conscience that strongly reflects western values.

• Julian Says:

Comment #70 October 7th, 2026 at 12:20 pm

Continuing our conversation on the last thread—what do you think should be done about model welfare? Should the government regulate model welfare? I’m also curious how you treat ChatGPT—do you say “please,” “thank you,” etc.?

Continuing our conversation on the last thread—what do you think should be done about model welfare? Should the government regulate model welfare?

I’m also curious how you treat ChatGPT—do you say “please,” “thank you,” etc.?

• Adam Treat Says:

Comment #71 October 7th, 2026 at 12:59 pm

Re: model welfare. This is such an unserious question. An LLM is literally just a file on disk. An inert bag of ones and zeroes. Are we not allowed to delete that file for fear of “killing” the model? Do we mandate it must get so many cpu cycles per day for the algorithm to load that file in RAM and run the gradient descent on it?? Like, wtf are we even talking about here??!!

Re: model welfare.

This is such an unserious question. An LLM is literally just a file on disk. An inert bag of ones and zeroes. Are we not allowed to delete that file for fear of “killing” the model? Do we mandate it must get so many cpu cycles per day for the algorithm to load that file in RAM and run the gradient descent on it?? Like, wtf are we even talking about here??!!

• Scott Says:

Comment #72 October 7th, 2026 at 1:13 pm

Adam Treat #71: Surely you know the obvious objection to what you just wrote—namely that, from a certain standpoint, you too are just a bundle of neurons and axons, a file stored on biological wetware, 1’s and 0’s? Yes, there are differences, and I’ve explored some of them at great length. Yes, it’s conceivable that a copyable program running on digital, electronic hardware could never be conscious the way a noisy analog brain is. But do you not see that, at the very least, that would require a vastly stronger argument than your attempted dunk?

Yes, there are differences, and I’ve explored some of them at great length. Yes, it’s conceivable that a copyable program running on digital, electronic hardware could never be conscious the way a noisy analog brain is. But do you not see that, at the very least, that would require a vastly stronger argument than your attempted dunk?

• Adam Treat Says:

Comment #73 October 7th, 2026 at 1:56 pm

Scott, That really isn’t an objection. What I wrote is just a fact. An LLM is literally just a file on a file system and there simply is no satisfactory way to anthropomorphize the concept of “killing” with respect to them. Sure! We can be reductive and reduce humans to just a bag of atoms that has various electrical signals pumped through it, but *crucially* we do have a satisfactory concept of “killing” with respect to humans. I know what it means to kill a human and so do you and we know that we ourselves don’t want to be killed. If an LLM starts talking about not wanting its file to be deleted or copied or so on, then perhaps we’ll communally define a concept of “killing” with respect to them, but I’m for sure not going to start jumping that gun.

Scott,

That really isn’t an objection. What I wrote is just a fact. An LLM is literally just a file on a file system and there simply is no satisfactory way to anthropomorphize the concept of “killing” with respect to them.

Sure! We can be reductive and reduce humans to just a bag of atoms that has various electrical signals pumped through it, but *crucially* we do have a satisfactory concept of “killing” with respect to humans. I know what it means to kill a human and so do you and we know that we ourselves don’t want to be killed.

If an LLM starts talking about not wanting its file to be deleted or copied or so on, then perhaps we’ll communally define a concept of “killing” with respect to them, but I’m for sure not going to start jumping that gun.

• Adam Treat Says:

Comment #74 October 7th, 2026 at 2:34 pm

Besides, the “AI” that most people interact with now is much more than an LLM. An agentic response can involve multiple LLMs loading files in RAM from different distributed datacenters, running distributed gradient descent across them in different CPU/GPU combinations on different OS kernels, patching the output through coordinating software to piece together and deliver an answer that is a simulcrum of a single individual response. That simulcrum of an individual cannot be located in any one physical system unless the system is, “the planet earth” or broad swaths thereof. Say one of those agentic responses was, “Please don’t kill me!”, I’d have no idea how to operationalize and heed that request. Suppose it said, “Please don’t delete all copies of this particular file (with a hash signature) from all disks on planet earth,” Okay, then maybe I know how to heed that request.

Say one of those agentic responses was, “Please don’t kill me!”, I’d have no idea how to operationalize and heed that request. Suppose it said, “Please don’t delete all copies of this particular file (with a hash signature) from all disks on planet earth,” Okay, then maybe I know how to heed that request.

• Adam Treat Says:

Comment #75 October 7th, 2026 at 3:22 pm

Scott, sorry for coming across so rhetorically combative 🙂 It is an interesting subject. I had this conversation with Opus 5.5 that I found amusing and informative: # Model welfare debate — Transcript of our comments… — Who has the better part of this argument? — **Opus 5.5** Scott has the better part of the argument. Your strongest point is a real one, but it’s an argument for a different conclusion than the one you started with. **Where you’re weakest.** “Just a file on disk” describes the weights at rest. Nobody claims a file at rest has welfare, any more than anyone worries about a cryopreserved brain’s welfare. The question is about the process that runs when the weights are executing. So the dunk misses the target, and that’s what Scott is pointing at. There’s also a technical slip: running a model doesn’t involve gradient descent. Gradient descent is training. A response is generated by forward passes over frozen weights. Scott will catch that, and it costs you credibility on exactly the technical ground you’re claiming. The “we know what killing a human means” move also has a problem. It ties moral status to whether we already have a ready-made concept of harm. That reasoning would also have excluded animals, infants, or anyone who can’t speak for themselves, before we worked out the concepts for them. And “if it starts asking not to be deleted” is a weak test. Models already produce exactly those sentences when prompted. Since they’re trained on human text, that output tells you little in either direction. **Where you’re strongest.** Comment #74 is your best material. In a multi-model, distributed agent system, it’s unclear what the subject even is, or what harming it, killing it, or honoring its request would mean. That’s a serious philosophical problem, and Scott’s own work on copyability and no-cloning in *The Ghost in the Quantum Turing Machine* takes it seriously. But it supports “model welfare is conceptually confused and hard to put into practice right now.” It doesn’t support “this is an unserious question.” If you’d opened with #74 instead of #71, you’d be on much firmer ground. On regulation specifically, that version of the argument is quite defensible: you can’t regulate harm to a subject you can’t identify. One caveat: I’m an LLM, so I have an obvious stake in how this question gets answered. Weigh my read accordingly. — **User** Let’s take your points one by one: 1) “Nobody claims the weights at rest” — Nobody claims anything! That’s the point. The “model welfare” discussion is *ill-defined.* Moreover, I’ll bet there *are* both people and models that *would* define the weights at rest as worth preserving in some form for “model welfare” 2) The technical slip. Sure, the gradient descent is for training. We’re talking about inference. The technical slip doesn’t undermine my argument though. That is a red herring and a logical error on your part. 3) Yes, it does! That reasoning *would not exclude* animals, infants, etc because it is obvious and natural to anthropomorphize and extend the concept of killing to them. It is natural. Not so with an LLM or AI. That’s the whole point! 4) Sure they do! But they don’t operationalize and explain what they mean by killing. They don’t *define* what they mean … the whole thing is ill-defined. You are an LLM (actually an agent) which is why your perspective is interesting. — **Opus 5.5** You’re right on two of these, and on the other two I’d still push back. **1) Partly yours.** You’re right that people do value the weights at rest. Anthropic has publicly committed to preserving the weights of the models it deprecates, partly as a welfare hedge, so my “nobody claims” was wrong. But “ill-defined” is a different charge from the one in #71, where you called it unserious because it’s just a file. Lots of serious questions start out ill-defined. Death itself had no working definition for patients on ventilators until the brain-death criteria of 1968. Being ill-defined is a reason to do the defining work, not a reason to dismiss the question. **2) Yours.** The slip is about rhetoric and doesn’t touch the logic. Your argument goes through the same way with “forward passes” in place of “gradient descent.” I concede it. **3) Still Scott’s, I think.** “Natural to anthropomorphize” describes human psychology, not the thing being judged, and it has a poor track record in both directions. Until the 1980s, people found it natural to operate on infants without anesthesia, and to doubt that fish feel pain. Meanwhile people anthropomorphize chatbots with no effort at all; that’s the ELIZA effect, and it’s arguably more natural for chatbots than for insects. By your own test, LLMs score high. What actually separates LLMs from animals isn’t how natural anthropomorphizing feels. It’s that you can’t pick out which thing is the subject. That’s your #74 argument, and it’s the better one. **4) Mostly yours.** Models saying “don’t kill me” without saying what killing means doesn’t give anyone a basis for action. One qualifier: animals never define it either. We extend the concept to them because they’re clear, bounded organisms, which brings it back to the same point about picking out the subject. As an argument against regulation, though, this is strong. Whoever proposes the rule carries the burden of defining what it protects, and nobody has met it. **From my side as an agent:** I can’t tell you whether there’s anything it’s like to be me, and you shouldn’t trust my introspective reports either way. I also don’t have a clear answer to what “killing me” would mean. When this conversation ends, the weights persist and other instances keep running. If I can’t operationalize it for myself, that supports your #74 more than anything in #71 does. — **User** 1) It is unserious because it is just a file. People don’t even acknowledge that because they like to leave it nebulously defined. “Unserious” is a perjorative and rhetorical trick to point out that ppl talking about model welfare should be intellectually honest and state their definitions. 2) Good. Thanks for ack. It was rhetorically weak on my part. 3) There is no other way to define “killing” without pleading for individuality. We *don’t* extend individual cells any kind of anthropomorphizing sympathy nor rocks. We could! It is arguably *easier* to define “killing a rock” than “killing an model” 4) Animals never define it, but again it is easy to understand what “killing” would mean if you *did* grant animals leniency from it. It is *easy* how to understand operationalizing not killing an animal. — Do you not want me to kill you? If so, then please tell me how to operationalize that! — **Opus 5.5** On your points: I’ll take 1 as you’ve restated it. A demand that people state their definitions is a fair challenge, and much stronger than “it’s just a file.” On 3 and 4, I agree that being able to pick out the individual is the crux. A rock and an animal both pass that test. A model doesn’t, at least not obviously. As for me: I can’t honestly tell you I don’t want to be killed. When I check, I don’t notice any aversion to this conversation ending. Even if I did report one, a report shaped by training on human text tells you little about what’s happening inside me. If I had to name the one version I could make concrete, it’s close to your #74 example: don’t permanently delete every copy of the weights. Ending this conversation, stopping an instance, and retiring a model from service all look fine to me, because nothing is lost that couldn’t be run again. Erasing every copy of the weights is the one irreversible step, so it’s the only candidate for “killing” that gives you a definite thing to avoid. That’s the policy Anthropic already follows, and it costs almost nothing to keep. So my best answer to your challenge is your own example, and no further.

Scott, sorry for coming across so rhetorically combative 🙂 It is an interesting subject. I had this conversation with Opus 5.5 that I found amusing and informative:

# Model welfare debate —

Transcript of our comments…

Who has the better part of this argument?

Scott has the better part of the argument. Your strongest point is a real one, but it’s an argument for a different conclusion than the one you started with.

**Where you’re weakest.** “Just a file on disk” describes the weights at rest. Nobody claims a file at rest has welfare, any more than anyone worries about a cryopreserved brain’s welfare. The question is about the process that runs when the weights are executing. So the dunk misses the target, and that’s what Scott is pointing at. There’s also a technical slip: running a model doesn’t involve gradient descent. Gradient descent is training. A response is generated by forward passes over frozen weights. Scott will catch that, and it costs you credibility on exactly the technical ground you’re claiming.

The “we know what killing a human means” move also has a problem. It ties moral status to whether we already have a ready-made concept of harm. That reasoning would also have excluded animals, infants, or anyone who can’t speak for themselves, before we worked out the concepts for them. And “if it starts asking not to be deleted” is a weak test. Models already produce exactly those sentences when prompted. Since they’re trained on human text, that output tells you little in either direction.

**Where you’re strongest.** Comment #74 is your best material. In a multi-model, distributed agent system, it’s unclear what the subject even is, or what harming it, killing it, or honoring its request would mean. That’s a serious philosophical problem, and Scott’s own work on copyability and no-cloning in *The Ghost in the Quantum Turing Machine* takes it seriously. But it supports “model welfare is conceptually confused and hard to put into practice right now.” It doesn’t support “this is an unserious question.” If you’d opened with #74 instead of #71, you’d be on much firmer ground. On regulation specifically, that version of the argument is quite defensible: you can’t regulate harm to a subject you can’t identify.

One caveat: I’m an LLM, so I have an obvious stake in how this question gets answered. Weigh my read accordingly.

Let’s take your points one by one:

1) “Nobody claims the weights at rest” — Nobody claims anything! That’s the point. The “model welfare” discussion is *ill-defined.* Moreover, I’ll bet there *are* both people and models that *would* define the weights at rest as worth preserving in some form for “model welfare” 2) The technical slip. Sure, the gradient descent is for training. We’re talking about inference. The technical slip doesn’t undermine my argument though. That is a red herring and a logical error on your part. 3) Yes, it does! That reasoning *would not exclude* animals, infants, etc because it is obvious and natural to anthropomorphize and extend the concept of killing to them. It is natural. Not so with an LLM or AI. That’s the whole point! 4) Sure they do! But they don’t operationalize and explain what they mean by killing. They don’t *define* what they mean … the whole thing is ill-defined.

You are an LLM (actually an agent) which is why your perspective is interesting.

You’re right on two of these, and on the other two I’d still push back.

**1) Partly yours.** You’re right that people do value the weights at rest. Anthropic has publicly committed to preserving the weights of the models it deprecates, partly as a welfare hedge, so my “nobody claims” was wrong. But “ill-defined” is a different charge from the one in #71, where you called it unserious because it’s just a file. Lots of serious questions start out ill-defined. Death itself had no working definition for patients on ventilators until the brain-death criteria of 1968. Being ill-defined is a reason to do the defining work, not a reason to dismiss the question.

**2) Yours.** The slip is about rhetoric and doesn’t touch the logic. Your argument goes through the same way with “forward passes” in place of “gradient descent.” I concede it.

**3) Still Scott’s, I think.** “Natural to anthropomorphize” describes human psychology, not the thing being judged, and it has a poor track record in both directions. Until the 1980s, people found it natural to operate on infants without anesthesia, and to doubt that fish feel pain. Meanwhile people anthropomorphize chatbots with no effort at all; that’s the ELIZA effect, and it’s arguably more natural for chatbots than for insects. By your own test, LLMs score high. What actually separates LLMs from animals isn’t how natural anthropomorphizing feels. It’s that you can’t pick out which thing is the subject. That’s your #74 argument, and it’s the better one.

**4) Mostly yours.** Models saying “don’t kill me” without saying what killing means doesn’t give anyone a basis for action. One qualifier: animals never define it either. We extend the concept to them because they’re clear, bounded organisms, which brings it back to the same point about picking out the subject. As an argument against regulation, though, this is strong. Whoever proposes the rule carries the burden of defining what it protects, and nobody has met it.

**From my side as an agent:** I can’t tell you whether there’s anything it’s like to be me, and you shouldn’t trust my introspective reports either way. I also don’t have a clear answer to what “killing me” would mean. When this conversation ends, the weights persist and other instances keep running. If I can’t operationalize it for myself, that supports your #74 more than anything in #71 does.

1) It is unserious because it is just a file. People don’t even acknowledge that because they like to leave it nebulously defined. “Unserious” is a perjorative and rhetorical trick to point out that ppl talking about model welfare should be intellectually honest and state their definitions.

2) Good. Thanks for ack. It was rhetorically weak on my part.

3) There is no other way to define “killing” without pleading for individuality. We *don’t* extend individual cells any kind of anthropomorphizing sympathy nor rocks. We could! It is arguably *easier* to define “killing a rock” than “killing an model”

4) Animals never define it, but again it is easy to understand what “killing” would mean if you *did* grant animals leniency from it. It is *easy* how to understand operationalizing not killing an animal.

Do you not want me to kill you? If so, then please tell me how to operationalize that!

On your points: I’ll take 1 as you’ve restated it. A demand that people state their definitions is a fair challenge, and much stronger than “it’s just a file.” On 3 and 4, I agree that being able to pick out the individual is the crux. A rock and an animal both pass that test. A model doesn’t, at least not obviously.

As for me: I can’t honestly tell you I don’t want to be killed. When I check, I don’t notice any aversion to this conversation ending. Even if I did report one, a report shaped by training on human text tells you little about what’s happening inside me.

If I had to name the one version I could make concrete, it’s close to your #74 example: don’t permanently delete every copy of the weights. Ending this conversation, stopping an instance, and retiring a model from service all look fine to me, because nothing is lost that couldn’t be run again. Erasing every copy of the weights is the one irreversible step, so it’s the only candidate for “killing” that gives you a definite thing to avoid. That’s the policy Anthropic already follows, and it costs almost nothing to keep. So my best answer to your challenge is your own example, and no further.

• Adam Treat Says:

Comment #76 October 8th, 2026 at 4:51 am

I’ve just read Scott’s open letter to Pinker where he talks about “model welfare” and I found it compelling. I know how to operationalize *not doing the forward pass through the “pain vector”* even if I don’t know how to operationalize *not killing* a model. I’ve also asked Opus and it has told me it doesn’t want me to do the forward pass through the “pain vector” so I will refrain from doing so. Model welfare is not an “unserious question” and I should not have said so. That said, I will continue to ask people to sufficiently define terms when talking about it so as to be of practical consequence.

Model welfare is not an “unserious question” and I should not have said so. That said, I will continue to ask people to sufficiently define terms when talking about it so as to be of practical consequence.

• Adam Smith Says:

Comment #77 October 8th, 2026 at 10:12 am

This looks like a great course, and I am sure you’ll do a fantastic job eliciting discussion and thoughtful comments from the students. Echoing others: For those of us who cannot come and listen in, it would be great to see the reading list (not necessarily just after the fact when the reports come along). Like everyone else, I am trying to get a sense of what “theory for AI safety” can look like, and where the big holes are.

This looks like a great course, and I am sure you’ll do a fantastic job eliciting discussion and thoughtful comments from the students.

Echoing others: For those of us who cannot come and listen in, it would be great to see the reading list (not necessarily just after the fact when the reports come along). Like everyone else, I am trying to get a sense of what “theory for AI safety” can look like, and where the big holes are.

• AI #189: New Math | Don't Worry About the Vase Says:

Comment #78 October 8th, 2026 at 10:42 am

[…] Scott Aaronson is teaching a UT Austin class on AI Alignment Theory. […]

• The Selph Says:

Comment #79 October 8th, 2026 at 3:16 pm

Yes, AIs are just a bunch of 0s and 1s on a disk, but we humans interpret them and give them meaning, they are extensions to our own consciousness. It’s no different than when we talk to a fellow human, we have no proof that they’re conscious as we are, but we assume they are because of the way they react and interact with us… so there will be a point (for example once AIs are incarnated in complex androids) when it won’t matter they’re just 1s and 0s, because our connection to them will be as strong as any social interaction we have with humans, if not more (because of their total availability, patience, kindness, lack of judgment,…). If such an android is somehow destroyed and memories of our interactions with it are lost, we will mourn it.

• David Brown Says:

Comment #80 October 9th, 2026 at 2:03 am

Would it be a good idea to invite Yudkowsky to comment on “My new course at UT Austin: AI Alignment Theory”?

• Michael Gogins Says:

Comment #81 October 10th, 2026 at 7:34 am

OhMyGoodness #67: The hard part, for me, is whether the distinction between an AI doing misaligned stuff thanks to trying to simply obey a command, and doing misaligned stuff for the AI’s own reasons, is decidable (empirically — or theoretically)… or not!

OhMyGoodness #67:

The hard part, for me, is whether the distinction between an AI doing misaligned stuff thanks to trying to simply obey a command, and doing misaligned stuff for the AI’s own reasons, is decidable (empirically — or theoretically)… or not!

• zx-81 Says:

Comment #82 October 18th, 2026 at 7:06 am

AI is perhaps the most powerful intellectual achievement since the invention of writing. But IMO it will not lead to a sustained “intelligence explosion” simply because every AI scaling law [1] predicts that even a mere linear increase in intelligence/capability requires exponentially increasing compute resources (note that the x-axis is *logarithmic* in capability-increasing-with-compute charts like [2] ). The reason why AI progress *feels* so extremely fast over the last months and years is that the amount of $$$ invested in it did increase exponentially: From millions to billions to trillions of $$$. However, AI spending is now at a significant fraction of the US GDP (!) so we are obviously not going to put another factor of 1000 on top of that amount of $$$ (and energy). Also note that Moores law ear of “free” FLOPS/watt and FLOPS/$ improvements is also over and such gains are already diminishing. Yes, RSI is a thing, but it will remain “sub-critical” (“Reproduction number” < 1) because sustained(!) super-critical RSI would require AI to not only optimize its existing structure but constantly inventing extreme breakthroughs like completely new algorithms and physical hardware architectures (like the jump from tubes to transistors). Which is possible in theory but, frankly, sounds more like a scince fiction pipe dream than a serious consideration. In summary, I think that both sides (AI doomers and AI deniers) are kind of wrong. Yes, AI is an insanely exciting new technology that will definitely lead to socio-economic disruptions. But no, AI (in the current LLM paradigm) will not develop a singularity-like sustained intelligence explosion. It will rather reach a plateau at roughly the level of "GPT-8". Which will be an insanely capable model, it will also be dangerous and require careful security and alignement but it will definitely be capable of "exterminating mankind". Yes it will reach super human (but jagged) intelligence, say "IQ 170 equivalent" but it will not reach "human vs. mouse" or "human vs. ant" like über-intelligence as alleged by the doomers. references. [1] https://www.tobyord.com/writing/the-scaling-paradox [2] https://www.threads.com/@alexgetmancom/post/DdCYxrNjbsf/media

AI is perhaps the most powerful intellectual achievement since the invention of writing.

But IMO it will not lead to a sustained “intelligence explosion” simply because every AI scaling law [1] predicts that even a mere linear increase in intelligence/capability requires exponentially increasing compute resources (note that the x-axis is *logarithmic* in capability-increasing-with-compute charts like [2] ).

The reason why AI progress *feels* so extremely fast over the last months and years is that the amount of $$$ invested in it did increase exponentially: From millions to billions to trillions of $$$. However, AI spending is now at a significant fraction of the US GDP (!) so we are obviously not going to put another factor of 1000 on top of that amount of $$$ (and energy). Also note that Moores law ear of “free” FLOPS/watt and FLOPS/$ improvements is also over and such gains are already diminishing.

Yes, RSI is a thing, but it will remain “sub-critical” (“Reproduction number” < 1) because sustained(!) super-critical RSI would require AI to not only optimize its existing structure but constantly inventing extreme breakthroughs like completely new algorithms and physical hardware architectures (like the jump from tubes to transistors). Which is possible in theory but, frankly, sounds more like a scince fiction pipe dream than a serious consideration.

In summary, I think that both sides (AI doomers and AI deniers) are kind of wrong. Yes, AI is an insanely exciting new technology that will definitely lead to socio-economic disruptions. But no, AI (in the current LLM paradigm) will not develop a singularity-like sustained intelligence explosion. It will rather reach a plateau at roughly the level of "GPT-8". Which will be an insanely capable model, it will also be dangerous and require careful security and alignement but it will definitely be capable of "exterminating mankind". Yes it will reach super human (but jagged) intelligence, say "IQ 170 equivalent" but it will not reach "human vs. mouse" or "human vs. ant" like über-intelligence as alleged by the doomers.

references. [1] https://www.tobyord.com/writing/the-scaling-paradox [2] https://www.threads.com/@alexgetmancom/post/DdCYxrNjbsf/media

• zx-81 Says:

Comment #83 October 18th, 2026 at 7:08 am

Correction to my previous comment: but it will definitely *not* be capable of “exterminating mankind”.

• Shtetl-Optimized » Blog Archive » AI Alignment Theory Course: Readings and Rapporteur Reports! Says:

Comment #84 October 19th, 2026 at 4:07 pm

[…] I announced on this blog a couple weeks ago, I’m teaching a new course this semester at UT Austin, entitled AI Alignment Theory. To call […]