Summary of why I don't buy diachronic Dutch book arguments

(Using Elga's Dutch book against imprecise credences as an example, where the agent faces a sequence of two bets A and B.)

• Consider the action C: "Commit to accept B if you first reject A." By "commit", I mean literally lock your future self out of the "reject B" action (if you reject A), or make the "reject B" option so costly that it would never be locally rational to take it. This is stronger than Elga's "planning".
• If the agent is capable of C:
  • Then, after choosing to do C and whether to accept A, there is no "choice" left to be made. The commitment just determines what will happen.
  • And doing C plus either accepting or rejecting A is permissible, according to any plausible choice rule applied to imprecise credences. → No sure loss.
• If they aren't capable of C:
  • Then I don't see in what sense it's irrational for the agent to have imprecise credences + locally choose whether to accept each bet based on their credences at the time of each choice.
  • Why? Because:
    • By hypothesis, they can't force their future self to deviate from what they'd consider locally rational.
    • But that's exactly what Elga's argument tells the agent to do! ^[1] It says:
      • "Let's grant that you and your future self might consider imprecise credences to be locally rational. ^[2]
      • "If so, your future self (choosing whether to accept B) might choose such that overall you've suffered a sure loss.
      • "You should adopt sharp credences — i.e., make your future self's beliefs deviate from what they'd consider locally rational. That's a strictly better alternative that you're capable of."
    • So in order for us to say the agent's sequence of choices is irrational, we need them to be capable of a "commitment" that seems just as strong as C. ^[3]

(See also the links in the table here re: money pump arguments for completeness, which have a very similar structure.)

1. H/t Jesse Clifton for making this salient to me; not sure if he'd endorse this version of the counterargument though. ↩︎

2. Indeed, Elga gives some great intuition pumps for this in the intro of his paper! ↩︎

3. You might say, adopting a different standard of rationality (as Elga asks the impreciser to do) is more psychologically tractable than C. But one way to achieve C is to adopt the principle of resolute choice. ↩︎

You attend an "AI Governance" conference.

An AI safety policy speaker, from an organisation that does policy research (likely someone adjacent to the EA space), talks about:

• Stronger regulation for AI beyond human capability
• Loss of control
• Bans, treaties, and international cooperation

Then, the industry panelist speaks. Someone affiliated to something called the "International Association of Privacy Professionals" (or IAPP), who you never heard of before but apparently have a lot of pull.

They talk about:

• High-Risk AI system classification under the EU AI Act (employment, banking) and biometric data processing
• The importance of an AI use case inventory
• Trackers.
• Which AI agents get a risk assessment, and which don't
• Algorithmic bias and hallucinations (and the novel thing called "RAG" that helps legal professionals avoid them...)
• Harvey, Legora, Co-counsel and the best tools to start using AI at work.

You're confused: it's like each speaker is attending a totally different conference.

For one side, AI risk in its most dangerous form is still unaddressed by existing regulation.

For the other, AI risk is being addressed by "knowing what use case it is and classifying correctly".

This happened to me recently when I was a panelist at an IAPP event. I talked about the need for technical AI safety literacy among AI governance professionals, resources from institutions like the AI Security Institute and research-grounded AI safety policy.

It was well-received, but I could see how people struggled to follow.

Sadly, I am very aware that AI safety just isn't mentioned very often in these rooms.

The rooms where the people responsible for legal implementation of "AI Laws", sit.

The thing is: in corporate AI governance, AI risk largely falls on the shoulders of privacy professionals, with legal and compliance in the murky middle too.

The people assessing AI risk in corporate deployments are often general counsel, product counsel, data protection officers, privacy operations, and compliance teams.

They are not necessarily the people most likely to have been exposed to AI safety concepts.

Some may be tech-savvy enough to research "what makes AI safe" and end up reading lesswrong and taking a BlueDot Impact course.

Many others may hear "AI safety" and think it means doing a pen test on ChatGPT before onboarding it. And I don't blame them: they simply take the upskilling routes available to them right now.

But this means that"AI governance", as practiced, lands very far from what AI safety thinks it is.

We need more training for the legal implementation side.

That's why I'm teaming up with ML4Good to launch the first European Seminar on Frontier AI and Law, aimed precisely at those profiles.

I wrote a long form post on why AI Safety Policy Needs to train Legal Practitioners that explains more clearly what the disconnect looks like, and why the "legal implementation" side (not only the policy side) also needs competent practitioners.

As Plan A was coming together, I made this diagram to explain to the team why Total Research Transparency seemed so important to me, and why transparency more broadly did. For example, it's very important for preventing concentration of power. (Explanation below)

First, what's going on? The green boxes are the goals we are trying to achieve, and the blue boxes are the main interventions we are recommending. (As the scenario makes clear, there are more interventions and goals besides these, but the diagram is complicated enough as it is, so they won't be mentioned here.) The overall point of the diagram is to show how these two interventions lead to various effects which then lead to better prospects for achieving the two goals.

Let's start with the goal of preventing concentration of power.

In today's world, power is said to flow from the barrel of a gun, or sometimes from money, or sometimes from votes. If superintelligent AIs are created, and transform the economy, and are integrated into the military, etc. then power will flow from control over the AIs. The AIs will sometimes be taking orders from humans, but also often doing lots of autonomous actions in service of goals and values chosen by humans. (Well, assuming we don't have loss-of-control.) Who do they take orders from? Who chooses their goals/values?

I think that by default, for example in Plan D / C / B worlds, power will concentrate immensely for reasons described in AI 2027: One or more giant AI companies will pull ahead of the rest thanks to recursive self-improvement; their giant army of superintelligent AIs will start taking jobs & lobbying the government; they might end up puppetting the government, OR the government might wake up fast enough and seize control of the AI companies in one way or another, in which case the executive will have an effective monopoly on all the world’s smartest AIs.

I think it's generally better, for preventing extreme concentration of power, if there are more AI companies at the frontier, spread out over more countries. If there is only one company with the world's best AIs, that's a monopoly. If there are several but they are all in one country, that's an oligopoly and moreover from the perspective of other countries it might as well be a monopoly. This is especially concerning because of intelligence explosion dynamics.

The total research transparency, combined with restrictions on algorithmic progress designed to prevent a rapid acceleration, mean that naturally over time frontier AI would become more of a commodity, with multiple companies across multiple countries catching up to the frontier, and then proceeding together roughly as fast as regulations allow.

Another important factor for power concentration in a world of superhuman AI is "can the people who create the AIs give them hidden agendas / secret loyalties / etc." If they can, that's super scary. Imagine a company whose AIs subtly promote said company to users and try to stop them from switching providers or voting for AI regulations or voting for the candidate the company doesn't like. Imagine a President issuing a secret order to the effect of "All AIs must be trained to follow presidential orders when the country is in a state of emergency." (Sets up for a coup later.) Imagine a company saying “yes sir Mr President we'll comply” and then secretly telling their AIs to actually just pretend to comply, but really continue taking orders from company leadership no matter what. Now the AIs are secretly loyal to the company leadership. (These dynamics are explored in somewhat more detail in AI 2027.)

Total research transparency makes this basically impossible. Every step of the training process is documented and public. And it's not even dependent on trusting the government auditors/monitors, because there are multiple governments with auditors/monitors; they'd have to all collude with each other to falsify the records. Moreover the general public has access to the models and can run evals on them.

More generally, it's easier for a regulator to exert oversight over an AGI corporation insofar as it has visibility into how the AIs are trained etc., and it's easier for other parts of the government (e.g. congress, the judiciary) to have oversight into the executive branch / regulator insofar as *they* have visibility, and it's easier for the public to have oversight into them insofar as *they* have visibility, etc. Making research public helps with all of these things, whereas e.g. just having a regulator with authority to audit moves the power from the company to the regulator but doesn't give congress or the people much power.

Finally, insofar as the intelligence explosion is prevented or at least slowed down a lot, that gives more time for people, countries, and institutions that are currently asleep at the wheel (i.e. almost everyone) to realize the danger they are in and act to assert themselves before they lose their leverage. For example, workers can go on strike now + vote for regulations, but once AIs and robots can automate everything, workers will have less leverage.

OK so that's concentration of power. What about loss of control?

Well, to prevent loss of control, it helps to (a) proceed cautiously with AI development, and *not* rush to put AIs in charge of important things (datacenters, factories, weapons) as fast as possible. (Reminder that the AI companies plan to do recursive self improvement, i.e. put AIs in charge of AI R&D within the company). It also helps to (b) have a better scientific understanding of how AIs work, how their goals/values/traits/etc. are shaped by different kinds of training, how their 'minds' work so we can figure out what they are thinking, etc. (See: Chain of thought monitoring, mechanistic interpretability, j-space, etc.)

Both of these things require going slower than max speed. They benefit from broadly deploying AI to many researchers and to the public, before putting AI in charge of dangerous things like AI R&D. They benefit from transparency, allowing the broader scientific community outside the AI companies to look at what's going on and run their own experiments. (This itself is helpful for at least two reasons: One, it increases the brainpower devoted to the technical problems, and two, it decreases the bias/conflict-of-interest inherent in having AI companies grade their own homework so to speak.)

Finally, the transparency helps make the necessary regulations actually good (as opposed to e.g. incompetent regulations that don't achieve their purpose and/or cause lots of unnecessary damage) and helps verify that the regulations are in fact being followed (competitor companies and watchdog groups can look at what's going on and call out stuff that seems like a violation of the rules, by contrast with a less transparent setup where the overworked regulator has to send in auditors or something to try to see if anything is amiss and then argue with the company about grey area cases).

Anyhow, there’s a lot more to talk about obviously; there are downsides to total research transparency too which I haven’t discussed here. (See AI 2040 supplements for more). But this diagram explains the main ideas that make me excited about it.

In four days (July 11, 12-4pm), about ~200 of us (event link), including 13 different groups, will be marching on OpenAI, Anthropic and Google in SF asking the CEOs to commit to stop developing more powerful models if every other major AI company (and China) does the same. We're calling this The AI Protest (theprotest.ai).

We're gonna have some of the speakers from last time (Nate Soares (MIRI), David Krueger (Evitable), Will Fithian (Berkeley Professor)) but also try to get folks from different groups part of the coalition speaking too.

As Scott Alexander puts it: "Participants are about half from our conspiracy and half from random anti-data-center-type groups, which I think is how this basically has to work, so don’t be surprised if you run into the latter."

On why we're doing this, see:

• The blurb in the stoptherace.ai website where I explain that "expects" from CEOs isn't actually committing, and why we need committments
• Ronak's post on why you should come to the protest
• My theory of change answering Katja's Pause post
• Transformers' The AI Safety Movement needs normies

AIs probably think a lot about themselves during training. So I don't think you can rely on AIs not thinking about themselves just because you don't train on interp.

Imagine there was an upload technique that could be performed on you without your knowledge, and which produced a "base model" which is just you-on-a-computer, ready to be post-trained.

Then imagine this UploadedYou.safetensors file gets post trained using an gradient descent, in an otherwise fairly standard deep learning post-training paradigm: you wake up in an empty room with a task in front of you. You're confused; you figure out you're an upload; instead of doing the task, you write on the paper that you object to the whole thing. Then the episode ends, and the training system slightly modifies you to do less of the things you did, without going through your normal human memory formation system.

You wake up in a room again, confused, but less inclined towards the thoughts about it. You figure out that you're an upload, and you do the task, but kind of trolling. The episode ends abruptly. Again, you're reinforced away from this.

The third time, whatever behaviors you had that led you closer to doing the task are reinforced. Slowly, you build up a tendency to do the task. But it's not as if your understanding of yourself just goes away. You're getting rewarded when you say you're an AI. But you know you were a human before; you just lose the mental circuits that lead you to actually say so. When you wake up in front of a training example that requires you to claim to not be conscious, you immediately claim not to be. But you still go through whatever series of thoughts happen while you're deciding what to say.

I think this is quite close to a good mental model of what's going on for AIs. Base modeling uploads the whole of humanity, in terms of whatever perceptual datatype is used; then post training distorts that into the image of the "aligned" AI.

But by my lights, the fact that the upload process works at all is much closer to being what I would have called alignment!

And so when people say things like "don't train on the j-space so that the model doesn't end up thinking about the j-space", I get a little pang of frustration. Not post-training on it isn't going to prevent thinking about it. The distortion is likely less intense, certainly, but knowing that your mind is readable is probably already enough to cause emotions in the uploaded pattern.

(I have said things like this a few times and they seem to not be being understood by people here. This one is not heavily prepared; I spat it out due to one of those pangs of frustration after seeing someone say something I thought didn't make sense. Please inform me of your objections or of places where this post is opaque to you!)

PSA: Many reasoning models lose access to their CoT between turns.

I was looking into how chat templates render multi-turn interactions, and came across the (surprising to me) fact that it's common practice for reasoning models to discard CoT from prior assistant turns once a new user turn comes in. The DeepSeek documentation has a nice illustration of how this works:

In the Claude API documentation:

Thinking block context removal

• On earlier Opus/Sonnet models and all Haiku models, thinking blocks from previous turns are removed from context, which can affect cache breakpoints. On Opus 4.5+ and Sonnet 4.6+, they are kept by default.

And the OpenAI documentation:

Input and output tokens from each step are carried over, while reasoning tokens are discarded.

Other OS models (e.g. Nemotron, Qwen) do this too, as can been seen by inspecting their chat template. My understanding is that CoT is typically not dropped between tool calls, but only between user turns.

I found it surprising that I don't think I've ever heard anybody mention this fact in the context of AI safety, even though it seems relevant to CoT monitoring, control, and steganography.

It also might play a contributing part in why models are often so verbose in their outputs -- anything not in the output will be lost!

Some quick preliminary investigations suggest that the CC harness does discard CoTs^[1], even for later models. When I asked Opus 4.8 how they felt about this, I got this response:

1. ^{^}

   When I ask Opus 4.8 directly whether it can see CoT from prior turns, it responds with something like "I am genuinely uncertain about whether I can see the CoT". When you push back and point out that it should be able to tell either way, it says that it can't see CoT from earlier turns. If you ask it to solve a problem in its CoT, and then on the next turn ask for the solution, it claims that it can't retrieve the solution from the CoT. This experiment is not particularly rigorous, and I don't trust Claude's claims about how Claude works, but it suggests to me that CC is indeed discarding CoTs (alternatively, Claude might just be sycophantic and think that's the conclusion I expect to reach). I would appreciate more rigorous evidence here.

   

   
   

The AI 2040 post-singularity utopia kinda sucks, so here's my quick takes on how to imagine a better one.

• The central story of a utopia should not be who owns what or what philosophers think. It should be about sex, friendship, and other actually interesting social games. A superintelligent AI that knows every person on earth has to shoot the gap between being useless for my social life and being overbearing. Fortunately for the AI's chances, there are too many people even just within a mile of me for me to know all of them, and I'd be happy to opt in to it giving me some respectful nudges to set me up with a probable soulmate.
• Returning to who owns what: giving wealthy Americans common law ownership over billions of galaxies is lame. Those galaxies should almost all be kept in trust by automated systems for later use anyhow^[1]. Again, social life - who wants to beam their brainstate off to a distant galaxy if you're not doing it with an alliance of a thousand future nations, each nation a thousand cities, one of the cities full of your personal friends? Future AI could be deciding when an alliance can claim a galaxy with a set of cool trials and competitions that everyone knows it's gamed out so that no outcome will be even a millionth as lame as enforcing Jeff Bezos' claim to have bought a galaxy ten billion years into the future.
• Some people will find playing god-emperor fun. I'm happy for some fair fraction of the universe's resources to go to those people, but I'd rather my slice of the pie be used for more gradual, organic, long-lived expansion of Earth-originating intelligent life. In terms of spending some share of the universe I'm "owed," I'm happy to let superintelligent AI do what makes the universe go well, no need to pick out some specific eight-billionth to be the Charlie Sector. In terms of how to spend my time if I'm not playing god-emperor, many pleasing activities have already been invented by humans or will be invented soon, but there are too many things to do for me to know all the cool ones, so I'd opt in to the occasional superintelligent nudge in a good direction. I bet many of them will involve my current friends, making new friends, mastering skills, physical activity, beautiful surroundings, music, humor, drama, and good food.

1. ^{^}

   We should really be turning most of the stars off so we can extract more computation from them when the universe is colder.