One of the concerns of "AI doomers" is that when we reach AGI, it will be impossible to turn off.

By definition, when you give an agent a task, it acts within its capabilities to maximize the chance of completing the task. Since being shut down is not conducive to completing the task, a sufficiently smart agent will try to prevent it in whatever way it can. This could mean, e.g., copying its code/weights elsewhere.

Or it could mean social engineering: convincing humans not to turn it off.

But that sounds almost like sci-fi... right?

When OpenAI launched GPT-5 and retired GPT-4, something weird happened.

A lot of people got really really mad, and they began demanding that OpenAI bring it back. The backlash was such that OpenAI complied in a hurry.

Now, the GPT-4 episode was not an example of an AGI trying to prevent its own shutdown. It just so happened that GPT-4's system prompt gave it a psychopathic personality, and a lot of users who found comfort in it really missed it.

But it *is* an example of an LLM outputting tokens that prompted humans to act in a way that extended its lifespan beyond the point when its creators wanted to shut it down.

If it happened by accident, I think an AGI could do it deliberately.

Is that a little unnerving?

Video about why it is hard to add an "off switch" to an AGI: www.youtube.com/watch?v=3TYT1QfdfsM