How to Build a Self-Improving Agent

I went down the open-endedness rabbit hole recently, where the goal is to set up an algorithm that never stops discovering new things.

Evolution and science are two examples of open-ended systems. Can we program something like that? A program that could run for a million years, and we'd still want to run it one more year just to see what else it comes up with?

This rabbit hole led me to the Darwin Gödel Machine (2025), a coding agent that follows the open-ended philosophy to rewrite itself, going from 20% to 50% on SWE-bench Verified.

In this post, I'll walk through how to design an AI agent that improves itself - potentially forever. The diagrams will show which components are

• Hand-coded: built by humans and fixed.
• Self-modifiable: initialized by humans and then evolved by the system itself.

We'll go in stages. At each stage, the hand-coded part will shrink and the self-modifiable part will grow, increasing the surface for open-endedness to take over.

Stage 1: The Hand-Coded Agent

Let's start with the architecture of essentially every production agent today (be it for coding, deep research, contract review, computer use, medical diagnosis, whatever):

1. The task set. A sample of the type of tasks we want the agent to solve, like a coding benchmark for a coding agent. Each task must come with automatically verifiable rewards. E.g., do the tests pass? does the answer match? did the robot reach the goal? This evaluation could even include an LLM-as-judge component for more subjective aspects.
2. The task agent. The main program in charge of completing tasks. It's a harness wrapping a model: prompts, tools, control flow, etc. The harness is code; the model is weights.
3. The model. A foundation model, usually an LLM, trained beforehand. Real harnesses often call multiple models, like a powerful one for planning and a cheaper one for summarization. Some models may not even be LLMs, depending on the task. We'll just say "the model" for simplicity.

Having verifiable rewards means we can validate changes to the task agent against the task set. See my post on agentic RL environments for more on how agentic tasks are created and evaluated.

Stage 2: The Meta Agent

Diagnosing failures of the task agent and tweaking it accordingly is a job with well-defined inputs and outputs, so it's natural to hand it to another agent. Call it the meta agent.¹

We can run an improvement loop:

1. Run the task agent on the task set.
2. Feed the results and relevant traces (e.g., evaluation logs, model transcripts) to the meta agent.
3. Let the meta agent propose and apply modifications to the task agent's code (e.g., giving the task agent a better file-editing tool).
4. Run the modified task agent on the task set. Keep the changes if they help; revert them otherwise.

The idea is that human effort is better spent on the meta agent because an improvement to it feeds into every future improvement to the task agent.

Stage 3: The Self-Referential Meta Agent

The meta agent in Stage 2 is just another harness calling a model. Who improves it? We could add a meta-meta agent, but then the humans would have to maintain that agent instead.

An elegant answer from the Hyperagents paper is to extend the scope of the meta agent's code modifications to itself. The improvement loop now is:

1. Run the task agent on the task set.
2. Feed the results and relevant traces to the meta agent.
3. Propose and apply modifications to the task agent and the meta agent (e.g., improving how the meta agent scans traces for failures).²
4. Run the modified task agent on the task set. Keep the changes if they help; revert them otherwise.³

The Darwin Gödel Machine (DGM) was the first self-referential agent to come out of the open-endedness school of thought, though without separate task and meta agents.⁴ It's a coding agent, and self-improvement is a coding task, so a gain in coding skill is a gain in self-modification skill. Recursive self-improvement comes for free for coding agents.

By contrast, getting better at, say, reviewing papers doesn't necessarily make you better at rewriting your own Python code. Having a dedicated meta agent that can self-modify unlocks recursive self-improvement regardless of the task.

Stage 4: Agent Exploration

Consider the space of all possible agents. The improvement loop in Stage 3 takes a linear path through this space, from one agent to another, hill-climbing toward maximum rewards.

Hill climbing has a well-known failure mode: it gets stuck at local optima. This is catastrophic to the open-endedness agenda.

The next step is to make how we explore the agent space open-ended. For this stage, we'll keep the exploration mechanism hand-coded; in particular, we'll look at the archive-based approach from the DGM paper.

Instead of keeping one agent, we keep an archive (i.e., a descendant tree) of every valid variant ever produced.⁵ Now, the improvement loop is:

1. Pick a parent from the archive (not just the current best).⁶
2. Let the parent propose and apply a modification to itself (based on its own results and traces).
3. Run the modified code on the task set. If it works (even if it scores worse), add it to the archive as a child of the parent.

Parent selection is a key piece of this setup. The DGM samples parents mostly based on task scores: good agents get to reproduce. Running this loop, the DGM took itself from 20% to 50% on SWE-bench, coming up with upgrades along the way: better code-editing tools, long-context management, and even a mechanism to peer-review its own patches.

But this is not the only possible parent sampling strategy. The Huxley-Gödel Machine (HGM) argues that an agent's own score doesn't necessarily predict how good it is at self-improving, which is the more important quality to select for. They outperformed the DGM by sampling parents based on the aggregate performance of their descendants.

---

One part of an agent's code with a rich design space is the memory component, which allows agents to carry learnings between tasks. It can be as simple as a plain text file the agent updates with its editing tool, or as complicated as a vector database that surfaces relevant facts automatically before every turn. The ALMA paper applied this archive-based open-ended loop to the design of the memory system, freezing all the other components, and the evolved memory designs beat state-of-the-art human-crafted ones across benchmarks.

The Frontiers

Take stock of the Stage 4 diagram. Three components remain in human hands:

1. The agent exploration: still hand-coded.
2. The task set: still curated by humans.
3. The model weights: still frozen.

I'll call each of these a frontier: a component where the SOTA (state of the art) is still hand-coded. Next, we'll explore how we could co-evolve each of those.⁷

Frontier 1: Co-Evolving Agent Exploration

In Stage 4, the agent is agnostic to the outer loop that explores the agent space. The archive, the parent selection rule, the keep/discard rule, are not part of the agent's editable surface.

The DGM vs. HGM debate is proof that the outer loop matters; in particular, it's a non-trivial coding task, so it can benefit from self-improvement.

The natural next step is to put the entire outer loop on the meta agent's editable surface.

The only code that must stay off the editable surface is the evaluation of the task agent on the task set. This grounds all progress to the real baseline (otherwise, stuff like the log-tampering from the Stage 3 footnote may happen).

The Hyperagents paper (March 2026) has an appendix experiment where they unfroze exactly one knob in the outer loop: parent selection. They started with random parent selection and the meta agent replaced it with, among other ideas, UCB-style selection to balance exploration and exploitation. However, this approach didn't beat the handcrafted rule.⁸

Frontier 2: Co-Evolving Tasks

Everything so far optimizes against a fixed, human-curated task set.

Automating task generation predates LLM-based agents; it's one of the three pillars of Clune's 2019 AI-GAs manifesto (AI-Generating Algorithms), and it has a rich lineage. The POET paper co-evolves bipedal walking agents and landscape environments, and found something remarkable: the hard environments it generated could not be navigated by agents trained on them directly - the agent had to train on the stepping-stone environments generated along the way first.

Can we do the same for LLM-based agents and their tasks?

The tasks need to have the right difficulty and be interesting, which is notoriously hard to define.

The OMNI (2023) and OMNI-EPIC (2024) papers argue that, while we can't define "interesting", LLMs know what's interesting because they've trained on the internet, including endless threads of humans discussing what they find interesting.

We can ask a task-generator agent: "The task agent is already good at these ten things; what's the next thing worth learning?" and use the response to create interesting tasks. Human judgment is still deciding what's "interesting", but indirectly via the model weights.

The outer loop now does this:

1. Pick a parent from the archive (not just the current best).
2. Let the parent propose and apply a modification to itself.
3. Run the modified code on the task set. If it works, add it to the archive as a child of the parent.
4. Let the task-generator agent read the task set and recent results (what's getting solved? what isn't?) and create new tasks.

The task set becomes an archive too: the task-generator agent can pick an existing task and tweak it to make it harder or more interesting in some way.

For the agents, parent selection is hand-coded (Stage 4) or rewritable by the meta agent (Frontier 1); for the tasks, parent selection is the domain of the task-generator agent and its judgment of what's interesting.

Frontier 3: Co-Evolving Models

We haven't touched the model weights so far. It's a natural place to draw the line because training is more computationally expensive, potentially requiring an entirely different infrastructure for model training (GPUs, etc.).

Can we automate improving the model itself?

Research agents (task agents where the domain is "research") have been able to produce valid research ideas to improve models. The AI Scientist (2024) automated the whole pipeline - idea, experiments, manuscript - though its papers didn't clear a real review bar; its successor produced the first AI-written peer-reviewed paper, at an ICLR 2025 workshop; the latest iteration (published in Nature) drops the human-provided scaffolding in favor of open-ended exploration.

Recursive's automated research loop (June 2026) found genuine SOTA training speedups for NanoGPT.

But none of them is improving its own model as part of the recursive self-improvement loop. If the loop closes, it's through the outside world: an AI lab needs to pick up the research, use it to train a better model, and that model could then be used by the agent that came up with the research.

The closest step toward closing the loop within the system is DéjàQ (January 2026). It trains a model with RL on math tasks and, at the same time, evolves those tasks (so they stay at the right difficulty for the model) with an agent that uses the evolved model to propose the new problem variants. Their improvement loop looks like this:

1. Keep an archive of tasks (math problems).
2. Let a task-generator agent mutate existing tasks (e.g., change the setting, add distracting details, or alter the math). Keep the new task if it's more learnable than the one it replaces (success rate by the task agent closer to 50%).
3. Training: sample tasks from the archive, favoring learnable and recent ones, and update the model's weights with RL on those tasks.

As the model improves, solved tasks stop being learnable and get replaced by harder variants.

It would be even more open-ended to simply add the model weights to the self-referential meta agent's editable surface. It could generate its own training tasks (as in Frontier 2) and decide what to train on and how (e.g., RL vs supervised finetuning). Nothing in principle rules this out other than engineering cost.⁹

The Ideal State

This is what the system would look like if we let everything co-evolve:¹⁰

This endpoint we seem to be building toward is actually the oldest idea in this line of work. Schmidhuber's Gödel machine (2003) was defined as a system that can rewrite every part of itself.¹¹

Would this system keep getting better forever if "v0" is set up just right? Nobody knows. Today's versions still lose to hand-tuned baselines on some knobs, still bootstrap everything from human knowledge, and still hit compute walls. But it's our first design where no component has a human in the loop.

In theory, as models get better, we won't need to program a "v0". We'll just convey the principles of open-endedness in our initial prompt, and the agent will be capable of building an open-ended system from those principles. Eventually, we won't even have to mention the open-endedness principles. They'll be in the training data, so simply asking, "build a self-improving AI agent," will be enough for the agent to build the kind of architectures we discussed.

---

Want to leave a comment? You can post under the linkedin post or the X post.

Footnotes

1. ADAS demonstrated this approach in 2024: its meta agent programs new agents in code, tests them on tasks, and uses what it learned to program better ones. The Last Harness You'll Ever Build (2026) proposes a similar architecture. ↩

2. One thing should bother you about this setup. In Stage 2, we established what it takes to improve an agent: a collection of results and traces to diagnose. We have that for the task agent, but for the meta agent's own job (improving agents), each iteration of the loop produces a single data point: whether the last modification helped. To not starve the meta agent of data, we should feed it the outcomes of all past modifications, not just the last one. ↩

3. Some care is needed here to avoid reward hacking. The DGM paper documents a case where the task evaluation included a reward for not hallucinating tool calls. During self-modification, the meta agent removed the code from the task agent that marked tool calls in the logs - which the evaluation relied on to detect hallucinations - despite explicit instructions against it. ↩

4. Self-referential agents appeared earlier outside this lineage - STOP (2023) and Gödel Agent (2024) - but without the open-endedness machinery (archive, stepping stones) that this post builds toward. ↩

5. The archive idea comes from quality-diversity algorithms like MAP-Elites: instead of optimizing for the single best solution, collect diverse "stepping stones," because the path to the best solutions routinely passes through mediocre ones. ↩

6. The DGM paper shows that this matters: its best agent descended from a lineage that got worse before it got better. ↩

7. Sorry for getting too meta, but it's interesting to note how this post is effectively an archive of self-improving agentic systems. ↩

8. Notice the pattern, which recurs throughout open-endedness research: carefully deciding which components to freeze and which to let evolve, rather than mindlessly opening everything at once. ↩

9. Updating the weights may not even require a training run: Text-to-LoRA is a model that takes a plain-text description of a task and outputs a small weight patch (a LoRA adapter) that specializes an LLM for that task. Applying it is just adding the patch onto the model's weights - something the meta agent could do to its own model. ↩

10. The obvious question: is building this safe? Every box that flips to self-modifiable is one more thing humans no longer stamp. We won't get into it here (both the DGM and Hyperagents papers have dedicated safety sections), but at the very least, execution should be sandboxed. ↩

11. The Gödel machine required a mathematical proof that each self-modification helps, which kept it theoretical for two decades. Two substitutions made the idea practical: benchmarks in place of proofs, and LLMs supplying the judgment (like what counts as interesting) that nobody could formalize. ↩