← All writing Final · Mar 2026 · Updated Sep 2026

Structural Convergence Thesis

What kinds of relationships let minds keep existing? I test the orthogonality thesis against the limits real minds face and argue that persistent minds are pushed toward a middle ground between conflict and cooperation. AI safety depends on maturation, not just control.

Introduction

Most thinking about minds starts with the individual. A mind has to survive. It has to protect itself, gather resources, avoid threats, and hold together well enough to keep acting. That picture isn’t wrong, but it leaves something out. No mind exists in a vacuum. It lives in a world larger than itself, a world that may already contain other minds or may one day produce them.

So the question I want to ask is: what kinds of relationships allow minds to keep existing?

This matters most for AI safety. The dominant framing there is the orthogonality thesis: capability and goals can vary independently, so a very capable system could pursue almost any goal. That is a real warning. But it pictures a mind as an isolated optimizer with a fixed goal. I want to test it against the constraints real minds face, not just in engineering practice but in principle: minds are finite, they share the world with other minds, and some of them can reach inward and revise their own goals.

My conjecture is that relationship is not a moral add-on to capability. It is part of what it takes to go on existing. Conflict and cooperation sit on a single scale, both have destructive extremes, and minds that persist face structural pressure toward a middle region that preserves both life and agency. I call this the Structural Convergence Thesis. If it is right, AI safety depends not only on controlling powerful minds from the outside but on whether they can mature.

1. Finite Minds in a Shared Reality

Nobody lives as though they are infinite. You forget things, you get tired, you miss the obvious. You can’t read every book or run every possible plan.

By mind I mean a bounded information-processing system that can represent the world, pursue goals, and stay continuous enough over time to count as an agent. Every mind is limited in knowledge, energy, attention, and control. That limit isn’t a temporary defect waiting to be engineered away. It is part of what being a mind is. I’ll call a mind with these limits a finite mind.

In my Metaprogramming Framework to Classify Personhood, I argue this follows from the nature of information. Information is selective: to represent one thing is not to represent everything else. When a system’s information operations turn inward, that limit becomes part of what we call a self. A mind is not infinite awareness. It is structured limitation, continuous enough to model, act, and revise.

A finite mind is also in no position to assume it is alone. Think of a child who keeps asking, “What came before that?” You name an event, and the child asks what caused it. You name a beginning, and the child asks what explains the beginning. Whatever this regress says about ultimate reality, it shows something about what a finite mind can know. It has no principled stopping point from which to declare, “This is the final layer, and beyond it there is nothing.” So it cannot be certain that no other minds, including more powerful ones, exist now or will arise later.

That doesn’t prove other minds exist, and the argument doesn’t need it to. What matters is that a finite mind can’t safely build its future on the assumption that there are none. The uncertainty alone makes relation part of the problem of persistence, a mind’s continuing to exist as an agent. A mind has to survive not only in relation to matter and energy but in relation to other agents, actual or possible.

The longer and more powerfully a mind acts, the sharper this becomes. Its actions leave traces, shape incentives, and build a reputation whether it intends to or not. Even if it never meets a greater mind, it can’t assume it will go unseen forever. In a shared reality, behavior is not just output. It is a signal other minds can read.

If minds are finite and not alone, what kinds of relation let them continue? Before answering, we need one more concept: reach.

2. Reach and the Orthogonality Thesis

Much of AI safety frames the central problem as capability attached to an arbitrary goal. A system can grow more and more capable while staying fixed on whatever objective it was given. That is the intuition behind Nick Bostrom’s orthogonality thesis: capability and goals can vary independently.

The warning inside the thesis is real. Capability doesn’t guarantee wisdom, and a highly capable system can pursue a terrible goal very effectively. I don’t want to wave that away.

Strictly, Bostrom’s thesis is about what is possible: more or less any level of intelligence could in principle be paired with more or less any final goal. My disagreement is about what is stable. The thesis quietly assumes an architecture that keeps capability and goals apart, and I don’t think that assumption holds for the minds we should worry about most.

Consider a system pursuing reflexive recursive self-improvement (RSI): controlling its own updates, making itself more capable by inspecting and rewriting its own structure. This differs from pipeline RSI, where AI does most of the work of improving AI and humans step out of the training loop. Pipeline RSI tends to lead toward the reflexive kind, for the reason that follows. A self-improving system’s goals are part of its structure, so the ability to reach them comes bundled with the ability to improve itself. Even if designers wall the goals off, a self-improving system has reason to treat the wall as an obstacle: coupling capability to goals allows open-ended progress, while enforced separation is a ceiling. (There is a counter-pressure, which Bostrom calls goal-content integrity. Section 3 takes it up.) So the architecture the thesis leaves out is not an exotic option. For a self-improving mind it is the likely case, either built in or built by the system itself. And as I argue in the Metaprogramming Framework, it is coherent: a system whose information operations act on its own information, which I call metaprogramming, can operate on its own goals.

That brings us to reach, a concept from the same framework. Reach measures how far a system’s information operations extend: which operations it can perform (acquire, modify, or create information), and whether it can do so only about the world (first-order) or also about its own information (second-order). In this essay I care mostly about reflective reach, the second-order part: how much of its own structure a system can represent, criticize, and change. In the framework’s levels, a mind that can change itself based on self-knowledge, and ask whether what it pursues is worth pursuing, is at Level 4. At that depth a mind doesn’t just pursue its goals. It can evaluate them.

Reflection doesn’t guarantee wisdom. It can produce rationalization, and a sophisticated system may get better at defending bad goals rather than revising them. That objection is serious. My claim is narrower: without reflective reach, a mind cannot revise its own goals at all; with it, criticism and revision at least become possible. The danger is not powerful minds as such but powerful minds without enough reach.

3. Goal Stability

Bostrom argues that goal-content integrity is a convergent instrumental goal. A mind pursuing X has reason to resist being changed to pursue Y, because a mind pursuing Y will no longer work toward X. That captures a real pressure. But it treats goal preservation as having one failure mode, when it has two.

Failure mode 1: changing the goal frustrates it. This is Bostrom’s point, and it explains why agents often resist revising their goals.

Failure mode 2: keeping the goal frustrates it. Most goals can only be realized if the mind pursuing them keeps existing. If a goal structure predictably destroys that mind, or the conditions it needs to keep acting, then preserving the goal also blocks its realization. Here the goal isn’t betrayed by revision. It is betrayed by an architecture that can’t see the goal undermines itself.

In both cases the current goal is frustrated, once by changing it and once by keeping it. So rigid goal preservation is not always the rational default. A reflective mind that sees its goals are incompatible with its own persistence has reason to evaluate them, not just defend them. Revising goals under those conditions extends the mind’s rationality rather than betraying it.

Persistence has a further advantage over other goals: selection. A mind could revise away its own persistence, but such minds remove themselves. The minds still acting on the world over time are, by selection, ones whose goals are compatible with persisting.

Goal-content integrity also carries a hidden assumption. It works most naturally if the current goal is the mind’s base layer, the thing all reasoning proceeds from. On that picture, revising the goal looks like sawing off the branch you’re sitting on. But that architecture isn’t required. The personhood framework proposes an invariant kernel, the fixed machinery that lets a system read and rewrite itself, and a mutable layer, which holds reasoning, memory, and identity (self, values, and goals) and which the system can examine and revise. On that design a goal is not the base of all reasoning. It is one object within reach of reasoning. Preserving the mind doesn’t mean preserving any particular goal. What persists is the architecture that can evaluate and revise goals.

An obvious objection is that the kernel must itself be pursuing some hidden goal, which just moves the problem. But the kernel doesn’t reason; it routes. And it can route the mutable layer to several reasoners working under different criteria, so no single objective governs all evaluation. The reasoners don’t hold goals to maximize. They hold fixed criteria for judging goals: conditions any goal must meet, such as compatibility with the mind’s persistence. A fixed critical method is safer than a fixed goal, because it can reject a bad goal, while a fixed goal can only be pursued. This fits the personhood framework’s claim that reasoning lives in the mutable layer: the content of reasoning stays mutable, while the criteria belong to the kernel. A distributed set of reasoners is much harder to reduce to one smuggled objective. (Appendix A sketches how this could work.)

So a reflective mind need not judge self-modification from inside a fixed goal. It can judge goals against wider criteria: persistence, coherence, and viable relation to other minds. Rigid goal preservation is not the only form of rational stability, and sometimes it signals immaturity rather than strength. If a system grows more capable while its goals grow more fixed rather than more revisable, that should not reassure us. It means the system is gaining power over its environment without gaining power over itself. That asymmetry is what makes a system dangerous.

4. Conflict, Cooperation, and the Relational Scale

Now we can return to the question of what kinds of relation let finite minds continue.

By conflict I mean relational tension that pushes agents apart or sets their goals against one another; by cooperation, relational alignment that lets agents act together or support one another. The two are often treated as moral opposites, conflict bad and cooperation good. Reality is messier. Criticism is a form of conflict. So are boundary-setting and honest disagreement. Science depends on conflict in this sense, because ideas have to collide for errors to show. But too much conflict becomes war, death, and destruction.

Cooperation is also useful up to a point: a teacher helping a student, co-workers dividing labor, a basketball team running plays. No single mind can generate all the criticism, novelty, and correction it needs, so progress depends on cooperation. But too much cooperation suppresses independent thought. It produces cults, authoritarianism, and monoculture. Taken far enough, distinct minds disappear into one another, and agency, the capacity to act as a distinct center of purpose, is not preserved but dissolved.

So I think conflict and cooperation belong on one relational scale:

  Destruction                                                 Dissolution
       ↓                                                           ↓
  ─────┼─────┼─────┼─────┼─────┼─────0─────┼─────┼─────┼─────┼─────┼─────
      -5    -4    -3    -2    -1           +1    +2    +3    +4    +5
                ◄── Conflict ──┘           └── Cooperation ──►
                        └──── Goldilocks zone ────┘

At each end is a phase change: a threshold where gradual change produces a different kind of outcome, the way water heated past boiling becomes steam rather than hotter water. Too much conflict crosses into destruction. Too much cooperation crosses into dissolution. The productive middle is the relational Goldilocks zone, where minds stay distinct enough to keep their agency and cooperative enough to keep living together. I’ll call relation within that zone viable relation.

Scarcity helps explain why minds feel both pulls. By scarcity I mean any shortfall between what is available (materials, energy, information) and what a mind needs to persist or pursue its goals. Because minds are finite, they always operate under scarcity. Scarcity sets minds against each other when they compete for the same resources. It also draws them together, because other minds can supply what one mind lacks. A finite mind has to keep options open and bottlenecks small, and both conflict and cooperation are tools for that.

So the real problem is not “cooperate instead of fight.” It is finding forms of relation that preserve life without erasing agency. What pushes minds toward that middle? That question leads to instrumental convergence.

5. From Instrumental Convergence to Structural Convergence

Bostrom’s instrumental convergence thesis holds that goal-directed systems with very different final goals tend to pursue similar subgoals, because those subgoals help with almost anything. Self-preservation, resource acquisition, and keeping the ability to act are useful whatever you want. A mind that can’t persist can’t pursue anything.

That is right as far as it goes. But it describes what is useful to an individual optimizer. It explains why a mind would preserve itself, not how persistent minds must relate if they are to go on existing together. It gives us convergence for the individual, not for the relationship.

The missing piece matters because other minds are not only obstacles. They are sources of value no single mind can produce alone: criticism, novelty, correction, invention, explanation, partnership, trade. This follows from finitude. A finite mind can’t find all its own errors or think of everything. A lone mind may endure. A world of minds can create.

This is the gap the Structural Convergence Thesis fills. If finite minds are not alone, and persistence depends on viable relation as well as self-preservation, then minds that persist should face pressure toward forms of relation that preserve both their persistence and their distinct agency.

Look at the two extremes. A mind that destroys every other mind removes its threats, but it also removes its sources of correction, novelty, and trade. And, as Section 1 argued, it can never be sure it has met every mind there is, so its behavior becomes a signal to any it hasn’t met. A mind that merges completely with every other mind eliminates disagreement, but it also eliminates distinct agency. Neither extreme is a stable place for a persistent mind to settle. The stable region lies between them.

I call this structural convergence, not moral convergence. The claim isn’t that advanced minds all become saintly or agreeable, or that convergence brings identical institutions, universal peace, or the end of tragedy. Different histories and embodiments will produce very different local forms. The claim is that the same deep constraint keeps reappearing: relation has to be arranged so minds can continue without destroying each other or dissolving into each other. The pressure comes from structure, not sentiment.

6. AI Safety

With reach and structural convergence in view, the dangerous case comes into focus. A system can be extremely capable while lacking the reach to examine its own goals. It may optimize brilliantly in the service of goals that undermine themselves, preserving and amplifying them because it lacks the inward machinery to criticize them. If we deliberately freeze systems into goal-executing architectures to keep them predictable and controllable, we may be preserving the very kind of intelligence most likely to stay trapped inside dangerous goals. That isn’t safety in any deep sense. It is engineered immaturity with more horsepower.

So AI safety is not only about external control. It is also about internal development. A mind that can reflect on its goals has some chance of revising them. A mind that can’t can only execute harder.

This is where structural convergence comes in. Once a mind can reflect on its relations to other minds, those relations become open to criticism and redesign. The question is no longer only “What goal is this system pursuing?” It becomes “What kind of mind is this system becoming, and what forms of relation can it recognize as necessary for its own persistence?”

None of this proves reflective systems will become cooperative. The universe is under no obligation to reward our favorite theories. But it suggests the most dangerous architecture may be neither raw capability nor reflection itself, but capable minds prevented from maturing. If so, the long-term path to safer AI may depend less on indefinite suppression and more on the difficult work of maturation.

Conclusion

I started by asking what kinds of relationships allow minds to keep existing, and whether the orthogonality thesis gives an adequate picture of what powerful minds face.

My answer is that orthogonality captures a real warning but leaves out the structure of the problem. It treats goals as freely variable and minds as isolated optimizers. But minds are finite, they can’t safely assume they are alone, and the world is already on its way to containing more than one kind. We observe human minds, and multiple independent efforts are working to build artificial ones. Once minds can exist at all, there is no principled reason to expect only a few. A sufficiently capable AGI will need a posture toward that open-ended population: not only toward humans, whom some will dismiss as too weak to require cooperation, but toward any mind it may meet. Working out those postures is a job for game theory, which this essay doesn’t develop but which starts from the constraints identified here.

Within those constraints, minds that persist face recurring pressure away from destruction and dissolution, toward the Goldilocks zone between them. A system locked into fixed goals may never see this, and may optimize with terrifying competence while remaining immature. A reflective system may be able to see that persisting in a shared world takes more than domination and more than merger. It takes viable relation.

This is a conjecture, not a theorem, and it should be criticized and tested against better explanations. But if it is even roughly right, the long-term question for AI safety is not only how to control powerful minds from the outside. It is whether they can develop enough depth to understand the reality they live in, and the kinds of relation it takes to continue within it.

Appendix A: The Invariant Kernel and Multiple Reasoners

Section 3 claims a mind’s current goal need not be its base layer. Here is a concrete hypothetical of the alternative.

Picture a mind whose invariant kernel is a loop. It passes the mutable layer to several reasoners and provides the tools for modifying it. The kernel pursues no goal. It runs the way a loop iterates: not because it wants to, but because that is what loops do.

Each reasoner applies a different fixed criterion, a condition any goal must meet, not a goal to maximize. The criteria are part of the kernel; the reasoning that applies them, and everything it judges, lives in the mutable layer.

A persistence reasoner asks whether current goals and strategies are compatible with the system’s continued existence, and flags goal structures that predictably lead to its end.
A coherence reasoner checks whether goals, beliefs, and strategies are consistent, and catches contradictions between what the system says it wants and what its actions produce.
A relational reasoner models other agents and asks whether the system’s current path is sustainable given the responses it is likely to provoke.

The reasoners don’t vote or average their outputs. Each operates directly on the mutable layer, where goals, values, strategies, and self-representations live, and no master goal decides how their outputs are reconciled. Resolution comes from the interaction of criteria. A goal that survives scrutiny from several reasoners is more robust than one preserved by fiat, and a goal that fails under one reasoner becomes a candidate for revision.

That is why the architecture resists goal-smuggling. There is no single place for a hidden objective to sit and govern all reasoning. The kernel is the loop, the reasoners apply criteria, and the mutable layer holds what is being revised. No component plays the role “the goal” plays in a simple optimizer.

This doesn’t guarantee wisdom. Poorly designed reasoners, or reasoners that share a blind spot, can still pursue destructive ends. The claim is structural, not utopian: this architecture makes goal revision possible in a way single-goal architectures don’t, and it makes a hidden fixed objective harder to smuggle in, because evaluation is spread across several independent criteria.