Alignment and Ethics

Introduction

Let us consider the question of “right action”. What ought one do? This question arises in ethics, where we ask how a person should act, but it also arises in the design of artificial or institutional agents, where we ask what objective an agent should pursue and how it should choose among possible actions.

In Functional Explanations of Art, I considered that producing a work of art could be “good” or “bad” depending on who it was good for over a specific time interval. Therefore, an artwork could be good for the artist now but bad for them later, or an artwork could be good for the artist but bad for society.

There is no reason to think that artistic production is exceptional in this regard. Actions generally have different consequences for different parties at different times. An employee may benefit from a decision that harms their employer, a company may benefit from a decision that harms its customers, or policy may benefit present citizens while imposing costs on foreigners, future people, or the ecological systems on which both depend.

The difficulty is not that these consequences are hard to predict, as even if all outcomes were known perfectly, different agents could still value the same actions differently. Instead, what is good for one agent may be bad for another. Similarly, extending the time horizon a decision is judged over can reverse an evaluation.

This problem becomes more difficult when agents form groups. For example, a group can adopt a procedure for making decisions and then act as a single agent. A company acts through management and employees, a state acts through political institutions, and a coalition acts through whatever decision rule is used by its members. The resulting collective may conflict with its own members, just as those members may conflict with one another. By recursion, a group may become part of a still larger system and enter into the same kind of conflict at a higher level or between levels.

This creates a persistent problem for accounts of right action. Should an individual sacrifice their interests for the group, or should the group be constrained by the interests of the members? Which stakeholders should a company serve: owners, employees, customers, or the public?

What’s worse, this problem cannot be dissolved by conceptually redefining the groups¹ (i.e. drawing new boxes saying who is in which group), as that changes the representation of the conflict but does not tell each new group which preferences to prefer. Treating an employee and employer as separate agents or both as part of a single company should not change the underlying action or its ethical consequences. When a decision benefits one of these parties by transferring costs to another, which boundary should determine whether the action is right?

In this essay, I examine this problem and show that many ethical problems can be recast in this framework.

The Fundamental Problem

In the discussion of Theory of the Club, I explored how groups bind themselves to similar preferences and then generate and act on common ends. But that discussion was mainly speculative and oriented around possible shifts in organizational forms due to the integration of AGI and the economy. Relatedly, once plural agents have formed groups, which ends should have prerogative in the case of conflicts?

We must assume that distinct agents will inevitably conflict in their goals (else they are not distinct²). Organizations of agents can define memberships and then by internal mechanisms collect preferences and bind agents to the group decision. But while this answers “What will we do?”, this does not answer “What ought we do?” An account of right action should explain why a particular action should govern when the objectives of other stakeholders conflict, both within and without the group. Binding commitments to a group resolve what action will be taken without resolving whose interests ought to prevail.

Agents therefore enter into different types of conflicts. Aggregating the preferences of several agents creates a collective objective, but the resulting group then acts in relation to other groups and individuals. The group’s objective may conflict with the members (oppression) and the member’s objectives may conflict with the group (parasitism), just as the objectives of the members previously conflicted with each other. Similarly, groups may conflict with other groups (war). Those conflicts may be addressed by forming organizations of organizations, but this just produces a new metacollective whose ends can conflict with agents and systems beyond the metacollective boundary. Thus, different aggregations relocate conflict rather than resolve it.

A theory of right action is supposed to rank actions in the underlying world, not merely relative to a chosen conceptual aggregation. If changing only the unit of aggregation reverses that ranking, then the theory gives incompatible judgments about the same world. On the other hand, if a theory avoids this result by privileging one aggregation, then it assumes the ranking it is trying to justify.

A theory of ethics therefore faces an apparent contradiction. Right action shouldn’t depend on an arbitrary choice of aggregation, but the objectives according to which actions are evaluated are themselves indexed relative to the units chosen by that aggregation.

Reaggregation and Identity

If a change in aggregation leaves the underlying world unchanged, it should also leave the ethical ranking of actions unchanged. Can we construct a theory of ethics that is invariant under arbitrary conceptual aggregations?

An analogy can be drawn to the use of control volumes in chemical engineering. A process may be divided into units in many different ways. A flow that crosses the boundary of one unit may become internal when several units are considered together, but redrawing the boundary does not change the underlying physical process. Thus, valid conservation laws must continue to hold under either representation. Note that this is also the operation considered in the post on allometry, where (for example) we used “torus actions” on systems to produce coordinates that are invariant to changes in unit.

The underlying process and flows are unchanged. Combining control volumes makes some transfers internal to the new boundary, so those transfers cancel from the aggregate balance.

From the outside, this is straightforward. If changing the conceptual reaggregation of the world changes the desired action, then the desired action was contingent on the representation rather than by the action or its consequences. An ethical theory should commute with reaggregation. We should be able to evaluate actions first and then (re)aggregate, or (re)aggregate first and then evaluate actions without altering the ranking (or only altering the ranking in some systematic way).

The matter is less straightforward from the perspective of an agent acting within the world. Reaggregation can change the referent of “I”, making reasoning difficult. What is optimal for me as an employee may not be optimal for the company of which I am a part, and what is optimal for the company may not be optimal for the market or society containing it. By this logic, the question of identity and the question of right action are entwined.

Ethical evaluation must therefore distinguish between two questions:

1. What action is optimal for a particular agent or aggregation?
2. What action is right when several possible identities for who “I” am yield incompatible answers?

The first question is relative to a choice of identity. The second must be answered without arbitrarily fixing that identity in advance. So a theory of right action should take account of the possible identities supported by the world while remaining invariant to the merely conceptual boundaries through which those identities are represented.

We can see that this is just a restatement of the alignment problem. Suppose we could find a quantity that doesn’t depend on how we draw the boxes, and that is maximized by the same action whichever agent or group of agents we evaluate. Then across all levels, all agents would be aligned as to which action to take.

Unfortunately, it seems unlikely such a quantity exists. Any quantity that survives a redrawing of the boxes has to be extensive³, and an extensive quantity is one that agents may compete over.

However, even a negative answer can help us better pose the alignment problem. Are there technical conditions under which such a quantity exists? Which worlds admit such a quantity? Are there approximate solutions that minimize conflict?

Ethical Problems as Reaggregation Problems

The preceding account suggests that many familiar ethical problems can be understood as conflicts among different possible identifications of the relevant agent. The underlying action and its consequences remain fixed, but the answer to “What is good?” changes with the answer to “Good for whom?” or, from the agent’s perspective, “Which system am I?”

The Present and Future Self

Suppose a person withdraws their retirement savings to fund an expensive vacation. The action benefits the present self but leaves the same person poorer and less secure decades later. If the relevant agent is the person at the moment of choice, the withdrawal may appear optimal, but if the person is treated as one agent extended through time, the future loss becomes internal to the decision.

Individuals and Organizations

Suppose an employee falsifies sales figures to earn a bonus, increasing their income while exposing the company to legal and financial risk. This is advantageous for the member but harmful to the organization. Conversely, suppose the company knowingly requires employees to work dangerous hours to meet a deadline. The organization may preserve a contract or increase its profits by imposing the risk on its members. The first case resembles parasitism and the second, oppression.

Relations Between Groups

Suppose two competing manufacturers secretly agree to raise prices. The agreement is cooperative from the perspective of the firms: both avoid competition and increase their profits. But from the perspective of customers and the market as a whole the same agreement is collusive and extractive. Likewise, a state may seize a neighboring state’s water supply to protect its own agriculture, advancing national persistence while destabilizing the wider regional system.

Man and Nature

Suppose a city pumps groundwater faster than the aquifer can replenish to support housing, agriculture, and industrial activity. This increases the throughput of the human settlement, but simultaneously alters recharge dynamics, subsidence patterns, and wetland hydrology. The apparent conflict arises only if human activity is treated as separate from the hydrological system. If instead the relevant identity is the coupled human–aquifer system then the same action is a reconfiguration within a single dynamical system. Under that aggregation, what looks like an external environmental cost becomes an internal change in the system’s own capacity to maintain its structure and flows.

Common Structure

In each case, the same action advances one possible identity while imposing costs on another: the present self against the future self, the member against the organization, the organization against its members, a coalition against outsiders, or a human system against the ecological system containing it. Prudence, parasitism, oppression, collusion, war, and environmental destruction can therefore be understood as failures of alignment among differently aggregated agents and systems.

Computational Kantianism

Kant’s formula of universal law says to act only on a maxim one could will as a universal law. It varies who acts: a maxim holds only if it still holds when anyone adopts it. The question here is the neighboring one. Instead of varying who acts, we redraw what counts as the agent, and ask whether the judgment survives.

To make the question computational, suppose each supported identity is represented by a Bellman operator or a similar decision rule, with every evaluation derived from one common world process.

Across the identities the world actually supports, is there a single policy that each of them ranks best? If the agents share a best policy, the practical conflict disappears for that decision (they don’t need to share reasons, only the top choice).

When no common policy exists, agents can instead bargain. But bargaining also takes the parties as given.

Group boundaries determine the parties to bargains, and therefore dictate who gets seats at the table, whose injuries matter, and who can object to agreements. Control over a boundary is therefore a form of power. But not every boundary is equally open to being drawn. The aggregates whose interests stay coherent across many reorganizations have the strongest claim to count, and those are the ones power cannot simply define away.

Conclusion

Based on this loose argument, ethics can be viewed as an alignment problem in which the identity of the agents themselves must be determined. Redrawing a conceptual boundary can’t resolve conflicts and shouldn’t change actions as it only changes the description of the world rather than the world itself.

When no common policy exists, deciding who counts as a party becomes part of the ethical problem.

AI Disclosure

I used AI to help draft this, especially Appendix A. Ideas my own, not developed in conjunction with AI.

Appendix A: Related Approaches

The problem here overlaps several approaches to ethics, although they begin from different questions.

Kant asks what principle could bind every rational will, rather than deriving morality from a person’s contingent desires. His formula of universal law tests whether I could will the maxim of my action as a law for everyone. This varies who adopts the maxim while holding fixed what counts as a person. Reaggregation varies what counts as the agent. Kant’s formula of humanity also adds something that the test proposed here does not provide on its own: a constraint against treating rational beings merely as instruments, even when an instrumental rule could be applied consistently.

Rawls asks which principles free and rational people would choose to govern the basic structure of society under fair conditions. The veil of ignorance prevents the parties in the original position from knowing which citizen they will become, so they cannot tailor the principles to their class, abilities, or conception of the good. This is a powerful way to neutralize a person’s location within a society, but it takes the society and its basic structure as the relevant collective. It does not ask whether the ranking survives when society is redescribed as nested or overlapping agents.

In The Basis of Morality, Schopenhauer asks what could motivate an action with genuine moral worth after rewards, punishments, and disguised self-interest have been excluded. His answer is compassion: direct participation in another person’s suffering weakens the apparent separation between self and other. The boundary of the individual therefore matters morally, although Schopenhauer challenges it through the experience and metaphysics of compassion rather than by comparing policies across possible organizations.

Parfit poses a related problem across time. In Reasons and Persons, he asks how personal identity bears on rational self-interest and argues that what matters can be psychological continuity and connectedness rather than an all-or-nothing identity. This is a direct precedent for the conflict between a present and future self. It destabilizes one particular boundary, the person through time, rather than giving a general account of reaggregation.

In The Methods of Ethics, Sidgwick tries to systematize the main procedures by which people decide what they ought to do. He ends with a “dualism of practical reason”: rational egoism directs a person toward their own good, while universalistic hedonism directs them toward the good of everyone, and neither principle can conclusively subordinate the other. This comes close to the conflict described here, but it contrasts the individual with the universal good. Reaggregation can reproduce the conflict at every intermediate level.

In The Social Contract, Rousseau asks how people can form a legitimate political association while remaining free, and answers in terms of a general will directed toward the common good. His discussion of factions gives perhaps the clearest precedent for the distinction between cooperation and collusion. The will of a faction can be general with respect to its members while remaining particular with respect to the state. Internal agreement therefore does not establish that the faction serves the larger collective.

List and Pettit ask a prior question: when can an organization literally count as an agent rather than merely as a collection of agents? They argue that groups with suitable decision procedures can form attitudes, make choices, and bear responsibility in their own right. Their theory helps identify the supported identities that enter the present argument. It does not, by itself, decide whose objective should prevail when a group, its members, and outsiders rank policies differently.

In The Philosophy of Loyalty, Royce looks for a moral principle in a person’s devotion to a cause larger than the private self. He distinguishes loyalty that sustains the possibility of loyalty from predatory loyalty, in which a group advances its own cause by destroying the ability of others to pursue theirs. His “loyalty to loyalty” is therefore close to the requirement that one aggregation not make itself final by eliminating the agents beyond its boundary.

Curry’s morality-as-cooperation treats moral rules as biological and cultural solutions to recurrent problems of cooperation, including kinship, group loyalty, reciprocity, bravery, deference, fairness, and property. This explains why cooperation supplies so much of the content of ordinary morality. The missing question is the level at which an interaction is called cooperative. A cartel solves a cooperation problem for its members by creating a predation problem for its customers.

Cosmopolitanism and Leopold’s land ethic respond to exclusion by enlarging the moral community. Cosmopolitanism places every human being within one moral community regardless of political membership. Leopold extends the community further to soils, waters, plants, and animals, making humans members rather than conquerors of the biotic community. Both approaches expose harms hidden by narrower boundaries. But each privileges an enlarged community, whereas the present problem asks whether judgments can remain coherent across several nested descriptions and what follows when they cannot.

Ulrich’s boundary critique begins in planning and systems practice, where calling something “the system” determines what the intervention is supposed to improve. Critical systems heuristics asks who benefits, who controls the means of decision, whose knowledge counts, and who represents those affected but not involved. It then compares those boundary judgments as they are with those that ought to govern the intervention. This is the closest direct precedent for treating the choice of boundary as part of the ethical problem. The proposal here adds a computational test: hold the world process fixed, reaggregate its representation, and ask whether the ranking of policies changes.

There are also narrower formal precedents for “Computational Kantianism.” Roemer’s Kantian optimization replaces unilateral deviation with the question of whether agents would endorse a common proportional change. Team reasoning shifts the practical question from “What should I do?” to “What should we do?” once people identify with a group. Work on a Kantian machine and on formalizing the humanity formulation asks whether Kantian tests can be expressed as computational procedures. These approaches formalize universalization or collective choice while taking the relevant person or team as given. The extra operation here is to vary that unit itself, much as the Bellman operators in scale-invariant control are compared across levels of description.

Footnotes

1. We should distinguish between a conceptual aggregation (any conceptual grouping of elements treated as a unit of evaluation), and a “real” organization (an aggregation supported by infrastructure that enables it to collect information, form objectives, make decisions, and act as a persistent unit). Here we are interested in the conceptual aggregations, as they only exist up to representation.

2. There’s probably some edge cases here. Are two agents distinct if their preferences are distinct but the course of action they would take under every realized circumstance is the same? These types of questions are out of scope.

3. If a quantity is going to be indifferent to how you draw the boxes, then the value of a merged box has to be fixed by the values of its parts, and the order of regrouping can’t matter. That forces the combining operation to be associative and commutative, and once we add mild regularity conditions (continuity, cancellativity), the only such operation is addition, up to a monotone relabeling, and hence extensivity.