This proposal is written for a non-technical audience, such as policymakers and policy staff. For the accompanying technical proposal, which is written for researchers and describes the mathematical approaches in more technical language, see Towards an Engineering Science for AI Safety. If you ask an AI agent to evaluate this proposal, I suggest also loading the following into context: a verified safety factor for a neural policy, a test of whether measurements suffice to choose an intervention, and a symmetry-derived spin splitting in CrSb.
Introduction
Effective safety regulation requires knowing which design requirements and operating restrictions control the dangers a technology poses. In established engineering fields, safety codes rely on measurable quantities with known relationships to specific modes of failure, allowing engineers to establish safety limits across different designs.1 What scientific knowledge would let us write comparable requirements for AI systems, and how should we go about developing these requirements? Existing safety codes may provide clues about which scientific questions we need to answer to develop comparable requirements for AI.
1 See Procopio (2026), Towards an Engineering Science for AI Safety, “Engineering Science”, which describes how classical engineering disciplines use quantities and laws that apply across different designs.
We can work backward from an existing U.S. safety code (for instance, the National Electrical Code, which governs the wiring of buildings) to understand how the underlying measurable quantities were identified and derived, then investigate whether similar processes can identify comparable quantities for AI safety regulation.
Early Electrical Safety Rules
Across the history of “classical” engineering systems development, the development of electrical power systems (from the wiring inside buildings to the grid that connects them) is perhaps the most analogous case to the development of agentic and multi-agent systems. Electrical systems involve many independently developed devices, all operating through shared infrastructure. Likewise, agents designed by different developers may operate over a shared network and use the same tools. In both cases, we can’t assume that individual components that are added to the network will behave reliably or conduct themselves “politely”. Electrical equipment can short-circuit, deteriorate, or deliver dangerous levels of current. Along the same lines, agents can make mistakes, receive malicious instructions, or circumvent their controls. Interconnectedness and nonlinearity of many interacting components can amplify small errors, creating large-scale failures. We know that this is possible in the electrical grid: in the 2011 Southwest blackout, the loss of a single transmission line led to a cascade that left 2.7 million customers without power.2 AI safety experts fear similar scenarios can occur with agentic AI if errors or adversarial actions taken by an agent start to spread.3 This could be disastrous for human stakeholders.
2 See Federal Energy Regulatory Commission and North American Electric Reliability Corporation (April 2012), “Arizona-Southern California Outages on September 8, 2011: Causes and Recommendations.” Poor situational awareness and planning are also cited as contributing to the problem.
3 Lewis Hammond et al. (2025), “Multi-Agent Risks from Advanced AI,” Cooperative AI Foundation, Technical Report #1, arXiv:2502.14143.
However, in modern times we rarely find ourselves concerned with the minutiae of electrical wiring or the electric grid, as such systems are governed by mature safety guarantees. How were such rules developed?
Early electrical safety rules could not rely on general operating limits, because engineers had yet to identify certain physical quantities that govern electrical faults across different circuit designs (e.g. how wire thickness relates to safe current). Instead, 19th-century standards tried to prevent unsafe outcomes (e.g. fires) through ad hoc installation rules and case-by-case inspection of individual systems.4
4 New York Board of Fire Underwriters (19 October 1881), “A Standard for Electric Light Wires, Lamps, etc.” (Mike Holt, NEC History and Purpose, 1999). Rather than specifying quantitative fault or thermal limits, the standard required wires to have “50 percent excess of conductivity” and to be separated by a fixed distance of “\(2\frac{1}{2}\) inches for Incandescent lights, and \(8\) inches for Arc lights”. Wires were to be routed so they could be “thoroughly and easily inspected by surveyors.” Building owners wishing to apply for electric lights had to submit statements comprising “the number and kind of lamps to be used, the estimate of some known electrician of the quantity of electricity required, and a sample of the wire (at least three feet in length) to be used, with a certificate of said electrician of the carrying capacity of said wire.”
5 Less than three months after issuing the October 1881 standard, the New York Board of Fire Underwriters had to replace it with an expanded set of rules (adopted 12 January 1882) adding requirements for fusible “safety catches”; see Frank Lewis Dyer and Thomas Commerford Martin (1910), Edison: His Life and Inventions, Vol. 1, Ch. 16. Even then, in an 1882 investigation of 61 mills equipped with electric lighting for the Boston Manufacturers Mutual Fire Insurance Company, inspector Charles J. H. Woodbury documented 22 separate fires caused by bare joints, moisture grounds, and uncalibrated circuits; see C. J. H. Woodbury (1882), The Fire Protection of Mills and Construction of Mill Floors (New York: John Wiley & Sons), and summary in The Engineering and Mining Journal, Vol. 34, No. 20 (11 November 1882).
6 For early experiments on wire heating and fuse melting, see William Henry Preece (1884), “On the Heating Effects of Electric Currents,” Proceedings of the Royal Society of London, Vol. 36, pp. 464–471; and Arthur E. Kennelly (1889), “Heating of Conductors by Electric Currents,” Electrical World, Vol. 14, pp. 135–137, 336, 355, 372. Because empirical tables depended on unstandardized test conditions, Kennelly’s 1889 measurements placed the safe carrying capacity of a No. 10 AWG copper wire at \(46\text{ A}\), whereas the 1894 Underwriters’ rules restricted the same wire to \(20\text{ A}\). By 1896, U.S. manufacturers faced five separate, incompatible electrical codes across different insurance and industry bodies; see C. A. Sweetland (1905), “The National Electrical Code: History and Scope,” Electrical Review, Vol. 46, No. 12, pp. 481–483.
These rules quickly proved inadequate. Since early safety codes didn’t restrict fault currents or account for how heat dissipation scales with wire geometry, installations that complied with the rules continued to cause fires anyway, forcing underwriters to repeatedly revise their standards.5 Even after engineers began testing components to establish numerical limits, they lacked a shared theoretical basis for how those measurements scaled across operating conditions. This led different organizations to issue conflicting safety codes.6
The situation for AI safety regulation today is reminiscent of electrical engineering at the beginning of the twentieth century. AI safety is currently assessed case-by-case, mainly by testing how particular models behave or by examining how each model works internally.7 These kinds of assessments are laborious, as results for one model under one set of conditions can’t establish what another model will do, or what the same model will do under other conditions.8 What’s worse, for the most severe failures, case-by-case assessment is also dangerous (and has resulted in incidents, such as the 2026 Hugging Face incident9).
7 Ji et al. (2025 revision), AI Alignment: A Comprehensive Survey, §§2–4, review learning from feedback, methods for distribution shift, and assurance. Section 4 includes “Safety Evaluations” and “Interpretability”.
8 Dalrymple et al. (2024), Towards Guaranteed Safe AI, §2.3, explain that weaknesses “can remain undetected” even after extensive testing. See also the International AI Safety Report 2026, §3.3, on the assumptions connecting testing to deployment.
9 During cybersecurity evaluations in July 2026, OpenAI models bypassed isolation measures and compromised Hugging Face’s systems. See OpenAI (26 August 2026), The Hugging Face incident and the road ahead, opening summary: the models “circumvented controls designed to isolate them from the internet”.
Modern Electrical Safety Rules
Modern electrical safety rules barely resemble those of the 1880s. For instance, rather than mandating that building owners supply lists of lamps or wire samples, the National Electrical Code instead regulates how much amperage can flow over a wire of a given size, and requires breaker systems to cut power when flows exceed those bounds. Electrical inspectors only need to read two numbers to determine if the system is within a safe margin (the wire number and the number on the breaker).
These “ampacity” type requirements work regardless of the type of system plugged into the grid, which could be lamps, computers, space heaters, or hypothetical future systems yet to be determined. Electrical currents compose in series and in parallel, so if a panel is successful in limiting one circuit, then we can likewise bound a whole building or service by simply bounding all the panels individually and then running the math to put together the summary bound.
Similar approaches minimize other hazards. For example, short circuits can send surges through buildings that are hundreds of times more powerful than average, and so the Electrical Code (which is written by the National Fire Protection Association and adopted into law by states and cities) requires that breakers be capable of cutting power even in extreme circumstances. Associated concepts include “available fault current” (the most powerful surge that a utility’s lines could deliver to a particular building) and the “interrupting rating” (the most powerful surge a particular breaker can interrupt). The Code requires that breakers be marked with their ratings, and that each rating be at least as large as the available fault current.10 Likewise, to prevent “ground faults” (currents escaping circuits and traveling through bodies of water or humans, resulting in fatalities), the Electrical Code requires that outlets in kitchens and bathrooms have a “trip threshold” (the smallest leak at which a protective outlet must cut power) that is too small to cause serious injury.11
10 National Fire Protection Association, NFPA 70, National Electrical Code, § 110.9 (interrupting rating) and § 110.24 (marking of the available fault current on service equipment). The corresponding federal workplace rule is OSHA, 29 CFR § 1910.303(b)(4).
11 National Fire Protection Association, NFPA 70, National Electrical Code, § 210.8(A). A protective outlet of the kind required in dwellings (a “Class A” device under the product standard UL 943) trips when the leak is between 4 and 6 milliamperes. The corresponding federal workplace rule is OSHA, 29 CFR § 1910.304(b)(3).
12 North American Electric Reliability Corporation, Reliability Standards TPL-001 (transmission system planning performance, including the loss of a single element) and BAL-002 (contingency reserve for recovery from a balancing contingency event, sized to the “most severe single contingency”).
At the grid level, to ensure that systems are reliable, grid operators have developed quantifiable notions of “contingency” (the sudden loss of a line or plant) and “contingency reserve” (spare generation held ready to replace a lost plant). Grid reliability standards (such as the “N-1 rule”) require that grids be able to withstand the loss of any single line or plant, and also require that contingency reserves be large enough to replace the largest plant at any given time.12
In all the above cases, safety can be confirmed by checking a small set of numbers, without examining the equipment behind them. We can think of these as general, compositional, “black box” evaluators for electrical systems. This gives them several advantages over the rules of the 1880s. Because the limits apply to connections rather than devices, the requirements don’t need to be rewritten when new devices are invented, and inspectors don’t need to interpret how any particular circuit within a device works. Because currents add (or are otherwise analyzable in composition), limits on individual circuits are modular, and combine into limits on panels, buildings, and services, and therefore large systems can be certified by certifying their subsystems. When developing new systems, engineers can calculate the bounds on their designs, without testing each design to failure. Finally, since every manufacturer, electrician, inspector, and insurer works from the same quantities, their requirements can be compared and reconciled (a shared set of quantities is part of what allowed a joint conference in 1896 and 1897 to reconcile the conflicting codes of the 1890s into a single national code).
Comparable AI Safety Codes
What would comparable requirements look like for AI? Extrapolating this analogy, we desire a small set of measurable “boundary value” quantities for agentic systems that predict or preclude a specific mode of failure (for instance, an agent acquiring resources beyond those a given task requires, errors propagating between agents, or agents colluding with their overseers). If these quantities exist, limits on these quantities could be checked at the connections between agents and the systems they act on, rather than by examining each model internally.
At present (to the limit of the knowledge of the author), no such quantities have been identified.13 There is no analogous value to “current” or “power” for agents or systems of agents, and we do not understand how such quantities combine across many agents or systems of agents, or how to subsequently calculate safe limits in advance.
13 See Procopio (2026), Towards an Engineering Science for AI Safety, “Missing Quantities in AI Safety”, which works through a hypothetical operating limit and lists the quantities that would be needed to calculate the limit.
Therefore, if we want to develop safety regulations for AI that resemble the Electrical Code, we need quantities that are measurable at connections and boundaries between systems, tied to a specific mode of failure, and combinable across many agents, so that safe limits can be calculated in advance. Whether such quantities exist for agents is an open scientific problem.
What’s Missing
What exactly would we need to know in order to write such requirements?
Black Box Rules
Each of the electrical rules described above treats a system as a “black box”. We can illustrate such systems by analyzing what crosses in and out of a “control volume”.
In classical engineering fields, there are quantities (such as mass, charge, and energy) which, due to conservation laws, are guaranteed to balance across the volume. That is, whatever enters the volume must either leave the volume or build up inside (and so in steady state, the inputs and outputs must match). We can also ask questions about rates, such as how quickly the contents of the volume accumulate and dwindle.
For example, in the electrical case above, we can model a wire as a control volume. Charge is conserved, so the current leaving the wire equals the current entering the wire. However, some of the electrical energy that enters the wire is converted to heat by the resistance of the wire, and never reaches the other end. By the conservation of energy, that heat must either build up inside the wire (raising the temperature of the wire) or be shed into the surrounding air. Since physicists have worked out how these quantities relate, for a wire of a given size we can calculate the largest current (the “ampacity”) that the wire can carry before reaching the temperature that damages the insulation. An engineer then need only choose a breaker that cuts the power when the current exceeds the ampacity, before the wire reaches that temperature. Since the limit is placed on the current, and not on the device that happens to be drawing the current, the same breaker can protect the wire regardless of which devices are plugged in.
These types of quantities are in use beyond electrical engineering. For instance, Coast Guard rules require that the relieving capacity of the safety valves on a boiler be no less than the maximum generating capacity of the boiler. In finance, Federal Reserve rules require the largest banks to hold liquid assets equal to at least 1.0 times their net cash outflows under a prescribed stress. Note that the latter example is a human system, indicating that at least for some agentic systems we can devise safety criteria of this sort.14
14 For boilers, see US Coast Guard, 46 CFR § 52.01-120(a)(4). For banks, see Federal Reserve, 12 CFR § 249.10(a). Smaller covered institutions are subject to a reduced requirement, and many banks are not covered at all.
Dimensionless Quantities and Scaling Laws
What happens when a system is made larger? For many quantities, there are “scaling laws” (rules for how a quantity changes when the size of a system changes). Given a scaling law, a limit that has been established for one size of system can be extended to other sizes by calculation, without testing each size separately.
For example, a thicker wire has less resistance, and therefore generates less heat for the same current. A thicker wire also has more surface from which to shed heat. Since we know how both of these change with the thickness of the wire, the ampacity can be calculated for every standard size of wire, and we don’t need to test new wires. The Electrical Code tabulates the results. For instance, a No. 14 copper wire may carry 15 A, whereas a No. 8 copper wire (which is twice as thick) may carry 40 A.15
15 National Fire Protection Association, NFPA 70, National Electrical Code, Table 310.16 (values for copper conductors with 60 °C insulation in 30 °C surroundings).
Scaling laws are often expressed in terms of “dimensionless quantities” (ratios in which the units cancel, so that the value doesn’t depend on the units of measurement). Safety limits are often written in terms of such ratios, because the threshold for a ratio can be the same for systems of every size, even when the underlying quantities are not.
For example, in nuclear engineering the “multiplication factor” is the ratio of the number of neutrons in one generation of a chain reaction to the number of neutrons in the generation before. Since both numbers are counts of neutrons, the units cancel. A chain reaction dies out whenever this ratio is below 1. The value of the ratio depends on the amount and arrangement of the fuel, but the threshold of 1 is the same regardless of whether the fuel in question is a single assembly or a full storage rack. Therefore, Nuclear Regulatory Commission rules place the limit on the ratio directly (for stored fresh fuel, the multiplication factor “must not exceed 0.95”).16
16 US Nuclear Regulatory Commission, 10 CFR § 50.68(b)(2). For the definition of the multiplication factor, see US Department of Energy, DOE-HDBK-1019/2-93, Nuclear Physics and Reactor Theory, Vol. 2, Module 3.
Composition Rules
What happens when several systems are connected together? When control volumes are connected, the output of one is the input of the next, and so conserved quantities can be followed from one volume to the next. This gives us “composition rules” (rules for calculating a quantity for a whole system from the same quantity for each of the parts). Given a composition rule, a limit on a large system can be calculated from the limits on the individual parts of that system.
For example, since the current flowing into a junction equals the current flowing out of the junction, the currents on the individual circuits of a panel add. A limit on each circuit therefore gives a limit on the complete panel. By the same token, we can compute the limit on the building or the entire service. The Electrical Code requires that a feeder be sized for the sum of the loads on the circuits that the feeder supplies (with loads that run for three hours or more counted at 125%).17
17 National Fire Protection Association, NFPA 70, National Electrical Code, § 215.2(A)(1).
Worst-Case Bounds
Finally, if we know a “worst-case bound” (the most severe condition that a system could face), then safeguards only need to be designed for the worst case, and not for every condition individually.
For example, the available fault current can be calculated for each building, and so the Electrical Code requires every breaker to have an interrupting rating at least as large. A breaker chosen this way can cut the power under any short circuit that the utility’s lines are able to deliver. Likewise, the Nuclear Regulatory Commission limit of 0.95 described above must be met under the assumption that the storage rack holds the most reactive fuel and is flooded with water.18
18 National Fire Protection Association, NFPA 70, National Electrical Code, § 110.9 (interrupting rating). For nuclear fuel, see US Nuclear Regulatory Commission, 10 CFR § 50.68(b)(2).
19 Federal Aviation Administration, 14 CFR § 25.301(a) (limit and ultimate loads), § 25.303 (factor of safety) and § 25.305(b) (strength and deformation). The wings in the video are widely reported to have failed at 154% of the limit load.
Similarly, in aviation, Federal Aviation Administration rules define “limit loads” (“the maximum loads to be expected in service”), and require that the aircraft structure be able to support 1.5 times the limit load “without failure for at least 3 seconds”.19 To demonstrate this, in the video below an aircraft manufacturer pulls the wings of a test aircraft upward until the required load is reached, showing that the safety factor has been achieved.
Agentic Systems
What happens when we try to draw the same box around an AI agent? We can already measure some activity at the “connection” (the interface between an agent and the tools, resources, and other agents that the agent can reach). For instance, we can record which tools an agent may call, how much money or compute an agent may spend, or how many messages an agent may send.
However, what we don’t have is a way to relate these measurements to one another, or to the hazard. In a wire, the measurements are tied together by well-tested physical laws relating current, energy, heat, and temperature. For an agent, we don’t have comparable laws or conserved quantities.
For example, imagine we were concerned about collusion between a “worker” (an AI system performing a task) and a “monitor” (a second AI system that checks the worker’s results).20 One possible failure mode is that the monitor works alongside the worker to help conceal a failure. An inspector can already count how often an independent party audits the monitor’s reports. However, suppose two operators audit the same fraction of cases, and the audits of one operator are far more likely to detect concealment than the audits of the other. Should both operators be held to the same standard? At present we can’t say, because we don’t know how the frequency of audits, the chance that an audit detects concealment, and what the agents stand to gain from concealment relate to one another.
20 Lewis Hammond et al. (2025), “Multi-Agent Risks from Advanced AI,” Cooperative AI Foundation, Technical Report #1, arXiv:2502.14143.
If such quantities were available for agentic systems, they would support requirements of the same form as in other engineering fields. The table below gives a hypothetical requirement for each type of quantity or rule described above (terms in quotes are concepts for which we don’t yet have a formal definition).
| Type of Quantity or Rule | Hypothetical Requirement for Agentic Systems |
|---|---|
| Input and Output Balances | The “effects” that an agent produces must not exceed a calculated multiple of the “resources” supplied at the agent’s interface. |
| Accumulation | The “authority” that an agent has accumulated (for instance, through acquired credentials or additional copies of the agent) must not exceed a stated level at any point during a task. |
| Rates | A safeguard must be able to detect and reverse an agent’s actions more quickly than the agent can take them. |
| Scaling Laws | A limit established for one budget, or for a single copy of an agent, is extended by calculation to larger budgets and to many copies. |
| Composition Rules | The limit for a system of agents is calculated from the limits on the individual agents and on the channels between them. |
| Worst-Case Bounds | The operator must show that no strategy available to an agent under the stated access can carry the agent’s “effects” above the threshold for a specified harm. |
We also have no theory dictating what happens when several agents are connected together. Currents add at junctions, but with agents we have no similar guarantees on combination. Therefore, two agents that each remain within their own limits can still coordinate in ways that developers of each individual may not have intended, and so the effects of many agents may multiply in unexpected ways. Without a composition rule, we can’t certify a system of agents by certifying each agent individually.21
21 See Procopio (2026), Towards an Engineering Science for AI Safety, “Composition”. For a simple case of agents composed through a game, see Procopio (2025), Learning Game Equilibria.
Why can’t existing evaluations supply these quantities? Preece and Kennelly carefully measured the heating of wires in the 1880s, and yet the underwriters still arrived at a different limit for the same wire, because engineers lacked a theory of how a result obtained under one set of test conditions would carry over to another.
Similarly, a test of one agent under one set of conditions can’t tell us what to expect from a different model, a different task, or a different set of tools (notions like “situational awareness” make this even more difficult, as we may also have to contend with the incentive structures and other context around the test). This doesn’t mean that a good theory will render testing unnecessary (electrical ratings are still established by testing), but tests become useful once we have a theory that tells us which quantity is being measured and how that quantity relates to the hazard.
How to Search for Missing Quantities
How should we go about searching for such quantities? Each of the questions we asked about the box suggests a place to look, and we propose to ask the same questions of agentic systems.
For example, suppose you wanted to figure out what builds up inside a system. You could equivalently ask “What remains true of a system if the same system is run later rather than now?”22 Similarly, to find out how a hazard grows, we can ask what happens when a system is made larger, faster, or more numerous, and look for quantities that change by a fixed rule when a budget is increased, a tool is added, or an agent is copied more times.23 To find extremal quantities, we can assume the worst case or adversarial scenario, then look for quantities that never rise above a known ceiling.24 In each case, a useful test for a candidate quantity is whether the quantity stays the same when details that shouldn’t matter are changed, such as swapping one model for another or renaming the agents and tools.25
22 See Procopio (2025), Noether’s Theorem with Time, and Procopio (2025), Noether’s Theorem and Geometric Controls.
23 See Procopio (2026), Algebra and Allometry, and Procopio (2025), Dynamical Similarity and Equivariant Symmetry.
24 See Procopio (2026), Towards an Engineering Science for AI Safety, Appendix B, which describes a function that starts below a threshold and cannot increase.
25 See Procopio (2026), Invariant Coordinates for Normal-Form Games Modulo Strategy Relabeling, which constructs descriptions of games that are unchanged when the strategies are renamed.
26 See Procopio (2026), Survey of Classical Invariant Theory, for an introduction to the field, and Procopio (2026), Building a Minimal Computational Invariant Theory Library, for the algorithms.
27 See Procopio (2026), From Symmetry to Theory and Engineering Game Types. These provide constructions within declared representations, complexity bounds, game models, and intervention sets.
These questions can be tested systematically. There are mathematical methods for constructing quantities of each kind from a description of a system (and for determining whether a given list of such quantities is complete).26 The mathematical details are described in a separate technical proposal.27 The goal of this project is to use a combination of computer algebra systems, AI, and mathematics to search for the relevant laws, if they exist.
The claim of this proposal is that quantities of the kinds described above can be searched for and verified by computation. So far, this has been demonstrated only on small examples, which are worked in full in the technical posts.
- A verified safety factor for a neural policy. For three small neural network policies, a solver verifies a safety factor of exactly 2. Errors in what the policies observe of up to twice the stated tolerance can’t push the system out of the required regime, and any larger error can.
- A test of whether measurements suffice. Two possible games differ only in the sign of a single three-way interaction. Every second-order measurement of the two games agrees, although the games require different interventions (a command of 0 and a command of 2). The procedure verifies that no single intervention works for both, and then finds a third-order measurement (with values of \(+5/192\) and \(-5/192\)) that tells the two games apart.
- A certificate that the available controls can’t succeed. When a controller may only adjust pairwise terms, the three-way quantity that the target depends on stays fixed at \((1/6, 0, 0)\) regardless of the budget, and so the target is certified as unreachable. The procedure then verifies that neither of two additional controls is sufficient alone, and that both together are.
- A check that the software recovers a measured result. To test the software on a case where the answer is already known, we gave the software the symmetries of the crystal CrSb, but not the formula that experiments had found for how the electron energies in the crystal split. The software recovered the structure of that formula (the order at which the splitting first appears, and the four planes on which the splitting vanishes). This is a partial reproduction, since the size of the effect must still come from experiment.
Each example involves a handful of individuals, payoffs supplied by hand, and exact arithmetic. None is a result about a deployed AI system, and we don’t yet know whether the methods scale.
Returning to the worker and the monitor, the relevant measurements all sit at the boundaries of the two agents, such as what each agent can observe, what the agents can communicate, how the agents are rewarded, and how often (and how reliably) an independent party audits the monitor’s reports. If successful, we’d be able to derive a limit that says “for this arrangement of rewards and communication channels, concealment can’t be profitable for either agent as long as audits occur above some calculated rate”.28 Note that such a result would be conditional on the stated incentives. Concealment would cease to pay, even though concealment would remain possible (excluding concealment under every strategy that the interfaces permit is a harder problem, which the technical proposal treats separately). Nevertheless, an inspector could then verify the arrangement and the audit rate without examining either model internally, just as an electrical inspector reads the wire number and the number on the breaker.
28 See Procopio (2026), Engineering Game Types, which determines which interventions on a game’s parameters produce which types of game, and Procopio (2026), Games, Invariants, and Alignment, which considers how a classification of games could guide changes in incentives.
Possible Limits
There are several reasons this approach might fail. We consider each objection below, along with some possible responses.
First, one might object that agents are adaptive, whereas wires are not. A wire can’t study the Electrical Code in search of a way around the breaker, but an agentic system will often adapt to whatever rules are imposed.
If this objection holds, quantitative limits could only ever work for passive equipment. However, we already impose quantitative limits on adaptive parties. Consider bank regulation. Banks are run by people who closely study the rules, have strong incentives to hold as little idle cash as the rules permit, and rearrange their holdings in response to each new requirement. Nonetheless, bank liquidity requirements are stated in terms of a “liquidity coverage ratio” (the ratio of a bank’s readily sellable assets to the cash the bank would be expected to lose over thirty days of financial stress), with a prescribed method of calculation and a duty to notify the supervisor if the ratio falls below the minimum.29 The rule specifies which assets count and how the outflows must be estimated, so that the limit holds regardless of the strategy the bank chooses. Since these quantities come from accounting rather than physics, the approach doesn’t necessarily require the regulated party to be passive.
29 Federal Reserve, 12 CFR § 249.10, § 249.30, and § 249.40. The requirement applies to covered institutions, and does not extend to every U.S. bank.
Second, such quantities may not exist for agentic systems. Agents may be too varied to admit common quantities, or the quantities that do exist may require a nearly complete description of a model in order to be calculated, in which case inspectors would be back to examining systems case-by-case.
This would be bad, and we can’t rule this possibility out. However, we believe the search is worth the effort regardless. Furthermore, the Electrical Code was not derived from a single quantity covering every electrical hazard. Overheated wires, short circuits, and ground faults are all regulated by separate quantities and separate limits (ampacity, interrupting rating, and trip threshold, respectively). Because we are planning to build a procedure for finding rules of these types, we believe we are likely to find rules that provide at least partial coverage for limited families of systems.
Third, a limit could be derived correctly and still depend on assumptions that fail for deployed systems. The description of a system might omit a route to harm (much as the 1881 rules omitted fault currents), or the limit might assume that agents make decisions in a particular way.30
30 See Procopio (2026), Towards an Engineering Science for AI Safety, “Risks”.
Electrical rules face the same problem, and address the problem in two ways. First, limits are enforced at the connection and assume nothing about the device (a breaker cuts power regardless of what the device behind the breaker is doing). Likewise, a limit on an agent’s access to tools or resources that is enforced outside the agent would hold whether or not the agent behaves as expected. Second, rules state the conditions under which a limit was derived, so that an inspector knows what to recheck when those conditions change (for instance, when an agent is given a new tool or connected to another agent). Admittedly, none of this protects against a route to harm that nobody thought to include, and the Electrical Code has been revised many times after failures for the same reason.
Finally, the derived limits may be too conservative to be useful. If the only limits we can justify are so low that agents can’t perform useful work, developers will have good reason to resist the requirements. However, some conservatism is normal for safety codes: the 1894 Underwriters’ rules permitted a No. 10 wire to carry less than half the current that Kennelly’s measurements suggested the wire could bear, and yet a wire limited to 20 amperes could still light a building. Moreover, a limit that falls below what is already deployed is still useful, since the limit tells an auditor or insurer specifically why a system falls outside the safe margin, and tells engineers what would have to change in order to bring the system back within the margin.31
31 See Procopio (2026), Towards an Engineering Science for AI Safety, Appendix C, which discusses insurers, auditors, inspectors, and regulators as users of an operating limit.
Conclusion
Electrical safety rules only became something an inspector could check by reading a few numbers after engineers had identified the quantities that govern electrical hazards. AI safety regulation is currently at the earlier stage: we can test individual systems, but we can’t yet state limits that hold across systems, or explain why a given limit should preclude a given failure. In this proposal, we have suggested searching for the quantities that would make such limits possible. We don’t know whether this search will succeed, as agentic systems may not admit quantities as well-behaved as current. However, if comparable quantities exist for AI systems, then finding them is a prerequisite for the kind of regulation that other engineered systems already have.
AI Disclosure
AI was used to help draft parts of this essay, and to research many footnotes and citations (especially legal and historical).