The Anatomy of Optimisation

Yesterday, mathematicians Steven Strogatz and Alex Townsend argued in The Guardian that optimisation can make systems less robust. Their examples range from hospital beds to traffic roundabouts and financial markets.¹ If optimisation is simply the process of finding the best solution, how can it go wrong in so many different ways?

»Any optimisation process has at least four components. An objective function specifies what is being optimised. A search space defines which possibilities can be considered. A move rule determines how the search proceeds. A stopping rule decides when to stop searching«

Strogatz and Townsend define optimisation as the pursuit of efficiency: minimising or maximising something. They identify »at least three traps to beware of in the relentless pursuit of optimality«. But these are not three versions of the same problem. They point to different parts of the process.

Any optimisation process has at least four components. An objective function specifies what is being optimised. A search space defines which possibilities can be considered. A move rule determines how the search proceeds. A stopping rule decides when to stop searching. Each of those also raises a question of governance: who gets to make the decisions, and who has to live with the consequences?

Their hospital example illustrates the stopping problem. Capacity is often planned around average occupancy, which looks efficient until demand surges and no beds remain. A hospital cannot be planned for the average day when it must also cope with the exceptional one. Safety researcher Erik Hollnagel distinguishes between eliminating known failures and building systems that can adapt to conditions nobody anticipated. Hospitals need both.²

The roundabout example illustrates a different trap: a poorly specified objective function. Milton Keynes’s roundabouts improve traffic flow but make coach journeys unpleasant for passengers. Traffic flow is easy to measure; passenger comfort is not. Engineers optimised what they could measure, overlooking what travellers experience. The optimisation succeeds on terms that exclude an important part of the picture. Who, then, gets to decide what counts as a good outcome? Political scientist Elinor Ostrom’s research on shared resources showed that giving affected people a role in making and revising rules can produce more durable arrangements.³ Optimisation is never entirely neutral. Someone decides what matters; someone else lives with the result.

The third trap appears when attempts to optimise collide – the trap here is less about any one part than who answers for all of them together. During the 36-minute 2010 Flash Crash, automated trading systems reacted to one another so rapidly that individually rational actions helped produce a market-wide collapse. What made sense for each trader did not make sense for the market as a whole. Circuit breakers, which halt trading during extreme movements, impose limits on this collective behaviour. Unlike Ostrom’s communities, whose participants can shape the rules, financial markets depend on external authorities to intervene when the system works against itself.

Two further traps lie in the search itself. The first is looking in too narrow a space. In 2020, researchers used machine learning to identify halicin, a known drug candidate whose antibacterial properties had gone unnoticed. Its structure differed from conventional antibiotics, showing what can be found by extending the search beyond familiar compounds.⁴

The second is how the search proceeds. In 2024, researchers showed that AI models trained on data generated by earlier models progressively lose information about the original distribution. Rare features disappear; repetition replaces exploration.⁵ Kenneth Stanley and Joel Lehman make a related point: searching for measurable progress towards a predetermined goal can prevent the discovery of possibilities nobody knew to seek.⁶

Strogatz and Townsend conclude that the challenge »is not simply finding the best answer. It is deciding what “best” really means – and knowing what not to optimise away«. But the anatomy of optimisation is larger: what we seek, where we look, how we search, when we stop, and who decides.

Architecture makes the consequences unusually clear. A trading algorithm can try again a millisecond later; a building can’t. Structural safety factors, fire escapes, spare capacity and provisions for future adaptation aren’t inefficiencies to be optimised away, but acknowledgements that some decisions can’t be reversed.

Architects have to ask what they might be wrong about before the concrete sets. Everything else gets to find out later. Much of it never does.

References