3

There's a pattern that pops up in a wide variety of systems - biological, economic, social, etc - and can occur whenever survival and propagation within the system is determined by how well members within it embody some particular quality. Probably the most well-known example is evolution, where species survive to the extent that they are capable of reproducing. This eventually leads to species like the black-tailed antechinus, where the male literally f**ks itself to death during mating season. In short, evolution may incidentally produce beautiful species or happy species incidentally, but leave the system running long enough and it will eventually sacrifice these qualities in species if doing so gives them a reproductive edge. Non-biological examples include businesses being selected for ruthless focus on profit which end up producing worse goods and services for their customers or satisfaction for their employees, and religions which focus mainly on increasing and retaining membership at the cost of spiritually enrichment for those involved.

The only place I've managed to find it discussed is this blog, where the author calls it "the multipolar trap". But that appears to be just their name for it, as searching for this term mostly just yields references to the same blog.

It seems similar to tragedy of the commons, prisoner's dilemma, and other sad phenomena that emerge from basic game theory. But none of those quite capture this pattern where the optimizing principle of the system favors the emergence of members which cast aside other, desirable qualities in favor of the one the system actually selects on.

6
  • "Paperclip maximizer" is a term I've heard, though it does tend to revolve around the hypothetical AI that would sacrifice everything to the manufacture of paperclips.
    – Mary
    Dec 8 '21 at 0:30
  • 1
    Your examples do not fit the phenomenon you describe. The point of the prisoner's dilemma and the tragedy of the commons is that individually optimal choices lead to collectively inferior outcome, there is nothing that the "system" optimizes in those cases. The same is true of evolution, it is typically a game of good enough. Even assuming that changing environment does not shift the "optimum" for a long time natural selection still does not achieve it. Species persist if their adaptations are enough to survive, even if they are suboptimal.
    – Conifold
    Dec 8 '21 at 5:51
  • 2
    Instrumental convergence, "an intelligent agent with unbounded but apparently harmless goals can act in surprisingly harmful ways", paperclip maximizer is a popular example.
    – Conifold
    Dec 8 '21 at 5:57
  • One term to describe that phenomenon is the maladaptive schema. Dec 10 '21 at 3:26
  • Added tags like evolution since selection is present in the question and complexity related to systems thinking.
    – J D
    Dec 10 '21 at 3:46
1

You are referring to a perverse incentive, "an incentive that has an unintended and undesirable result that is contrary to the intentions of its designers."

Also under the heading of Goodhart's law:

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

Or, expressing similar sentiment, Campbell's law:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

1

Caveat

As Ted Wrigley noted, there's a difference between a system that develops to have a clash in values between the part and the whole, and one that develops parts that might lead to the failure of the whole. Let's try both.

Short Answer

Is there a name for the phenomenon where a system that selects for one quality eventually leads to optimizing that quality at the cost of others?

Yes. It's called specialization. A person who spends all of their time designing databases stands to benefit from the supply-demand curve, but might find himself homeless if the tech-bubble quickly bursts and lacks the social skills to get help from others. But you seem interested in particular cases in specialization where said specialization leads to unintended/collateral negative consequences. Your examples are cast wide, so we'll examine a few cases to suggest some terms.

Long Answer

Specialists, Generalists, and Fragility

In systems thinking from an ecological perspective, the best terms might be fragility/antifragility.

A fragile system is one that develops in such a way that certain portions of a system specialize in a way to open up the system as a whole to failure, particularly through black swan events. Instrumental convergence is nothing but puffery to characterize AI systems that avoid specialization and hedge their bets using a variety of general-purpose strategies. From WP quoting Bostrom:

Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent's goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by a broad spectrum of situated intelligent agents.

Going back to the example of a database programmer, the adage cited is "The jack of all trades is a master of none, but oftentimes better off than a master of one."

In general systems theory championed early by Bertalanffy, and added to by Nassim Nicholas Taleb, the notion that a system might evolve in such a way to open the system as a whole to failure starts with the example of how it's easy to believe all swans are white, only to find out that such a claim is a failure with the advent of the discovery of a single black swan raising the specter of the scandal of induction. The black swan example is practically an idiom in epistemology.

Black swan theory is largely epistemological in so far as it seeks to characterize systems of knowledge. From WP:

Taleb's problem is about epistemic limitations in some parts of the areas covered in decision making. These limitations are twofold: philosophical (mathematical) and empirical (human known) epistemic biases. The philosophical problem is about the decrease in knowledge when it comes to rare events as these are not visible in past samples and therefore require a strong a priori, or an extrapolating theory; accordingly predictions of events depend more and more on theories when their probability is small. In the fourth quadrant, knowledge is uncertain and consequences are large, requiring more robustness.

So, in this context, an optimization in game-theoretic terms, might make a system outperform in one domain, and be susceptible to failure in another. Cognitive biases and informal fallacies are an example of this according to behavioral economist Daniel Kahneman, the main thesis of his Thinking, Fast and Slow where he defends a perspective of the human brain and mind based on evolutionary psychology. To wit:

The book's main thesis is that of a dichotomy between two modes of thought: "System 1" is fast, instinctive and emotional; "System 2" is slower, more deliberative, and more logical. The book delineates rational and non-rational motivations or triggers associated with each type of thinking process, and how they complement each other,

The usual explanation rests in the intuition of a human on the plains of Africa whose brain has to choose between paranoid and alive, and logical and eaten when the brushes are rustling and possibly populated with hungry lions. That same paranoia today might be counterproductive since human paranoia combined with in-group-out-group thinking might lead to war and death of humans who might be better off collaborating.

Competition, Cooperation, and Coopetition

In biological systems that develop to preserve some genes over others or the organism, is related to selfish gene, biological altruism, and reciprocal altruism. Sociobiology suggests that psychological altruism is a consequence of the biological sort.

In your business example, you might conceive of a system developing so that the system collapses to benefit a part of the system. When corporate profits are put above environmental sustainability (externalizing costs and Tragedy of the Commons), a business may destroy the source of it's profit, such as the eradication of the environment, as in Dr. Seuss's The Lorax, or it may destroy it's customer base by maximizing short-term profits for shareholders and destroying the company, a phenenomeon related to the golden parachute:

A golden parachute is an agreement between a company and an employee (usually an upper executive) specifying that the employee will receive certain significant benefits if employment is terminated. These may include severance pay, cash bonuses, stock options, or other benefits. Most definitions specify the employment termination is as a result of a merger or takeover,15 also known as "change-in-control benefits",6 but more recently the term has been used to describe perceived excessive CEO (and other executive) severance packages unrelated to change in ownership

In these cases, systems clearly specialize in direct relation and at the expense of parts, wholes, collaborators, or competitors. It might be possible to categorize this as some unified class of parasitism-predation. From WP's article on parasitism:

The entomologist E. O. Wilson has characterised parasites as "predators that eat prey in units of less than one".2

Summary

Specialization has three manifestations that can lead to negative outcomes in some regard. On the one hand, a system might specialize allowing it to perform optimally under one set of conditions, and fail under another. This is fragility. The second manifestation of specialization is when in the part-whole relationship, the specialization of one part benefits or externalizes expenses in relation to other parts. This is altruism and parasitism. And if it's between whole systems, the terms altruism and predation apply. Is there a single term to describe these various exemplars? I'd say corporate employment, but I don't think that's the term you're looking for.

0

There's a category error invoked here that you ought to be aware of. There are two basic dynamics that you're referring to indiscriminately:

  1. The tension between specialization and adaptability which is a common feature of biological evolution. The first is a tendency to maximize utilization of best available resources, while the second leans towards sub-optimal utilization of a wide variety of resources. The first is an optimal strategy in a stable environment; the second an optimal strategy in a variable or changing environment.
  2. The tension created by value-preferences within human society. These are an assortment of cognitive dichotomies: good/evil, right/wrong, better/worse, moral/immoral, etc. As with the first point, there are different strategies: some people seek out long-term values, some short-term, some settle for sub-optimal individual values to create collective values, etc. Acculturation and socialization play large roles in mediating these strategies.

However, the fact that these two general dynamics seem parallel does not imply that they are equivalent or interrelated. Evolution is value-free. The fact that (say) female praying mantises cannibalize their mates, or that male ducks attack and force sexual activity on females, isn't intrinsically bad or good except in the human mind. Mantises and ducks are not happy or unhappy with these conditions; that is simply what they do, perfectly normal, with some (perhaps deeply) historical reason why that behavior allowed those species to thrive and persist. But value-preference dynamics are an entirely different order of activity. Value-preferences do not strive to reach some optimal strategy within an established environment; value-preference dynamics strive to coerce the environment into an optimal state for some optimal state for a (perceived) optimal strategy.

I think the best term you're going to find for this concept is path dependence, though the term doesn't normally imply the negative value judgement that you are reaching for. The basic idea is that committing to a particular path lowers the costs of continuing and raises the costs of switching paths. For instance, in the earliest days of the automobile there was a competition between internal combustion engines, steam-powered engines, and (yes) electric engines. Steam-powered engines were an already-established technology, and quite efficient, but had shortcomings for personal use (for instance, a long start-up period). Electric vehicles were hampered by the poor battery technology or the late 19th century, and only found effective use on trams or trolleys that could be wired to external power sources. internal combustion engines, though dirty and inefficient, maximized convenience and utility, and so took off. After that, a range of industries shot up to provide the necessities for IC engines, and the social and economic costs of switching back to steam or electric vehicles became progressively more steep.

Part of the reason the linked author points at capitalism as a problem is that capitalism — at least in Rightist ideology — tries to reduce value-preferences to mere questions of specialization and adaptability. Hard-line capitalism is only interested in optimizing success strategies, but as with evolution that creates a value-free dynamic in which any outcome (even the extinction of the species itself) is morally neutral.

6
  • +1 bc you're calling out the contours of the analogy. Hmmm. Evolution is built on natural selection, which along genetic lines, seems to imply teleological expression of genes; wouldn't any disposition towards an end imply a value-preference? Or is your contention that the purpose-driven architecture of proteins is only anthropomorphization, and that dispositions of physical structure, be they organs or snowflakes, lack value bc value in a literal sense is constrained to byproducts of agency?
    – J D
    Dec 10 '21 at 3:53
  • @JD: Teleology implies that means are ends driven: that the system has goals 'in mind' and pushes to have those goals met. Natural selection is (ostensibly) more along the "throw stuff at the wall" model: throw a bunch of stuff at the wall; see what sticks; toss out what doesn't stick; rinse and repeat with the stuff that's left. Dec 10 '21 at 4:20
  • @JD: You can think of evolution a bit like traffic. No one tries to make traffic. Traffic is what happens when a whole bunch of people in cars compete for the limited resource or roadways. If we didn't control traffic the way we do (no stoplight, lanes, right of way, etc), competition would get wild, people would die, and the ones who were left — the best at navigating lethal traffic — would go on to breed a new generation. Would we be trying to breed a generation of stunt car drivers as a goal? No, that's merely what would happen over time. Dec 10 '21 at 4:25
  • @JD: I could make a serious argument that we are creating environmental pressures which give advantages to two groups: people with autistic tendencies (highly intelligent and focused people with poor socialization and communication skills, capable of doing sophisticated autonomous tasks; think Elon Musk and Mark Zuckerberg) and people with psychopathic tendencies (emotionally stunted individuals incapable of seeing others as completely 'real', who treat all social interactions as 'games' that they are determined to 'win'; think Jeff Bezos and most of the top Trumpists)... Dec 10 '21 at 4:33
  • @JD: does anyone think that is an ideal goal for the human race? Dec 10 '21 at 4:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.