Back to Articles
What optimization theory, evolutionary biology, competitive markets, and machine learning all predict — and why the answer is the same
---
Those who follow Dharma AI already know that we view specialization as one of the defining principles of effective AI systems, shaping everything from cost and performance to reliability and sovereignty. Few papers have articulated that case as rigorously as the 2026 work by Goldfeder, Wyder, LeCun, and Shwartz-Ziv.
In this article, we explore and interpret ideas from AI Must Embrace Specialization via Superhuman Adaptable Intelligence (Goldfeder, Wyder, LeCun, & Shwartz-Ziv, 2026). The paper's convergence case — spanning optimization theory, biology, organizational economics, and machine learning — provides both the evidential structure and the intellectual foundation for the discussion that follows. The framing, organization, and editorial synthesis presented here are Dharma's.
---
The conventional expectation is reasonable: as AI systems grow more capable, they should also grow more general. Greater capability and broader applicability seem like natural companions — more resources, better methods, and expanded training should produce systems that approach more tasks with increasing confidence.
The pattern that actually appears is different. The systems that achieve the most significant results in any given domain tend to be the ones most narrowly focused on it. The breakthrough in protein structure prediction came from a system engineered for a single scientific task. The historical milestones of AI, examined closely, reflect intense domain targeting rather than expanding generality.
This pattern recurs. It recurs across domains, across decades, across architectural choices that have almost nothing in common. A pattern this consistent suggests a common cause — one that does not originate inside AI research at all.
---
An Algorithm Wins by Fitting Its Target
In 1997, Wolpert and Macready proved something that rarely surfaces in discussions of AI architecture: no single, general-purpose optimization algorithm outperforms all others across all possible problems (Wolpert & Macready, 1997). The proof is mathematical, not philosophical. Averaged across every conceivable problem a learner might face, every algorithm performs equally well — and equally poorly. An algorithm that gains on one distribution of problems necessarily concedes on others. The performance is redistributed, not multiplied.
The practical implication is direct: “an algorithm wins by being a good fit for the target problem” (Goldfeder et al., 2026). The theorem does not say generality is impossible — it says generality is not a performance advantage. The consistent structural path to outperformance is concentration: trading breadth for fit.
This becomes sharper when finite resources enter the picture. Any real system operates under constraints — finite compute, finite data, finite development time. Given finite energy, an approach that directs available resources toward learning a finite set of tasks will outperform one that distributes those same resources across an unlimited range. The arithmetic is unforgiving: as the task set expands without bound, the resources available per task shrink toward zero. Universal coverage and meaningful performance are, under finite resources, in direct tension.
The conclusion the theorem points toward is not that generality is bad. It is narrower and more operational than that: as the paper states, "universal generality is a theoretical concept, but in practical terms it is a myth" (Goldfeder et al., 2026). What survives contact with real constraints is not the system that tries to do everything — it is the system that fits its target.
The mathematics establishes this as a prediction, not a preference. Whether that prediction holds in the world beyond optimization theory is a different question.
---
What Biology and Markets Already Know
Two other domains arrived at the same prediction before optimization theory gave it a name.
As the paper describes the biological case: every performance gain in one niche comes at a cost elsewhere. A generalist carries traits suited to many environments but optimal for none — competence spread too thin to dominate any particular condition. There are no performance gains without trade-offs; the resources invested in one capability are unavailable for another. Selection favors designs matched to local conditions over those optimized for uniform coverage across all possible environments. The organisms that survive to reproduce are not the most generally capable — they are the most specifically matched. The result, accumulated over evolutionary timescales, is not generalists dominating — it is specialists filling niches. As the paper states: "Specialization is not an accident of biology; it is a predictable consequence of limited resources, competing objectives, and environments that reward performance on a small subset of evolutionarily relevant challenges" (Goldfeder et al., 2026).
Competitive markets follow the same dynamic through different means. Organizations and strategies that fail to meet performance thresholds are eliminated — not through extinction, but through exit, defunding, and replacement by better-matched alternatives. Competition acts as a selection mechanism: it amplifies effective strategies and eliminates ineffective ones. The mechanism has nothing in common with biological selection — no inheritance, no mutation, no evolutionary timescale. The unit of selection is not the organism but the organization, the product, the strategy. Yet the structural pressure is the same: finite resources, performance requirements, and the systematic removal of entities too broadly distributed to excel where it counts. Concentrated capacity outcompetes distributed capacity when performance standards are clear and consistent.
Evolution and markets operate through entirely different mechanisms — different timescales, different units of selection, different inheritance mechanisms. Yet both produce the same outcome under resource pressure: fit over breadth. The theorem predicts this. Biology and markets arrive at it independently. When a third domain arrives at the same finding through different means entirely, the pattern ceases to look like a theorem and begins to look like something more general about how constrained systems behave.
---
Machine Learning Keeps Rediscovering Specialization
The same pattern has emerged inside machine learning — not derived from optimization theory, but arrived at through the accumulated experience of building systems and watching what improves them.
The clearest form is negative transfer: a measurable degradation that occurs when a system trained on multiple tasks suffers because those tasks compete rather than cooperate (Ruder, 2017). When tasks share structure, training together helps. But when tasks compete for representational capacity, or impose conflicting gradients during training, performance on individual tasks falls below what a dedicated system would achieve. The gain from breadth becomes a cost to depth. It is a documented consequence of dividing finite capacity across tasks that pull against each other. The specialist, facing no such competition, does not pay this cost.
The architecture of frontier models offers a different form of evidence. Mixture-of-experts systems achieve their breadth not through uniform generality across all parameters, but by routing each input to a specialized subset of the network — activating different experts for different tasks. The paper's authors read this as a structural concession: a system designed to be general achieving its results by recovering specialization internally. This is an argued interpretation, not a demonstrated theorem — these architectures were designed for computational efficiency, and what they imply about generality's limits is a reasonable inference rather than a stated intent. But it is a notable one: the most capable general-purpose systems reach their performance by doing internally what specialist systems do by design.
The clearest historical example follows the same logic. AlphaFold achieved a step change in protein structure prediction by targeting that specific task with task-specific architecture and training choices (Jumper et al., 2021). Its gains came from narrower focus, not broader coverage. The paper uses AlphaFold as an archetypal case — not as evidence that all specialized systems achieve equivalent gains, but as an unusually clear illustration of the mechanism. That mechanism has appeared repeatedly: the history of AI milestones, the paper notes, frequently reflects intense domain targeting rather than broad competence, even when the results look like demonstrations of general intelligence.
Three distinct places. Three different mechanisms. The same finding.
---
What Scaling Doesn't Change
The picture would be incomplete without addressing one of AI research's most cited observations. Sutton's Bitter Lesson holds that methods relying on domain knowledge are consistently outperformed by methods that scale computation (Sutton, 2019). On its face, this appears to complicate the case for specialization: if scale and generality win, perhaps specialization is only a useful heuristic under resource constraints that will ease as compute becomes cheaper.
The objection rests on a conflation between two distinct concepts. Domain knowledge refers to hand-coded features, engineered priors, and rules designed to give a system insight into a particular area. The Bitter Lesson targets this — and it is correct to do so. Systems that encode explicit domain knowledge have been consistently outperformed as scale increases.
Domain specialization is different: the decision to direct a system's resources, architecture, and training toward a bounded set of tasks rather than distributing them broadly. This is not the encoding of knowledge about a domain. It is a decision about scope.
The paper draws the distinction precisely:
"The diminishing usefulness of domain knowledge is distinct from the usefulness of domain specialization. As scaling progresses, we will need to know less about proteins to build a system that does protein folding; however, such a system still benefits from focusing specifically on proteins." (Goldfeder et al., 2026)
Scaling changes what systems can learn from data. It does not change whether concentrating resources on a finite task set outperforms distributing them across an unlimited range. The Bitter Lesson and the specialization argument operate on different dimensions — one describes how knowledge should be acquired, the other describes what a system should be pointed at. Both can be true simultaneously. Scaling changes the mechanisms by which systems learn; it does not dissolve the constraint that makes fit more valuable than breadth.
---
Across four analytical traditions, the same pattern emerged through different paths. This is not a coincidence that demands explanation. It is the evidence.
When finite resources meet selection pressure — in an optimization problem, an ecosystem, a market, or a training run — fit consistently beats breadth. The specific mechanisms differ. The timescales differ. The units of selection differ. But the structural dynamic is the same, and it produces the same result.
The theorem does not cause this pattern in biology. Biology does not cause it in markets. Neither causes it in machine learning. They all face the same underlying constraint: performance under scarcity requires concentration. What the theorem establishes mathematically, evolutionary history confirms empirically, competitive markets demonstrate institutionally, and machine learning rediscovers architecturally.
Specialization is not a preference. It is what emerges when finite resources meet the requirement to perform.
---
If you're evaluating how domain focus affects AI performance in your organization — or building the case internally for a specialization strategy — we'd like to hear about your context. Get in touch with Dharma AI.
---
Primary Source
- Goldfeder, S., Wyder, M., LeCun, Y., & Shwartz-Ziv, R. (2026). AI must embrace specialization via superhuman adaptable intelligence. arXiv:2602.23643.
Sources
Wolpert, D.H. & Macready, W.G. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67–82.
Forister, M.L., Novotny, V., Panorska, A.K., Baje, L., Basset, Y., Butterill, P.T., & Dyer, L.A. (2012). Global distribution of diet breadth in insect herbivores. Proceedings of the National Academy of Sciences, 109(2), 418–423.
Futuyma, D.J. & Moreno, G. (1988). The evolution of ecological specialization. Annual Review of Ecology and Systematics, 19, 207–233.
Hannan, M.T. & Freeman, J. (1977). The population ecology of organizations. American Journal of Sociology, 82(5), 929–964.
Loasby, B.J. (1983). Knowledge, learning and the firm. As cited in Goldfeder et al. (2026).
Ruder, S. (2017). An overview of multi-task learning in deep neural networks. arXiv:1706.05098.
Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120), 1–39.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., & Hassabis, D. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596, 583–589.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., & Hassabis, D. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419), 1140–1144.
Sutton, R.S. (2019). The bitter lesson. Retrieved from http://www.incompleteideas.net/IncIdeas/BitterLesson.html
---
Further Reading
---
Explore Dharma AI on Hugging Face to try our interactive demos, download our open-source models, and discover how specialized AI systems outperform general-purpose models in real enterprise applications.
Instagram Engineering
Shopify Engineering