The last common ancestor (LCA) of chimpanzees and humans lived in a dominance hierarchy: high-ranking bullies used strength to assert control over scarce resources, and lower-ranking victims submitted deferentially (Boehm, 1999). Yet, sometime between the LCA (6–7 million years ago) and Homo sapiens (300,000 years ago), our hominin ancestors transitioned from dominance hierarchy (DH) to a society in which bullies were actively suppressed (Dubreuil, 2010). This transition to a reversed dominance hierarchy (RDH) separates our society from that of other great apes and was a major step in the evolution of the uniquely extensive social cooperation of humankind (Sterelny, 2021). Although human societies have since developed new forms of hierarchy, understanding this initial reversal of DH is critical to understanding the origins of human cooperation.

Because weak challengers lacking weapons can only defeat bullies by combining their strength, understanding the RDH transition requires us to identify conditions that favored the evolutionary selection of anti-bullying coalitions. However, many factors worked against the formation of coalitions among the earliest hominins. Early hominins had limited cognition and communication, thus restricting their ability to plan and coordinate challenges. Bystanders could also refrain from helping, so challenging came with the risk of receiving no help and suffering severe harm. How could bully suppression have evolved under these conditions?

This paper presents a theoretical analysis of the RDH transition among early hominins facing these conditions. It draws from the archaeological and anthropological literature to construct a stylized model of early-hominin bullying. It then uses formal mathematical and numerical analysis to identify the evolutionary path of an RDH transition and the parametric configurations that generate that path. Finally, it reassesses the timing of the RDH transition in light of both the model and the literature on hominin evolution during the Miocene, Pliocene, and Pleistocene epochs.

Ultimately, the paper makes two contributions to the literature on the RDH transition. The first is a set of new theoretical insights into the RDH transition. The analysis reveals how the RDH transition depends on the costs and benefits of helping a challenger. Paradoxically, an increase in bullies' extraction capabilities can spark this process by giving helpers and challengers less to lose by working together against bullies. Evolutionary analysis also shows that DH is sufficiently stable and that an additional factor is needed beyond large fitness benefits of helping in a challenge against a bully. A just-right level of evolutionary drift provides this extra ingredient. Finally, the analysis finds that the RDH transition will go through a transitional phase characterized by intense violence before RDH is realized: as the rate of helping increases, challenging becomes more widespread, violence replaces submission as the common response to bullying, and the rate of conflict peaks as bullies are violently suppressed by coalitions. Evolutionary selection then disfavors bullying as a strategy, and the RDH transition is completed.

Beyond these technical findings, the insights from the model also inform our understanding of the timing of the RDH transition. In particular, by calibrating the parameters of the model to fit the conditions faced by early hominins, we gain insight into when the RDH occurred. According to this calibration, the conditions faced by the first hominins right after the LCA were not conductive to an RDH transition. Later developments should have triggered an RDH transition by the time of Homo erectus (1.8 million years ago), but an earlier transition is also possible. According to the analysis, the emergence of weapons provides the strongest trigger for the RDH transition (Bingham, 1999, Bingham, 2000), yet the analysis reveals that a weapons-free RDH transition is also possible given other evolutionary developments.

The literature on human cooperation is large, spans multiple disciplines, and offers several potential explanations for this distinctive aspect of human social life (Axelrod, 1984; Bowles & Gintis, 2011; Nowak & Highfield, 2011). The first influential explanation was a disposition to help one's kin (Hamilton, 1964a, Hamilton, 1964b), but another explanation is needed to explain the non-kin coalitions in RDH. A preference to conform to social norms can suppress bullies (Boyd & Richerson, 2005; Heath, 2011), yet these preferences probably did not reach their modern forms until later in the Homo lineage, perhaps only within the last 120,000 years (Sterelny, 2021). The RDH transition thus most likely occurred with simpler predispositions.

The anti-bullying dispositions studied here are a form of reciprocity. Direct (Trivers, 1971), indirect (Alexander, 1979, Alexander, 1987), and strong (Fehr et al., 2002; Gintis, 2000) reciprocity play important roles in human cooperation, but my focus is on the evolution of two distinct reciprocity dispositions that specifically concern bully suppression: a disposition to challenge a bully and a disposition to help form a coalition against a bully once a challenge has been initiated. One motivation for this focus is that the RDH transition is an early and specific innovation in cooperation that enabled the later emergence of more extensive forms of cooperation (Dubreuil, 2010; Sterelny, 2021). Another motivation is that the cognitive limitations of early hominins limited their ability to draw on stronger enablers of cooperation (Kurokawa & Ihara, 2017) including language (Bishop & Lerch, 2023; Smith, 2010) and norm conformity.

Considerable attention has been given to modeling coalitions more generally, with significant attention given to modeling coalition formation (Bissonnette et al., 2015; Mesterton-Gibbons & Sherratt, 2021).1 Economists have developed models of endogenous coalition formation for human interaction (Sánchez-Pagés, 2007; Skaperdas, 1998; Tan & Wang, 2010), and models have also been used to study animal coalitions (Connor & Whitehead, 2005; Mesterton-Gibbons & Sherratt, 2009; Noë, 1994; van Schaik et al., 2004; van Schaik et al., 2006; Whitehead & Conner, 2005). Other work uses formal evolutionary models. For example, the formation of coalitions depends on an investment cost and relative strengths (Mesterton-Gibbons & Sherratt, 2021), and cultural inheritance can spur the emergence of large coalitions (Gavrilets et al., 2008). Gavrilets (2012) shows how an RDH transition can occur when bullies impose sufficiently negative indirect fitness costs on bystanders. Hooper et al. (2011) identify when strategies that suppress hierarchy and support egalitarianism are evolutionarily stable.

My paper contributes to this literature by focusing on the RDH transition among early hominins. Bullying, challenging, and helping a challenge are treated as distinct dispositions that must coevolve over time to achieve an RDH transition. This paper also contributes to the smaller literature on the evolutionary reversal of DH. Anthropologists long ago identified bully suppression in small-scale human societies (Fried, 1967), but Boehm, 1993, Boehm, 1997, Boehm, 1999 was the first to emphasize the significance of the RDH transition in hominin evolution. Dubreuil (2010) identified key steps in the RDH transition but did not model the transition itself. My work provides a formal model for studying the mechanisms and timing of the RDH transition.

Bully suppression among non-kin likely emerged in real-time, mutualistic settings (Boehm, 2009; Tomasello, 2009), and my findings show that the conditions conducive for the RDH transition were potentially present in the late Pliocene. The intent of my work is not to disprove or discredit other cooperation-enhancing developments in hominin evolution, such as stochastic interactions (Kurokawa & Ihara, 2009), jointly sharing punishments (Deng et al., 2012), leadership (Grabo & van Vugt, 2016), language (Wilson & Harris, 2017), or theory of mind (Qi & Vul, 2022). Rather, the goal is to clarify the existence and potential timing of an evolutionary path for the RDH transition.