Group selection and the development of the biological species concept

James Mallet


The development of what became known as the biological species concept began with a paper by Theodosius Dobzhansky in 1935, and was amplified by a mutualistic interaction between Dobzhansky, Alfred Emerson and Ernst Mayr after the second world war. By the 1950s and early 1960s, these authors had developed an influential concept of species as coadapted genetic complexes at equilibrium. At this time many features of species were seen as group advantages maintained by selection to avoid breakdown of beneficial coadaptation and the ‘gene pool’. Speciation thus seemed difficult. It seemed to require, more so than today, an external deus ex machina, such as allopatry or the founder effect, rather than ordinary within-species processes of natural selection, sexual selection, drift and gene flow.

In the mid-1960s, the distinctions between group and individual selection were clarified. Dobzhansky and Mayr both understood the implications, but their views on species changed little. These group selectionist ideas now seem peculiar, and are becoming distinctly less popular today. Few vestiges of group selectionism and species-level adaptationism remain in recent reviews of speciation. One wonders how many of our own cherished views on evolution will seem as odd to future biologists.

1. Introduction

One of the greatest Darwinists and Darwin-promoters of the twentieth century, Ernst Mayr, made what seems today an extremely off-message claim about Darwin's Origin of Species: … Darwin failed to solve the problem indicated by the title of his work. … In retrospect, it is apparent that Darwin's failure, as well as that of the antievolutionists, resulted to a large extent from a misunderstanding of the true nature of species.(Mayr 1963, p. 12, 14)More recently, it has been argued that Darwin's species concept served well as a means of promoting evolutionary ideas in 1859 (McOuat 1996; Stamos 2006; Mallet 2008a, in press; Kohn 2009; Sloan 2009; Ereshefsky in press). It is therefore of interest to explain why Darwin's view of species was perceived to be so faulty, in the 1930s–1960s, that it needed to be replaced, rather than fine-tuned.

One likely reason for the perception that Darwin had failed was that the biological species concept of the mid-twentieth century became imbued with a number of group properties such as ‘cohesion’ and adaptiveness which were not present, and indeed were actively resisted by Darwin, who used a purely operational definition. The distinction between adaptation at the individual (or gene) and at the population or species level had not been clearly distinguished, at least until the mid-1960s (Hamilton 1964; Williams 1966), and was publicized widely only by the 1970s (Wilson 1975; Dawkins 1976). I propose the thesis that the biological species concept was conceived in terms of group properties and became amplified to a peak of group adaptationism in the 1950s and early 1960s, before gradually losing most of this by today.

The aim of this paper is to trace the history of group selectionist ideas in the origination and subsequent development of the reproductive isolation view of species. By ‘group selectionism’, I mean the hypothesis that traits evolve for the benefit of the group (in this case, the species), as opposed to for purely individual benefit, which is termed ‘individual selection’, equivalent to Darwinian natural selection (Maynard Smith 1964). The group adaptationism I discuss here is ‘naive group selectionism’ sensu Wilson & Wilson (2007), because the proposer has not considered carefully whether a group trait can actually evolve as a group adaptation or will be counteracted by natural selection.

I do not mean to imply that group adaptations can never evolve (Wilson & Wilson 2007). Indeed, one of today's likely, or at least possible explanations for the evolution of sex is still that recombination confers an advantage to whole species (Maynard Smith 1978). Provided mutations to selfish asexuality are extremely rare, sex may be favoured because sexual species survive and proliferate more than asexual species (Nunney 1989).

I do not attempt to draw any conclusions here about whether today's version of the biological species concept is useful. I am aware that I run the risk of being accused of being ‘unwilling to separate the isolation concept of species from its historical adaptationist baggage’ (Coyne et al. 1988), so I have strenuously attempted to make this history palatable to most evolutionary biologists, whatever their favourite species concept. In this Darwin year of 2009, the history of group selection in the biological species concept seems to me of great interest because it shows how even poorly justified ideas about evolution can become accepted when they agree with the prevailing mood. It suggests we should be cautious also about popular evolutionary ideas of our own time.

2. Why was darwin's view of species rejected?

Darwin's major thesis in ‘The Origin’ was that species were not ‘essentially’ different from varieties within species, and that species evolve readily from varieties. To Darwin, the difference between species and varieties was that species are delineated by gaps in morphology that persist when they overlap in space, whereas varieties are less definite, and show intermediates. Hereafter, we shall be compelled to acknowledge that the only distinction between species and well-marked varieties is, that the latter are known, or believed, to be connected at the present day by intermediate gradations, whereas species were formerly thus connected.(Darwin 1859, p. 485)In contrast, Mayr and Dobzhansky viewed Darwin's species ideas as inadequate, and promoted a new ‘Modern Synthesis’ view of species, the ‘biological species concept’, in which species were characterized by ‘reproductive isolation’ from other species; varieties, such as geographic subspecies within species, do not (Dobzhansky 1935, 1937; Mayr 1940, 1942, 1963). Appreciating Mayr's holistic views on species, as well as those of Theodosius Dobzhansky, is important for understanding their views on speciation, for example in their beliefs in the requirement for allopatry, or the founder effect.

Curiously, both Darwin (1859) and Wallace (1865) had argued against a view of species based on reproductive isolation, even though both clearly understood that ‘intercrossing’ (today's gene flow) would act against divergent selection and inhibit speciation (e.g. Darwin 1859, p. 42, 159), and that geographical isolation would enhance the probability of speciation by preventing intercrossing (Darwin 1859, pp. 104–107). Darwin devoted a whole chapter, ‘Hybridism’, to dismissing the creationist idea that reproductive isolation (in the form of inviability and sterility of hybrids) constituted a useful definition of species.

Historians of science have long agreed that Darwin understood the biological nature of species (Ghiselin 1969; Kottler 1978; Sulloway 1979), but many biologists (e.g. Futuyma 1998; Coyne & Orr 2004) have continued to accept uncritically Mayr's argument for ‘Darwin's failure’ (Mallet in press). This twentieth century view that Darwin failed on species and speciation is particularly perplexing, because it came about as a result of the Modern Synthesis, a movement which championed Darwinian natural selection as compatible both with Mendelian genetics and with the data on natural biodiversity. Just as Darwin's ideas were again becoming broadly accepted in evolution, Mayr and Dobzhansky seem to have influenced several more generations of biologists to believe that there was another major problem at the very heart of Darwin's work: his supposedly erroneous view of species and speciation.

A number of important reasons led to the development of the biological species concept in opposition to Darwin's ideas on species. I highlight group selectionism in this paper, but other likely causes are dealt with in detail elsewhere (Mallet in press). I outline these other causes briefly here for completeness.

  • —The philosophical preferences of Theodosius Dobzhansky and Ernst Mayr seem to have been on the concrete side, possibly because of their educational backgrounds in pre-war Russia and Germany. They preferred to deal with concepts about the ‘reality’, or ‘true nature’ of species, rather than merely operational definitions. Both Dobzhansky and Mayr considered themselves well-versed in continental European philosophy, and Dobzhansky also peppered his prose with philosophical Latin, such as conditio sine qua non and mutatis mutandis. Dobzhansky fired the opening salvo for the biological species concept in a philosophical journal (Dobzhansky 1935).

  • —Mayr, at least, had a very incomplete understanding of Darwin's view of species, which he apparently obtained from greatly abbreviated quotations from ‘The Origin’ (Ghiselin 1969, 2004; Kottler 1978; Sulloway 1979; Mallet in press).

  • —Dobzhansky and Mayr were both influenced by a long unresolved debate about ‘physiological selection’ (i.e. sterility and inviability of hybrids, sometimes supplemented with preferential mating or fertilization). This had been rumbling on since 1859, starting with a critique by Darwin's own ‘bulldog’, who questioned whether Darwin had explained the evolution of hybrid sterility (Huxley 1860). It was exacerbated later when a friend and former disciple of Darwin argued for physiological selection (Romanes 1886) as a kind of group selection for reproductive isolation to prevent species blending. These critics were unconvinced by Darwin's chapter VIII on ‘Hybridism’, which argued that sterility evolved as a sporadic and uncertain by-product of evolution (see below), and therefore was not a valid essence of species (Darwin 1859). This alternative train of thought tended therefore to default back to sterility and other forms of reproductive isolation as an argument for the reality of species. Of course, one can believe that hybrid sterility and inviability are important in speciation without believing that they are group adaptations, and many do today; this debate has yet to be resolved. The topic is discussed elsewhere (Wallace 1886; Coyne & Orr 2004; Mallet 2008b; Gourbière & Mallet 2010), and is underemphasized here except insofar as it intersects with group selectionism.

  • —Related to this is another issue sparked off by the mutationists and saltationists of the early twentieth century, such as William Bateson and Hugo De Vries, and later Otto Schindewolf and Richard Goldschmidt. Goldschmidt, for instance, argued that species were separated by ‘bridgeless gaps’ which could be breached only by ‘systemic mutations’ rather than natural selection (Goldschmidt 1940). Examples of such important gaps were hybrid sterility and inviability. According to Schindewolf, the fossil record also seemed to suggest both stasis and occasional punctuated change that he associated with rapid speciation. Both Dobzhansky and Mayr argued strongly against the saltationists' mechanisms of speciation. However, they accepted their argument that species were more ‘real’ and discrete (genetically and ‘physiologically’) than Darwin had claimed. Dobzhansky cited Bateson to this effect as late as 1951 (p. 259).

  • —Finally, Dobzhansky and Mayr felt they had no alternative to using biological ‘mechanisms’—i.e. functional, and supposedly adaptive group characteristics to delimit species. When morphology failed, as in the case of cryptic ‘sibling’ species, they could not, in the 1930s–1960s at least, turn to the abundant molecular markers we now have available.

Hence, I hope it is clear I do not claim that group selectionism was the only or even main reason why Dobzhansky and Mayr regarded the biological species concept important and Darwin's ideas wrong. Nonetheless, I do argue here that group adaptationism was an important component, especially of their post-war views of species. The new view of species that stemmed from this group selectionism and the terminology that reflects it remain influential today, even though the catalyst itself has largely disappeared from current species literature.

3. The origins of the biological species concept

The idea that sterility (or inviability) of hybrids provides a useful criterion of species long pre-dates Darwin. Intersterility was assumed to be provided by the Creator as a means of maintaining the separateness of created kinds of animals and plants. By the 1850s, the most famous of such arguments was Buffon's from the eighteenth century: ‘We should regard two animals as belonging to the same species if, by means of copulation, they can perpetuate themselves and preserve the likeness of the species; and we should regard them as belonging to different species if they are incapable of producing progeny by the same means’ (Buffon, as translated in Lovejoy 1968).

Darwin attempted to rebut this creationist view by arguing, in his chapter ‘Hybridism’, that hybrids between species were only sometimes infertile, and that infertile crosses occurred within species also. To Darwin, hybrid sterility evolved as an incidental and sporadic by-product of evolution, not as an advantageous means of speciation. Now do these complex and singular rules indicate that species have been endowed with sterility simply to prevent their becoming confounded in nature? I think not. For why should the sterility be so extremely different in degree, when various species are crossed, all of which we must suppose it would be equally important to keep from blending together? Why should the degree of sterility be innately variable in the individuals of the same species?(Darwin 1859, p. 260)Therefore sterility did not, in Darwin's view, constitute a useful definition of species. The same argument rebuts also the ‘physiological selection’ argument of George Romanes (1886), as well as later group selectionist ideas.

Many post-Darwin evolutionists were nonetheless attracted to the idea that species could be defined by ‘physiological isolation’ (Hey 2006). One of the clearest of these early biological species concepts was that of Edward Bagnall Poulton. A fervent supporter of Darwinian evolution and a personal friend of Alfred Russel Wallace, he did however argue differently from Darwin and Wallace on this issue. For Poulton, the true criterion of species was ‘syngamy’, or reproductive compatibility including both pre- and post-mating compatibility, as opposed to ‘asyngamy’ between species (Poulton 1904). Poulton had been, in his turn, influenced by Wallace's (1865) discussions of species concepts and the discrete, female-limited, mimicry polymorphisms in Papilio butterflies (Mallet 2004). Papilio polymorphisms seemed to be an exception to a strict interpretation of Darwin's morphological definition, which would apparently have argued for separating each morph into a separate species, even though these morphs formed a reproductive community. Darwin, however, himself learned of the Papilio example from Wallace, and added discussions of Papilio polymorphisms to subsequent editions of ‘The Origin’ without altering his view of species (Darwin 1872).

Dobzhansky probably formed his view of species as reproductively isolated groups in part as a result of scientific discussions of ‘physiological isolation’ among the entomological community he left behind in Russia, especially A. P. Semenov-Tian-Shansky (Krementsov 1994), to whom he often referred. But it seems likely that he also read Poulton. The latter is conjectured (Mallet 2004) because Dobzhansky did refer to Poulton (Dobzhansky 1937) in a section about mimicry. Given the main topic of Dobzhansky's own book, it is hard to imagine that his eyes did not stray over the reprinted 1904 ‘What is a species?’ essay in the very book he cited (Poulton 1908). Even if Poulton was not the immediate source, Dobzhansky had ample opportunities from the ‘physiological selection’ literature for other, similar influences (Krementsov 1994; Hey 2006); this literature also gibed with his own experimental work on hybrid sterility in Drosophila. That Poulton presaged Mayr's own biological species concept is more certain: Mayr revived Poulton's (1904) term ‘sympatry’ with acknowledgement (Mayr 1942), though he rechristened Poulton's antonym, ‘asympatry’ with today's more familiar ‘allopatry’.

Mayr himself was also informed by an encyclopaedic knowledge of bird systematics. By around 1900, systematists had developed the practice of labelling as ‘subspecies’ the geographic taxa that differ in multiple morphological characters, but which intergrade at contact zones (Jordan D. S. 1905; Jordan K. 1905; Mayr 1982; Rothschild 1983). Previously, these geographic replacement forms were often regarded as species. This new systematics reinforced the saltationists' argument for ‘bridgeless gaps’ between species, because a whole class of ‘species’ that really do intergrade with other such ‘species’ as Darwin had argued were suddenly no longer considered real species at all (Mayr 1942, pp. 113–114). The higher level sympatric forms now considered separate species were more divergent, and intergraded less than their component subspecies (Mayr 1942). Dobzhansky expressed himself similarly, albeit with less encyclopaedic knowledge of taxonomic literature (e.g. Dobzhansky 1937, pp. 311–312). Mayr often conflated his biological ‘concept’ of species, based on a lack of local interbreeding, with the idea of the ‘polytypic species’ as a taxon, that is the idea of species having multiple subspecies (Mayr 1992; Bock 1995; Haffer 2007).

Dobzhansky's first stab at the biological species concept (Dobzhansky 1935) built upon his interpretation of the ‘syngameon’ idea, or pure-breeding population, of the botanist Johannes Lotsy. (Dobzhansky's interpretation of Lotsy was apparently erroneous, and it seems to refer more to Poulton's (1904) ‘syngamy’.) Dobzhansky extended this idea to whole groups of populations, proposing that ‘a species is a group of individuals fully fertile inter se, but barred from interbreeding with other similar groups by its physiological properties (producing either incompatibility of parents, or sterility of the hybrids, or both)’ (p. 352). It was in this publication that he also developed the concept of prezygotic and postzygotic ‘isolating mechanisms’. He included geographical isolation as an isolating mechanism, but argued that only ‘physiological isolating mechanisms’ (including assortative mating and ecological isolation, as well as hybrid unfitness) were important in defining species.

How did this physiological isolation evolve? ‘We are almost forced to conjecture that the isolating mechanisms are merely by-products of some other differences between the organisms in question, these latter differences having some adaptive value and consequently being subject to natural selection’ (p. 349). At this time, there was only a weakly expressed belief in group advantage for reproductive isolation, inherent in the use of the word ‘mechanisms’ and a hint of wistfulness in ‘we are almost forced …’. Dobzhansky apparently felt more strongly that isolating mechanisms were important in species designation because species were thereby made verifiable via experimental, genetic data.

Dobzhansky quickly admitted that there would always be grey areas where the definition broke down. ‘… in some cases classification of two groups of individuals as separate species or as varieties of the same species remains arbitrary’ (p. 352). He felt that the problem was that one tended to want to see taxonomic terms like species as ‘static’ entities, whereas in reality evolution was continuous, and ‘dynamic’. However, he argued that there were few conflicts in practice between his new, ‘genetic’ concept of species, and the original ‘taxonomic’ one, because isolating mechanisms caused real discontinuities in nature. Based on these ideas, he enabled a return to a somewhat essentialist, Buffonian species concept. In doing so, he had to appeal to a logic which allowed a paradox to be ignored: he felt comfortable arguing that species were real, cohesive groups in a modern, dynamic and genetic sense, in spite of being sometimes indistinct and dubious (albeit rarely, in his view) in a practical, taxonomic sense.

These ideas were amplified, as is well known, in Dobzhansky's ground-breaking 1937 book. Again he made the Darwinian argument that physiological isolation is mainly a by-product of adaptive evolution, in particular with regard to his famous proposal of two-locus epistasis (pp. 254–256) now often called ‘Dobzhansky–Muller’ incompatibility. He also developed for the first time his argument for ‘reinforcement’ a few pages later: ‘… isolation becomes advantageous for species whose distributions overlap, provided that each species represents a more harmonious genetic system than the hybrids between them. Under these conditions, the genes that produce or strengthen isolation become advantageous on that ground alone, and may be favoured by natural selection’. He clearly intended his theory to apply to postzygotic as well as prezygotic mechanisms, for example: What, for example, is the rôle played within Crepis tectorum by the gene that causes the death of the hybrids with C. capillaris? …only those genetically distinct types that have developed isolation can subsequently coexist without a breakdown of the differences between them… Therefore, isolation becomes advantageous for species whose distributions overlap, provided that each species represents a more harmonious genetic system than the hybrids between them. Under these conditions, the genes that produce or strengthen isolation become advantageous on that ground alone, and may be favoured by natural selection.(Dobzhansky 1937, p. 258).Whereas reinforcement is today widely recognized as a possible route for the evolution of sexual barriers or prezygotic isolation (Noor 1999; Coyne & Orr 2004), postzygotic isolation cannot usually evolve in this way because it is never beneficial to individuals to produce sterile or inviable offspring, as Darwin had himself argued (but see Barton 2010, in this issue, for a potential counter-example). Many other statements in Dobzhansky's book show the beginnings of both group selectionist views found in his later ‘balance theory’ for polymorphisms (p. 127), as well as his views on reproductive isolation as a protective device for species (pp. 257–258). His ideas on reinforcement leading to ‘physiological isolation’ were later amplified, and were also then clearly envisaged to apply to postzygotic as well as to prezygotic isolating mechanisms. For example, ‘the very fact that isolating mechanisms are as diversified as they are is strong evidence for the prevention of interbreeding being an essential characteristic of the process of speciation’ (Dobzhansky 1940, p. 320).

At this time, Dobzhansky was strongly under the influence of Sewall Wright (Crow 1987; Lewontin 1987). He reproduced and promoted Wright's ‘adaptive landscape’ diagrams in his 1937 book. Whereas Wright had written mainly about populations on different adaptive peaks, and the spread of these adaptive peaks to other populations (Wright 1932), Dobzhansky applied Wright's ideas of multilocus adaptation to whole species (Plutynski 2008). By 1940, Dobzhansky argued that ‘each species, genus and probably each geographical race is an adaptive complex which fits into an ecological niche somewhat distinct from those occupied by other species, genera and races. The adaptive value of such a complex is … a property of the genotype as a whole’. Epistatic coadaptation, as well as gene flow, united populations of species, and isolating mechanisms were now seen as devices that protected the ‘harmonious genetic systems’, or adaptive peaks that were separate species. Speciation began to require ‘genotypical reorganization’ (Dobzhansky 1937, p. 126). Foreshadowing his later ‘balance hypothesis’ ideas (see also Provine 1986, p. 344), genetic variation itself also began to be seen as a beneficial trait at the species level: there was a ‘necessity for the species to possess at all times a store of concealed potential variability’ (Dobzhansky 1937, p. 127), because it enhanced its future evolutionary flexibility.

Dobzhansky settled on a proposed biological function of the species, and of species properties, as a definition of species in part also because of a new problem created by general acceptance of Mendelian genetics. As pointed out above, this was the difficulty of applying a Darwinian definition of species based on gaps in continuous morphological variation. Single Mendelian elements, genes or whole chromosomes, can determine major morphological gaps between groups of individuals, individual morphs and rarer mutants, which everyone wanted to classify within species. Some examples are mimetic Papilio butterfly morphs, or separate sexes (Poulton 1904). Another perceived problem was the exact converse: reproductively isolated forms, later called ‘sibling species’ (Mayr 1942, pp. 150–151), can coexist in sympatry in spite of a lack of morphological gaps. With molecular markers not yet available, there seemed no possibility of classifying such species or forms correctly without appealing to a presumed biological property of species, reproductive isolation (Dobzhansky 1937, pp. 309–316).

Mayr frequently interacted with Dobzhansky at this time (Haffer 2007), and attempted to improve on Dobzhansky's ideas with a more practical, taxonomic definition, based on his knowledge of polytypic bird species. ‘A species consists of a group of populations which replace each other geographically or ecologically and of which the neighbouring ones intergrade or hybridize wherever they are in contact or which are potentially capable of doing so …’ (Mayr 1940, p. 256). Citing Dobzhansky, he added his weight to the latter's argument for the reality of species: ‘the opinion is very prevalent among non-taxonomists that ‘species’ are just idealized abstractions …, but in reality the species themselves are remarkably real natural phenomena’ (p. 257). These ideas were amplified in his important book two years later, where he presented an alternative, shorter definition: ‘species are groups of actually or potentially interbreeding natural populations, which are reproductively isolated from other such groups’ (Mayr 1942, p. 120).

4. The biological species concept: why ‘biological?’

Since all possible species definitions are about organisms, and therefore are by definition biological, it may seem odd that this idea became known as the ‘biological species concept’. Mayr (1963, p. 20) defended the term in part because ‘the underlying concept is based on the biological meaning of the species, that is, to serve as a protective device for a well-integrated, coadapted set of gene complexes’. However, the term seems to originate with Dobzhansky (1937, p. 316), who used ‘biological species’ (his ‘dynamic’ definition) to contrast with taxonomists’ character-based species (the ‘static’ definition). At the time, Dobzhansky was intensively studying the intersterility of races A and B of Drosophila pseudoobscura (race B is now known as D. persimilis). Dobzhansky argued these ‘races’ were actually good biological species, even though he encountered strong opposition from Alfred Sturtevant and taxonomists in so doing. In the same passage Dobzhansky also cited a well-known review of ‘biological races’ in insects (Thorpe 1930). Thorpe showed that insects may specialize on different hosts or ecological niches, and can separate into cryptic ‘biological races’ that become physiologically isolated. Cryptic biological species can therefore exist even where morphology reveals no gap in variation detectable with traditional taxonomic methods and a set of museum specimens.

5. Group selectionism before hamilton and williams

Soon after the second world war, Dobzhansky and his followers began to develop a new brand of population genetics that seemed, in its emphasis on holistic multilocus interaction and balanced polymorphisms, to improve on the prewar ‘bean-bag genetics’ (Mayr 1959) of Wright, Haldane, Fisher and Muller, which Mayr interpreted as largely about single loci. The group selectionist idea that populations develop regulatory mechanisms, such as refraining from reproduction to avoid violent fluctuations, had been long held in ecology (Elton 1927). One of the first and most successful postwar ecology textbooks in the USA also promoted this viewpoint (Allee et al. 1949). A key author of this book, Alfred E. Emerson, studied termite sociality, and was particularly keen on the ‘superorganism’ concept to explain insect sociality. Presumably in part due to his influence, the superorganism concept was expanded further in the textbook—to the species level, and upwards even to whole ecological communities (Allee et al. 1949).

Emerson also wrote about species and species concepts, and it was he who originally coined the term ‘reproductive isolation’ in a work describing the discovery of cryptic species of termites (Emerson 1935). Emerson was a close associate of Dobzhansky's. In 1939, Emerson was the organiser of the ‘Society for the Study of Speciation’ with Dobzhansky and Julian Huxley (Cain 2000). The influence on Dobzhansky's thinking became particularly clear by 1951: ‘Sexual reproduction has brought about a new form of biological integration. Individuals are combined into reproductive communities, Mendelian populations. These supraindividual entities are considered supraorganisms by some authors (Allee et al. 1949). … in sexual organisms, Mendelian populations, rather than individuals, have become the units of the adaptively most decisive forms of natural selection’ (Dobzhansky 1951, p. 260).

Kenneth Mather in the UK and, particularly, Isadore Michael Lerner in the USA informed the more genetic aspects of these holistic views. Lerner, a Russian émigré like Dobzhansky, was an animal breeder who worked on poultry. He developed a general hypothesis of ‘genetic homeostasis’ (Lerner 1954). Lerner adapted the ideas of Conrad Waddington and another Russian, Ivan Schmalhausen, whose work ‘has in no way lost its significance today’ according to Dobzhansky (1981, p. 240), on canalization of phenotypes, and argued that developmental buffering against environmental variation, and also inbreeding depression and the limits often reached in artificial selection, were all caused by genome-wide advantages of heterozygosity. These ideas were particularly favoured by the new ‘balance’ genetics school. Dobzhansky's group, including former student Bruce Wallace and co-worker James C. King, were also obtaining by then empirical evidence for strong heterozygous advantage of chromosomal inversions in Drosophila. By 1951, this led to a much more concrete view of the advantages of variation within populations than in 1937 (Dobzhansky 1951). Soon thereafter came the distinction between the ‘classical’ school, exemplified by Muller's views that most mutations were deleterious (Crow 1987; Lewontin 1987), and the ‘balance’ school of Dobzhansky's circle (Dobzhansky 1955). In the balance school, Wrightian multilocus coadaptation became coupled with near-universal heterozygous advantage, which maintained abundant polymorphisms in natural populations. Heterozygous advantage at many loci, as explained by the balance hypothesis, also came to be viewed as an adaptation which buffered whole populations from environmental vicissitudes (Lerner 1954).

This new selectionist and indeed group adaptationist population genetics led to major changes in Dobzhansky's views on speciation (Provine 1986, p. 452). The third edition of his Genetics and the Origin of Species (Dobzhansky 1951) now had a chapter on ‘Adaptive Polymorphism’ in place of ‘Variation in Natural Populations’. Genetic variation in natural populations had previously been interpreted as largely neutral (Dobzhansky 1937). In the new chapter, he argued: ‘polymorphism within a species, or any other kind of diversity of sympatric forms, increase[s] the efficiency of the exploitation of the resources of the environment by the living matter’ (Dobzhansky 1951, p. 110). The advantages were also due to coadaptation among genes both within and among genotypes in populations, and reproductive isolation became a protective mechanism to preserve these beneficial gene interactions: ‘if life is to endure, the gene combinations whose adaptive value has been vouchsafed by natural selection must be protected from disintegration. Without isolation the ravages of natural selection would be too great’ (p. 180). Soon after a section on two locus Dobzhansky–Muller epistasis, nearly identical to that in the 1937 book, Dobzhansky now, confusingly, argued against Muller's hypothesis for hybrid sterility as a mere by-product of evolution. Instead, ‘isolating mechanisms appear to be ad hoc contrivances that prevent the exchange of genes between nascent species, rather than incongruities originating in accidental changes in the gene functions’ (p. 208). This new view of species as cohesive wholes, integrated by coadaptation and heterozygous advantage, and protected by isolating mechanisms, went hand in hand with the announcement of Dobzhansky's (1955) balance hypothesis for genetic variation (Lewontin 1987).

Mayr at around this time had had many informal conversations with Dobzhansky acolytes Bruce Wallace and James King. Previously, Mayr had learned his genetics largely from Dobzhansky's 1937 book (Mayr 1999), but now he was enthralled by the possibilities of this new school of population genetics to explain speciation (Provine 2004). ‘… a gene pool is not an unconnected ‘pile of genes’, but a well-integrated, balanced system’ (Mayr 1954). The idea of extensive population-level coadaptation at multiple genes fitted both with recent work on chromosome polymorphisms from the Dobzhansky group (Crow 1987; Lewontin 1987), and also with his own knowledge of island and mainland birds in New Guinea. Mayr began to see gene flow and natural selection in large populations on continents as largely conservative forces which prevented further evolution, and therefore speciation also. ‘A well integrated genetic system may come into perfect balance with its environment and become so well stabilized that evolutionary change will no longer occur’ (Mayr 1963, p. 555).

These new ideas, added to an older idea about the ‘founder principle’ he had already broached in his 1942 book (Mayr 1942, p. 237), led to Mayr's innovative hypothesis of founder effect speciation (Lewontin 1974, p. 159; Provine 2004). According to Mayr (1954), genes on continents exposed to abundant gene flow are selected for compatibility to this variation. They ‘do well on a great variety of genetic backgrounds … A ‘good mixer’ rather than a good ‘soloist’, has a tremendous advantage in such a system’. Yet, this slows down evolution. However, when a few individuals found a new, isolated colony, the sudden reduction in population size and loss of alleleles causes the frequency of homozygotes to rise. ‘As a consequence, homozygotes will be much more exposed to selection… Thus, the ‘soloist’ is now the favorite rather than the good mixer’. This change ‘may affect all loci at once. Indeed, it may have the character of a veritable ‘genetic revolution’. Furthermore, this ‘genetic revolution’ may well have the character of a chain reaction. Changes in any locus will in turn affect the selective values at many other loci, until finally the system has reached a new state of equilibrium’ (Mayr 1954).

Mayr felt that ecotypic variation involving clinal adaptation (produced by standard natural selection), which may occur in the face of gene flow, could not lead readily to speciation. ‘Clines indicate continuities, but since species formation requires discontinuities, we might formulate a rule: The more clines are found within a region, the less active is species formation’ (Mayr 1942, p. 97, Mayr's italics). Citing Goldschmidt's (1940) argument for the impotence of natural selection along a cline to effect speciation, he agreed, and wrote: ‘Owing to the never-ceasing gene-flow through such a system these [clinal] populations are merely variations on a single theme’ (Mayr 1954, p. 159).

Genetic revolutions in founder populations, in contrast, produced in Mayr's thesis a qualitatively different kind of evolution permitting a shift between alternative stable equilibria. The idea apparently fitted also with the fossil record, which showed ‘typostrophic’, or punctuated changes documented by Schmalhausen, and with Goldschmidt's ideas about ‘bridgeless gaps’ between species. Mayr had discovered, to his satisfaction, that there were really two kinds of evolution, and only one of them could readily lead to speciation. Therefore, Darwin did not understand speciation, and indeed did not really deal with it in ‘The Origin’ because of his insistence on ordinary natural selection as the cause of speciation. Mayr felt his insight was novel and important, and was aghast when the manuscript he submitted took two years to be printed. He worried that someone else (Dobzhansky, perhaps?) would come up with the same idea (Mayr 1989).

Unlike Dobzhansky's 1951 book, neither Mayr's 1942 book nor even his 1954 paper explicitly discussed group adaptation, as far as I can discern. In 1942, although Mayr had agreed that sympatric species were almost always separated by ‘bridgeless gaps’, he had argued that the new views he was promoting specifically downgraded the species to a less important category than hitherto realized (Mayr 1942, p. 7). By the mid-1950s, however, he was a convert to Dobzhansky's new view (Haffer 2007, pp. 273–275) of species as real, balanced, adaptive units: Each species is a delicately integrated genetic system… Hybridization would lead to a breakdown of this system and would result in the production of disharmonious types. It is the function of the isolating mechanisms to prevent such a breakdown, and to protect the integrity of the genetic system of species. Any attribute of a species that would favour the production of hybrids is selected against, since it results in wastage of gametes. Such selection maintains the efficiency of the isolating mechanisms and may indeed add to their perfection.(Mayr 1963, p. 109).He also was very much an adherent of Dobzhansky's ‘balance’ school, believing that variation was a good thing per se, an important basis for the above view of species (Lewontin 1974, p. 159). Here, Mayr uses almost exactly Dobzhansky's words: ‘… numerous devices have actually been discovered that protect variation against the relentless ravages of natural selection’ (p. 214).

It is clear that both scientists can be criticized for naive group-adaptationist thinking, although both also clearly understood that they were proposing mechanisms that would generally be opposed by ordinary natural selection. They were not specialists in the area of altruism, and they were products of their time. Lerner (1954, p. 5) was perhaps more cautious, arguing only that ‘… it is possible to construct a genetic model on the basis of which the more complex integrative properties of Mendelian populations can emerge from evolutionary forces operating on individual genotypes’. Both Mayr (1963) and Dobzhansky (1951, 1955) cited countervailing literature. Mayr, perhaps not recognizing that he was contradicting his own stance on the adaptive nature of reproductive isolation, was even rather sceptical of Dobzhansky's group selectionist reinforcement hypothesis (Mayr 1963, pp. 548–554; 1970, pp. 325–329).

Yet, in this era that also produced the balance-of-nature environmentalism of The Silent Spring (Carson 1962), the mindset of all three authors was to view many processes that occurred in nature as harmonious, and for the general good. Other findings were interpreted as protective against bad influences such as ‘errors of selection’ or ‘ravages of natural selection’. Their view of species not only conformed to the group selectionism of the time, the whole package was seen as a revolutionary new idea which made sense of 100 years' of confusing evolutionary biology. ‘Hardly any one, until Ernst Mayr …, really understood that the species was a unique unit, having quite a different significance in the evolutionary system from that of any other category. … We now realise that the species is something which is not just a matter of judgement but has a quite definite objective reality’ (Hardy 1965).

6. The species concept since the sociobiology revolution

Group selection became a popular idea for explaining patterns in nature after the war until a decline in the mid-1960s. Perhaps, the most famous example of naive group selectionism in behavioural ecology argued for altruistic population regulation by territoriality, aggression and abnegation of breeding (Wynne-Edwards 1962. Interestingly, there is excellent evidence that Wynne-Edwards was himself strongly influenced by Elton, Allee and Dobzhansky, as well as by Sewall Wright (Borrello 2009)). This orgy of wishful thinking soon led to an inevitable reaction. ‘Cheats’ in populations are likely to increase at the expense of ‘altruists’, and group fitness therefore provides a poor predictor of evolutionary direction. Both Dobzhansky and Mayr had partially understood this, as we have seen. Particularly influential were Bill Hamilton and George C. Williams (Hamilton 1964; Williams 1966). Hamilton, an entomologist, is best known (among many important achievements) for establishing an extension of Darwinian natural selection, ‘inclusive fitness’, which could explain evolution of altruistic behaviour in kin groups of social insects. Williams' work is perhaps more relevant for speciation, as he also addressed more generally the topic of adaptation. (However, see Hamilton's brief critique of isolating mechanisms (Hamilton 1965): ‘Whereas a mating aversion could easily be an isolating mechanism sympatrically evolved, hybrid sterility certainly could not’, since it cannot be directly favoured by individual-based natural selection.) He found fault with Waddington's canalization, which formed an important influence on Lerner's (1954) genetic homeostasis theory. Williams writes in his 1996 preface that his book Adaptation and Natural Selection: a Critique was triggered after going to a lecture by a revered ecologist. Williams was particularly exasperated by claims that ageing was an adaptation ‘to cull the old and impaired from populations so that fitter youthful individuals could take their places’. The lecturer was none other than the already mentioned Alfred Emerson of superorganism fame (Williams 1966, 1996 edition, p. ix).

The emergent Dobzhansky and Mayr view of species and speciation escaped severe criticism at this time perhaps in part because their group selectionism was not as blatant as that of some of the behavioural and population ecologists. The great advances promised by the new biological species concept seemed so important, and indeed so obvious and true to a majority of evolutionary biologists by the 1960s, that this emerging view of species did not seem a valid target. Perhaps the main flaw of the new species ideas, viewed in retrospect, was to endow what are really inevitable, trivial outcomes of evolution with beneficial or detrimental values. For example: ‘The freedom of the gene exchange between Mendelian populations is so regulated by natural selection that adaptive plasticity is preserved at the price of destruction and elimination of the smallest possible numbers of ill-adapted individuals’ (Dobzhansky 1951, p. 297). Mayr (1963, pp. 110–135) similarly regarded hybridization between species a ‘breakdown in isolating mechanisms’. Emergent, inconsequential properties of populations became viewed as important characteristics and adaptations, or alternatively as threats to species integrity to be overcome by evolution for the benefit of the group.

As already mentioned, Dobzhansky and Mayr were aware of the dangers of group adaptationism, and had presciently laid out the battleground they were traversing. Indeed, they came only to hesitant conclusions about the validity of population adaptation (Dobzhansky 1955, pp. 11–13; Mayr 1963, pp. 182–214, particularly 197–203). By 1970 they had both become much more aware of the threat to their ideas about species, given the new individualist understanding of adaptation introduced by Hamilton and Williams (Dobzhansky 1970; Mayr 1970). Dobzhansky (1970, p. 425), for instance, discussed and applauded Williams' (1966) use of ‘Occam's razor to argue that group selection should not be assumed to exist if a form of adaptedness can conceivably be ascribed to intrapopulational selection’, although he added ‘he wields this razor with some abandon’. Yet, Dobzhansky never shook off the idea that ‘it is advantageous for populations that occupy different adaptive peaks to steer clear of the production of gene combinations in the progeny that would fall into adaptive valleys’. Populations belonging to separate species ‘avoid disruption of their genetic systems’ by means of ‘a variety of methods, among which two classes are easily distinguishable: geographic and reproductive isolation’ (Dobzhansky 1970, p. 312).

Mayr in the same year again discussed the same problem in detail. He agreed (1970, p. 115) that ‘all attempts to establish group selection as a significant evolutionary process are unconvincing’. In spite of this unequivocal rejection of group selectionism, Mayr reproduced almost verbatim the above (1963, p. 109) arguments on the protective function of isolating mechanisms (1970, p. 68). He believed in this group-adaptive function of reproductive isolation till the end of his life. For example: ‘it is now clear that the isolating mechanisms of a species are a protective device for well-integrated genotypes. Any interbreeding between different species would lead to a breakdown of well-balanced, harmonious genotypes, and would quickly be counteracted by natural selection’ (Mayr 1988).

Somewhat later, Hugh Paterson explicitly recognized the inherently group selectionist nature of ‘isolating mechanisms’. He cited Williams (1966) in support of his argument, and proposed his own ‘recognition concept of species’ as a better alternative to the biological species concept that avoided the problem of group selectionism (Paterson 1985). Paterson's reason for doing this was that in his view both postzygotic and prezygotic isolation evolved only as indirect side-effects of evolution and could not evolve under natural selection. He believed that cohesive mechanisms within the species, ‘specific mate recognition systems’, in contrast, evolved under stabilizing selection for intraspecific compatibility. Species defined via ‘isolating mechanisms’ were therefore, in his view, illogical groupings based on non-adaptive group traits (Raubenheimer & Crowe 1987; Coyne et al. 1988), and he argued instead that mate recognition within species would qualify as a more suitable functional trait for defining species.

Paterson did not apparently consider the counter-argument that specific mate recognition systems, just like reproductive isolating mechanisms, could also form as by-products of individual selection, rather than as a designed-in function, so to speak, of the species as a whole. Are ‘recognition systems’ really ‘specific’? Are they really about what species individuals belong to? Are they even ‘systems’ at all? Or are they more a matter of individuals recognizing other suitable individuals as mates, a kind of good-genes sexual selection (Mallet in press) that may break up species, as well as bind them together? Paterson perhaps correctly identified a group adaptationism problem with reproductive isolation, but he sought to solve it using other dubious group properties, ‘specific mate recognition systems’, in their place.

Since Mayr and Dobzhansky, various evolutionists have revised the Dobzhansky/Mayr population cohesion/isolation view of species and suggested similar or related founder effect mechanisms for speciation, under the names ‘founder-flush’ and ‘genetic transilience’ (Carson 1968; Templeton 1980, 1981). ‘Punctuated equilibrium’ theory to explain gaps in the fossil record was firmly grounded on the supposed ubiquity of Mayr's founder effect speciation (Eldredge & Gould 1972; Gould & Eldredge 1977). The ‘cohesion concept’ of species, while attempting to engage both reproductive isolation and specific mate recognition, continued very much in the Mayr–Dobzhansky tradition of a group-functional definition (Templeton 1989).

The consensus among practising speciation biologists today appears to be that the species generally has few group adaptations or functional implications (Coyne & Orr 2004; Gavrilets 2004; Bolnick & Fitzpatrick 2007; Futuyma 2009), and that founder effects are neither necessary nor effective in causing speciation (Barton & Charlesworth 1984; Coyne & Orr 2004). There are few vestiges of group selectionism remaining in today's ‘relaxed’ biological species concept (Coyne & Orr 2004). These debates are not over, but recently there has been a distinct subsidence of interest in the holistic, cohesive idea of species so popular in the 1950s–1980s. Gene flow and epistasis are today considered among the few cohesive ‘forces’ affecting speciation. ‘Selfish genes’, gene genealogies, sexual selection and genetic conflict have largely replaced ‘harmonious gene complexes’ in our thinking. The group-adaptive view of species, so popular in the mid-twentieth century, is starting to seem as peculiar to us as orthogenesis, or ‘hopeful monster’ theories of speciation. One wonders which of today's cherished views on evolution will seem similarly odd to future biologists.


I thank the Wenner-Grenn Foundation for an invitation to the ‘Origin of Species—150 years later’ Symposium in Kristineberg, Sweden, June 2009. I am grateful for a generous sabbatical fellowship from the Wissenschaftskolleg zu Berlin, and in particular for conversations with colleagues in the Schwerpunktgruppe on Sympatric Speciation: Axel Meyer, Jeff Feder, Patrik Nosil and Francisco Úbeda. The article was extensively revised while I was in receipt of another generous fellowship from the Radcliffe Institute of Advanced Studies, Harvard University. I am especially grateful for discussions with Anya Plutynksi and Richard Lewontin, as well as the comments from two suitably hostile and anonymous reviewers, all of which helped me to clarify the manuscript.



View Abstract