When the fitness of a trait is influenced by how common it is (i.e., its frequency) relative to different traits within a population, this is referred to as frequency-dependent selection. Frequency-dependent selection may occur between species or within a single species. This type of selection can either be positive—with more common phenotypes having higher fitness—or negative, with rarer phenotypes conferring increased fitness.

Positive Frequency-Dependent Selection

In positive frequency-dependent selection, common phenotypes have a fitness advantage. This scenario is often seen in interactions where mimicry is involved. In the Neotropical region of Central America, the butterfly species Heliconius cydno and Heliconius sapho are involved in a Müllerian mimicry partnership. Both butterflies are black and white, a common aposematic signal in the animal kingdom that warns of toxicity, venom, bad taste, or other predator deterrents.

Interestingly, H. cydno can hybridize with a closely related sister species, H. melpomene, and produce offspring. H. melpomene is predominantly black and red. The resulting mixed white-red-black hybrid offspring are significantly less fit. In addition to the female hybrids being sterile, predators do not recognize the colors as deterrent warnings, and butterflies of either parent species do not recognize the hybrids as potential mates. Therefore, the most common phenotype—black and white—is selected for. However, the more frequent the white-red-black hybrids become, the more relatively fit the phenotype becomes because predators are more likely to have learned about the warning pattern through a previous encounter with another hybrid individual.

Negative Frequency-Dependent Selection

Negative frequency-dependent selection is a form of selection in which common phenotypes are selected against. One type of negative frequency-dependent selection occurs when rare phenotypes of a prey species confer higher fitness because predators do not recognize the organisms as prey. This is known as apostatic selection.

A classic example of apostatic selection is found in the grove snail and one of its predators, the thrush. The grove snail displays polymorphic shell patterning, but the predatory thrushes tend to focus on one or two common forms of shell patterning when searching for prey. These common phenotypes, therefore, experience stronger negative selection pressure.

Another example of negative frequency-dependent selection is found in plant self-incompatibility systems. In angiosperms, homomorphic self-incompatibility is crucial to prevent self-fertilization that typically involves genetic mechanisms that prevent pollen germination or tube growth if the pollen and pistil express identical alleles. This is controlled by a multiallelic genomic region called the S-locus. Because of this, plants expressing common forms of the S-locus will often encounter false “selves”—where a potential reproductive event, and therefore gene flow, is blocked due to the self-incompatibility genes. This means that rarer forms of the S-locus are under positive selection, while common forms are selected against.