Speciation is the evolutionary process resulting in the formation of new, distinct species—groups of reproductively isolated populations.

The genetics of speciation involves the different traits or isolating mechanisms preventing gene exchange, leading to reproductive isolation. Reproductive isolation can be due to reproductive barriers that have effects either before or after the formation of a zygote. Pre-zygotic mechanisms prevent fertilization from occurring, and post-zygotic mechanisms reduce the viability, or reproductive capacity, of the hybrid offspring.

For example, pre-zygotic mechanisms act early in the life cycle of an organism, imposing the strongest impediment to gene flow, and preventing unfavorable mating combinations. Some mating combinations produce hybrid individuals. Natural selection can work against the production of hybrids with low fitness, thereby increasing reproductive isolation between two species.

Post-zygotic reproductive barriers can be due to the intrinsic inviability of hybrids. Genetic complications resulting from aberrant ploidy levels, different chromosomal arrangements, or gene incompatibilities where the alleles do not function properly contribute to different genetic makeup and alternative developmental pathways in hybrids. These genetic alterations affect both plants and animals, leading to post-zygotic isolation and speciation.

Epistasis, or non-allelic gene interactions, is a distinctive feature contributing to speciation. The effect of a gene variant is dependent on the genetic background in which it appears. For example, an allele giving rise to a normal phenotype in members of the same species may function poorly in the genetic environment of hybrids. This hybrid weakness can also lead to reproductive isolation and speciation.