Lineages undergoing rapid evolutionary diversification may evolve a set of mixed traits as a result of hybrid speciation, gene flow, or incomplete lineage sorting (ILS). However, how best to quantify these alternative processes impacting phenotypic variation remains unclear. Here, we examined trait evolution in two sister genera of endangered Asian primates,
Trachypithecus
and
Semnopithecus
. By assembling a de novo genome of the
Trachypithecus pileatus
group, which is geographically located in a transitional zone between these two genera, our integrated phylogenomic analyses clarified the evolutionary relationship of the
T. pileatus
group with other
Trachypithecus
species. We also identified that morphological similarities shared between this species group and
Semnopithecus
are the result of ILS, rather than gene flow or hybrid speciation. Across all chromosomes of the
T. pileatus
group, ILS contributed 8.9% of whole genome segments. Across these segments, we distinguished 77 genes such as
FGFBP1
and
FOXO1
that are involved in pathways of bone development and osteoblast differentiation. Functional experiments indicate that
FGFBP1
genotypes shared by species of the
T. pileatus
group and
Semnopithecus
appear to enhance osteogenic capability and mineralization, possibly resulting in larger body size and similarities in skull morphology compared with other species of
Trachypithecus
. The study reveals that ILS has shaped the evolution of mixed traits within gene-regulatory networks, driving phenotypic diversity during periods of rapid species divergence in this highly successful primate radiation. ILS appears to be more common than previously thought and represents a critical step in accurately assessing the phylogeny of closely related taxa.