A key question in ecology is what limits species richness. Coexistence theory presents the persistence of species amidst heterospecifics as a balance between niche differences and fitness differences that favour and hamper coexistence, respectively. With most applications focusing on species pairs, we know little about how niche and fitness differences respond to species richness, i.e. what constraints richness most. We present analytical proof that, in absence of higher-order interactions, the average fitness difference increases with richness, while the average niche difference stays constant. Analysis of a simple model with higher-order interactions, extensive simulations that relaxed all assumptions, and analyses of empirical data, confirmed these results. Our work thus shows that fitness differences, not niche difference, limit species richness. Our results contribute to the expansion of coexistence theory towards multi-species communities.