Grazing by livestock can affect plant biodiversity and topsoil functions. However, experimental evidence on whether these impacts are limited to the topsoil or penetrate into deep layers (via changes in soil environment and resource locations) of soil is lacking, especially for soil microbial biomass and diversity. Here, we used paired grazed and ungrazed (fenced) plots at 10 locations across the Mongolian Plateau to investigate how long-term (>10 years) livestock grazing affects the biomass, diversity, composition, and function of microbial communities in surface (0–20 cm) and deep soil layers (40–60 cm). Livestock grazing increased bacterial diversity by 5–9% in both soil layers but increased fungal diversity by 10% only in the topsoil. Livestock grazing also strongly altered bacterial and fungal community composition in both soil layers. Livestock grazing decreased soil C mineralization rates by 11–25% in both soil layers, and decreased soil N mineralization rates by 16% and bacterial biomass by 20% only in the topsoil. The grazing-induced increase in microbial diversity in both soil layers was mainly explained by the changes in plant C:N ratio and plant biomass rather than by soil abiotic variables, especially for the deep soil layer. The grazing-induced negative effects on ecosystem functions (soil C and N mineralization) were mainly associated with soil abiotic variables together with plant variables or microbial diversity in the surface soil layer and were mainly associated with plant variables and soil microbial diversity in the deep soil layer. Overall, our regional field experiment provides the first evidence that the strong effects of livestock grazing on soil microbial biomass, diversity, composition, and function can penetrate the deep soil in arid and semi-arid grasslands. This knowledge suggests that models should consider the dynamic interactions between land use and both soil microbial diversity and biomass across soil depths in global drylands. Conceptual diagram showing the effects of livestock grazing on the soil microbial community structure and functions in surface and deep soil layers on the Mongolia Plateau. We investigated how large herbivore grazing affects biomass and diversities of soil bacterial and fungal communities and ecosystem functions in both surface and deep soil layers using paired grazed and ungrazed plots at 10 locations across the Mongolian Plateau. Our results provide the first evidence that the strong effects of livestock grazing on soil microbial biomass, diversity, composition, and function can penetrate to a deep soil layer on arid and semi-arid grasslands. The findings suggest that models should consider the dynamic interactions between land use and both soil microbial biomass and diversity across soil depths in global drylands. • Livestock grazing increased soil bacterial diversity in both topsoil and deep soils. • Livestock grazing increased soil fungal diversity only in the topsoil. • Livestock grazing decreased soil bacterial biomass only in the topsoil. • Grazing effects on soil bacteria were stronger than on soil fungi across soil depths. • Grazing effects on soil microbes and functions can penetrate the deeper soils.