To determine the effects of parent material and land use on the concentration of trace elements in the agricultural topsoil of Guizhou Province, China, a total of 584 agricultural topsoil samples were collected in a typical region. The results indicate that the contents of trace elements (As, Cd, Cr, Hg, and Pb) in agricultural soils were greater than in the uncultivated soils, and the paddy fields exhibited higher contents of trace elements than dry lands. The enrichments of most trace elements in agricultural topsoil derived from carbonate rock were more serious. In paddy fields, Cd, Cr, and As showed positive relationships with soil organic matter (p < 0.01) but were not affected by pH, carbon-to-nitrogen (C/N) ratio, and clay (p > 0.05). Lead and Hg formed the second component in principal component analysis (PCA) and were closely related to pH and clay content. In dry lands, the trace elements were well correlated with pH, C/N, and clay (p < 0.05). Analysis of PCA and correlation showed that Cd, Cr, and Hg were mainly derived from inorganic fertilizers, whereas Pb and As were primarily from organic manures. These results suggest that the effect of anthropogenic activities on paddy fields is more serious than on dry lands. Parent materials not only serve as sources of soil trace elements but also control the loss and accumulation of trace elements by affecting soil physicochemical properties, especially in dry lands.