Due to the approved applicability of differential evolution (DE) in geophysical problems, the algorithm has been widely concerned. The DE algorithms are mostly applied to solve the geophysical parametric estimation based on specific models, but they are rarely used in solving the physical property inverse problem of geophysical data. In this paper, an improved adaptive differential evolution is proposed to solve the lp norm magnetic inversion of 2D data, in which the perturbation direction in the mutation strategy is smoothed by using the moving average technique. Besides, a new way of updating the regularization coefficient is introduced to balance the effect of the model constraint adaptively. The inversion results of synthetic models demonstrate that the presented method can obtain a smoother solution and delineate the distributions of abnormal bodies better. In the field example of Zaohuoxi iron ore deposits in China, the reconstructed magnetic source distribution is in good agreement with the one inferred from drilling information. The result shows that the proposed method offers a valuable tool for magnetic anomaly inversion.