Scattering of magnetic vortices and vortex pairs by a magnetic defect in two-dimensional easy-plane ferromagnets was theoretically studied using classical equations of magnetization dynamics and the approximation of collective variables for the coordinates of vortices. A defect model was proposed as a local region of a magnet whose exchange interaction differs from that of the remaining part. The rotation of the magnetic vortex around the defect and the associated scattering of vortex pairs (vortex-antivortex bound states) were considered. In the limit of small-angle scattering, analytical expressions were obtained for the differential scattering cross section and the dependence of the scattering angle on the parameters of the vortex pair, defect and impact distance. The scattering of vortex pairs by a defect was studied, numerically and qualitatively, in the entire range of scattering parameters. Our study indicated that the impact parameter has critical values at which the scattering pattern changes in a qualitative manner and defect-localized rotational states of vortex pairs possibly exist.