Camera distortion is a critical factor affecting the accuracy of camera calibration. A conventional calibration approach cannot satisfy the requirement of a measurement system demanding high calibration accuracy due to the inaccurate distortion compensation. This paper presents a novel camera calibration method with an iterative distortion compensation algorithm. The initial parameters of the camera are calibrated by full-field camera pixels and the corresponding points on a phase target. An iterative algorithm is proposed to compensate for the distortion. A 2D fitting and interpolation method is also developed to enhance the accuracy of the phase target. Compared to the conventional calibration method, the proposed method does not rely on a distortion mathematical model, and is stable and effective in terms of complex distortion conditions. Both the simulation work and experimental results show that the proposed calibration method is more than 100% more accurate than the conventional calibration method.