This paper represents a novel analysis method to improve the measurement accuracy of an optical stereo deflectometry system. A novel imaging model is proposed to research the relation between the phase uncertainty and the normal uncertainty of a stereo deflectometry system. By comprehensively considering the influence on the sampling phase error and the normal error, the system's screen pixel size and the period of fringe displayed on the screen are analyzed to enhance the measurement accuracy. To verify the proposed method, experiments using a common LCD (liquid crystal display) monitor and a screen with Retina Display have been investigated respectively. Experimental results demonstrate using the screen with Retina Display can significantly improve the measurement accuracy of a stereo deflectometry 2.38 times in accuracy due to the ultra-fine pixel size. A stereo deflectometry system based on the proposed method has been built to measure a standard flat mirror and a standard concave mirror. The global measurement accuracy of the flat mirror and the concave mirror can reach 69.7 nm and 96.8 nm respectively.