In the gear transmission system, tooth root crack often occurs due to the impact of machining technology and cyclic load. The appearance of tooth root cracks will affect its mesh stiffness and change the vibration characteristics of the system, thereby reducing system reliability and service life. In previous studies on the modeling of the mesh stiffness of cracked gears, the cracks were often supposed to be in a closed state, and the elastoplastic deflection of the fault gear teeth was also neglected, resulting in significant errors in the estimation of the mesh stiffness. When a tooth root crack occurs, the engaged gear teeth will deviate from the theoretical meshing position due to the elastoplastic deflection of the fault tooth. Therefore, an accurate mesh stiffness estimation model in view of the opening crack and elastoplastic deflection of the cracked tooth is put forward in this study. Firstly, the actual meshing position of the tooth pairs is derived considering the elastoplastic deformation in the crack opening state; Then, the improved potential energy method is adopted to calculate the mesh stiffness, and the finite element method (FEM) is applied to verify it. At last, the influence of failure severity and elastoplastic deflection degree on mesh stiffness and load sharing is analyzed. The results show that the severity of the crack failure and elastoplastic deflection has a significant impact on the mesh stiffness and load-bearing. This research can provide stiffness input for the study of fault dynamics of the gear transmission system.