As one of the most common gear failure modes, tooth wear can produce nonlinear modulation sidebands in the vibration frequency spectrum. However, limited research has been reported in monitoring the gear wear based on vibration due to the lack of tools which can effectively extract the small sidebands. In order to accurately monitor gear wear progression in a timely fashion, this paper presents a gear wear condition monitoring approach based on vibration signal analysis using the modulation signal bispectrum-based sideband estimator (MSB-SE) method. The vibration signals are collected using a run-to-failure test of gearbox under an accelerated test process. MSB analysis was performed on the vibration signals to extract the sideband information. Using a combination of the peak value of MSB-SE and the coherence of MSB-SE, the overall information of gear transmission system can be obtained. Based on the amplitude of MSB-SE peaks, a dimensionless indicator is proposed to assess the effects of gear tooth wear. The results demonstrated that the proposed indicator can be used to accurately and reliably monitor gear tooth wear and evaluate the wear severity.