This paper presents a novel method for diagnosing the gradual deterioration of gears using modulation signal bispectrum (MSB) and vibration measurements. A nonlinear model was derived to understand dynamic forces applied to gears that are excited by quadratic terms, e.g., shaft rotating speeds and gear meshing frequencies. Owing to its sensitivity to those quadratic terms, MSB is powerful in recovering less noisy condition related features from the measured vibration signals, e.g., gear meshing and multiples of shaft rotating speed. This allows a more pronounced representation of gear dynamic forces and makes it more effective for detecting early gear deterioration. The proposed method was verified through a run-to-failure test based on a helical gearbox system. The results show small gears at low-speed stages deteriorate faster and fail at 838 hours. This was because they prone to wear more severe due to poorer lubrication conditions compared with gears at high-speed stages. Moreover, fault detectability of the developed MSB-based method outperforms that of time synchronous averaging (TSA). Compared to TSA, clearer signs of early gear deterioration were captured using MSB, which makes it a more powerful tool for monitoring the condition of gearboxes.