Gearbox is a key component in mechanical transmission and faults on gears will lead to breakdowns and unscheduled downtime. Health condition monitoring and remaining useful life (RUL) prediction can provide sufficient leading time for gearbox timely maintenance. To some degree, the RUL prediction accuracy relies on the performance of the diagnostic features on reflecting the degradation of gears during its lifetime. However, most current commonly used features fail to reveal the fault mechanism hidden behind vibration signal and hold poor capability on noise cancellation. In this paper, modulation signal bispectrum (MSB) is proposed to reveal the signal modulation mechanism and monitor the health condition of gears. Then, an improved relevance vector machine (IMRVM) is introduced to realize the process of RUL prediction. Last, a run-to-failure test rig is designed to verify the effectiveness the MSB features for RUL prediction. Results show that MSB possesses better performance on denoising and capturing the weak variation due to modulation in gear system. The optimal MSB features after selection show better performance on reflecting the degradation of the gear and have higher prediction accuracy for gear RUL prediction comparing with RMS, kurtosis and so on. These findings provided more useful and practical information for gear RUL prediction and gearbox maintenance.