The planetary gearbox is at the heart of most rotating machinery. The premature failure and subsequent downtime of a planetary gearbox not only seriously affects the reliability and safety of the entire rotating machinery but also results in severe accidents and economic losses in industrial applications. It is an important and challenging task to accurately detect failures in a planetary gearbox at an early stage to ensure the safety and reliability of the mechanical transmission system. In this paper, a novel method based on wavelet packet energy (WPE) and modulation signal bispectrum (MSB) analysis is proposed for planetary gearbox early fault diagnostics. First, the vibration signal is decomposed into different time-frequency subspaces using wavelet packet decomposition (WPD). The WPE is calculated in each time-frequency subspace. Secondly, the relatively high energy vectors are selected from a WPE matrix to obtain a reconstructed signal. The reconstructed signal is then subjected to MSB analysis to obtain the fault characteristic frequency for fault diagnosis of the planetary gearbox. The validity of the proposed method is carried out through analyzing the vibration signals of the test planetary gearbox in two fault cases. One fault is a chipped sun gear tooth and the other is an inner-race fault in the planet gear bearing. The results show that the proposed method is feasible and effective for early fault diagnosis in planetary gearboxes.