Transient impulses are important information for machinery fault diagnosis. However, the transient features contained in the vibration signals generated by planetary gearboxes are usually immersed by a large amount of background noise and harmonic components. Even mathematical morphology (MM) is an excellent anti-noise signal processing method that can directly extract the geometry of impulse features in the time domain, but the four basic operators of MM can only extract one-way impulses while cannot extract the bidirectional impulses effectively at the same time. To accurately extract the impulse feature information, a novel method for fault detection of planetary gearbox based on an enhanced average (EAVG) filter and modulated signal bispectrum (MSB) is proposed. Firstly, the properties of the extracted impulses based on the four basic operators of MM will be divided into two categories of enhanced average operators. The four EAVG filters consist of the average weighted combination of enhanced average operators, and then the best EAVG filter is selected based on correlation coefficient to implement on the original vibration signal. It allows EAVG filter to extract positive and negative impulses of vibration signal, thereby improving the accuracy of planetary gearbox fault detection. Subsequently, the performance of the EAVG filter is influenced by the length of its structural element (SE), which is adaptively determined using an indicator based kurtosis. Then, the EAVG filter selects the optimal SE length to eliminate the interference of background noise and harmonic components to enhance the impulse components of the vibration signal. However, the nonlinear modulation components that are related to the fault types and severities are not extracted exactly and still remained in the filtered signal by EAVG. Finally, the MSB is utilized to the EAVG filtered signal to decompose the modulated components and extract the fault features. The advantages of EAVG over average (AVG) filter are clarified in the simulation study. In addition, the EAVG-MSB is validated by analyzing the vibration signals of planetary gearboxes with sun gear chipped tooth, sun gear misalignment and bearing inner race fault. The results indicate that the EAVG-MSB is effective and accurate in feature extraction compared with the combination morphological filter-hat transform (CMFH) and average combination difference morphological filter (ACDIF), and the feasibility of the EAVG-MSB are proved for planetary gearbox condition monitoring and fault diagnosis.