Vibration-based fault diagnosis is widely used for gearbox monitoring. However, it often needs considerable effort to extract effective diagnostic feature signal from noisy vibration signals because of rich signal components contained in a complex gear transmission system. In this paper, an adaptive fractional Fourier transform filter is proposed to suppress noise in gear vibration signals and hence to highlight signal components originated from gear fault dynamic characteristics. The approach relies on the use of adaptive filters in the fractional Fourier transform domain with the optimised fractional transform order and the filter parameters, while the transform orders are selected when the signal have the highest energy gathering and the filter parameters are determined by evolutionary rules. The results from the simulation and experiments have verified the performance of the proposed algorithm in extracting the gear failure signal components from the noisy signals based on a multistage gearbox system.