In this paper, we propose a methodology for the enhancement of small transients in gear vibration signals in order to detect local tooth faults, such as pitting, at an early stage of damage. We propose to apply the optimal denoising (Wiener) filter based on the spectral kurtosis (SK). The originality is to estimate and apply this filter to the gear residual signal, as classically obtained after removing the mesh harmonics from the time synchronous average (TSA). This presents several advantages over the direct estimation from the raw vibration signal: improved signal/noise ratio, reduced interferences from other stages of the gearbox and easier detection of excited structural resonance(s) within the range of the mesh harmonic components. From the SK-based filtered residual signal, called SK-residual, we define the local power as the smoothed squared envelope, which reflects both the energy and the degree of non-stationarity of the fault-induced transients. The methodology is then applied to an industrial case and shows the possibility of detection of relatively small tooth surface pitting (less than 10%) in a two-stage helical reduction gearbox. The adjustment of the resolution for the SK estimation appears to be optimal when the length of the analysis window is approximately matched with the mesh period of the gear. The proposed approach is also compared to an inverse filtering (blind deconvolution) approach. However, the latter turns out to be more unstable and sensitive to noise and shows a lower degree of separation, quantified by the Fisher criterion, between the estimated diagnostic features in the pitted and unpitted cases. Thus, the proposed optimal filtering methodology based on the SK appears to be well adapted for the early detection of local tooth damage in gears.