To develop accurate diagnostic techniques, this study examines the dynamic responses of spur gear transmission system with including frictional effects on a tooth mesh process. An 8-degree-of-freedom model is developed to include the effects of supporting bearings, a driving motor and a loading system. Moreover, it takes into account not only the time-varying stiffness, but also the time-varying forces and moments due to the frictional effect. The latter causes additional vibration responses in the direction of the off-line-of-action (OLOA). To show the quantitative effect of the friction, vibration responses are simulated under different friction coefficients. It shows that an increase in friction coefficient value causes a nearly linear increase in the vibration features of diagnostics. However, features from torsional responses and the principal responses in the line-of-action show less changes in the vibration level, whereas the most significant increasing is in the OLOA direction. Furthermore, the spectral peaks at the rotational and sideband frequencies are influenced significantly by small breakage defects, especially when the friction effect is taken into account. In addition, the second and third harmonics of the mesh frequency are more influenced than the first harmonic component for all motions, which can be effective features for both indicating lubrication deterioration and improving conventional diagnostic features.