Planetary gearbox is numerously used in various mechanical transmission systems because of their extensive bearing range and high reliability. However, due to limitations in manufacturing technique and economic considerations, the gear tooth surface roughness inevitable exists in practical manufacturing process. To analyze the influence of the tooth surface roughness on the vibration signals of a planetary gear system, a nonlinear dynamic model considering multi-factor coupling is established. The dynamic model takes into account gear tooth surface roughness, the gear backlash, time-varying meshing stiffness (TVMS) and vibration transfer path. Via this nonlinear model, the effects of different tooth surface roughness on the system dynamics response are analyzed. Furthermore, the dynamic responses in the time domain and frequency domain are used to examine the influences in the main dynamic parameters, such as rotational speed and meshing force. The results show that the tooth surface roughness significantly affect the system dynamic characteristics, and with the increase of the roughness, the influence on the response of the system will enlarge. This paper can offer some theoretic guidance for the development, operation and fault identification of planetary gear transmission system.