The transmission system of a motor car directly affects the operating reliability and running safety of a high-speed train. This study examines the coupled torsion vibration responses of a transmission system in a vehicle–track vibration environment. It considers the time-varying mesh stiffness, nonlinear damping characteristics, wheel flexible deformation and wheel–rail interactions. The results are compared to those of previous works as a means of validation. Then, the dynamic characteristics of the transmission system and its coupling effects are investigated along with gear eccentricity, which describes the torsional vibration properties in the vehicle–track coupled system more comprehensively. The results show that the traction torque and gear eccentricity affect the gear meshing, vibration amplitudes and frequency multiplication of the transmission system during the vehicle acceleration process. The eccentricity, especially of the pinion, has a great effect on vibration in the transmission system at high speed ranges. Also, the dynamic responses of the wheels of the same motor car wheelset are different under traction conditions, which should not be neglected when assessing the torsional dynamic performance. The model provides a useful source of reference for the design and condition monitoring of traction systems for high-speed trains.