The dynamic performance of the trailer bogie brake system is essential for the safe operation of a high-speed train during the braking process. In this work, we developed a novel ten-degree-of-freedom (ten-DOF) dynamics model of a trailer bogie brake system based on dynamic and tribological theories. The non-linear factors including disc-pad non-linear friction and wheel-rail non-linear interactions were considered. The field test results demonstrated that more intense vibrations occur at low vehicle speeds, which is also supported by numerical simulations based on the novel dynamics model. The abnormal vibration is caused by the limit cycle oscillation of the brake unit, and it leads to the instability of the whole system. To further understand the instability of the trailer bogie brake system, the equilibrium points and the linear stability of the system under different braking conditions and suspension parameters were investigated based on the Lyapunov theory. Then, the non-linear vibration response and stability of the trailer bogie brake system affected by the suspension parameters were further calculated by numerical integration methods. The results demonstrated that the mounting stiffness of the brake unit approached 3.4×10 7N/m and the primary suspension stiffness approached 6.0×10 5N/m or remained within the range of 10×10 5-14×10 5N/m, causing more violent vibrations and increased system instability. A primary suspension damping exceeding 2.4×10 4N·s/m was conducive to improving system stability.