Operational modal analysis (OMA) is a powerful vibration analysis tool and widely used for structural health monitoring (SHM) of various system systems such as vehicles and civil structures. Most of the current OMA methods such as pick-picking, frequency domain decomposition, natural excitation technique, stochastic subspace identification (SSI), and so on are under the assumption of white noise excitation and system linearity. However, this assumption can be desecrated by inherent system nonlinearities and variable operating conditions, which often degrades the performance of these OMA methods in that the modal identification results show high fluctuations. To overcome this deficiency, an improved OMA method based on SSI has been proposed in this paper to make it suitable for systems with strong nonstationary vibration responses and nonlinearity. This novel method is denoted as correlation signal subset-based SSI (CoS-SSI) as it divides correlation signals from the system responses into several subsets based on their magnitudes; then, the average correlation signals with respective to each subset are taken into as the inputs of the SSI method. The performance of CoS-SSI was evaluated by a simulation case and was validated through an experimental study in a further step. The results indicate that CoS-SSI method is effective in handling nonstationary signals with low signal to noise ratio (SNR) to accurately identify modal parameters from a fairly complex system, which demonstrates the potential of this method to be employed for SHM.