As a non-contact and full-field testing method, high-speed camera-based modal analysis has become a feasible and acknowledged approach. However, extracting small displacements from noisy images has experienced high level of difficulty, especially in high frequency range. This paper proposes a novel adaptive spatial filtering (beamforming) algorithm to extract the displacement signals using high-speed camera. In the proposed algorithm, one pixel is considered as a sensor measuring displacement and a set of pixels are therefore taken as the elements of sensor array. Then, an adaptive spatial filtering acting on this sensor array is proposed. The proposed approach mainly includes three steps. Firstly, a set of pixels are selected to compose a sensor array according to signal to distortion and noise (SINAD). Secondly, a node/antinode searching scheme is proposed based on sinusoid-based piecewise functions, which works as an adaptive filter to match mode shape and enhance modal displacement. Finally, the output of spatial filtering is adopted as the system response for the identification of modal parameters. To validate the performance of the proposed method, simulation and experiment studies are conducted based on measuring the vibration model properties of a free-free beam, which includes a comparison with LK optical flow method and conventional accelerometer-based method. The results show that the SNR of estimated displacement and computational efficiency is significantly improved without using additional sensors. The proposed method paves a way for broadening the applications of using high-speed camera for full-field vibration measurements.