A vibration data-based machine learning architecture is designed for structural health monitoring (SHM) of a steel plane frame structure. This architecture uses a Bag-of-Features algorithm that extracts the speeded-up robust features (SURF) from the time-frequency scalogram images of the registered vibration data. The discriminative image features are then quantised to a visual vocabulary using K-means clustering. Finally, a support vector machine (SVM) is trained to distinguish the undamaged and multiple damage cases of the frame structure based on the discriminative features. The potential of the machine learning architecture is tested for an unseen dataset that was not used in training as well as with some datasets from entirely new damages close to existing (i.e., trained) damage classes. The results are then compared with those obtained using three other combinations of features and learning algorithms—(i) histogram of oriented gradients (HOG) feature with SVM, (ii) SURF feature with k-nearest neighbours (KNN) and (iii) HOG feature with KNN. In order to examine the robustness of the approach, the study is further extended by considering environmental variabilities along with the localisation and quantification of damage. The experimental results show that the machine learning architecture can effectively classify the undamaged and different joint damage classes with high testing accuracy that indicates its SHM potential for such frame structures.