This study focuses on a comprehensive investigation of the inter-modular connection shear behaviour under lateral load using theoretical, experimental, and numerical analyses. Initially, three design configurations of proposed inter-modular connection with varying bolt sizes and hole tolerances were tested in shear, and their load-deformation behaviours were studied. Finite element models were then developed in ANSYS and validated against the test results obtained from the experiments. The connections were identified as slip critical connections for serviceability design, as they tend to fail in slippage even at a very small lateral load. Further, evaluation of combined tension and shear effects on the connections confirmed that the failures were due to the combined effect not purely by shear, and therefore connections of this type should consider this as the most critical design check. Based on findings, this paper then describes a methodology for estimating the overall stiffness of inter-modular connections, such that those stiffness values can be employed in modelling the inter-modular connections as a link or spring type elements in the global model of modular buildings. This paper also presents recommendations and suggestions for future enhancement of inter-modular connection designs highlighting the shear slip behaviour and onsite installation constraints.