In this letter, a compact Ridge Substrate Integrated Waveguide (RSIW) filter with transmission zeros (TZs) is developed by employing the higher-order mode coupling and the mixed electric and magnetic coupling. A novel RSIW cavity with triangular ridge is designed to implement the coupling of the higher-order mode in order to achieve a TZ above the passband. The mixed coupling is employed to obtain another TZ below the passband without increasing the size of the filter. A prototype of the mixed-coupled RSIW filter with triangular ridge is designed and tested. It is shown that high selectivity and compact size are achieved for the proposed filter.