This study proposes a novel filter, namely areal RrVMD based on Variational Mode Decomposition (VMD), for decomposing surface areal texture into the form, waviness and roughness. VMD is one of the latest signal decomposition techniques and has been introduced into the field of surface metrology recently. The paper develops a residual-restrained method to further improved the VMD algorithm. It consists of three processing steps: firstly, calculating the robust weight function; secondly, decomposing the surface into the corresponding k modes and a residual by using the devised areal residual-restrained VMD; thirdly, identifying different surface topography features by different wavelengths of modes. This study also proposes a robust algorithm to handle outliers and defects on the measured surface. The experimental results demonstrate that the robust areal residual-restrained VMD can precisely separate form, waviness and roughness and eliminate outliers efficiently.