Fast and accurate peak wavelength extraction of chromatic confocal signals is of vital importance for efficient chromatic confocal measurement. Existing peak extraction algorithms do not effectively balance calculation efficiency and extraction accuracy. For example, centroid algorithms have high computational efficiency but low accuracy; model-based fitting algorithms have high extraction accuracy but low calculation efficiency. In this paper, a fast and accurate mean-shift vector based peak wavelength extraction algorithm is proposed. In this algorithm, the initial peak wavelength is preset with a centroid algorithm; then the corresponding mean-shift vector is calculated, after which, a new peak wavelength can be calculated by shifting the peak wavelength along the mean-shift vector. This process is then iteratively carried out until a preset ending condition is met. Our experimental results show that the mean-shift vector based peak extraction algorithm has an accuracy equal to that achieved with Gaussian fitting while the efficiency improves by over 70 times.