Nonlinear error compensation algorithm for signal resolution of chromatic confocal measurements

Jiacheng Dai, Wenhan Zeng, Wenbin Zhong, Jane Jiang, Wenlong Lu

Research output: Contribution to journalArticlepeer-review


The mechanisms of the nonlinear error, primarily stemming from the deviation of the polynomial model fitting the wavelength-distance mapping relationship in chromatic confocal measurement, have been elucidated. A compensation method is proposed for the nonlinear error, based on a regression tree model. Through calibration experiments using our chromatic confocal sensor, an impressive improvement of over 40 % in precision could be achieved by our proposed compensation method, which advances from ±1 μm to ±0.6 μm within a 4 mm measurement range.
Original languageEnglish
Article number114091
Number of pages8
JournalMeasurement: Journal of the International Measurement Confederation
Early online date3 Jan 2024
Publication statusPublished - 28 Feb 2024

Cite this