The surface integrity of a material has a significant impact on fretting wear performance. Based on the analysis of the material surface integrity, the machining parameters were optimized to obtain the best fretting wear performance of CuNiAl alloy. First, the effects of ultrasonic vibration-assisted milling (UVAM) processing parameters on the surface integrity of CuNiAl alloy were studied. Consequently, the relationship between UVAM processing parameters and CuNiAl alloy surface integrity parameters was established through the optimized support vector machine (PSO-SVM) model. The relationship between the surface integrity parameters, processing parameters, and fretting wear performance was established. The best fretting wear performance was obtained at a spindle speed of 147.4 m/min, feed rate of 76 mm/min, and vibration current of 199.5 mA.