This paper presents optical duobinary and dicode signalling, as alternatives to the binary format, in order to improve the transmission performance in the presense of non-linear effects in a dense wavelength division multiplex (WDM) optical system. Duobinary signalling is applied to an optical system to explore the reduction of stimulated Brillouin scattering (SBS) effects. Duobinary signalling suppresses the SBS effects, and an eye-opening improvement of 0.25 to 1.2 dB is achieved relative to binary transmission over a range of input power levels. An experimental study demonstrates that duobinary modulation suppresses the four wave mixing (FWM) products of a dense WDM system by a maximum of 3 dB. The suppression is maintained over a range of channel spacings. An investigation of the impact of fibre dispersion on FWM products under binary, duobinary and dicode modulation in a dense WDM system is then performed, with interchannel spacing and optical power variation. This leads to the development of a set of guidelines for the application areas, in which it is appropriate to use duobinary or dicode modulation in WDM systems as a means of mitigating the impact of FWM.