In this work, rapid thermal annealing was performed on InGaAsGaAs quantum dot infrared photodetectors (QDIPs) at different temperatures. The photoluminescence showed a blueshifted spectrum in comparison with the as-grown sample when the annealing temperature was higher than 700 °C, as a result of thermal interdiffusion of the quantum dots (QDs). Correspondingly, the spectral response from the annealed QDIP exhibited a redshift. At the higher annealing temperature of 800 °C, in addition to the largely redshifted photoresponse peak of 7.4 μm (compared with the 6.1 μm of the as-grown QDIP), a high energy peak at 5.6 μm (220 meV) was also observed, leading to a broad spectrum linewidth of 40%. This is due to the large interdiffusion effect which could greatly vary the composition of the QDs and thus increase the relative optical absorption intensity at higher energy. The other important detector characteristics such as dark current, peak responsivity, and detectivity were also measured. It was found that the overall device performance was not affected by low annealing temperature, however, for high annealing temperature, some degradation in device detectivity (but not responsivity) was observed. This is a consequence of increased dark current due to defect formation and increased ground state energy.