The two-photon luminescence (TPL) of small 10 nm × 40 nm colloidal gold nanorods (GNR) is investigated at the single object level, combining polarization-resolved TPL and simultaneously acquired topography. A very high dependence of the TPL signal with both the nanorods longitudinal axis and the incident wavelength is observed, confirming the plasmonic origin of the signal and pointing to the limit of the analogy between GNRs and molecules. The maximum two-photon brightness of a single GNR is measured to be a few millions higher than the two-photon brightness of fluorescein molecules. The spectral analysis of the TPL evidence two emission bands peaks: in the visible (in direct connection with the gold band structure) and in the infrared (IR). In both bands, the TPL signal is observed to vary quadratically with the excitation beam; the signal emitted in either the visible or the IR exhibits, however, different polarization properties. We show that the important TPL observed in these small gold nanorods results from resonance effects both at the excitation and emission level: local field enhancement at the longitudinal surface plasmon resonances (LSPR) first results in an increase of the electron–hole generation. Further relaxation of electron–hole pairs then mostly leads to the excitation of the GNR transverse plasmon mode and its subsequent radiative relaxation.