Temperature dependence of polarization relaxation in semiconductor quantum dots