Yes, the panel temperature can be hugely different due to absorption of radiation (I don't think it's the thermal mass, which as you say is going to be negligib

Yes, the panel temperature can be hugely different due to absorption of radiation (I don't think it's the thermal mass, which as you say is going to be negligible). Ambient air temperature is not a good way to approximate the panel's temperature. There's a huge difference between ambient air temperature and an object's surface temperature due to (among other things) the latter's absorption/emittance of radiation. As I mentioned in another comment, I think what's happening is the dew (fog) on the surface of the panel is evaporating and the panel is cooling due to the evaporative cooling impact. Specifically, the surface gets cooler as latent heat is used to phase change liquid to vapour, and degrades the air film on the surface (which also contributes to thermal resistance of panel). This would also explain why you're getting such a huge peak, since the latent heat impact is nonlinear. So the temperature of the solar panel is increasing linearly as the air/sun rises, then the dew starts evaporating and the temperature increase will stop.