Additional evidence against the second hypothesis: almost nothing in LLM-land is double precision anyway, weights are generally half-precision (or something lik

Additional evidence against the second hypothesis: almost nothing in LLM-land is double precision anyway, weights are generally half-precision (or something like bfloat16 with more range by less precision than IEEE float16).