4 ms·
I thought all current implementations accumulate into a fp32 instead of accumulating in fp16.
by jiggawatts 2y ago
I thought all current implementations accumulate into a fp32 instead of accumulating in fp16.
- KeplerBoy 2y agoI haven't looked at all implementations, but the hardware (tensor cores as well as cuda cores) allows you to accumulate at fp16 precision.
- janwas 2y agoWe (gemma.cpp) recently started accumulating softmax terms into f64. There is at least one known case of this causing differing output, but after 200 tokens, hence unlikely to be detected in many benchmarks. Does anyone have experience with higher-precision matmul and whether it is worthwhile?
- ComputerGuru 2y agoIsn’t 200 tokens basically nothing? Did you mean to say 2000?
- janwas 2y agoThat's indeed short for some actual uses such as summarization, but AFAIK many/most? evals involve generating less than 200.