2 ms·
There's a glimmer of hope with ROCmFP4 which seems to double the current throughput: https://github.com/charlie12345/rocmfp4-llama https://github.com/charlie123
by prima-facie 3mo ago
There's a glimmer of hope with ROCmFP4 which seems to double the current throughput: https://github.com/charlie12345/rocmfp4-llama https://github.com/charlie12345/rocmfp4-llama
- syntaxing 3mo agoI saw that but end of the day, the chips themselves don’t have hardware support for FP4. There’s smart ways around this limitation but it will never natively be close to true FP4 performance like MXFP4 and NVFP4 (happy to be proven wrong though).