4 ms·

X:https://x.com/deepseek_ai/status/1893836827574030466 https://x.com/deepseek_ai/status/1893836827574030466 BF16 support Paged KV cache (block size 64) 3000

X:https://x.com/deepseek_ai/status/1893836827574030466 https://x.com/deepseek_ai/status/1893836827574030466 BF16 support Paged KV cache (block size 64) 3000 GB/s memory-bound & 580 TFLOPS compute-bound on H800

WithinReason 2y ago
That's 90% bandwidth efficiency and 60% compute efficiency https://www.nvidia.com/en-us/data-center/h100/ https://www.nvidia.com/en-us/data-center/h100/
- helloericsf 2y ago
  They don't have h100. wink,wink.
  - rfoo 2y ago
    They have H800s which have exactly same memory bandwidth and max FLOPS.
    - pk-protect-ai 2y ago
      What about NVLink? Does it plays a role here?
      - rfoo 2y ago
        For FlashMLA? No. The code here runs on one GPU only and do not have a builtin communication part.
        pk-protect-ai 2y ago
        But for the training it does. You need to communicate gradient changes between GPUs.
    - deleted 2y ago
      [deleted]