5 ms·
MK1 Flywheel Unlocks the Full Potential of AMD Instinct for LLM Inference
- hereme888 3y agoIsn't this an unfair comparison, given they test the A6000 Ampere (half the price of MI210) instead of the Ada generation (still a cheaper than the MI210, but ~2x the CUDA cores as Ampere)?
- paul_mk1 3y agoComparisons between different chip architectures are imperfect. In our opinion the most fair thing to do is 1) match the TFLOPs (since these workloads are compute bound), and 2) find a similar card that can run the same size models. Since MI210 has 64GB, the closest one is the A6000 with 48GB. Many of the less expensive ADA cards can't even fit a 13B model in VRAM, so a comparison could not even be made.
- wintercharm 3y agoThis is sick!
- bloopernova 3y agoIt will be fascinating to see if AMD's open source ML platform beats Nvidia's closed one. (Usually, the more open platform becomes more popular.)
- jjn2009 3y agoit should at least provide some better price to performance options once the AMD software/drivers get up to speed.
- fbdab103 3y agoBeen waiting over a decade for AMD to get serious about their software.
- roenxi 3y agoUnlikely; that is almost in no-one's interests. The story down that path is ROCm gets 95% of the way there feature wise, then Nvidia catches a sudden case of community spirit and opens up CUDA. Although that is mostly wishful thinking. I expect ROCm probably works quite well on the server, the issues in my experience is that it assumes that the AMD card is dedicated to processing and once it tries to run graphics and compute at the same time it becomes Too Hard to manage memory for both and then the driver locks up. So their buggy drivers basically kill any sort of hobbyist or researcher effort unless there is a dedicated supercomputer on hand. Much better plan to go with Nvidia I assume.
- pjmlp 3y agoNot really, CUDA and DirectX are more popular than Khronos stuff because of tooling, SDKs and overall developer experience. Simple stuff that matters more for most folks outside FOSS circles. It is up to Intel and AMD to provide similar experiences.
- matiasdwek 3y agoCan I try this on AWS? Do they have any instances with AMD GPUs?
- jadbox 3y agoIs it open source? What is MK1?
- kristianp 3y agoAccording to [1,2], MI210 Memory Bandwidth is 1,638 GB/s, vs RTX A6000 of 768.0 GB/s, that HBM2e at 4096 bits really beats GDDR6 at 384bits on bandwidth anyway. So I would expect the MI210 to have better utilization for bandwidth-heavy workloads. [1] https://www.techpowerup.com/gpu-specs/radeon-instinct-mi210.c3857 https://www.techpowerup.com/gpu-specs/radeon-instinct-mi210.... [2] https://www.techpowerup.com/gpu-specs/rtx-a6000.c3686 https://www.techpowerup.com/gpu-specs/rtx-a6000.c3686
- paul_mk1 3y agoThis is a good observation, the cards do have different memory bandwidth with the MI210 having more than double the bandwidth via HBM2e. Note that the comparisons between the two cards (MI210 and A6000) are being made for high throughput workloads, and in this regime the performance is compute bound. So as long as the memory bandwidth is decent (as it is for GDDR6 with 768.0 GB/s) the lower memory bandwidth is not the main bottleneck. There are also other architectural differences so any comparison will ultimately be imperfect, but we found A6000 to be the closest match for the workloads that we cared about (ie high throughput workloads at reasonable latencies). Also worth noting that there are still more stack optimizations on the table, which again can shift the bottleneck between compute and memory. In those cases it might makes sense to compare with another card with matched memory bandwidth. paul @ mk1
- nagemsley 3y agoimpressive
- tinoargentino 3y agoNice!!!
- pholos 3y agoLooks like good performance on AMD and NVIDIA, nice job!
- z3phyr 3y agoA little off-topic, but I am rather disappointed by software taking cool stuff names. As soon as I looked at the title, my first thought went to an actual flywheel (A mechanical power storage system). A flywheel in space powering some unnatural energy beam is a Vader-esque evil-cool-evil system I frequently imagine.
- nabakin 3y agoConsidering how well their last post turned out[0] (read the comments), forgive me for not having much confidence in what this company claims. I'll need independent testing. [0] https://news.ycombinator.com/item?id=37016413 https://news.ycombinator.com/item?id=37016413
- paul_mk1 3y agoYou can try it yourself on SageMaker using NVIDIA. There's a free trial. https://aws.amazon.com/marketplace/seller-profile?id=seller-wkonsb76xsee6 https://aws.amazon.com/marketplace/seller-profile?id=seller-... For AMD, you'll have to wait until these cards become available on the cloud.
- Zetobal 3y agoDoubt. Especially after the last announcement. Probably again optimised for the benchmark the green and low comment fanboys are also a bit sus.
- tucnak 3y agoPlease stop manipulating HN frontpage for publicity.
- westhanover 3y agoThey definitely are. Weird one word comments on a closed source product that is a wrapper of GGML, trying to cash in on AI hype. Even if we don't have a problem with your grift, please don't clog up this space with your nonsense.
- kkielhofner 3y agoI support any progress to erode the Nvidia monopoly. That said from what I'm seeing here the free and open source (less other aspects of the CUDA stack, of course) TensorRT-LLM[0] almost certainly bests this implementation using the Nvidia hardware they reference for comparison. Compare to real (datacenter) Nvidia GPUs that aren't three years old and prepare to get your hair blown back. I don't have an A6000 but as an example with the tensorrt_llm backend for Nvidia Triton Inference Server (also free and open source) I get roughly 30 req/s with Mistral 7B on my RTX 4090 with significantly lower latency and I'm in the early stages of tuning. Comparison benchmarks are tough, especially when published benchmarks like these are fairly scant on the real details. TensorRT-LLM has only been public for a few months and if you peruse the docs, PRs, etc you'll see they have many more optimizations in the works. In typical Nvidia fashion TensorRT-LLM runs on any Nvidia GPU (from laptop to datacenter) going back to Turing (five year old cards) assuming you have the VRAM. It even works on their Jetson line of hardware. You can download and run this today, free and "open source" for these implementations at least. I'm extremely skeptical of the claim "MK1 Flywheel has the Best Throughput and Latency for LLM Inference on NVIDIA". You'll note they compare to vLLM, which is an excellent and incredible project but if you look at vLLM vs Triton w/ TensorRT-LLM the performance improvements are dramatic. Of course it's the latest and greatest ($$$$$$ and unobtanium) but one look at H100/H200 performance[3] and you can see what happens when the vendor has a robust software ecosystem to help sell their hardware. Pay the Nvidia tax on the frontend for the hardware, get it back and then some as a dividend via the software especially when anything close (assuming this even is) is another paid product/SaaS/whatever their monetization strategy is. At the risk of this turning into an Nvidia sales pitch Triton will do the same thing for absolutely any model via the ONNX, TensorRT, Pytorch, Tensorflow, OpenVINO, etc backends. I have an implementation generating embeddings via bge-large-v1.5 that's also the fastest thing out there. Same for Whisper, vision models, whatever you want. I feel like MK1 must be aware of TensorRT-LLM/Triton but of course those comparison benchmarks won't help sell their startup. [0] - https://github.com/NVIDIA/TensorRT-LLM https://github.com/NVIDIA/TensorRT-LLM [1] - https://github.com/triton-inference-server/tensorrtllm_backend https://github.com/triton-inference-server/tensorrtllm_backe... [2] - https://mkone.ai/blog/mk1-flywheel-race-tuned-and-track-ready https://mkone.ai/blog/mk1-flywheel-race-tuned-and-track-read... [3] - https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/Falcon180B-H200.md#llama-70b-on-h200-up-to-67x-a100 https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source...