felarof

13 ms·

1. 151.▲
   by felarof 2y ago

   Oops, my calculation was wrong. Let me add an edit to the blog, thanks for pointing it out! My train step was taking 30s. And I was using a batch size of 16 and seq length of 64, making the training speed as (16*64/30) tokens per sec =
2. 152.▲
   by felarof 2y ago

   Good question. We used a new AI+math-based testing tool (benchify.com) to run comparison tests, but we are working on building more robust infrastructure for this. Translating models from PyTorch to JAX is core to our strategy. That said, t
3. 153.▲
   by felarof 2y ago

   Thanks for the feedback! I appreciate you pointing that out. My understanding was based on the PyTorch documentation for scaled_dot_product_attention ( https://pytorch.org/docs/stable/generated/torch.nn.functio
4. 154.▲
   by felarof 2y ago

   Ahh, interesting, will take a look! Curious what are the steps to run PyTorch on AMD (does it work out-of-box with PyTorch+rocm docker image)? Does torch.compile work smoothly?
5. 155.▲
   by felarof 2y ago

   Good question. No good metric give performance depends on software stack (JAX vs PyTorch) + optimizations. But my take performance per dollar of TPU > AMD > NVIDIA.
6. 156.▲
   by felarof 2y ago

   Haha, TPUs are not slow :) All of Google's training (including Gemini models) is done on TPUs. There are good 1p [a] and 3p [b] benchmarks comparing TPUs vs NVIDIA GPUs. [a] - https://github.com/GoogleCloudPlatform&#x2F
7. 157.▲
   by felarof 2y ago

   Yeah, used Obsidian Publish. But struggling to get custom domain to work with it (have emailed support).
8. 158.▲
   by felarof 2y ago

   A few weeks ago, I did a Show HN explaining our journey: https://news.ycombinator.com/item?id=41512142 . We initially started with the goal of fine-tuning LLaMA 3 on TPUs, but PyTorch XLA was clunky, so we decided to rewrite
9. 159.▲
   by felarof 2y ago

   Nice! I need to calculate MFU. GPU, VRAM details can be found in the repo: https://dub.sh/amd-405b-res . I plan to reattempt the training run next weekend and JIT the entire training step to calculate MFU then
10. 160.▲
    by felarof 2y ago

    (author here, sorry for the delay in replying, was stuck in back-to-back meetings) I updated our github repo to include GPU, VRAM utilization data ( https://github.com/felafax/felafax?tab=readme-ov-file#amd-40... ) Note:
11. 161.▲
    by felarof 2y ago

    Hey HN, we recently fine-tuned the llama3.1 405B model on 8xAMD MI300x GPUs using JAX instead of PyTorch. JAX's advanced sharding APIs allowed us to achieve great performance. Check out our blog post to learn about the cool sharding tr
12. 162.▲
    We fine-tuned Llama 405B on AMD GPUs(publish.obsidian.md)

    495 points by felarof 2y ago|101 comments
13. 163.▲
    by felarof 2y ago

    Thank you for the detailed analysis. We need to spend some time thinking and coming up with a price comparison like this. We’ll use this as inspiration!
14. 164.▲
    by felarof 2y ago

    Thanks for the detailed feedback! Yes, supporting LoRA fine-tuning is one of the things we are already working on. btw, we have LoRA supported with Llama3 PyTorch-XLA model. Check that out in meanwhile.
15. 165.▲
    by felarof 2y ago

    Unsloth is great! They focus on single-GPU and LoRA fine-tuning on NVIDIA GPUs. We are initially trying to target multi-node, multi-TPU, full-precision training use cases. That said, in terms of single-GPU speed, we believe we would be behi
16. 166.▲
    by felarof 2y ago

    Haha, yeah, good point. I'll remove it.
17. 167.▲
    by felarof 2y ago

    Thank you! Yeah, there are a few complexities and very little documentation around JAX, plus a lot of missing libraries.
18. 168.▲
    by felarof 2y ago

    Totally agree, thanks for feedback! This is one of the TODOs on our radar.
19. 169.▲
    by felarof 2y ago

    Check out this benchmark where they did an analysis: https://github.com/GoogleCloudPlatform/vertex-ai-samples/blo... . At the bottom, it shows the calculations around the 30% cost efficiency of TPU vs GPU. Our rang
20. 170.▲
    by felarof 2y ago

    I think you slightly misunderstood, and I wasn't clear enough—sorry! It's not a 30-70% speedup; it's 30-70% more cost-efficient. This is mainly due to non-NVIDIA chipsets (e.g., Google TPU) being cheaper, with some additional
21. 171.▲
    by felarof 2y ago

    30% is a conservative estimate (to be precise, we went with this benchmark: https://github.com/GoogleCloudPlatform/vertex-ai-samples/blo... ). However, the actual difference we observe ranges from 30-70%. Also, cal
22. 172.▲
    by felarof 2y ago

    Ahh, the reddit thread is referring to edge TPU devices, will check it out. Google also has Cloud TPUs, which are their server-side accelerators, and this is what we are initially trying to build for!
23. 173.▲
    by felarof 2y ago

    Google published this benchmark a year or so ago comparing TPU vs NVIDIA ( https://github.com/GoogleCloudPlatform/vertex-ai-samples/blo... ) Conclusion is at the bottom, but TLDR was TPUs were 33% cheaper (performan
24. 174.▲
    by felarof 2y ago

    It took roughly 2-3 weeks to translate Torch to JAX, but I had past experience writing JAX from my time at Google. We spent nearly 4 weeks getting PyTorch XLA working on TPU. Hope that answers your question!
25. 175.▲
    Show HN: Tune LLaMa3.1 on Google Cloud TPUs(github.com)

    189 points by felarof 2y ago|52 comments
26. 176.▲
    by felarof 2y ago

    Very interesting!
27. 177.▲
    by felarof 2y ago

    Take my money, if someone can build an obsidian plugin similar to this!
28. 178.▲
    by felarof 2y ago

    I have a small YT channel that teaches JAX bit-by-bit, check it out! https://www.youtube.com/@TwoMinuteJAX
29. 179.▲
    by felarof 2y ago

    If you want to a/b GPT and Claude3 test, try out http://www.clashofgpts.com
30. 180.▲
    by felarof 3y ago

    I built https://clashofgpts.com to use Gemini and GPT4 side-by-side. A good way to use your Gemini Ultra 2months trial and see if you want to switch to Google AI Premium subscription.