4 ms·
Amazon's Trainium2 AI Accelerator Features 96 GB of HBM, 4x Training Performance
- dvdbloc 3y agoAs someone from outside the AI space, how useful are chips like this from Amazon? Meaning am I going to have to rewrite my entire stack to make use of this and then be locked into Amazon forever? Or are workloads at least portable in some capacity?
- pram 3y agoThere’s semi lock-in. You can use torch, but you also have to use their framework. Not much different than using cuda I guess.
- belval 3y agoIt's a bit of a mixed bag. NeuronSDK is implemented with Torch XLA so there's not real lock-in there. Your code will usually just work on CUDA if it works in Trainium. However for larger models (ZeRO, Sharding, etc...) they have their own libraries that are more coupled to the actual accelerator topology because using something like DeepSpeed was not possible. Overall the challenge is often more with making your code work on their accelerator, not porting it away from their accelerators so I wouldn't be too concerned about lock-in.
- yeldarb 3y agoWe tried Inferentia & though they claim their compiler should "just work" on your existing models it was producing a bunch of NaNs for us & was opaque enough that we didn't really have a path forward so we gave up on it. If we were operating at 100x or 1000x the scale it might make sense to spend the extra engineering time, but we just pay for the NVIDIA chips (especially because a big chunk of our customers run at the edge where NVIDIA compatibility is even more important so we'd be doing double the engineering on an ongoing basis).
- bfeynman 3y agonot very useful, but it's getting better with XLA. It does not just "work" by default for any given model, and if you're missing an HLO that has an optimized kernel written you lose benefits of accelerated computing thus people won't use it. I would not put any effort into it until they truly have it where you just cast `to_device(..)`.
- latchkey 3y agoAMD is coming out with the MI300x on Dec 6th, it has 192 GB of HBM3 and you put 8 of them in a chassis all connected to each other with super fast connections (5.2 TB/s of memory bandwidth) and each chassis is networked with 800-900G connections. We don't know what the tflops will be for the 300x yet, but it seems like the limitations here are more about memory than it is about tflops.
- joshhart 3y agoWill be a long time before a large number of these are available in clouds though, not until summer at least.
- latchkey 3y agoI predict sooner, but that's cause it is what I'm working on now. ;-)
- buildbot 3y agoNot sure of the relationship between MI300x and MI300, but MI300 is claimed to be 2.2 PFlops FP8 (with structured sparsity, which I did not know AMD was implementing!!!): https://www.amd.com/en/claims/instinct https://www.amd.com/en/claims/instinct
- latchkey 3y agoThere are 3 versions of MI300. X is pure gpu, A is more like an APU (cpu+gpu), C is pure cpu.
- belval 3y agoThose are all impressive numbers, but did they actually fix their software stack? Are there people here that have actually trained with those and if so how was the experience? I'm genuinely curious because it seems like every other AI Accelerator/GPU thread on HN mentions MI300X, yet I haven't talked to/met anyone who trained anything with AMD.
- 3y ago
- Vt71fcAqt7 3y agoI hope the development of all these accelerators drives developers and frameworks to move away from CUDA only and even better to move to a open standard of some sort (eg. ROCM, if it can atain parity with CUDA in terms of usability). It would be such a waste for each framework to need to have a seperate backend for each hardware vendor. Then we will end up in the same desperate situation as game engines with Vulkan, Metal and DX12 with a side of OpenGL and Intel OneAPI (for things like eg. Blender). That sadly seems to be the way it's going right now.
- joshhart 3y agoTrainium and Inferentia sadly aren't going this way, they have their own approach https://github.com/aws-neuron/transformers-neuronx https://github.com/aws-neuron/transformers-neuronx . I think the best case scenario is some middleware like Triton https://openai.com/research/triton https://openai.com/research/triton becomes the standard and we can build adapters to all of these backends. It is disappointing though.
- janalsncm 3y agoIn my experience AWS Nvidia machines are extremely overpriced. $30/hour for 8 A100s, and you have to pay for all 8 at a time. Most competitors I have seen are around half that price. At $30/hour I’m literally spending a dollar or two just installing dependencies. Whether AWS custom hardware is reasonably priced, and how much of a pain in the butt it will be to port my code, and how training speed compares to Nvidia GPUs will be my main deciding factors. I’m not locked in to AWS so Trainium will need to be faster and cheaper (cheaper per epoch) to be competitive for me.
- thrtythreeforty 3y agoCan't you install dependencies on a potato instance then change the instance to the GPU when you're ready to roll?
- dylan604 3y agoI was wondering the same thing of why not have an image ready to go, but then realized why take away the easy complaint with rational logic?
- buildbot 3y agoThat only helps if your dependencies are slower moving targets than whatever research/product you are working on. Otherwise, the dependencies may be different each experiment or attempt. Or whatever cloud provider you use does something funky and the setup on one machine is less portable to a GPU machine than you might expect.
- janalsncm 3y agoThe $2 dependency installation cost isn’t the problem, it’s the $30/hour training cost.
- cbsmith 3y agoA month with an a2-ultragpu-8g is going to cost you ~$6400/month or about $36.92. Where are you getting half price?
- buildbot 3y ago650 TOPs Is surprisingly low for narrow precision datatypes - I assume they are quoting the fastest performing datatype (Int8 or maybe even lower!). That's on par with an A100, an H100 hits 2 PETA TOPs (4 PTOPs 2:4 sparse!) of Int8!
- teruakohatu 3y agoEveryone was trying to get their hands on A100s right until the H100 was released. If Amazon can replace A100s that’s a big win.
- 2OEH8eoCRo0 3y agoWhat happens to all this data center hardware when Amazon or others get rid of it? Do they sell it or do they shred it?
- sumtechguy 3y agoThe answer is 'yes'. It probably depends on how they accounted for it on their taxes. Some sold, some destroyed. You can usually find bunches of it on ebay. Unless it is 'custom' then it will most certainly be destroyed as there is no market for it.
- 2OEH8eoCRo0 3y agoHow do they know there's no market for their outdated in-house silicon? It's a shame that all this compute is being manufactured only to go to the shredder.
- riku_iki 3y agowho would need something very used/abused without software support?..
- sumtechguy 3y agoAlso some companies consider the bespoke ones 'IP' and do not want it getting out.
- riku_iki 3y agoIts not necessary they don't want, IP has cost attached in accounting, and they would need to explain for tax and investment reporting why they give this IP away.