4 ms·

I am surprised that they could have you up and running so fast. Giventhe worldwide shortage of AI compute, and extreme demand, How is Amazon able to onboard n

• leetharris 3y ago
  There's a surprising amount of A100s available right now, everyone wants H100s very badly. Those are still really hard to find. We also need GPUs for two things: training and inference. Our training needs a bunch of A100/H100 or large TPU nodes. But our final models are all pretty small (we're in ASR, TTS, translation, etc) relative to most commercial LLMs. So we run a large amount of inference on GPUs like T4s, L4s, etc that are much easier to obtain. Our intent was to try to find a unified, scalable platform that we could train/infer on that didn't have the cold boot problem AWS GPU instances have when our inference needs go beyond our dedicated hardware. With AWS the P99 cold boot times were over 5 minutes which means we'd have to queue inference and we'd end up paying for 5 extra minutes of GPU time. This isn't that big of a deal, but when you're spinning thousands of GPUs up and down it adds up to a ton of money every month. Since our models require a lot less kernels than a lot of modern LLMs, TPUs seemed like a really good fit. But after dealing with GCP the last few months we abandoned our plans to port our model to JAX and decided to stick with GPUs for now.