5 ms·
How do you go about running a model this large?
by aquajet 6y ago
How do you go about running a model this large?
- andrewnc 6y agoAzure
- ganstyles 6y agoThis post may be tongue in cheek, but realistically that's right. I ran the Facebook chatbot that was posted maybe a month ago and didn't get good performance until I was using many Tesla v100 GPUs, which are $8k each. Thankfully modern CSPs like Azure allow renting rigs that can handle this pretty easily.
- aquajet 6y agoHow much does that cost?
- ganstyles 6y agoThe largest AWS instances are $30-something/hr on demand, less with reserved time. So pretty expensive.
- cambalache 6y ago21.3 USD/h in vast.ai (8X Tesla V100, 118.8 TFLOPS)
- lumost 6y agoMore or less the cost of a well trained, experienced customer service representative?
- FeepingCreature 6y agoYeah but the cost will come down.
- Jack000 6y agothat's only 256GB, which isn't enough. I'm not sure it's even possible to nvlink 16 v100s. I'd love to try it out for $40/hr if it were possible though.
- robkop 6y agoI would hazard a guess that they will release versions that will be a smaller size (they have in the past). But in order to run this, you'd just have to use a cloud provider, first guesses say it'll be 500GB+ of just weights that ideally you want in memory.
- ganstyles 6y agoEven the Teslas I use have 256GB memory and those are pretty cutting edge. 500GB in GPU memory is insane.
- whymauri 6y agoCould we bank on the Lottery Ticket Hypothesis, distillation, or other model compression algorithms to make these models smaller?
- aquajet 6y agoI would guess so, but compressing it by 1/3rd it's size (ie. distilgpt) would still be quite large. To be fair, I don't know if distillation scales like that.
- sdan 6y agoRealistically you'd be able to train up to 760M param models. For that you'd need 32gb+ VRAM GPUs which I think AWS might have. You can try iwth 16gb VRAM GPUs, but you would need to figure out FP16. https://github.com/shawwn https://github.com/shawwn is doing some work in the GPT-2 space including using TPUs instead -- which has given him pretty good results.
- jaredtn 6y ago32GB RAM is nowhere near what's necessary. It can barely fit the GPT-2 models on there with a batch size in the single digits. We'll need extensive model parallelism libraries (Zero2 from Microsoft) to run this at all.
- jaredtn 6y agoThey use extensive model parallelism when training. Even TPUs (64GB) or Tesla V100 GPUs (32GB) don’t have enough memory to fit a model into a single child, so you’ll need activation checkpointing or model parallelism.