4 ms·
OP mentions the failure rate of GPUs as "If this were in GPU land, it would have failed within the first few days for sure.". In my humble opinion, we never ha
by ganeshkrishnan 3y ago
OP mentions the failure rate of GPUs as "If this were in GPU land, it would have failed within the first few days for sure.".
In my humble opinion, we never had failures of GPU even for large scale training. Our current training batch job is a 20GB json file which takes 6 hours just to load and has been running for more than 15 days with not a hiccup. And we are using the older Tesla T4.
GPUs have memory constraint issues but if you can plan and work around it, I havent seen it crash in real life.
- teaearlgraycold 3y agoHa! We’re also committing great sins of computation against T4s at our company. Hopefully, as I learn, things get less janky.
- gwern 3y ago> And we are using the older Tesla T4. That's an undemanding and well-debugged chip by this point (6 years ago!). So you aren't experiencing any of the pain people using A100s or H100s (never mind people who have to stand up clusters with B100s soon) are going through now.
- icpmacdo 3y ago> never mind people who have to stand up clusters with B100s soon Gwern(or anyone else) do you have any resources on this?
- shrubble 3y agoHave you checked if there is a faster way to parse your JSON? 3Gbytes/hour to load a file seems slow on today's CPUs...
- flybarrel 3y agoWhat would be an ideal (or more appropriate) speed?
- shrubble 3y agoWell it would depend on the specifics of the JSON file but eyeballing the stats at https://github.com/miloyip/nativejson-benchmark/tree/master https://github.com/miloyip/nativejson-benchmark/tree/master seems to indicate that even on a 2015 MacBook the parsing proceeds using e.g. Configuru parser at several megabytes per second.
- mc10 3y agosimdjson can parse JSON files at ~2.5-3GB/s: https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson
- deleted 3y ago[deleted]
- nl 3y ago> 20GB json file… takes 6 hours just to load Err you definitely should be doing something about that. 20GB on T4s (how many?) isn’t really comparable to terabytes on thousands of A100s.