3 ms·
A recent ex-googler here: quality of Google3 in general is pretty good, but the LLM training bits are so abysmal that I know people who have resigned instead of
by ein0p 3y ago
A recent ex-googler here: quality of Google3 in general is pretty good, but the LLM training bits are so abysmal that I know people who have resigned instead of working on it. And it’s also extra slow because getting a couple local GPUs is not really an option. So you’re forced to “develop in Colab” which works for some things and not for others and in general sucks ass if you’re working on anything substantial. For anything more substantial you’ll be launching stuff on some resource pool, waiting for like 10-15 minutes until it starts (much longer for large models), and then trying to divine why it failed from voluminous and sometimes indecipherable crash logs which also hang your browser when cluster UI tries to load them.
Rumors of Google’s AI code superiority are vastly overblown in 2024. I’m currently at another major AI lab, and the code here can actually be understood and worked on, which I consider to be a massive advantage.
- alsoworkedthere 3y agoFinally, an accurate portrayal! Google has superb robustness and code quality, with garbage-level usability. Once you're setup, you can kick off many massive training jobs and compare results easily. However, getting to that point is really hard. You'll never figure out how to use the ML infrastructure and libraries on your own. You can only get it to work by meeting with the teams that wrote the infra so they can find and fix every error and misconfiguration. Usually, there is one single way to get things working together, and neither the documentation nor the error messages will get you to that brittle state. It's near impossible to get a VM with a TPU or GPU attached, so there's no way to debug issues that happen between the library and the accelerator. Plus somehow they've made Python take longer to build (??!!) and run than C++ takes, so your iteration cycle is several minutes for what would take seconds at any other place. Fun stuff! Somehow it's still one of the best places to do ML work, but they sure try to make it as difficult as possible.
- ein0p 3y agoGoogle doesn’t use VMs internally to run workloads. But yeah, seconds-long dev iteration cycles take minutes or even tens of minutes there.