3 ms·
Finally, an accurate portrayal! Google has superb robustness and code quality, with garbage-level usability. Once you're setup, you can kick off many massive t
by alsoworkedthere 3y ago
Finally, an accurate portrayal!
Google has superb robustness and code quality, with garbage-level usability. Once you're setup, you can kick off many massive training jobs and compare results easily. However, getting to that point is really hard. You'll never figure out how to use the ML infrastructure and libraries on your own. You can only get it to work by meeting with the teams that wrote the infra so they can find and fix every error and misconfiguration. Usually, there is one single way to get things working together, and neither the documentation nor the error messages will get you to that brittle state.
It's near impossible to get a VM with a TPU or GPU attached, so there's no way to debug issues that happen between the library and the accelerator. Plus somehow they've made Python take longer to build (??!!) and run than C++ takes, so your iteration cycle is several minutes for what would take seconds at any other place. Fun stuff! Somehow it's still one of the best places to do ML work, but they sure try to make it as difficult as possible.
- ein0p 3y agoGoogle doesn’t use VMs internally to run workloads. But yeah, seconds-long dev iteration cycles take minutes or even tens of minutes there.