4 ms·
Vertex has a training service, rather than spinning up a notebook. If you can dockerize the training job you just need to upload the container and data to gcs t
by te_chris 4y ago
Vertex has a training service, rather than spinning up a notebook. If you can dockerize the training job you just need to upload the container and data to gcs then it’s point and click to run once - assume it’s some sort of kubeflow or whatever in the background.
- tysam_and 4y agoThat sounds nice for the more involved big runs. I'll try to take a look at it sometime. I have some training runs that loop in just under 7 seconds at the shortest for the actual training portion of the run (maybe 9-10 including notebook spinup). After doing this for so many years, I've found that keeping it as short as humanly possible is usually the way to go. :D That said, I can definitely take a look at it for scale once I need to go over something like 1-2 hours maybe! :D
- te_chris 4y agoYep, definitely best to do somewhere else if you can and just need to iterate, but it’s a good option when you have to do a big one and eat some real GPU time.