4 ms·
This is great - especially for small projects and learning. However, there are a few limitations to be aware of if you’re thinking about using it for real proje
by spaceandthyme 9y ago
This is great - especially for small projects and learning. However, there are a few limitations to be aware of if you’re thinking about using it for real projects.
First, is memory. I’ve seen reports of practical limitations being around 500mb. Call me a spoiled millennial noob but the vast majority of my real-world ML projects use a lot more - especially when working with image data.
Second, is getting data into your notebook. Most ML is data heavy and you want a fast way of working with it. For example, the Google Landmark challenge on Kaggle has nearly half a terabyte of unaugmented images just for the test set. You could easily push several terabytes for that one-off challenge alone if you were not careful.
Note: it is possible to work with much less data in many cases by resizing and preprocessing the images as you pull them down but that has problems too. Also, you can use Google Drive but it’s not ideal for large datasets.
Related:
https://www.reddit.com/r/MachineLearning/comments/84532y/n_google_colab_gives_you_free_usage_of_a_k80_gpu/ https://www.reddit.com/r/MachineLearning/comments/84532y/n_g...
https://stackoverflow.com/questions/48750199/google-colaboratory-misleading-information-about-its-gpu-only-5-ram-available/49225460 https://stackoverflow.com/questions/48750199/google-colabora...
- singularity2001 9y ago[[deleted]]
- londons_explore 9y agoThey give you 12GB of GPU ram. The 500mb is simply the host ram for the python runtime. As long as you aren't converting everything to numpy arrays, you'll be fine.
- singularity2001 9y agofantastic, thanks!
- londons_explore 9y agoThe memory limitation is on the host (ya know - the stuff you don't actually need). You get 12GB of GPU memory I think. You can mount a remote ssh disk as a working directory, and for your actual data stream it in from cloud storage direct from tensorflow.