3 ms·

So what happen when a notebook on a given container needs more resources, how do you handle that ?

Jugurtha 6y ago
We specify resources as parameters to limit usage both for notebook servers, and for long-running notebooks we schedule[0][1]. In other words: for notebook servers, we can select the resources we want when starting a notebook, say 5GB of RAM for exploring something and dry-running some code on a sub-sample. Then we schedule the notebook with more resources (GPU, RAM, etc). We can set the timeout to kill the job. We also shut down the notebook servers automatically, and use resources only for the duration of the long-running notebook when they're scheduled. OP might want to look into JupyterHub spawners[2]. They can use the API to make custom spawners that fit their specific needs (handling storage, using specific images, etc). One limitation in ZTJPH in one comment above if I recall correctly is the locality of the cluster. OP also might want to take a look at JupyterHub Enterprise Gateway[3] which may solve their need. It depends on the experience OP wants to provide for their users. We wanted to let users roam around, have files, use the console, collaborate in real-time, load data, and fresh environments... Similar to what they'd have on their own laptops, except with a better experience. Not just "one notebook view" or a rigid pipeline. That's what we disliked in most of the products we've seen so far. That and the fact many of them focused way too much on pretty stylesheets and animations instead of actual problems ML people struggle with. - [0]: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ https://kubernetes.io/docs/concepts/configuration/manage-res... - [1]: https://docs.docker.com/config/containers/resource_constraints/ https://docs.docker.com/config/containers/resource_constrain... - [2]: https://jupyterhub.readthedocs.io/en/stable/reference/spawners.html https://jupyterhub.readthedocs.io/en/stable/reference/spawne... - [3]: https://jupyter-enterprise-gateway.readthedocs.io/en/latest/ https://jupyter-enterprise-gateway.readthedocs.io/en/latest/

We specify resources as parameters to limit usage both for notebook servers, and for long-running notebooks we schedule[0][1]. In other words: for notebook servers, we can select the resources we want when starting a notebook, say 5GB of RAM for exploring something and dry-running some code on a sub-sample. Then we schedule the notebook with more resources (GPU, RAM, etc). We can set the timeout to kill the job. We also shut down the notebook servers automatically, and use resources only for the duration of the long-running notebook when they're scheduled. OP might want to look into JupyterHub spawners[2]. They can use the API to make custom spawners that fit their specific needs (handling storage, using specific images, etc). One limitation in ZTJPH in one comment above if I recall correctly is the locality of the cluster. OP also might want to take a look at JupyterHub Enterprise Gateway[3] which may solve their need. It depends on the experience OP wants to provide for their users. We wanted to let users roam around, have files, use the console, collaborate in real-time, load data, and fresh environments... Similar to what they'd have on their own laptops, except with a better experience. Not just "one notebook view" or a rigid pipeline. That's what we disliked in most of the products we've seen so far. That and the fact many of them focused way too much on pretty stylesheets and animations instead of actual problems ML people struggle with. - [0]: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ https://kubernetes.io/docs/concepts/configuration/manage-res... - [1]: https://docs.docker.com/config/containers/resource_constraints/ https://docs.docker.com/config/containers/resource_constrain... - [2]: https://jupyterhub.readthedocs.io/en/stable/reference/spawners.html https://jupyterhub.readthedocs.io/en/stable/reference/spawne... - [3]: https://jupyter-enterprise-gateway.readthedocs.io/en/latest/ https://jupyter-enterprise-gateway.readthedocs.io/en/latest/