4 ms·

Hosting Jupyter Notebooks for 100k Users [video]

151 points by jbredeche 8y ago

sandGorgon 8y ago
there is Dash (https://plot.ly/products/dash/ https://plot.ly/products/dash/) and there is Jupyter. I wish there was some abstraction to generate a Dash like output from Jupyter. There are a lot of people who would pay serious money for that. Even Airbnb built a framework to extract code from Jupyter notebooks and push them into a machine learning pipeline (https://medium.com/airbnb-engineering/using-machine-learning-to-predict-value-of-homes-on-airbnb-9272d3d4739d https://medium.com/airbnb-engineering/using-machine-learning...). Jupyter can be so much more by going closer to how it fits within a production pipeline versus just competing against Rstudio.
- ssanderson11235 8y ago
  There have been a couple attempts to add dashboarding to Jupyter: https://github.com/jupyter/dashboards https://github.com/jupyter/dashboards was/is a dashboard system built by a team at IBM. I think the project stalled somewhat after IBM stopped funding it. There are a few long threads in the currently active jupyter repos about building dashboard systems as extensions: https://github.com/jupyterlab/jupyterlab/issues/1640 https://github.com/jupyterlab/jupyterlab/issues/1640, https://github.com/jupyter-widgets/ipywidgets/issues/2018 https://github.com/jupyter-widgets/ipywidgets/issues/2018.
  - deven88 8y ago
    I was searching for such dashboard utility and I found this: https://github.com/oschuett/appmode https://github.com/oschuett/appmode It may be useful for some of the cases.
- minimaxir 8y ago
  > I wish there was some abstraction to generate a Dash like output from Jupyter. There are a lot of people who would pay serious money for that. Notebook-to-dataviz is the value proposition of Mode Analytics, which has been doing well: https://modeanalytics.com/ https://modeanalytics.com/
- rb808 8y ago
  Also I always thought notebooks would be a great devops tool, kind of like a super command line that has easily observable steps grouped in chunks and graphical feedback. No one else seems to think so though so maybe I'm wrong.
  - jacquesm 8y ago
    That makes good sense, in case it helps your confidence. It's a middle ground between literate programming and a traditional IDE and there is nothing like that aimed at the ops space where it would actually be quite valuable.
    - nickbarnwell 8y ago
      emacs' org-mode / org-babel is well suited for it. This blog post [1] has some fantastic examples. The problem, of course, is that it's so closely tied to emacs. 1: http://howardism.org/Technical/Emacs/literate-devops.html http://howardism.org/Technical/Emacs/literate-devops.html
      - craigching 8y ago
        Nice, I just posted the video from that link above ;) Great minds think alike!
        nickbarnwell 8y ago
        Howard's blog and github are an amazing resource; I'm a huge fan of his demo-it package as well. Highly recommend checking it out if you ever give technical presentations or talks.
  - existencebox 8y ago
    As disclaimer/context, am a dev on the Azure hosted Jupyter Notebooks product, You're not wrong! (Or at the least, it's a topic that has come across our ears before, and is something I certainly agree with) Obviously I probably shouldn't go off spouting all the pipe dreams I have in this space, but given that I got my start doing Ops work and tried to keep an eye for things I might have liked back then, I can assure you there you're not alone. I always saw the similarity foremost as a direct upgrade to the "runbooks"/"firefighting/deploy checklists" that crop up all too often.
- peatmoss 8y ago
  The R community has a really nice answer for this: https://rmarkdown.rstudio.com/flexdashboard/ https://rmarkdown.rstudio.com/flexdashboard/
ssanderson11235 8y ago
Speaker here. If you want to follow along with the slides from the talk, you can find them at https://speakerdeck.com/ssanderson/hosting-notebooks-for-100-000-users https://speakerdeck.com/ssanderson/hosting-notebooks-for-100.... Also, happy to answer any questions that people might have.
- diabeetusman 8y ago
  Do you know if there's any way that we can see what's happening during the mini demo?
  - ssanderson11235 8y ago
    I gave an earlier version of this talk at JupyterCon 2017 https://www.youtube.com/watch?v=TtsbspKHJGo https://www.youtube.com/watch?v=TtsbspKHJGo, which captured my full screen output. The pgcontents demo starts around 19:30 in that video.
    - chupapuma 8y ago
      Hey Scott, how did you get your unlisted YouTube link for your presentation? I don't think I ever found mine from the same conference.
      - ssanderson11235 8y ago
        I found it in the YouTube playlist from the event: https://www.youtube.com/playlist?list=PL055Epbe6d5aP6Ru42r7hk68GTSaclYgi https://www.youtube.com/playlist?list=PL055Epbe6d5aP6Ru42r7h....
        chupapuma 8y ago
        Weird. Mine isn't there, or I am blind. shrugs Though I did find it by searching the O'Reilly user :)
- porterde 8y ago

151 points by jbredeche 8y ago

there is Dash (https://plot.ly/products/dash/ https://plot.ly/products/dash/) and there is Jupyter. I wish there was some abstraction to generate a Dash like output from Jupyter. There are a lot of people who would pay serious money for that. Even Airbnb built a framework to extract code from Jupyter notebooks and push them into a machine learning pipeline (https://medium.com/airbnb-engineering/using-machine-learning-to-predict-value-of-homes-on-airbnb-9272d3d4739d https://medium.com/airbnb-engineering/using-machine-learning...). Jupyter can be so much more by going closer to how it fits within a production pipeline versus just competing against Rstudio.

ssanderson11235 8y ago
There have been a couple attempts to add dashboarding to Jupyter: https://github.com/jupyter/dashboards https://github.com/jupyter/dashboards was/is a dashboard system built by a team at IBM. I think the project stalled somewhat after IBM stopped funding it. There are a few long threads in the currently active jupyter repos about building dashboard systems as extensions: https://github.com/jupyterlab/jupyterlab/issues/1640 https://github.com/jupyterlab/jupyterlab/issues/1640, https://github.com/jupyter-widgets/ipywidgets/issues/2018 https://github.com/jupyter-widgets/ipywidgets/issues/2018.
- deven88 8y ago
  I was searching for such dashboard utility and I found this: https://github.com/oschuett/appmode https://github.com/oschuett/appmode It may be useful for some of the cases.
minimaxir 8y ago
> I wish there was some abstraction to generate a Dash like output from Jupyter. There are a lot of people who would pay serious money for that. Notebook-to-dataviz is the value proposition of Mode Analytics, which has been doing well: https://modeanalytics.com/ https://modeanalytics.com/
rb808 8y ago
Also I always thought notebooks would be a great devops tool, kind of like a super command line that has easily observable steps grouped in chunks and graphical feedback. No one else seems to think so though so maybe I'm wrong.
- jacquesm 8y ago
  That makes good sense, in case it helps your confidence. It's a middle ground between literate programming and a traditional IDE and there is nothing like that aimed at the ops space where it would actually be quite valuable.
  - nickbarnwell 8y ago
    emacs' org-mode / org-babel is well suited for it. This blog post [1] has some fantastic examples. The problem, of course, is that it's so closely tied to emacs. 1: http://howardism.org/Technical/Emacs/literate-devops.html http://howardism.org/Technical/Emacs/literate-devops.html
    - craigching 8y ago
      Nice, I just posted the video from that link above ;) Great minds think alike!
      - nickbarnwell 8y ago
        Howard's blog and github are an amazing resource; I'm a huge fan of his demo-it package as well. Highly recommend checking it out if you ever give technical presentations or talks.
- existencebox 8y ago
  As disclaimer/context, am a dev on the Azure hosted Jupyter Notebooks product, You're not wrong! (Or at the least, it's a topic that has come across our ears before, and is something I certainly agree with) Obviously I probably shouldn't go off spouting all the pipe dreams I have in this space, but given that I got my start doing Ops work and tried to keep an eye for things I might have liked back then, I can assure you there you're not alone. I always saw the similarity foremost as a direct upgrade to the "runbooks"/"firefighting/deploy checklists" that crop up all too often.
peatmoss 8y ago
The R community has a really nice answer for this: https://rmarkdown.rstudio.com/flexdashboard/ https://rmarkdown.rstudio.com/flexdashboard/

ssanderson11235 8y ago

Speaker here. If you want to follow along with the slides from the talk, you can find them at https://speakerdeck.com/ssanderson/hosting-notebooks-for-100-000-users https://speakerdeck.com/ssanderson/hosting-notebooks-for-100.... Also, happy to answer any questions that people might have.

diabeetusman 8y ago
Do you know if there's any way that we can see what's happening during the mini demo?
- ssanderson11235 8y ago
  I gave an earlier version of this talk at JupyterCon 2017 https://www.youtube.com/watch?v=TtsbspKHJGo https://www.youtube.com/watch?v=TtsbspKHJGo, which captured my full screen output. The pgcontents demo starts around 19:30 in that video.
  - chupapuma 8y ago
    Hey Scott, how did you get your unlisted YouTube link for your presentation? I don't think I ever found mine from the same conference.
    - ssanderson11235 8y ago
      I found it in the YouTube playlist from the event: https://www.youtube.com/playlist?list=PL055Epbe6d5aP6Ru42r7hk68GTSaclYgi https://www.youtube.com/playlist?list=PL055Epbe6d5aP6Ru42r7h....
      - chupapuma 8y ago
        Weird. Mine isn't there, or I am blind. shrugs Though I did find it by searching the O'Reilly user :)
porterde 8y ago