12 ms·

Estimating Number of Jupyter Notebooks on Github

275 points by eoinmurray92 7y ago

xchaotic 7y ago
Python Notebooks will the be the PERL of 2010s - write once, pretty impossible to maintain long term
manaskarekar 7y ago
Off topic: Possibly something in my config, but I've recently got a lot of "Sorry, something went wrong. Reload?" when trying to view Jupyter notebooks on github itself. Seems to be working right now. I have used this as an alternate: https://nbviewer.jupyter.org/ https://nbviewer.jupyter.org/
- Willson50 7y ago
  I don't think it's just you, I've been getting them too. It's probably Github timing out trying to render the notebook.
- pequalsnp 7y ago
  I've gotten this as well after uploading notebooks to github. It usually goes away for me after ~10mins after I've uploaded the .ipynb. I'll check this out though, thanks!
- jparkie 7y ago
  I get it a lot, so I set a precommit hook to render all my notebooks to HTML. With GitHub Pages, I can access them easily.
  - pimlottc 7y ago
    Sounds useful, care to share that hook?
- semi-extrinsic 7y ago
  Re: this - it would be fun to set up a Beowulf cluster of Jupyter notebooks, if you will, and just upload them to Github for free parallell computations. (Obviously this would be for shits 'n giggles, not any serious work).
  - kam 7y ago
    GitHub and nbviewer don't actually execute the notebooks, they simply render the markdown, code, and saved output to HTML.
- eoinmurray92 7y ago
  This is what we made Kyso for - the linked post is actually a Jupyter notebook itself, the code is hidden by default to make it readable to non-technical people but you can click on the "code hidden" button on the top right to see the code in full. If github is not working for you well for your notebooks. You can try Kyso by signing up and importing your notebook from Github directly on this page: https://kyso.io/github https://kyso.io/github, or upload it using this page: https://kyso.io/create/study https://kyso.io/create/study Disclaimer - I am OP and founder of Kyso
  - aw3c2 7y ago
    You made Kyso for making GitHub-hosted Jupyter Notebooks render when accessed with a web browser on github.com?
    - KyleOS 7y ago
      Well, it's one of the ways to post to Kyso, but yeah, you can synchronize your Github repositories to Kyso and choose the notebooks you want to have rendered. When you push changes to the repo, the sister post on Kyso will be automatically updated.
- localhostdotdev 7y ago
  thanks for linking there, tried to make some ruby notebooks and works quite well https://nbviewer.jupyter.org/github/localhostdotdev/notebooks/blob/master/list%20of%20github%20repositories.ipynb https://nbviewer.jupyter.org/github/localhostdotdev/notebook... (very similar to the github interface though)
funkythingsss 7y ago
I hate jupyter notebooks. Joel Grus puts it perfectly: https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit#slide=id.g362da58057_0_1 https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh... past hn discussion: https://news.ycombinator.com/item?id=17856700 https://news.ycombinator.com/item?id=17856700
- localhost 7y ago
  What specifically do you hate about Jupyter? Is it out of order execution exacerbating the "hidden state" problem? If so, and if you already use VS Code, I encourage you to try out our Python VS Code extension. We have an "Interactive Python Window" mode https://code.visualstudio.com/docs/python/jupyter-support https://code.visualstudio.com/docs/python/jupyter-support that we showed a lot of folks at pycon last week and even among the "I don't like Jupyter" crowd, it was quite well received. The key thing about our experience is that it is an editor focused interactive programming experience vs. trying to just replicate Jupyter functionality in an editor (though we are also doing work here because we believe that folks want an experience where they can move seamlessly back and forth between and editor and a notebook experience). Disclaimer: I designed this experience in VS Code.
  - mike_ivanov 7y ago
    That surely is a step in the right direction, thank you.
  - euler_angles 7y ago
    Please allow me to thank you for designing an experience that hits the sweet spot between maintaining a good history of work through git while allowing for interactivity and ease of exploration. I started using VS Code on seeing a video of the jupyter support within the latest release of the Python extension. This has eased and sped up my work tremendously!
    - localhost 7y ago
      Thanks for the kind words! If you find things that you would like to see improved, please do open an issue on our Github - https://github.com/Microsoft/vscode-python/issues https://github.com/Microsoft/vscode-python/issues. We also need to work in the discoverability of this feature too. Lots of existing users of our extension had no idea it was there ... suggestions welcome!
gkwalton 7y ago
Jupyter notebooks and Github are a perfect match if one wants to signal activity. On a more serious note, I think he trend to stick documentation into notebooks is quite unfortunate and leads to further erosion of documentation quality.
formalsystem 7y ago
Maybe it's time to be able to run them implicitly on Azure cloud?
KyleOS 7y ago
I think it would be cool to run the same analysis on the number of R Notebooks on Github and compare the two.
- turingbike 7y ago
  Aren't Jupyter notebooks R notebooks? Jupyter stands for "Julia, Python, R" I believe
  - eoinmurray92 7y ago
    Yeah this is true but sometimes R notebooks also refers to R studio notebooks - https://rmarkdown.rstudio.com/lesson-10.html https://rmarkdown.rstudio.com/lesson-10.html
  - jsilence 7y ago
    But the user is choosing a Jupyter Kernel upon creation and the code is run on either one of the available Kernels. No code mixing within one notebook at the current point.
    - hulahoof 7y ago
      I thought you could with apache toree? Maybe I misunderstand the project
  - KyleOS 7y ago
    I think the file extension scraped on Github is only .ipynb, which is only python notebooks right?
    - rodonn 7y ago
      I think most people save under this extension even if they are using a different kernel (i.e. they are running R, Julia, Matlab, etc. code in the notebook).
      - 7y ago
lmeyerov 7y ago
We use notebooks heavily for onboarding devs & data scientists to Graphistry, and I only see that increasing. Interestingly, for initial use, we increasingly start teams on their existing internal NB servers, and for new ones, they either start on Jupyter included in their Graphistry AMI or use Google Colab. So, very little outside of our quick start notebook skeletons hits GitHub. So... How many notebooks are actually out there? Probably an even more interesting growth curve...!
- eoinmurray92 7y ago
  How do you share and collaborate on the notebooks internally - I'd love to get your thoughts on our Kyso for teams system [1] if you would be willing to chat? [1] https://kyso.io/for-teams https://kyso.io/for-teams
  - lmeyerov 7y ago
    Google Colab has solved the 90% for us. I wish it'd have context sharing across users and better default folder management, and probably other things, but free + usable + sharing has been amazing in + across partners. Feel free to ping in a ~couple weeks, happy to chat.
jjtheblunt 7y ago
exponentially?
- sp332 7y ago
  Yeah? https://imgur.com/a/J93nCXR https://imgur.com/a/J93nCXR
randomfool 7y ago
There's also the GitHub extracts table available in BigQuery which allows analysis of the contents of the notebooks themselves: https://bigquery.cloud.google.com/table/fh-bigquery:github_extracts.contents_ipynb?pli=1 https://bigquery.cloud.google.com/table/fh-bigquery:github_e...
martinzugnoni 7y ago
My two cents: We've been recently working in a FREE hosted version of Jupyter Lab mainly intended for education. Feel free to check it out. https://notebooks.ai/ https://notebooks.ai/ Would love to hear some feedback.
- jsilence 7y ago
  Wondering how you are planning to keep it free. Also wondering whether you would possibly consider shifting to Sagemath/CoCalc as a service.
  - martinzugnoni 7y ago
    We got support of the local university at my city and we got a bunch of free credits at AWS. Costs are very low and we want to keep it that way so we can support the most students we can with a free access.
    - deleted 7y ago
      [deleted]
    - ertemplin 7y ago
      What happens when your funding and AWS credits run out?
      - martinzugnoni 7y ago
        We might add bigger paid tiers later if we decide to support business usage of it. For now it's only educational and we can deal with the costs, even without the credits. Containers used are small and get shut down on inactivity. So, we only need to care about concurrent users. Hope it makes sense.
        zeptomu 7y ago
        > Containers used are small and get shut down on inactivity. How do you define inactivity? If I do $ nohup ./computational_intense_and_runs_for_100_hours.py & Do you just kill the process (or stop the container)? In essence Jupyter is a graphical rich shell, so you providing free *nix machines - don't underestimate how this feature can be exploited (e.g. CoCalc limits at least internet access for free instances).
ricardojoaoreis 7y ago
I've dabbled with the idea of using Jupyter notebooks for personal planning (like a kind of org-mode). Has anyone tried this?
eoinmurray92 7y ago
OP and Founder of Kyso here - we built Kyso to make it easier to blog your notebooks to the public and also to make them easier to share in teams. The linked post is actually a Jupyter notebook itself - analysing the number of notebooks on Github. A key element with Kyso is that the code is hidden by default to make it readable to non-technical people but you can click on the "code hidden" button on the top right to see the code in full. If you want to give Kyso a go - sign up and import from Github directly on this page: https://kyso.io/github https://kyso.io/github, or upload using this page: https://kyso.io/create/study https://kyso.io/create/study
- eoinmurray92 7y ago
  We actually have a few ways to publish your notebooks. Upload directly, import from Github or use our plugin to publish from Jupyterlab itself - all those methods are outlined on our docs page: https://kyso.io/docs https://kyso.io/docs
- aequitas 7y ago
  In my opinion hiding code is an anti feature as exposing the code by default gives you more incentive to write clean understandable and explaining (self documenting) code as it's always visible. By hiding it people might be more likely to paste in big blobs ugly code that would be much better put into a reusable function then in a notebook snippet.
  - eoinmurray92 7y ago
    Thats true if you sharing with someone who also understands the code - we think that the feature allows you to share the notebooks with a completely non-technical audience. Image you had a notebook to analyse sales data and you needed to present the results to your CEO (who perhaps cannot code) - this feature lets you present the notebook as is, without needing to prepare a report in some other format
    - mike_ivanov 7y ago
      Why would you share a notebook with somebody who doesn't understand the code?? What are they supposed to do with them anyways? Reading? Isn't that what PDF is for? Just give them PDFs, C*O people have got piles of other things to worry about besides shared notebooks.
airocker 7y ago
We built a runnable jupyter notebook website. Would someone be able to take a look and give us some feedback? https://datacabinet.systems https://datacabinet.systems We are VM based for now but are moving to be kubernetes based to make sharing better. Our initial market is classrooms.
- asdfman123 7y ago
  There are a few basic UI problems I ran into immediately. You should focus on that instead of rearchitecting things. It needs to look sleeker, too. Pay some designer to create a new design for you. Like I can't add a project. It says "To create project Please wait till hard drive button turn green." I don't know what that means. You've lost my attention.
  - airocker 7y ago
    Thanks. We are doing some enhancements. Here is some initial screen: https://mondaytest.datacabinet.systems/ https://mondaytest.datacabinet.systems/ system but the page inside is still the same. We have an elaborate design done.
    - airocker 7y ago
      Right now, for VMs we make a hard drive(mkfs, install things) for every user that takes some initial time. We will fix that in the kubernetes version.
- localhost 7y ago
  Something to be aware of (and a general comment about k8s in general) is that k8s is not suitable for use in hostile multi-tenant scenarios like the one that you're describing. Once an attacker escapes from the container (see HN archives for lots of examples of this), they can p0wn the entire cluster. Jessie Frazelle has a great post on this: https://blog.jessfraz.com/post/hard-multi-tenancy-in-kubernetes/ https://blog.jessfraz.com/post/hard-multi-tenancy-in-kuberne... There are expensive ways to deal with this today, e.g., running each user isolated in a separate VM. Hopefully we will have better solutions in the near future.
kbd 7y ago
If you ever put notebooks in source control, you owe it to yourself to try the text-based notebooks supported in Visual Studio Code[1]. They're round-trippable with real (i.e. browser-based) notebooks, yet are much better for collaboration, diffing, and editing. [1] https://code.visualstudio.com/docs/python/jupyter-support https://code.visualstudio.com/docs/python/jupyter-support
- claytonjy 7y ago
  How does this compare to Jupytext? I prefer pipenv to Conda, and I don't like having Jupyter(Lab) installed in each venv separately, so instead I only add `Ipykernel` to each venv and then use my system-level JupyterLab to access per-project kernels; seems like that wouldn't work here?
  - kbd 7y ago
    Thanks for mentioning that project. I'm wasn't familiar with it, but the difference seems to be that Jupytext is a tool to convert notebooks between ipynb and text formats, while VSCode supports the same format (#%% cell delimiters) but enables you to edit text-based notebooks similar to how you would in the browser, including keyboard shortcuts to execute individual cells, it'll render the markdown, and so on. I'm not sure if I understand your kernel question, but VSCode's Python extension has everything built-in. As soon as you add a #%% comment it considers what follows to be a notebook cell and automatically gives you a "run cell" button that uses your chosen Python interpreter.
  - bawana 7y ago
    I've noticed that if I dont install jupyter to each venv, then the pythonpath defaults to the jupyter install. As a result, the jupyter notebook cannot find the packages in the venv(for example, caffe, module not found errors)
    - claytonjy 7y ago
      huh, I've never used Caffe but wrapping my venvs with pipenv has never caused me this kind of grief. I tell pipenv to store venv stuff in the project folder (`export PIPENV_VENV_IN_PROJECT=1`), and then do the following to start a new project: pipenv --python 3.7 pipenv install ipykernel pipenv run python -m ipykernel install --user --name=`basename $(pipenv run dirname '$VIRTUAL_ENV') jupyter lab (all the magic is in that third command) Then in the list of kernels, in addition to the usual suspects I'll have one named for the folder I ran the above in. If I started a notebook before all that, I'll have to change it's kernel. Doing more `pipenv install` at the prompt makes new packages immediately available in the running notebook.
syntaxing 7y ago
On the topic of Jupyter Notebooks, Is there something similar to a paid version of Google's CoLab? CoLab is so awesome for creating prototypes and even better since it's free. However, there is no paid alternative that I have seen. I do not want to have to deal with setting up my own VM or server. The way that CoLab is perfect for what I need.
- eoinmurray92 7y ago
  The OP post is a Jupyter notebook itself and if you sign up to Kyso you can actually Jupyterlab on our cloud and the post the notebooks to the web, or make them private on the paid plan - is that what your looking for?
  - syntaxing 7y ago
    Yeah but unfortunately there is no GPU support. I wish there was!
    - eoinmurray92 7y ago
      Ah yeah ok - we are not planning GPU support soon - what if we created one of those one-click deploy a VM to aws/digital-ocean buttons and from there if you wanted to post to Kyso you could do it with git or our jupyter lab plugin. You get most of the same experience and you can even customise various of the steps?
      - syntaxing 7y ago
        Interesting! That would be super awesome. Like a one-click deploy with auto-shutdown after the end of run to save money. I would definitely pay for a service like that!
- enisberk 7y ago
  You can use a VM with Colab[1],also paperspace[2]and floydub[3] are other companies providing cloud gpu with notebooks. [1] https://colab.research.google.com/drive/1Xh0slMoD1dfi8iTQJbO18uuCilJMLoKc https://colab.research.google.com/drive/1Xh0slMoD1dfi8iTQJbO... [2] https://www.paperspace.com/ml https://www.paperspace.com/ml [3] https://www.floydhub.com/ https://www.floydhub.com/
tincholio 7y ago
If only more people would use org-babel... If you're on emacs and like Jupyter, there's https://github.com/dzop/emacs-jupyter https://github.com/dzop/emacs-jupyter , which is pretty nice. I've been using it for a few days with Julia, and it works really nice. It also allows you to use different kernels from the same org-mode file, though I haven't tried to pass data between them yet (should be possible, though, at least it works in plain org-mode).
- mark_l_watson 7y ago
  Also when using Term2 (on Mac) or a Linux terminal you can configure the backend of matplotlib to draw plots right in a SSH shell, term window, etc. I think this would work fine if you were running emacs with -nw no window option in a terminal.
- kadal 7y ago
  I agree. emacs-jupyter is great but I'm eagerly waiting for jupyter notebook server support since my work is on remote clusters now. For local use, it's pretty great.
- gnulinux 7y ago
  Emacs jupyter notebook mode (ein-mode) is pretty excellent too. That's my go-to way to connect to jupyter.
sytse 7y ago
We're seeing an explosion of Jupyter use as well on GitLab. GitLab already makes Jupyter easier to install on a Kubernetes cluster https://docs.gitlab.com/ee/user/project/clusters/#installing-applications https://docs.gitlab.com/ee/user/project/clusters/#installing... In response to the growing demand we're doing two things: 1. Adding better Jupyter support to GitLab 12.0 https://gitlab.com/gitlab-org/gitlab-ce/issues/47138 https://gitlab.com/gitlab-org/gitlab-ce/issues/47138 as suggested by my co-founder. 2. Making it easier to do the entire data lifecycle with Meltano https://meltano.com/ https://meltano.com/ which plans to include JupyterHub
hyperbovine 7y ago
Especially odd because they are so unsuitable for use with git. Someone needs to find a way to fix this.
- chrisjc 7y ago
  > because they are so unsuitable for use with git Can you going into a little more depth about this statement?
  - pwhitebelt 7y ago
    Diffs primarily I'm guessing - a, it's kinda hard to parse the jsons that you see when you look at a notebook in raw text b, every time I execute a cell, it shows up in the diff as a change. That being said, there are plugins and tools that deal with these issues quite well. check out https://nbdime.readthedocs.io/en/latest/ https://nbdime.readthedocs.io/en/latest/
    - hyperbovine 7y ago
      I have not had good luck with nbdiff. Minutes-long runtimes and huge memory consumption on fairly standard ipynb’s.
trpc 7y ago
nice marketing, kyso.io team
julienchastang 7y ago
In the same spirit as “Effective Java” and “Effective C++” we need to have a book entitled “Effective Jupyter Notebooks”. Here are some of my items below. Maybe this sub-thread can come up with an outline for this book. Item #1 Writing a notebook is foremost an exercise in expository writing. Make sure the writing is high quality is the first objective when writing a notebook. This is the Knuth’s literate programming idea where prose takes precedence over code, which is usually the reverse of the way we usually program; code first, comments second. Item #2 Don't use notebooks for general purpose programming. Notebooks are supposed to have an audience and clearly explain something. Item #3 Keep code cells simple and clear. If needing a comment in the code block, consider putting that verbiage in a markdown cell instead and elaborating on the idea the notebook is trying to convey. Item #4 Don't make notebooks a long series of extended code cells, or even worse, just one long cell. Explain what is going on or see Item #2.
- jimmar 7y ago
  For #2, I have used notebooks for personal data analysis. I am my own audience and need to remember what I did and why.
- arnioxux 7y ago
  For example Peter Norvig does an exemplary job with jupyter notebooks for expository writings: https://github.com/norvig/pytudes https://github.com/norvig/pytudes
- eoinmurray92 7y ago
  This is a really good idea, and I've been planning to write said guide for a while. I think there is a lot to do with organising and formatting the notebooks, and the %run command works a treat for me when breaking a project into multiple notebooks
- zimablue 7y ago
  I disagree massively with this, to me Jupyter is a repl+ development environment + rapid prototyping in ETL/visualisation. There's no rule that they have to be used for exposition, or any writing at all. I don't know why you think this is a rule, or what you'd use as an alternative data-repl.
hodder 7y ago
Very cool work here. This is a pretty epic post, so please do not take this the wrong way. I was under the impression that FB Prophet was optimal for significantly seasonal time series data. Honestly given the fickle nature of these kind of growth patterns beyond the very near term, an ARIMA with a flat vol or a simple eyeball extrapolation in my experience as a quant would likely generate just as reasonable/reliable results. While I understand this is likely intended as a standalone project, it would be interesting to run a comparison of ARIMA vs FB Prophet on out of sample trending Github tools/file types, as well as the general performance of these predictions beyond a one year time frame (especially vs the reported confidence intervals in Prophet). I am not that familiar with how Prophet works, so I am absolutely open to being humbled and corrected. I have a project myself that has a varying seasonal component and I am looking forward to diving into Prophet for a deeper understanding. I am attempting to model an Asian 2 asset spread option with a volume weighted average index price setting mechanism where the underlying exhibits seasonality in the volume traded over the trading time window. I am currently running a Monte Carlo on the valuation with a simple average settlement assumption, as opposed to a volume weighted average assumption, and I was thinking Prophet could help. Does anyone have experience in financial time series analysis and option valuation who would care to chime in? Also, what is everyone's thoughts on using prophet non seasonal vol clustering times series?
- KyleOS 7y ago
  Hey, I posted the notebook by the OP. Thank you for your feedback! You're correct in saying that FB Prophet is for forecasting time series with strong seasonal effects. FB Prophet was the model used in the original script I found & the main point here was simply to make the notebook more readable on kyso, which has quite a few non-technical readers. I've worked a lot with ARIMAs before for financial/economic data and I like the idea of comparing the results between the two, and maybe even extend the time frame. So 1. I think that'll be my next project and 2. if your project is public I'd love to give it a read when published.
  - jerednel 7y ago
    Did you try running the forecast with a log ceiling to control for the trajectory a bit? Or would that only be a concern of yours if you had to forecast past a couple of years? I find that when I use Prophet to forecast down to the day I end up creating initial forecasts with heavy log ceilings to prevent unreasonable estimates of the future and then end up removing the ceiling once enough history is established to provide a resaonable baseline.
alpb 7y ago
Why doesn't this just use the GitHub public dataset available on Google BigQuery to have much more accurate data rather than "scraping GitHub web search results"? https://cloud.google.com/bigquery/public-data/ https://cloud.google.com/bigquery/public-data/ There are a lot of examples of people analyzing public code on GitHub efficiently for patterns and usages with BigQuery and getting pretty accurate data out of it. https://medium.com/google-cloud/analyzing-go-code-with-bigquery-485c70c3b451 https://medium.com/google-cloud/analyzing-go-code-with-bigqu... If you use GitHub on a daily basis, you are unlucky enough to know that web search sadly can't even find words that exist in your repository.
- mmcniece 7y ago
  One limitation of the BigQuery dataset is they only look at repos with a license on them[1], the scraping approach can look at all public repos. [1] https://github.blog/2017-01-19-github-data-ready-for-you-to-explore-with-bigquery/ https://github.blog/2017-01-19-github-data-ready-for-you-to-...
JBorrow 7y ago
This seems like an incredibly complicated way to fit an exponential to data...
nsxwolf 7y ago
So I just learned they're not laptops.
- superdimwit 7y ago
  This comment made my day!
gus_massa 7y ago
Isn't the prediction too low? My (unsupported) prediction fitting a smooth curve in the graphic is https://imgur.com/a/ykeIxPm https://imgur.com/a/ykeIxPm

275 points by eoinmurray92 7y ago

Python Notebooks will the be the PERL of 2010s - write once, pretty impossible to maintain long term

Off topic: Possibly something in my config, but I've recently got a lot of "Sorry, something went wrong. Reload?" when trying to view Jupyter notebooks on github itself. Seems to be working right now. I have used this as an alternate: https://nbviewer.jupyter.org/ https://nbviewer.jupyter.org/

Willson50 7y ago
I don't think it's just you, I've been getting them too. It's probably Github timing out trying to render the notebook.
pequalsnp 7y ago
I've gotten this as well after uploading notebooks to github. It usually goes away for me after ~10mins after I've uploaded the .ipynb. I'll check this out though, thanks!
jparkie 7y ago
I get it a lot, so I set a precommit hook to render all my notebooks to HTML. With GitHub Pages, I can access them easily.
- pimlottc 7y ago
  Sounds useful, care to share that hook?
semi-extrinsic 7y ago
Re: this - it would be fun to set up a Beowulf cluster of Jupyter notebooks, if you will, and just upload them to Github for free parallell computations. (Obviously this would be for shits 'n giggles, not any serious work).
- kam 7y ago
  GitHub and nbviewer don't actually execute the notebooks, they simply render the markdown, code, and saved output to HTML.
eoinmurray92 7y ago
This is what we made Kyso for - the linked post is actually a Jupyter notebook itself, the code is hidden by default to make it readable to non-technical people but you can click on the "code hidden" button on the top right to see the code in full. If github is not working for you well for your notebooks. You can try Kyso by signing up and importing your notebook from Github directly on this page: https://kyso.io/github https://kyso.io/github, or upload it using this page: https://kyso.io/create/study https://kyso.io/create/study Disclaimer - I am OP and founder of Kyso
- aw3c2 7y ago
  You made Kyso for making GitHub-hosted Jupyter Notebooks render when accessed with a web browser on github.com?
  - KyleOS 7y ago
    Well, it's one of the ways to post to Kyso, but yeah, you can synchronize your Github repositories to Kyso and choose the notebooks you want to have rendered. When you push changes to the repo, the sister post on Kyso will be automatically updated.
localhostdotdev 7y ago
thanks for linking there, tried to make some ruby notebooks and works quite well https://nbviewer.jupyter.org/github/localhostdotdev/notebooks/blob/master/list%20of%20github%20repositories.ipynb https://nbviewer.jupyter.org/github/localhostdotdev/notebook... (very similar to the github interface though)

funkythingsss 7y ago

I hate jupyter notebooks. Joel Grus puts it perfectly: https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit#slide=id.g362da58057_0_1 https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh... past hn discussion: https://news.ycombinator.com/item?id=17856700 https://news.ycombinator.com/item?id=17856700

localhost 7y ago
What specifically do you hate about Jupyter? Is it out of order execution exacerbating the "hidden state" problem? If so, and if you already use VS Code, I encourage you to try out our Python VS Code extension. We have an "Interactive Python Window" mode https://code.visualstudio.com/docs/python/jupyter-support https://code.visualstudio.com/docs/python/jupyter-support that we showed a lot of folks at pycon last week and even among the "I don't like Jupyter" crowd, it was quite well received. The key thing about our experience is that it is an editor focused interactive programming experience vs. trying to just replicate Jupyter functionality in an editor (though we are also doing work here because we believe that folks want an experience where they can move seamlessly back and forth between and editor and a notebook experience). Disclaimer: I designed this experience in VS Code.
- mike_ivanov 7y ago
  That surely is a step in the right direction, thank you.
- euler_angles 7y ago
  Please allow me to thank you for designing an experience that hits the sweet spot between maintaining a good history of work through git while allowing for interactivity and ease of exploration. I started using VS Code on seeing a video of the jupyter support within the latest release of the Python extension. This has eased and sped up my work tremendously!
  - localhost 7y ago
    Thanks for the kind words! If you find things that you would like to see improved, please do open an issue on our Github - https://github.com/Microsoft/vscode-python/issues https://github.com/Microsoft/vscode-python/issues. We also need to work in the discoverability of this feature too. Lots of existing users of our extension had no idea it was there ... suggestions welcome!

gkwalton 7y ago

Jupyter notebooks and Github are a perfect match if one wants to signal activity. On a more serious note, I think he trend to stick documentation into notebooks is quite unfortunate and leads to further erosion of documentation quality.

formalsystem 7y ago

Maybe it's time to be able to run them implicitly on Azure cloud?

KyleOS 7y ago

I think it would be cool to run the same analysis on the number of R Notebooks on Github and compare the two.

turingbike 7y ago
Aren't Jupyter notebooks R notebooks? Jupyter stands for "Julia, Python, R" I believe
- eoinmurray92 7y ago
  Yeah this is true but sometimes R notebooks also refers to R studio notebooks - https://rmarkdown.rstudio.com/lesson-10.html https://rmarkdown.rstudio.com/lesson-10.html
- jsilence 7y ago
  But the user is choosing a Jupyter Kernel upon creation and the code is run on either one of the available Kernels. No code mixing within one notebook at the current point.
  - hulahoof 7y ago
    I thought you could with apache toree? Maybe I misunderstand the project
- KyleOS 7y ago
  I think the file extension scraped on Github is only .ipynb, which is only python notebooks right?
  - rodonn 7y ago
    I think most people save under this extension even if they are using a different kernel (i.e. they are running R, Julia, Matlab, etc. code in the notebook).
    - 7y ago

lmeyerov 7y ago

We use notebooks heavily for onboarding devs & data scientists to Graphistry, and I only see that increasing. Interestingly, for initial use, we increasingly start teams on their existing internal NB servers, and for new ones, they either start on Jupyter included in their Graphistry AMI or use Google Colab. So, very little outside of our quick start notebook skeletons hits GitHub. So... How many notebooks are actually out there? Probably an even more interesting growth curve...!

eoinmurray92 7y ago
How do you share and collaborate on the notebooks internally - I'd love to get your thoughts on our Kyso for teams system [1] if you would be willing to chat? [1] https://kyso.io/for-teams https://kyso.io/for-teams
- lmeyerov 7y ago
  Google Colab has solved the 90% for us. I wish it'd have context sharing across users and better default folder management, and probably other things, but free + usable + sharing has been amazing in + across partners. Feel free to ping in a ~couple weeks, happy to chat.

jjtheblunt 7y ago

exponentially?

sp332 7y ago
Yeah? https://imgur.com/a/J93nCXR https://imgur.com/a/J93nCXR

randomfool 7y ago

There's also the GitHub extracts table available in BigQuery which allows analysis of the contents of the notebooks themselves: https://bigquery.cloud.google.com/table/fh-bigquery:github_extracts.contents_ipynb?pli=1 https://bigquery.cloud.google.com/table/fh-bigquery:github_e...

martinzugnoni 7y ago

My two cents: We've been recently working in a FREE hosted version of Jupyter Lab mainly intended for education. Feel free to check it out. https://notebooks.ai/ https://notebooks.ai/ Would love to hear some feedback.

jsilence 7y ago
Wondering how you are planning to keep it free. Also wondering whether you would possibly consider shifting to Sagemath/CoCalc as a service.
- martinzugnoni 7y ago
  We got support of the local university at my city and we got a bunch of free credits at AWS. Costs are very low and we want to keep it that way so we can support the most students we can with a free access.
  - deleted 7y ago
    [deleted]
  - ertemplin 7y ago
    What happens when your funding and AWS credits run out?
    - martinzugnoni 7y ago
      We might add bigger paid tiers later if we decide to support business usage of it. For now it's only educational and we can deal with the costs, even without the credits. Containers used are small and get shut down on inactivity. So, we only need to care about concurrent users. Hope it makes sense.
      - zeptomu 7y ago
        > Containers used are small and get shut down on inactivity. How do you define inactivity? If I do $ nohup ./computational_intense_and_runs_for_100_hours.py & Do you just kill the process (or stop the container)? In essence Jupyter is a graphical rich shell, so you providing free *nix machines - don't underestimate how this feature can be exploited (e.g. CoCalc limits at least internet access for free instances).

ricardojoaoreis 7y ago

I've dabbled with the idea of using Jupyter notebooks for personal planning (like a kind of org-mode). Has anyone tried this?

eoinmurray92 7y ago

OP and Founder of Kyso here - we built Kyso to make it easier to blog your notebooks to the public and also to make them easier to share in teams. The linked post is actually a Jupyter notebook itself - analysing the number of notebooks on Github. A key element with Kyso is that the code is hidden by default to make it readable to non-technical people but you can click on the "code hidden" button on the top right to see the code in full. If you want to give Kyso a go - sign up and import from Github directly on this page: https://kyso.io/github https://kyso.io/github, or upload using this page: https://kyso.io/create/study https://kyso.io/create/study

eoinmurray92 7y ago
We actually have a few ways to publish your notebooks. Upload directly, import from Github or use our plugin to publish from Jupyterlab itself - all those methods are outlined on our docs page: https://kyso.io/docs https://kyso.io/docs
aequitas 7y ago
In my opinion hiding code is an anti feature as exposing the code by default gives you more incentive to write clean understandable and explaining (self documenting) code as it's always visible. By hiding it people might be more likely to paste in big blobs ugly code that would be much better put into a reusable function then in a notebook snippet.
- eoinmurray92 7y ago
  Thats true if you sharing with someone who also understands the code - we think that the feature allows you to share the notebooks with a completely non-technical audience. Image you had a notebook to analyse sales data and you needed to present the results to your CEO (who perhaps cannot code) - this feature lets you present the notebook as is, without needing to prepare a report in some other format
  - mike_ivanov 7y ago
    Why would you share a notebook with somebody who doesn't understand the code?? What are they supposed to do with them anyways? Reading? Isn't that what PDF is for? Just give them PDFs, C*O people have got piles of other things to worry about besides shared notebooks.

airocker 7y ago

We built a runnable jupyter notebook website. Would someone be able to take a look and give us some feedback? https://datacabinet.systems https://datacabinet.systems We are VM based for now but are moving to be kubernetes based to make sharing better. Our initial market is classrooms.

asdfman123 7y ago
There are a few basic UI problems I ran into immediately. You should focus on that instead of rearchitecting things. It needs to look sleeker, too. Pay some designer to create a new design for you. Like I can't add a project. It says "To create project Please wait till hard drive button turn green." I don't know what that means. You've lost my attention.
- airocker 7y ago
  Thanks. We are doing some enhancements. Here is some initial screen: https://mondaytest.datacabinet.systems/ https://mondaytest.datacabinet.systems/ system but the page inside is still the same. We have an elaborate design done.
  - airocker 7y ago
    Right now, for VMs we make a hard drive(mkfs, install things) for every user that takes some initial time. We will fix that in the kubernetes version.
localhost 7y ago
Something to be aware of (and a general comment about k8s in general) is that k8s is not suitable for use in hostile multi-tenant scenarios like the one that you're describing. Once an attacker escapes from the container (see HN archives for lots of examples of this), they can p0wn the entire cluster. Jessie Frazelle has a great post on this: https://blog.jessfraz.com/post/hard-multi-tenancy-in-kubernetes/ https://blog.jessfraz.com/post/hard-multi-tenancy-in-kuberne... There are expensive ways to deal with this today, e.g., running each user isolated in a separate VM. Hopefully we will have better solutions in the near future.

kbd 7y ago

If you ever put notebooks in source control, you owe it to yourself to try the text-based notebooks supported in Visual Studio Code[1]. They're round-trippable with real (i.e. browser-based) notebooks, yet are much better for collaboration, diffing, and editing. [1] https://code.visualstudio.com/docs/python/jupyter-support https://code.visualstudio.com/docs/python/jupyter-support

claytonjy 7y ago
How does this compare to Jupytext? I prefer pipenv to Conda, and I don't like having Jupyter(Lab) installed in each venv separately, so instead I only add `Ipykernel` to each venv and then use my system-level JupyterLab to access per-project kernels; seems like that wouldn't work here?
- kbd 7y ago
  Thanks for mentioning that project. I'm wasn't familiar with it, but the difference seems to be that Jupytext is a tool to convert notebooks between ipynb and text formats, while VSCode supports the same format (#%% cell delimiters) but enables you to edit text-based notebooks similar to how you would in the browser, including keyboard shortcuts to execute individual cells, it'll render the markdown, and so on. I'm not sure if I understand your kernel question, but VSCode's Python extension has everything built-in. As soon as you add a #%% comment it considers what follows to be a notebook cell and automatically gives you a "run cell" button that uses your chosen Python interpreter.
- bawana 7y ago
  I've noticed that if I dont install jupyter to each venv, then the pythonpath defaults to the jupyter install. As a result, the jupyter notebook cannot find the packages in the venv(for example, caffe, module not found errors)
  - claytonjy 7y ago
    huh, I've never used Caffe but wrapping my venvs with pipenv has never caused me this kind of grief. I tell pipenv to store venv stuff in the project folder (`export PIPENV_VENV_IN_PROJECT=1`), and then do the following to start a new project: pipenv --python 3.7 pipenv install ipykernel pipenv run python -m ipykernel install --user --name=`basename $(pipenv run dirname '$VIRTUAL_ENV') jupyter lab (all the magic is in that third command) Then in the list of kernels, in addition to the usual suspects I'll have one named for the folder I ran the above in. If I started a notebook before all that, I'll have to change it's kernel. Doing more `pipenv install` at the prompt makes new packages immediately available in the running notebook.

syntaxing 7y ago

On the topic of Jupyter Notebooks, Is there something similar to a paid version of Google's CoLab? CoLab is so awesome for creating prototypes and even better since it's free. However, there is no paid alternative that I have seen. I do not want to have to deal with setting up my own VM or server. The way that CoLab is perfect for what I need.

eoinmurray92 7y ago
The OP post is a Jupyter notebook itself and if you sign up to Kyso you can actually Jupyterlab on our cloud and the post the notebooks to the web, or make them private on the paid plan - is that what your looking for?
- syntaxing 7y ago
  Yeah but unfortunately there is no GPU support. I wish there was!
  - eoinmurray92 7y ago
    Ah yeah ok - we are not planning GPU support soon - what if we created one of those one-click deploy a VM to aws/digital-ocean buttons and from there if you wanted to post to Kyso you could do it with git or our jupyter lab plugin. You get most of the same experience and you can even customise various of the steps?
    - syntaxing 7y ago
      Interesting! That would be super awesome. Like a one-click deploy with auto-shutdown after the end of run to save money. I would definitely pay for a service like that!
enisberk 7y ago
You can use a VM with Colab[1],also paperspace[2]and floydub[3] are other companies providing cloud gpu with notebooks. [1] https://colab.research.google.com/drive/1Xh0slMoD1dfi8iTQJbO18uuCilJMLoKc https://colab.research.google.com/drive/1Xh0slMoD1dfi8iTQJbO... [2] https://www.paperspace.com/ml https://www.paperspace.com/ml [3] https://www.floydhub.com/ https://www.floydhub.com/

tincholio 7y ago

If only more people would use org-babel... If you're on emacs and like Jupyter, there's https://github.com/dzop/emacs-jupyter https://github.com/dzop/emacs-jupyter , which is pretty nice. I've been using it for a few days with Julia, and it works really nice. It also allows you to use different kernels from the same org-mode file, though I haven't tried to pass data between them yet (should be possible, though, at least it works in plain org-mode).

mark_l_watson 7y ago
Also when using Term2 (on Mac) or a Linux terminal you can configure the backend of matplotlib to draw plots right in a SSH shell, term window, etc. I think this would work fine if you were running emacs with -nw no window option in a terminal.
kadal 7y ago
I agree. emacs-jupyter is great but I'm eagerly waiting for jupyter notebook server support since my work is on remote clusters now. For local use, it's pretty great.
gnulinux 7y ago
Emacs jupyter notebook mode (ein-mode) is pretty excellent too. That's my go-to way to connect to jupyter.

sytse 7y ago

We're seeing an explosion of Jupyter use as well on GitLab. GitLab already makes Jupyter easier to install on a Kubernetes cluster https://docs.gitlab.com/ee/user/project/clusters/#installing-applications https://docs.gitlab.com/ee/user/project/clusters/#installing... In response to the growing demand we're doing two things: 1. Adding better Jupyter support to GitLab 12.0 https://gitlab.com/gitlab-org/gitlab-ce/issues/47138 https://gitlab.com/gitlab-org/gitlab-ce/issues/47138 as suggested by my co-founder. 2. Making it easier to do the entire data lifecycle with Meltano https://meltano.com/ https://meltano.com/ which plans to include JupyterHub

hyperbovine 7y ago

Especially odd because they are so unsuitable for use with git. Someone needs to find a way to fix this.

chrisjc 7y ago
> because they are so unsuitable for use with git Can you going into a little more depth about this statement?
- pwhitebelt 7y ago
  Diffs primarily I'm guessing - a, it's kinda hard to parse the jsons that you see when you look at a notebook in raw text b, every time I execute a cell, it shows up in the diff as a change. That being said, there are plugins and tools that deal with these issues quite well. check out https://nbdime.readthedocs.io/en/latest/ https://nbdime.readthedocs.io/en/latest/
  - hyperbovine 7y ago
    I have not had good luck with nbdiff. Minutes-long runtimes and huge memory consumption on fairly standard ipynb’s.

trpc 7y ago

nice marketing, kyso.io team

julienchastang 7y ago

In the same spirit as “Effective Java” and “Effective C++” we need to have a book entitled “Effective Jupyter Notebooks”. Here are some of my items below. Maybe this sub-thread can come up with an outline for this book. Item #1 Writing a notebook is foremost an exercise in expository writing. Make sure the writing is high quality is the first objective when writing a notebook. This is the Knuth’s literate programming idea where prose takes precedence over code, which is usually the reverse of the way we usually program; code first, comments second. Item #2 Don't use notebooks for general purpose programming. Notebooks are supposed to have an audience and clearly explain something. Item #3 Keep code cells simple and clear. If needing a comment in the code block, consider putting that verbiage in a markdown cell instead and elaborating on the idea the notebook is trying to convey. Item #4 Don't make notebooks a long series of extended code cells, or even worse, just one long cell. Explain what is going on or see Item #2.

jimmar 7y ago
For #2, I have used notebooks for personal data analysis. I am my own audience and need to remember what I did and why.
arnioxux 7y ago
For example Peter Norvig does an exemplary job with jupyter notebooks for expository writings: https://github.com/norvig/pytudes https://github.com/norvig/pytudes
eoinmurray92 7y ago
This is a really good idea, and I've been planning to write said guide for a while. I think there is a lot to do with organising and formatting the notebooks, and the %run command works a treat for me when breaking a project into multiple notebooks
zimablue 7y ago
I disagree massively with this, to me Jupyter is a repl+ development environment + rapid prototyping in ETL/visualisation. There's no rule that they have to be used for exposition, or any writing at all. I don't know why you think this is a rule, or what you'd use as an alternative data-repl.

hodder 7y ago

Very cool work here. This is a pretty epic post, so please do not take this the wrong way. I was under the impression that FB Prophet was optimal for significantly seasonal time series data. Honestly given the fickle nature of these kind of growth patterns beyond the very near term, an ARIMA with a flat vol or a simple eyeball extrapolation in my experience as a quant would likely generate just as reasonable/reliable results. While I understand this is likely intended as a standalone project, it would be interesting to run a comparison of ARIMA vs FB Prophet on out of sample trending Github tools/file types, as well as the general performance of these predictions beyond a one year time frame (especially vs the reported confidence intervals in Prophet). I am not that familiar with how Prophet works, so I am absolutely open to being humbled and corrected. I have a project myself that has a varying seasonal component and I am looking forward to diving into Prophet for a deeper understanding. I am attempting to model an Asian 2 asset spread option with a volume weighted average index price setting mechanism where the underlying exhibits seasonality in the volume traded over the trading time window. I am currently running a Monte Carlo on the valuation with a simple average settlement assumption, as opposed to a volume weighted average assumption, and I was thinking Prophet could help. Does anyone have experience in financial time series analysis and option valuation who would care to chime in? Also, what is everyone's thoughts on using prophet non seasonal vol clustering times series?

KyleOS 7y ago
Hey, I posted the notebook by the OP. Thank you for your feedback! You're correct in saying that FB Prophet is for forecasting time series with strong seasonal effects. FB Prophet was the model used in the original script I found & the main point here was simply to make the notebook more readable on kyso, which has quite a few non-technical readers. I've worked a lot with ARIMAs before for financial/economic data and I like the idea of comparing the results between the two, and maybe even extend the time frame. So 1. I think that'll be my next project and 2. if your project is public I'd love to give it a read when published.
- jerednel 7y ago
  Did you try running the forecast with a log ceiling to control for the trajectory a bit? Or would that only be a concern of yours if you had to forecast past a couple of years? I find that when I use Prophet to forecast down to the day I end up creating initial forecasts with heavy log ceilings to prevent unreasonable estimates of the future and then end up removing the ceiling once enough history is established to provide a resaonable baseline.

alpb 7y ago

Why doesn't this just use the GitHub public dataset available on Google BigQuery to have much more accurate data rather than "scraping GitHub web search results"? https://cloud.google.com/bigquery/public-data/ https://cloud.google.com/bigquery/public-data/ There are a lot of examples of people analyzing public code on GitHub efficiently for patterns and usages with BigQuery and getting pretty accurate data out of it. https://medium.com/google-cloud/analyzing-go-code-with-bigquery-485c70c3b451 https://medium.com/google-cloud/analyzing-go-code-with-bigqu... If you use GitHub on a daily basis, you are unlucky enough to know that web search sadly can't even find words that exist in your repository.

mmcniece 7y ago
One limitation of the BigQuery dataset is they only look at repos with a license on them[1], the scraping approach can look at all public repos. [1] https://github.blog/2017-01-19-github-data-ready-for-you-to-explore-with-bigquery/ https://github.blog/2017-01-19-github-data-ready-for-you-to-...

JBorrow 7y ago

This seems like an incredibly complicated way to fit an exponential to data...

nsxwolf 7y ago

So I just learned they're not laptops.

superdimwit 7y ago
This comment made my day!

gus_massa 7y ago

Isn't the prediction too low? My (unsupported) prediction fitting a smooth curve in the graphic is https://imgur.com/a/ykeIxPm https://imgur.com/a/ykeIxPm