7 ms·

Now Serving: Full-Text Sampler in IPython Notebook Format

42 points by ptwobrussell 13y ago

jyeee 13y ago
$1.15M (plus my opinion) says that IPython is the bee's knees: http://ipython.org/sloan-grant.html http://ipython.org/sloan-grant.html
ptwobrussell 13y ago
I think it's your opinion that is really the deciding factor there ;)
- jyeee 13y ago
  lol, don't just take my word for it: http://blog.fperez.org/2012/09/blogging-with-ipython-notebook.html http://blog.fperez.org/2012/09/blogging-with-ipython-noteboo...
  - leephillips 13y ago
    Really interesting blog. I would love to follow it but I don't see an RSS (or atom) feed: did I miss it?
    - jyeee 13y ago
      On bottom right: http://miningthesocialweb.com/feed/ http://miningthesocialweb.com/feed/ One of the fun examples shows how to mine an RSS feed ;) https://rawgithub.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/master/ipynb/html/Chapter%205%20-%20Mining%20Web%20Pages.html https://rawgithub.com/ptwobrussell/Mining-the-Social-Web-2nd...
mcburton 13y ago
YES! I would also love to see a publishing workflow that could render IPython Notebooks for print on dead trees.
- ptwobrussell 13y ago
  The 1.0.0 version started bundling nbconvert, which does a pretty good job of exporting to html, pdf, latex, etc. Have you tried it out yet?
  - mcburton 13y ago
    I haven't started meaningfully playing with nbconvert yet, but I think one of the really promising features are the nbconvert templates[1]. From what I gather you can make custom templates with jinja and invoke them on the command line. It would be interesting to create a docbook, msword, specifically formatted pdf templates which would allow IPython Notebooks to flow into existing print publication workflows (if I recall O'Reilly uses docbook, yes?). I haven't been able to find documentation about the nbconvert templates so I'm dragging this out of memory from an email or a tweet I saw once upon a time. One thing I'm hoping to figure out is a way to transform IPython Notebooks into PDFs that satisfy my university's dissertation formatting guidelines (one inch margins, 12pt times, etc). [1] https://github.com/ipython/ipython/tree/master/IPython/nbconvert/templates https://github.com/ipython/ipython/tree/master/IPython/nbcon...
    - ptwobrussell 13y ago
      Wow, you definitely have some great ideas here, and given the enthusiasm about ipynb, I would not be surprised to see these things happen in the next 6-12 months. O'Reilly does use docbook internally as part of the toolchain, but asciidoc is coming into style as well. I've done most of my work in docbook with a docbook editor, but did some of my latest boook in asciidoc, which was much easier. Just use a standard text editor of choice (like Vim) and get right to it. Much easier for me, anyway. One thing I'd really like to see is ipynb supporting asciidoc instead of or in addition to markdown. You can go to/from asciidoc and docbook in a lossless way, IIRC.
      - mcburton 13y ago
        asciidoc support would require an ansciidoc renderer written in javascript, which does seem to exist[1](although I haven't used it). This also highlights some of the problems with markdown because it is an insufficiently rich language for generating semantically meaningful markup for use-cases beyond blogging. There has been some really interesting work on "scholarly markdown"[2][3], but given the social and technical complexities of markdown implementations and the community[4] seems like adding asciidoc is easier than fixing markdown. [1] http://asciidoctor.org/news/2013/05/21/asciidoctor-js-render-asciidoc-in-the-browser/ http://asciidoctor.org/news/2013/05/21/asciidoctor-js-render... [2]https://github.com/scholmd/scholmd/wiki https://github.com/scholmd/scholmd/wiki [3]http://blog.martinfenner.org/2013/06/19/citations-in-scholarly-markdown/ http://blog.martinfenner.org/2013/06/19/citations-in-scholar... [4]http://www.codinghorror.com/blog/2012/10/the-future-of-markdown.html http://www.codinghorror.com/blog/2012/10/the-future-of-markd...
badhairday 13y ago
I purchased Mining the Social Web, 2nd Edition (O'Reilly, 2013) a few weeks ago and have been loving the IPython Notebook setup they're using to distribute the code examples. Anyone can clone the following git repo and play around with the code: https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Ed...
mherradora 13y ago
It is a very practical idea. How many times do you want to test an example? With pdf and other formats you could make a copy-paste but many things could go wrong. But the really important improvement is the use of Vagrant and a virtual machine to give an accurate workspace.
- jyeee 13y ago
  +1 mherradora - I got the first version right after twitter changed the API >_< and busted one of the examples in chapter 1. Not anymore with Mining the Social Web 2e :). Bonus style points for Vagrant so that the author doesn't have to host (i.e. pay for) a VM and the massive bandwidth.
hesdeadjim 13y ago
I would love to see this start to happen in the academic world too. Being able to play around with a very complicated algorithm you are trying to wrap your head around would be a godsend.
- ptwobrussell 13y ago
  There are some awesome notebooks out that demonstrate exactly that. Here are a couple of good links with combinatorial optimization and TSP: http://nbviewer.ipython.org/urls/raw.github.com/fonnesbeck/Bios366/master/notebooks/Section3_2-Combinatorial-Optimization.ipynb http://nbviewer.ipython.org/urls/raw.github.com/fonnesbeck/B... http://nbviewer.ipython.org/url/norvig.com/ipython/TSPv3.ipynb http://nbviewer.ipython.org/url/norvig.com/ipython/TSPv3.ipy...
  - sillysaurus2 13y ago
    This is the first I've looked into IPython Notebook, so apologies for the newbie question: Is a notebook an html page with Python code snippets interspersed? Are these supposed to be interactive at all? Or do you have to copy the Python locally, then install some packages before you can finally run it? I guess I'm asking, what's the difference between an HTML page that shows a Python implementation of an algorithm vs those links? Or what's the appeal? I'm certain IPython Notebooks are valuable, since everybody seems to love them. It's not clear from glancing at those examples what the big deal is, though, so I thought I should ask in case more newbies come along and incorrectly conclude that there's nothing especially interesting about those examples beyond their content. (To be clear, the content is excellent, but why is IPython Notebook to thank rather than the author? Could those examples not have been done via some other method, or is the purpose of IPython Notebook just to make it really easy to generate such pages?)
    - jyeee 13y ago
      The nbviewer links are just HTML pages, but if you were running your own IPython notebook server, it would be the actual python interpreter running your live code! Setting up that IPython notebook server is easy: check out this 2 min video (http://vimeo.com/72383764 http://vimeo.com/72383764). Bonus: if you use the one provided in github in that link, it comes with all the python imports you'd already need like networkx, response, etc. :)
julienchastang 13y ago
I work with scientists that love the IPython Notebook technology. Some claim the IP[y]: Notebook to be the best thing since the Mosaic web browser and the most important development in scientific computing in a decade. They are replacing a variety of tools (Perl, Matlab, R, etc.) with the notebook technology. I tend to agree, it is a revolutionary technology and the idea of executable papers is tantalizing. But I also see some limitations. Setting up a Python environment with all the necessary libraries can be a real pain in the neck even with technologies like pip. Also taking advantage of multicore parallelism is not trivial because of the Python Global Interpreter Lock. I have also worked in JVM environments where parallel computing is becoming significantly easier and I don't see that happening in Python anytime soon. I would love to be proven wrong, of course.
- ptwobrussell 13y ago
  I would totally agree on the first point about how much of a burden it can be to get any moderately complex Python environment up and running (whether IPython Notebook is part of the install or not.) Unfortunately, it's just not as easy as it should be yet. IMHO, this is what makes a duo like Vagrant + IPython Notebook so appropriate for creating a minimum barrier to entry. In other words, what you really need is a VM that has IPython Notebook and all of the necessary dependencies installed that you just start up as a guest machine. Then, on your host, you point your browser to http://localhost:8888 http://localhost:8888 and get to work with the the ipynb UI. The approach I described is exactly what I did with Mining the Social Web 2E, and I elaborated on it a bit here: http://miningthesocialweb.com/2013/08/24/reflections-on-authoring-a-minimum-viable-book/ http://miningthesocialweb.com/2013/08/24/reflections-on-auth... The cool thing about the way Vagrant works is that you can run a VM locally or you can run it in the cloud just as easily with the appropriate plugin. There's a video that demonstrates how easy it can be to use Vagrant and IPython Notebook to get started here - https://vimeo.com/72383764 https://vimeo.com/72383764
- ptwobrussell 13y ago
  By the way, I really like the metaphor of describing IPython Notebook as "executable paper" - that's clever and concise.
- 13y ago
carreau 13y ago
That's really amazing, I think I can speek for the all IPython team by saying that we are Really happy with this. We've cross a few authors that were writing books for O’Reilly with IPython Notebooks in the last few month and we have plan to help convert those to (and from) many format. We would be extremely happy to work with people from O'Reilly to simplify the conversion process. We also have plan to integrate with publication in academic world (see https://github.com/ipython/ipython/issues/4119 https://github.com/ipython/ipython/issues/4119) but we really need more people involved than the few of us (Core Team). Even if IPython seem like a big complicated project, moving forward in area like document conversion is not that hard, but just needs lots of eyes and tester to catch all cases. As for people wanting to convert notebook to other format than those currently supported, it is perfectly possible using nbconvert with template. We can spit the TeX that is just between begin/end document so you can include in your preferred dissertation template. It should be pretty easy to also just write a converter that export to asciidoc or doocbook. For those interested of the idea behing how nbconvert I wrote this quite some time ago: http://nbviewer.ipython.org/urls/raw.github.com/Carreau/posts/master/06-NBconvert-Doc-Draft.ipynb http://nbviewer.ipython.org/urls/raw.github.com/Carreau/post... and https://github.com/ipython/ipython-in-depth https://github.com/ipython/ipython-in-depth Offer some example too. Hope this will push some people to contribute, we'll always be happy do guide you in your first step.
- IanOzsvald 13y ago
  I'm working on an O'Reilly book using IPython Notebook, I'm basing my workflow on Olivier Grisel's script (with a couple of tiny fixes): https://github.com/ianozsvald/ipynbhelper https://github.com/ianozsvald/ipynbhelper which extracts the Notebook's 'markdown' blocks (which contain asciidoc which obviously won't render in the browser) & code blocks, these get exported as asciidoc for O'Reilly. It is clunky but I'm hoping we'll get better control as nbconvert evolves, so we're experimenting with this approach. Most of the code examples are not 'live', they're pasted in along with analysis results (the book is about high performance and parallel computing: http://shop.oreilly.com/product/0636920028963.do http://shop.oreilly.com/product/0636920028963.do ), as lots of the examples are best run from a fresh VM.

42 points by ptwobrussell 13y ago

jyeee 13y ago

$1.15M (plus my opinion) says that IPython is the bee's knees: http://ipython.org/sloan-grant.html http://ipython.org/sloan-grant.html

ptwobrussell 13y ago

I think it's your opinion that is really the deciding factor there ;)

jyeee 13y ago
lol, don't just take my word for it: http://blog.fperez.org/2012/09/blogging-with-ipython-notebook.html http://blog.fperez.org/2012/09/blogging-with-ipython-noteboo...
- leephillips 13y ago
  Really interesting blog. I would love to follow it but I don't see an RSS (or atom) feed: did I miss it?
  - jyeee 13y ago
    On bottom right: http://miningthesocialweb.com/feed/ http://miningthesocialweb.com/feed/ One of the fun examples shows how to mine an RSS feed ;) https://rawgithub.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/master/ipynb/html/Chapter%205%20-%20Mining%20Web%20Pages.html https://rawgithub.com/ptwobrussell/Mining-the-Social-Web-2nd...

mcburton 13y ago

YES! I would also love to see a publishing workflow that could render IPython Notebooks for print on dead trees.

ptwobrussell 13y ago
The 1.0.0 version started bundling nbconvert, which does a pretty good job of exporting to html, pdf, latex, etc. Have you tried it out yet?
- mcburton 13y ago
  I haven't started meaningfully playing with nbconvert yet, but I think one of the really promising features are the nbconvert templates[1]. From what I gather you can make custom templates with jinja and invoke them on the command line. It would be interesting to create a docbook, msword, specifically formatted pdf templates which would allow IPython Notebooks to flow into existing print publication workflows (if I recall O'Reilly uses docbook, yes?). I haven't been able to find documentation about the nbconvert templates so I'm dragging this out of memory from an email or a tweet I saw once upon a time. One thing I'm hoping to figure out is a way to transform IPython Notebooks into PDFs that satisfy my university's dissertation formatting guidelines (one inch margins, 12pt times, etc). [1] https://github.com/ipython/ipython/tree/master/IPython/nbconvert/templates https://github.com/ipython/ipython/tree/master/IPython/nbcon...
  - ptwobrussell 13y ago
    Wow, you definitely have some great ideas here, and given the enthusiasm about ipynb, I would not be surprised to see these things happen in the next 6-12 months. O'Reilly does use docbook internally as part of the toolchain, but asciidoc is coming into style as well. I've done most of my work in docbook with a docbook editor, but did some of my latest boook in asciidoc, which was much easier. Just use a standard text editor of choice (like Vim) and get right to it. Much easier for me, anyway. One thing I'd really like to see is ipynb supporting asciidoc instead of or in addition to markdown. You can go to/from asciidoc and docbook in a lossless way, IIRC.
    - mcburton 13y ago
      asciidoc support would require an ansciidoc renderer written in javascript, which does seem to exist[1](although I haven't used it). This also highlights some of the problems with markdown because it is an insufficiently rich language for generating semantically meaningful markup for use-cases beyond blogging. There has been some really interesting work on "scholarly markdown"[2][3], but given the social and technical complexities of markdown implementations and the community[4] seems like adding asciidoc is easier than fixing markdown. [1] http://asciidoctor.org/news/2013/05/21/asciidoctor-js-render-asciidoc-in-the-browser/ http://asciidoctor.org/news/2013/05/21/asciidoctor-js-render... [2]https://github.com/scholmd/scholmd/wiki https://github.com/scholmd/scholmd/wiki [3]http://blog.martinfenner.org/2013/06/19/citations-in-scholarly-markdown/ http://blog.martinfenner.org/2013/06/19/citations-in-scholar... [4]http://www.codinghorror.com/blog/2012/10/the-future-of-markdown.html http://www.codinghorror.com/blog/2012/10/the-future-of-markd...

badhairday 13y ago

I purchased Mining the Social Web, 2nd Edition (O'Reilly, 2013) a few weeks ago and have been loving the IPython Notebook setup they're using to distribute the code examples. Anyone can clone the following git repo and play around with the code: https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Ed...

mherradora 13y ago

It is a very practical idea. How many times do you want to test an example? With pdf and other formats you could make a copy-paste but many things could go wrong. But the really important improvement is the use of Vagrant and a virtual machine to give an accurate workspace.

jyeee 13y ago
+1 mherradora - I got the first version right after twitter changed the API >_< and busted one of the examples in chapter 1. Not anymore with Mining the Social Web 2e :). Bonus style points for Vagrant so that the author doesn't have to host (i.e. pay for) a VM and the massive bandwidth.

hesdeadjim 13y ago

I would love to see this start to happen in the academic world too. Being able to play around with a very complicated algorithm you are trying to wrap your head around would be a godsend.

ptwobrussell 13y ago
There are some awesome notebooks out that demonstrate exactly that. Here are a couple of good links with combinatorial optimization and TSP: http://nbviewer.ipython.org/urls/raw.github.com/fonnesbeck/Bios366/master/notebooks/Section3_2-Combinatorial-Optimization.ipynb http://nbviewer.ipython.org/urls/raw.github.com/fonnesbeck/B... http://nbviewer.ipython.org/url/norvig.com/ipython/TSPv3.ipynb http://nbviewer.ipython.org/url/norvig.com/ipython/TSPv3.ipy...
- sillysaurus2 13y ago
  This is the first I've looked into IPython Notebook, so apologies for the newbie question: Is a notebook an html page with Python code snippets interspersed? Are these supposed to be interactive at all? Or do you have to copy the Python locally, then install some packages before you can finally run it? I guess I'm asking, what's the difference between an HTML page that shows a Python implementation of an algorithm vs those links? Or what's the appeal? I'm certain IPython Notebooks are valuable, since everybody seems to love them. It's not clear from glancing at those examples what the big deal is, though, so I thought I should ask in case more newbies come along and incorrectly conclude that there's nothing especially interesting about those examples beyond their content. (To be clear, the content is excellent, but why is IPython Notebook to thank rather than the author? Could those examples not have been done via some other method, or is the purpose of IPython Notebook just to make it really easy to generate such pages?)
  - jyeee 13y ago
    The nbviewer links are just HTML pages, but if you were running your own IPython notebook server, it would be the actual python interpreter running your live code! Setting up that IPython notebook server is easy: check out this 2 min video (http://vimeo.com/72383764 http://vimeo.com/72383764). Bonus: if you use the one provided in github in that link, it comes with all the python imports you'd already need like networkx, response, etc. :)

julienchastang 13y ago

I work with scientists that love the IPython Notebook technology. Some claim the IP[y]: Notebook to be the best thing since the Mosaic web browser and the most important development in scientific computing in a decade. They are replacing a variety of tools (Perl, Matlab, R, etc.) with the notebook technology. I tend to agree, it is a revolutionary technology and the idea of executable papers is tantalizing. But I also see some limitations. Setting up a Python environment with all the necessary libraries can be a real pain in the neck even with technologies like pip. Also taking advantage of multicore parallelism is not trivial because of the Python Global Interpreter Lock. I have also worked in JVM environments where parallel computing is becoming significantly easier and I don't see that happening in Python anytime soon. I would love to be proven wrong, of course.

ptwobrussell 13y ago
I would totally agree on the first point about how much of a burden it can be to get any moderately complex Python environment up and running (whether IPython Notebook is part of the install or not.) Unfortunately, it's just not as easy as it should be yet. IMHO, this is what makes a duo like Vagrant + IPython Notebook so appropriate for creating a minimum barrier to entry. In other words, what you really need is a VM that has IPython Notebook and all of the necessary dependencies installed that you just start up as a guest machine. Then, on your host, you point your browser to http://localhost:8888 http://localhost:8888 and get to work with the the ipynb UI. The approach I described is exactly what I did with Mining the Social Web 2E, and I elaborated on it a bit here: http://miningthesocialweb.com/2013/08/24/reflections-on-authoring-a-minimum-viable-book/ http://miningthesocialweb.com/2013/08/24/reflections-on-auth... The cool thing about the way Vagrant works is that you can run a VM locally or you can run it in the cloud just as easily with the appropriate plugin. There's a video that demonstrates how easy it can be to use Vagrant and IPython Notebook to get started here - https://vimeo.com/72383764 https://vimeo.com/72383764
ptwobrussell 13y ago
By the way, I really like the metaphor of describing IPython Notebook as "executable paper" - that's clever and concise.
13y ago

carreau 13y ago

That's really amazing, I think I can speek for the all IPython team by saying that we are Really happy with this. We've cross a few authors that were writing books for O’Reilly with IPython Notebooks in the last few month and we have plan to help convert those to (and from) many format. We would be extremely happy to work with people from O'Reilly to simplify the conversion process. We also have plan to integrate with publication in academic world (see https://github.com/ipython/ipython/issues/4119 https://github.com/ipython/ipython/issues/4119) but we really need more people involved than the few of us (Core Team). Even if IPython seem like a big complicated project, moving forward in area like document conversion is not that hard, but just needs lots of eyes and tester to catch all cases. As for people wanting to convert notebook to other format than those currently supported, it is perfectly possible using nbconvert with template. We can spit the TeX that is just between begin/end document so you can include in your preferred dissertation template. It should be pretty easy to also just write a converter that export to asciidoc or doocbook. For those interested of the idea behing how nbconvert I wrote this quite some time ago: http://nbviewer.ipython.org/urls/raw.github.com/Carreau/posts/master/06-NBconvert-Doc-Draft.ipynb http://nbviewer.ipython.org/urls/raw.github.com/Carreau/post... and https://github.com/ipython/ipython-in-depth https://github.com/ipython/ipython-in-depth Offer some example too. Hope this will push some people to contribute, we'll always be happy do guide you in your first step.

IanOzsvald 13y ago
I'm working on an O'Reilly book using IPython Notebook, I'm basing my workflow on Olivier Grisel's script (with a couple of tiny fixes): https://github.com/ianozsvald/ipynbhelper https://github.com/ianozsvald/ipynbhelper which extracts the Notebook's 'markdown' blocks (which contain asciidoc which obviously won't render in the browser) & code blocks, these get exported as asciidoc for O'Reilly. It is clunky but I'm hoping we'll get better control as nbconvert evolves, so we're experimenting with this approach. Most of the code examples are not 'live', they're pasted in along with analysis results (the book is about high performance and parallel computing: http://shop.oreilly.com/product/0636920028963.do http://shop.oreilly.com/product/0636920028963.do ), as lots of the examples are best run from a fresh VM.