8 ms·

Interactive Go programming with Jupyter

292 points by kawera 9y ago

robochat42 9y ago
This looks very cool. There's no overview of how it actually works. It seems to launch a goroutine for each Jupyter code cell but any variables defined are also available for use in later cells so I'm not sure how that works. Just playing with the code in the mybinder.org it feels responsive enough. As well as for data analysis, this will be a great way to do those quick checks that I like to do in python to confirm how an api works. I've been meaning to start using golang more and this might be how I do it.
- vertis 9y ago
  My feeling is that the friction of learning a language has a large impact on the success and adoption of the language. Something like this definitely reduces friction.
- cosmos72 9y ago
  It compiles a plugin for each Jupyter cell code with "go build -buildmode=plugin" then loads it. It's easy to see with Linux "lsof -p LGO_PID": it lists, among others, the loaded shared libraries - there is one per Jupyter cell code.
  - bm1362 9y ago
    This is super cool, but I’d like to know more. You can’t reload plugins without changing the plugin path and any shared references won’t get GC’d, from what I remember. I tried to do server handler hot reloading via plugins and ran into this a while ago.
- chatmasta 9y ago
  As someone who hasn't worked with Jupyter beyond a simple iPython shell, this comment sent me down a bit of a rabbit hole. If anyone's interested, here are some links. Jupyter docs for making a kernel: http://jupyter-client.readthedocs.io/en/latest/kernels.html http://jupyter-client.readthedocs.io/en/latest/kernels.html List of jupyter kernels: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels https://github.com/jupyter/jupyter/wiki/Jupyter-kernels Jupyter seems like a cleanly designed piece of software that is easy to integrate with. I like the use of ZeroMQ for the wire protocol.
  - carreau 9y ago
    Sorry it's my job to nitpick. IPython is upper case I and P. We don't want to upset the fruit giant.
chillydawg 9y ago
I wonder what kind of dark arts might be required to have this talk to python within one notebook? Would it even be useful?
- mjburgess 9y ago
  In the simplest case, you would parse the text output of a previous cell for the other language.
- fredley 9y ago
  If this were possible, it would require the blackest of black magic, and mapping anything other than primitive types across the void would be essentially impossible, e.g. anything useful like a DataFrame.
  - catawbasam 9y ago
    Arrow is meant to help with that
- filmor 9y ago
  One could implement a kernel that forwards calls to other kernels depending on a prefix and explicit push-and-pull functions to move variables around. That last part would require an extension of the protocol or a parallel channel, but the forwarding kernel is actually not that difficult to implement.
- deleted 9y ago
  [deleted]
mar77i 9y ago
So, what about the obvious rename to "Jugoter"?
- TuringTest 9y ago
  That has the problem of sounding like "judgemental" in French ("jugeote")
  - staz 9y ago
    Jugeote means more "savvy" or "to have discernment" to me
  - akerro 9y ago
    So what? Jupyter sounds like a name of a planet in English.
cosmos72 9y ago
For information, it's not the only Go kernel for Jupyter. There is also https://github.com/gopherdata/gophernotes https://github.com/gopherdata/gophernotes which uses a different approach: contains a portable Go interpreter, instead of invoking the Go compiler for each Jupyter cell as lgo does (lgo is currently linux only)
- fredley 9y ago
  A comparison with gophernotes is included in the article. From the things they've chosen to compare, it looks like a step up!
  - cosmos72 9y ago
    Yes, from the things they've chosen to compare. I am gomacro author's (gophernotes uses gomacro as Go interpreter) and I find the comparison needs an update. Among other things, gomacro now supports interfaces - although gophernotes still has to catch up. It's also partially incorrect: gophernotes and gomacro are type safe I guess I will have to open an issue... Also, IMHO the comparison criteria are chosen based on what lgo author considers important - not necessarily what's important for someone else. Some quick examples that are important for me: portability on non-Linux systems, redefining a variable or function (lgo can only shadow them), redefining a method, allowing unused local variables... I have great respect for what yunabe accomplished with lgo, including the deep integration with Jupyter (displaying HTML, Javascript, images...), code completion, and many other features, but I think his comparison with gophernotes needs to be updated.
tekkk 9y ago
This is very fascinating, great job! One thing I was wondering how did you manage to build it? There's a lot of code there and I was wondering if I could make my own version for a different programming language.
gaff33 9y ago
Is there a pandas equivalent / Dataframe library for Go yet?
- jboynyc 9y ago
  This isn't really mature enough to be considered an equivalent to pandas, but that seems to be the aspiration: https://github.com/kniren/gota https://github.com/kniren/gota
- dm319 9y ago
  I just don't see Go ever becoming a good language for numerical computing. There was such a push with implementing [multidimensional arrays in Go](https://github.com/golang/go/issues/6282 https://github.com/golang/go/issues/6282) but it looks like it's been shot down more than once. When using Python and R interactively, the syntactic sugar and functions that allow for rapid analysis (subsetting, indexing, merging data objects etc etc) more than make up for the slowness of the language. If someone wanted to implement that workflow in Go for repeated analysis of very large datasets, I doubt they would need it to be interactive. But it looks like Julia might become the goto numerical language that is compiled and fast.
21stio 9y ago
I don't get why you want to use this over an IDE. All you get is cached intermediate variables and being able to print them, am I right? On the other hand you loose the capability to see the definition of everything externally defined and you can't use the debugger. Please let me know if I am missing something
- rufugee 9y ago
  I believe it's intended for teaching and guides, not development. For example, I'm currently working through a data science course which uses Jupyter notebooks to allow interactive code to be embedded within the course materials, instead of being kept in separate files. The student can read the material and experiment with the code without switching to another window.
- rhizome31 9y ago
  Jupyter serves a different purpose than IDEs and text editors. The main use cases are experimentation and presentation. If you want to try a library it can be more flexible than a REPL and more interactive than a program in its own file. Jupyter is also used to produce documents that mix code, prose and visualizations. A typical use case is to write a small amount of code that processes some data and produce a chart, providing explanation formatted in Markdown along the way. These documents can be exported in a variety of formats, including printable documents, presentation slides and blog posts. There's a very vibrant and high-quality ecosystem of projects developed around Jupyter which ranges from making interactive documents to running distributed notebook servers. In general you wouldn't develop a full software project in Jupyter. Note that Jupyter is not the only project that provides this type of notebook interface. You'll also find Zeppelin coming from the Java community, Observable for JavaScript and the very powerful Nextjournal SasS product.
- mschuetz 9y ago
  I love to use notebook style IDEs to prototype, test and benchmark "smaller" things. And with smaller I don't mean things that aren't complex or difficult, but things that don't need thousands of lines of code or ellaborate software architecture. They are great to try new algorithms or evaluate their performance. If something is not quite right the first time, just change a line and evaluate that block again, without having to run another, potentially expensive block, again. You get to see the changes to your latest block immediately, without having to go through all the previous steps again. They are also great if you want to put together some graphs or statistic displays. I'm still hoping for something like Mathematica for javascript, including ways to print graphs, meshes, point clouds, etc. to you notebook.
pressurefree 9y ago
ISNT IT PRONOUNCED 'GOH'? LIKE HOG BACKWARDS.
odd_noises 9y ago
This is def gonna help me understand Golang way better
fwdpropaganda 9y ago
Serious question: what's the point? Is there anything in the realm of interactive data analysis-type programming that isn't already better with Python and instead needs to be done in Go? I might be missing something. If so, what?
- Shakahs 9y ago
  Jupyter has uses outside of data science; I find it to be an extremely useful programming scratchpad for web development with Python and Node. It was renamed from IPython to Jupyter to reflect its broader scope, Python was the original kernel but there are many now: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

292 points by kawera 9y ago

This looks very cool. There's no overview of how it actually works. It seems to launch a goroutine for each Jupyter code cell but any variables defined are also available for use in later cells so I'm not sure how that works. Just playing with the code in the mybinder.org it feels responsive enough. As well as for data analysis, this will be a great way to do those quick checks that I like to do in python to confirm how an api works. I've been meaning to start using golang more and this might be how I do it.

vertis 9y ago
My feeling is that the friction of learning a language has a large impact on the success and adoption of the language. Something like this definitely reduces friction.
cosmos72 9y ago
It compiles a plugin for each Jupyter cell code with "go build -buildmode=plugin" then loads it. It's easy to see with Linux "lsof -p LGO_PID": it lists, among others, the loaded shared libraries - there is one per Jupyter cell code.
- bm1362 9y ago
  This is super cool, but I’d like to know more. You can’t reload plugins without changing the plugin path and any shared references won’t get GC’d, from what I remember. I tried to do server handler hot reloading via plugins and ran into this a while ago.
chatmasta 9y ago
As someone who hasn't worked with Jupyter beyond a simple iPython shell, this comment sent me down a bit of a rabbit hole. If anyone's interested, here are some links. Jupyter docs for making a kernel: http://jupyter-client.readthedocs.io/en/latest/kernels.html http://jupyter-client.readthedocs.io/en/latest/kernels.html List of jupyter kernels: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels https://github.com/jupyter/jupyter/wiki/Jupyter-kernels Jupyter seems like a cleanly designed piece of software that is easy to integrate with. I like the use of ZeroMQ for the wire protocol.
- carreau 9y ago
  Sorry it's my job to nitpick. IPython is upper case I and P. We don't want to upset the fruit giant.

chillydawg 9y ago

I wonder what kind of dark arts might be required to have this talk to python within one notebook? Would it even be useful?

mjburgess 9y ago
In the simplest case, you would parse the text output of a previous cell for the other language.
fredley 9y ago
If this were possible, it would require the blackest of black magic, and mapping anything other than primitive types across the void would be essentially impossible, e.g. anything useful like a DataFrame.
- catawbasam 9y ago
  Arrow is meant to help with that
filmor 9y ago
One could implement a kernel that forwards calls to other kernels depending on a prefix and explicit push-and-pull functions to move variables around. That last part would require an extension of the protocol or a parallel channel, but the forwarding kernel is actually not that difficult to implement.
deleted 9y ago
[deleted]

mar77i 9y ago

So, what about the obvious rename to "Jugoter"?

TuringTest 9y ago
That has the problem of sounding like "judgemental" in French ("jugeote")
- staz 9y ago
  Jugeote means more "savvy" or "to have discernment" to me
- akerro 9y ago
  So what? Jupyter sounds like a name of a planet in English.

cosmos72 9y ago

For information, it's not the only Go kernel for Jupyter. There is also https://github.com/gopherdata/gophernotes https://github.com/gopherdata/gophernotes which uses a different approach: contains a portable Go interpreter, instead of invoking the Go compiler for each Jupyter cell as lgo does (lgo is currently linux only)

fredley 9y ago
A comparison with gophernotes is included in the article. From the things they've chosen to compare, it looks like a step up!
- cosmos72 9y ago
  Yes, from the things they've chosen to compare. I am gomacro author's (gophernotes uses gomacro as Go interpreter) and I find the comparison needs an update. Among other things, gomacro now supports interfaces - although gophernotes still has to catch up. It's also partially incorrect: gophernotes and gomacro are type safe I guess I will have to open an issue... Also, IMHO the comparison criteria are chosen based on what lgo author considers important - not necessarily what's important for someone else. Some quick examples that are important for me: portability on non-Linux systems, redefining a variable or function (lgo can only shadow them), redefining a method, allowing unused local variables... I have great respect for what yunabe accomplished with lgo, including the deep integration with Jupyter (displaying HTML, Javascript, images...), code completion, and many other features, but I think his comparison with gophernotes needs to be updated.

tekkk 9y ago

This is very fascinating, great job! One thing I was wondering how did you manage to build it? There's a lot of code there and I was wondering if I could make my own version for a different programming language.

gaff33 9y ago

Is there a pandas equivalent / Dataframe library for Go yet?

jboynyc 9y ago
This isn't really mature enough to be considered an equivalent to pandas, but that seems to be the aspiration: https://github.com/kniren/gota https://github.com/kniren/gota
dm319 9y ago
I just don't see Go ever becoming a good language for numerical computing. There was such a push with implementing [multidimensional arrays in Go](https://github.com/golang/go/issues/6282 https://github.com/golang/go/issues/6282) but it looks like it's been shot down more than once. When using Python and R interactively, the syntactic sugar and functions that allow for rapid analysis (subsetting, indexing, merging data objects etc etc) more than make up for the slowness of the language. If someone wanted to implement that workflow in Go for repeated analysis of very large datasets, I doubt they would need it to be interactive. But it looks like Julia might become the goto numerical language that is compiled and fast.

21stio 9y ago

I don't get why you want to use this over an IDE. All you get is cached intermediate variables and being able to print them, am I right? On the other hand you loose the capability to see the definition of everything externally defined and you can't use the debugger. Please let me know if I am missing something

rufugee 9y ago
I believe it's intended for teaching and guides, not development. For example, I'm currently working through a data science course which uses Jupyter notebooks to allow interactive code to be embedded within the course materials, instead of being kept in separate files. The student can read the material and experiment with the code without switching to another window.
rhizome31 9y ago
Jupyter serves a different purpose than IDEs and text editors. The main use cases are experimentation and presentation. If you want to try a library it can be more flexible than a REPL and more interactive than a program in its own file. Jupyter is also used to produce documents that mix code, prose and visualizations. A typical use case is to write a small amount of code that processes some data and produce a chart, providing explanation formatted in Markdown along the way. These documents can be exported in a variety of formats, including printable documents, presentation slides and blog posts. There's a very vibrant and high-quality ecosystem of projects developed around Jupyter which ranges from making interactive documents to running distributed notebook servers. In general you wouldn't develop a full software project in Jupyter. Note that Jupyter is not the only project that provides this type of notebook interface. You'll also find Zeppelin coming from the Java community, Observable for JavaScript and the very powerful Nextjournal SasS product.
mschuetz 9y ago
I love to use notebook style IDEs to prototype, test and benchmark "smaller" things. And with smaller I don't mean things that aren't complex or difficult, but things that don't need thousands of lines of code or ellaborate software architecture. They are great to try new algorithms or evaluate their performance. If something is not quite right the first time, just change a line and evaluate that block again, without having to run another, potentially expensive block, again. You get to see the changes to your latest block immediately, without having to go through all the previous steps again. They are also great if you want to put together some graphs or statistic displays. I'm still hoping for something like Mathematica for javascript, including ways to print graphs, meshes, point clouds, etc. to you notebook.

pressurefree 9y ago

ISNT IT PRONOUNCED 'GOH'? LIKE HOG BACKWARDS.

odd_noises 9y ago

This is def gonna help me understand Golang way better

fwdpropaganda 9y ago

Serious question: what's the point? Is there anything in the realm of interactive data analysis-type programming that isn't already better with Python and instead needs to be done in Go? I might be missing something. If so, what?

Shakahs 9y ago
Jupyter has uses outside of data science; I find it to be an extremely useful programming scratchpad for web development with Python and Node. It was renamed from IPython to Jupyter to reflect its broader scope, Python was the original kernel but there are many now: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels https://github.com/jupyter/jupyter/wiki/Jupyter-kernels