5 ms·

Unlike most IDEs which are desktop based, Jupyter Notebook system is fundamentally network based with a standard HTML5 front end sending messages back and forth

Unlike most IDEs which are desktop based, Jupyter Notebook system is fundamentally network based with a standard HTML5 front end sending messages back and forth to the kernel. So fundamentally Jupyter is not a document based application that most of us software developers grow up with. You have to think in terms of messaging and the cells are just the input/output terminals. The rendering of cells is determined by the runtime container that parse the .ipynb (JSON) string. Here is an contrived example to explain it: https://nbviewer.jupyter.org/gist/ontouchstart/ea1631f69e507a81a9d9ec56b79e4d11 https://nbviewer.jupyter.org/gist/ontouchstart/ea1631f69e507... There is a lot going on depends on the container.

TuringTest 8y ago
That's good insight. Since the 50's, coding has been understood as creating text objects that contain the entire specification of a program in a programming language. Input data, parameters & configurations are considered external to the code object. It's the same with other activities like debugging, exploratory programming, debugging and building tests, which are considered peripheral to the "true coding" activity of altering the source code that will run on the production environment. The single successful exception to that model is the spreadsheet, were data and code are indissolubly merged in the same document; and, despite its success as a development tool for non-programmers, professional developers usually despise it. Could it be that Jupyter and notebooks are making developers comfortable with a new approach, where the distinction between "coding" and "running the code" is blurred, and where partial execution and mixing code with data are the norm? Coders are accustomed to using the REPL for this approach, but in the end only the code that is "commited" by copying it to the source file is considered the "final" version. Yet, with notebooks, code that has been written within an interactive session can be stored as the official definition for a part of a program, without having to transfer it to a separate object.
- ontouchstart 8y ago
  At semantic level, coding is still about creating objects and managing their relationships with the environment. An .ipynb file itself just an object tree. Since Jupyter people made the wise decision of using easy to parse JSON format instead of more efficient binary format, the platform is opened up to a big ecosystem that people can make tools for specific need. Like the document centric desktop application revolution in the 80s that gave us MS Office and Adobe Photoshop, the open, data centric web application revolution we are seeing in this decade will fundamentally change how we consume and produce software. It is a natural evolution of computing system production from hardware panels to punch card, to text code instructions to object graph management in network environments. (The buzzword is “the cloud”.)
  - TuringTest 8y ago
    Yeah, you're right that coding is creating computational objects. I was pointing out that "coding" is traditionally understood to refer only to the sequences of instructions in source and object code that remains unmodified during runtime execution, but there are other views that extend the definition to all the other objects involved in running the software in its environment. In this view, coding is about defining automated behaviors that can execute on their own without a human leading by hand every step. I'm thinking in particular about End-User Development approaches, which allow non-developers to create such automations without ever learning the syntax of a formal language. Under this approach, the abstraction level may be lower, since the actual instances of data used are as important to understanding the system as the code that processes them.
    - ontouchstart 8y ago
      Another type of coding is describing the desired state and let the runtime environment achieve the transition. Unix Makefile is one of the early examples. So are HTML/CSS etc. More recent examples are dataflow graph in tensorflow https://www.tensorflow.org/guide/graphs https://www.tensorflow.org/guide/graphs and kubernetes https://github.com/kubernetes/kubernetes/blob/release-1.3/docs/design/README.md https://github.com/kubernetes/kubernetes/blob/release-1.3/do... A new idea just occurred to me during a technical lunch meeting with a bunch of friends today: since Jupyter Notebook itself is nothing but a object graph, it might be possible to use a "computed" .ipynb for this kind of "coding", i.e., describing desired state. That would be TDD in another sense.
      - TuringTest 8y ago
        Yeah, I happen to work with declarative statements + problem solver for a living :-) In this context, a precise textual description helps to exactly define what you want to achieve, but a graphical view helps to inspect how many different cases are there in your data, and how available context are connected. Representing state machines with graphs is a relatively common thing in rapid prototyping apps, so I've thinking of building a prototyping app that would also allow embedding real code and data in them (similar to Visual GUI builders, but allowing much more code to be defined visually). There are several tools attempting to achieve that, but all fail at the abstraction problem of defining new instances for your objects (i.e. you can't create new instances of arbitrary visually defined objects, only for things in tables).
ontouchstart 8y ago
Make it even more convoluted, you can also load the code to another notebook on Google Colab and play with it yourself. https://colab.research.google.com/gist/ontouchstart/934da948842bff86d470ffe25d692e67/test.ipynb https://colab.research.google.com/gist/ontouchstart/934da948... Now it is hard to debug programs like this unless you trace all the links and cached assets. On the other hand, many API centered cloud native application behaves like this and you have to deal with it one way or another. So to some extend, Jupyter is the IDE for the age of cloud.

TuringTest 8y ago

That's good insight. Since the 50's, coding has been understood as creating text objects that contain the entire specification of a program in a programming language. Input data, parameters & configurations are considered external to the code object. It's the same with other activities like debugging, exploratory programming, debugging and building tests, which are considered peripheral to the "true coding" activity of altering the source code that will run on the production environment. The single successful exception to that model is the spreadsheet, were data and code are indissolubly merged in the same document; and, despite its success as a development tool for non-programmers, professional developers usually despise it. Could it be that Jupyter and notebooks are making developers comfortable with a new approach, where the distinction between "coding" and "running the code" is blurred, and where partial execution and mixing code with data are the norm? Coders are accustomed to using the REPL for this approach, but in the end only the code that is "commited" by copying it to the source file is considered the "final" version. Yet, with notebooks, code that has been written within an interactive session can be stored as the official definition for a part of a program, without having to transfer it to a separate object.

ontouchstart 8y ago
At semantic level, coding is still about creating objects and managing their relationships with the environment. An .ipynb file itself just an object tree. Since Jupyter people made the wise decision of using easy to parse JSON format instead of more efficient binary format, the platform is opened up to a big ecosystem that people can make tools for specific need. Like the document centric desktop application revolution in the 80s that gave us MS Office and Adobe Photoshop, the open, data centric web application revolution we are seeing in this decade will fundamentally change how we consume and produce software. It is a natural evolution of computing system production from hardware panels to punch card, to text code instructions to object graph management in network environments. (The buzzword is “the cloud”.)
- TuringTest 8y ago
  Yeah, you're right that coding is creating computational objects. I was pointing out that "coding" is traditionally understood to refer only to the sequences of instructions in source and object code that remains unmodified during runtime execution, but there are other views that extend the definition to all the other objects involved in running the software in its environment. In this view, coding is about defining automated behaviors that can execute on their own without a human leading by hand every step. I'm thinking in particular about End-User Development approaches, which allow non-developers to create such automations without ever learning the syntax of a formal language. Under this approach, the abstraction level may be lower, since the actual instances of data used are as important to understanding the system as the code that processes them.
  - ontouchstart 8y ago
    Another type of coding is describing the desired state and let the runtime environment achieve the transition. Unix Makefile is one of the early examples. So are HTML/CSS etc. More recent examples are dataflow graph in tensorflow https://www.tensorflow.org/guide/graphs https://www.tensorflow.org/guide/graphs and kubernetes https://github.com/kubernetes/kubernetes/blob/release-1.3/docs/design/README.md https://github.com/kubernetes/kubernetes/blob/release-1.3/do... A new idea just occurred to me during a technical lunch meeting with a bunch of friends today: since Jupyter Notebook itself is nothing but a object graph, it might be possible to use a "computed" .ipynb for this kind of "coding", i.e., describing desired state. That would be TDD in another sense.
    - TuringTest 8y ago
      Yeah, I happen to work with declarative statements + problem solver for a living :-) In this context, a precise textual description helps to exactly define what you want to achieve, but a graphical view helps to inspect how many different cases are there in your data, and how available context are connected. Representing state machines with graphs is a relatively common thing in rapid prototyping apps, so I've thinking of building a prototyping app that would also allow embedding real code and data in them (similar to Visual GUI builders, but allowing much more code to be defined visually). There are several tools attempting to achieve that, but all fail at the abstraction problem of defining new instances for your objects (i.e. you can't create new instances of arbitrary visually defined objects, only for things in tables).

ontouchstart 8y ago

Make it even more convoluted, you can also load the code to another notebook on Google Colab and play with it yourself. https://colab.research.google.com/gist/ontouchstart/934da948842bff86d470ffe25d692e67/test.ipynb https://colab.research.google.com/gist/ontouchstart/934da948... Now it is hard to debug programs like this unless you trace all the links and cached assets. On the other hand, many API centered cloud native application behaves like this and you have to deal with it one way or another. So to some extend, Jupyter is the IDE for the age of cloud.