24 ms·

How to Grow Software Architecture out of Jupyter Notebooks

189 points by jedwhite 8y ago

deleted 8y ago
[deleted]
im3w1l 8y ago
Gradually pulling stable code out of the notebook into a tested library has worked well for me.
marmaduke 8y ago
> it might be a good idea to skip unit tests barring some barriers (e.g. use caching where appropriate) I've found writing unit tests is a neat clean replacement for notebooks.
- TomBombadildoze 8y ago
  A thousand times this. Jupyter notebooks are an antipattern. They encourage bad habits and yield copy-pasted, hacked-together, unmaintainable code. The REPL is a tool for exploration, not for full-blown development. > barring some barriers (e.g. use caching where appropriate) Fixtures.
  - marmaduke 8y ago
    Yes thanks for the vocab. Notebooks are like a single mutable global namespace fixture. Definitely a regression in terms of engineering practice, but Python slow startup and reloading semantics are also barriers, and I’d like to see a nailgun (JVM startup accelerator) for Python someday.
    - TomBombadildoze 8y ago
      > nailgun (JVM startup accelerator) for Python someday It's not an exact corollary but pypy sort of fills that niche. Unfortunately, it's two major versions behind and it's not a 100% drop-in replacement.
laichzeit0 8y ago
So I'm software developer for 10 years that started using Jupyter Notebooks the last year. I absolutely love that the REPL that ipython gives you. Do a query that takes really long, store it in some variable, spend the next hour or two working on that dataset in memory, changing code, iterating, all the while never having to re-execute that query or load data because it's just a REPL. How can one get the same type of developer experience in Python without using Jupyter notebooks? What I'm talking about, to people that maybe do traditional development in lets say Java or C++ is, imagine running your code in a debugger, setting a breakpoint, when it hits that breakpoint, you see "ah here's the problem", you fix the code, and have it re-execute all the while not having stopped your program at all. No re-compiling, and then having to re-execute from the beginning. It's like once you've done things the Juypter way, how can you possibly want to back to writing code in a traditional sense, it's just too slow of a process. How do you get that same experience without using Jupyter? I tried the VSCode plugin [1] that tries to make things the same as a Juptyer Notebook, but it's no where near as smooth an experience and feels clunky. [1] https://marketplace.visualstudio.com/items?itemName=donjayamanne.jupyter https://marketplace.visualstudio.com/items?itemName=donjayam...
- tempay 8y ago
  I get a very similar experience using IPython from a terminal, sometimes copying code over from my main text editor (where I have linting, multiple cursors and other niceties). As I do a lot of plotting I use itermplot[1] to display inline graphics. When I'm done I copy everything using %history XX to a permanent home in libraries or standalone scripts. [1] https://github.com/daleroberts/itermplot https://github.com/daleroberts/itermplot
- source99 8y ago
  I’ve spent a fair amount of time architecting my software to be able to set a flag and instead of processing a full typical run lasting several hours of compute it processes a subset in minutes or seconds. It’s not as clean as a Jupiter notebook but it’s much more repeatable than working in a notebook. I have a hard time maintaining discipline in a notebook and therefore don’t find them very useful
- molteanu 8y ago
skybrian 8y ago
It seems weird that notebooks don't have built-in unit testing. All you'd have to do is compare the current output to the saved "known good" output and show the diffs. It could be a checkbox next to each item in the notebook.
- DarkCrusader2 8y ago
  nbval does something similar to what you want. https://github.com/computationalmodelling/nbval https://github.com/computationalmodelling/nbval
tarasmatsyk 8y ago
TL;DR Did not get any insights here except links to books. What I prefer to do is to use Jupyter for all R&D tasks then `productize` modules/parts that proved to be working. When wrapping solutions as a product we go through typical engineering process: TDD, CI/CD, Docker, code review etc. In any case, we end up with having a bunch of machine instructions and there is no point in breaking best practices of engineering craft, they have been established for years. No need to invent another wheel.
fabatka 8y ago
My first thought when I saw the title: why would you want to do that? I feel like the only justification it gave (faster feedback) is way too insignificant compared to the downsides of developing inside a notebook. Joel Grus has a presentation that he gave at the 2018 JupyterCon about this which I found very enlightening and convincing. It is about why not to use notebooks apart from the very beginning of any serious project. The slides are available here: https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit#slide=id.g3d168d2fd3_0_255 https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh... Of course I still use notebooks for experimenting (usually it is open beside my IDE), I just think that developing a whole software architecture inside notebooks is not something notebooks are for.
- ptd 8y ago
  If you don't mind elaborating a bit, how deep do you have to get into a project before you get past the point of experimenting? Is there value in moving away from notebooks all together while you are still learning? Do these comments apply to platforms such as MyBinder or Jupyterhub? Finally,(other than Grus, he is excellent) are there any resources, people, or websites you would recommend for maintaining a sterile workflow or for general knowledge? Thanks.
  - fabatka 8y ago
    Sorry, I'm afraid I'm much less experienced than you think I am. My comment's value is entirely in sharing the slides, I only put some of my thoughts there so that my comment isn't just a link to a google drive file.
- murphy214 8y ago
  Slide 21 is to real.
breatheoften 8y ago
I made a module to offer some more options for some of these problems recently - particularly useful when you have code that you _want_ to keep organized in a notebook for continued interactive experimentation and/or for explaining a prototype process — but you also want to reuse that code in an external script or program. Let’s you call any notebook as a function. Keyword arguments can be passed and there’s a simple way to receive these values within the notebook to allow the notebook to be parametrizable when invoked externally. https://github.com/breathe/NotebookScripter https://github.com/breathe/NotebookScripter
- davecap1 8y ago
  Is this similar to papermill? https://github.com/nteract/papermill https://github.com/nteract/papermill
  - breatheoften 8y ago
    I hadn’t come across papermill before - looks quite full featured. Seems to be oriented a bit towards “scripting Notebooks” with the ability to generate output notebook files with populated output cells. I think it must have to emulate some of the logic of the Notebook ui to do that. NotebookScripter is much more minimal and doesn’t support generating/updating .ipynb files. It has just one api allowing one to treat a notebook file as a function that can be called from another python program. I’m not entirely sure whether papermill can run the target notebook in the calling process — it looks like it spins up a Jupyter kernel and communicates with it via message passing. NotebookScripter creates an ipython interpreter context in process and execs the notebook code in process.
Myrmornis 8y ago
I’m a backend python software engineer in my day job, working on production code in a standard git, text editor, .py files workflow. I occasionally use Jupyter notebooks (e.g. yesterday for the AdventOfCode challenges). I am ashamed to say that despite the beautiful front-end that works so well, I absolutely hate using Jupyter notebooks. - The mutable state with global variables from all cells in scope drives me crazy. I just want to run an ephemeral script repeatedly so I can be sure what the state is during execution. - The process of starting the server and moving code from version-controlled .py files into notebook cells to become part of Untitled(25).ipynb, which can’t be sanely version controlled, drives me crazy. - Not being able to use my normal text editor drives me crazy. Instead of building up lines of well-tested python functions in a disciplined line-by-line fashion, periodically committed to git with readable diffs, I end up with a chaotic mess of code fragments in various cells with no confidence regarding what will happen when some subset of the cells are executed, and none of it in git let alone with a sane commit history. I’ve tried so many times over the last 10 years, and I feel bad because it’s such an amazing project, but I really dislike the experience of using Jupyter instead of standard python development tools. The reason I do it is for graphics. This isn’t a Jupyter gripe other than the diverted attention, but why the fuck can’t I just import matplotlib in a normal python file? (under macOS it throws something about “frameworks” which no-one cares to understand. I think there’s some incantation that makes it work but seriously, this is ridiculous). And maybe draw graphics to a GUI graphics widget from the (i)python shell like R does. (No need to reply "Because you haven't written it"! This is deliberately a rant; I contribute to open source projects.)
- rafiki6 8y ago
  Fair enough. Using Jupyter Notebooks for any of the intended purposes you mention above is really out of scope of what Jupyter offers tbh. It's meant to be a Notebook. Run calculations, create visualizations, experiment with code. It's not a software engineering tool so much as it's an analysis tool closer to an advanced excel.
- mlevental 8y ago
  pycharm scientific mode let's you do exactly what you're describing re matpotlib
TuringTest 8y ago
Web notebooks are the new system shell, it's just that most developers haven't realized it yet. The original shell was an evolution over the teletype. Developers would type their code on a typewriter machine connected to the CPU over a wire (maybe involving punched tape or cards); and the CPU spouted back the results of the commands executed. Later, when video screens were adopted, they emulated the same linear approach of the terminal on the new device, because developers tend to be an extremely conservative bunch when it comes to the approach taken by their tools of the trade. It took several decades to exploit the advantages of interactive sessions that the new media allowed, creating a new family of IDEs with live debugging, property inspectors and Intellisense/contextual help. We are at a similar turning point with respect to classic development tools. The notebook is still used mostly as an improved version of the bash shell "with graphics", but there's a lot of research to do on how to enhance the model with new tools to take advantage of the new medium (including literate programming, web-based collaboration, or instant feedback of command execution on persistent objects). People are exploring the possibilities of this approach, with Bret Victor possibly being the most influential. I'm sure the personal experiences of thousands of developers with their home-made workflows, involving lots of heterogeneous tools used in creative ways, could be studied to design a more general environment that will change how large software projects are built.
- solomatov 8y ago
  I think https://datalore.io/ https://datalore.io/ is further away than Jupyter in respect to being a better shell for developers. It has interactive recomputation, collaboration built-in, integrated version control, etc.
ontouchstart 8y ago
Unlike most IDEs which are desktop based, Jupyter Notebook system is fundamentally network based with a standard HTML5 front end sending messages back and forth to the kernel. So fundamentally Jupyter is not a document based application that most of us software developers grow up with. You have to think in terms of messaging and the cells are just the input/output terminals. The rendering of cells is determined by the runtime container that parse the .ipynb (JSON) string. Here is an contrived example to explain it: https://nbviewer.jupyter.org/gist/ontouchstart/ea1631f69e507a81a9d9ec56b79e4d11 https://nbviewer.jupyter.org/gist/ontouchstart/ea1631f69e507... There is a lot going on depends on the container.
- TuringTest 8y ago
  That's good insight. Since the 50's, coding has been understood as creating text objects that contain the entire specification of a program in a programming language. Input data, parameters & configurations are considered external to the code object. It's the same with other activities like debugging, exploratory programming, debugging and building tests, which are considered peripheral to the "true coding" activity of altering the source code that will run on the production environment. The single successful exception to that model is the spreadsheet, were data and code are indissolubly merged in the same document; and, despite its success as a development tool for non-programmers, professional developers usually despise it. Could it be that Jupyter and notebooks are making developers comfortable with a new approach, where the distinction between "coding" and "running the code" is blurred, and where partial execution and mixing code with data are the norm? Coders are accustomed to using the REPL for this approach, but in the end only the code that is "commited" by copying it to the source file is considered the "final" version. Yet, with notebooks, code that has been written within an interactive session can be stored as the official definition for a part of a program, without having to transfer it to a separate object.
  - ontouchstart 8y ago
    At semantic level, coding is still about creating objects and managing their relationships with the environment. An .ipynb file itself just an object tree. Since Jupyter people made the wise decision of using easy to parse JSON format instead of more efficient binary format, the platform is opened up to a big ecosystem that people can make tools for specific need. Like the document centric desktop application revolution in the 80s that gave us MS Office and Adobe Photoshop, the open, data centric web application revolution we are seeing in this decade will fundamentally change how we consume and produce software. It is a natural evolution of computing system production from hardware panels to punch card, to text code instructions to object graph management in network environments. (The buzzword is “the cloud”.)
InGodsName 8y ago
I no longer code anything. I've been working remote at a big company now. I write requirements of a function then i send it to my subcontractors on freelancer website. While i am waiting, i play with my kids, kiss my wife and take her out for shopping. I get back the functions, i paste them in the codebase... change a thing or two. I am writing 3000 lines of code with this approach daily. I used to write 200 lines of code alone. One key thing, my passion in programming is killed entirely because always I've to work according to what plans management devise. Management changes direction, randomly kills project, changes requirements - this makes me feel like a slave who has to dance to statisfy the masters, no matter how many landmines they place on the dance floor. So, now i just create outline and let cheap labor fill the blanks. My value proposition isn't code but because i can understand the problems at hand and create plans to tackle the problem. Choice of problem isn't in my hand tho area of problem is definitely in my hand. I am never going back to open office plan, sebum/ear wax filled noise cancelling headphones.
- clueless123 8y ago
  Doing about the same thing 50% of my day, except I have to spend another 50% on meetings explaining higher-up's the why's of every and each architectural decision. Then the third 50% of my time I spend on HR issues, like filling forms and compliance training.. They do pay me by the hour.. so no complaints here :)
- BeetleB 8y ago
  Are you an employee or a contractor? If the former, are you not violating IP by subcontracting?
  - clueless123 8y ago
    one price, one person All-tasks Contractor.. I farm out/purchase all basic graphics, use tons of open source libraries and farm out most tedious stuff. My employers know, as long as we deliver they do not care.
- PurpleRamen 8y ago
  > I am writing 3000 lines of code with this approach daily. Don't you mean you "collect" ~3000 lines of code per day? And how much does that even cost you? How do you ensure quality? Do you pay per hour or result?
rafiki6 8y ago
I argue that this misses the mark. Jupyter Notebooks is much more an evolution of Excel then it is of other development tools.
GChevalier 8y ago
Author here. Whoa, thank you all for the nice comments, I didn't expect to make such a buzz here nor today. I'm glad to see the reactions - even the bad ones, it seems aligned with what I thought. Yes, notebooks are very useful for the faster coding cycle, but they become easily heavy (I'd love to see a better multiline edit and a better autocompletion in Jupiter). Seems like I already posted my article 2 months ago, but renamed the GitHub repo since then, which may explain why someone else (jedwhite) could submit my article again: https://news.ycombinator.com/item?id=18339703 https://news.ycombinator.com/item?id=18339703 I didn't submit it twice to HN. Well, nice to see that in a parallel world my post did the 1st page on HN! :-) But could this mean that my HN account is like "shadow banned" or something? Strange to see that all my own submissions on HN haven't got much attention for months. Or maybe it's just the random factor... Well, thanks!

189 points by jedwhite 8y ago

deleted 8y ago

[deleted]

im3w1l 8y ago

Gradually pulling stable code out of the notebook into a tested library has worked well for me.

marmaduke 8y ago

> it might be a good idea to skip unit tests barring some barriers (e.g. use caching where appropriate) I've found writing unit tests is a neat clean replacement for notebooks.

TomBombadildoze 8y ago
A thousand times this. Jupyter notebooks are an antipattern. They encourage bad habits and yield copy-pasted, hacked-together, unmaintainable code. The REPL is a tool for exploration, not for full-blown development. > barring some barriers (e.g. use caching where appropriate) Fixtures.
- marmaduke 8y ago
  Yes thanks for the vocab. Notebooks are like a single mutable global namespace fixture. Definitely a regression in terms of engineering practice, but Python slow startup and reloading semantics are also barriers, and I’d like to see a nailgun (JVM startup accelerator) for Python someday.
  - TomBombadildoze 8y ago
    > nailgun (JVM startup accelerator) for Python someday It's not an exact corollary but pypy sort of fills that niche. Unfortunately, it's two major versions behind and it's not a 100% drop-in replacement.

laichzeit0 8y ago

So I'm software developer for 10 years that started using Jupyter Notebooks the last year. I absolutely love that the REPL that ipython gives you. Do a query that takes really long, store it in some variable, spend the next hour or two working on that dataset in memory, changing code, iterating, all the while never having to re-execute that query or load data because it's just a REPL. How can one get the same type of developer experience in Python without using Jupyter notebooks? What I'm talking about, to people that maybe do traditional development in lets say Java or C++ is, imagine running your code in a debugger, setting a breakpoint, when it hits that breakpoint, you see "ah here's the problem", you fix the code, and have it re-execute all the while not having stopped your program at all. No re-compiling, and then having to re-execute from the beginning. It's like once you've done things the Juypter way, how can you possibly want to back to writing code in a traditional sense, it's just too slow of a process. How do you get that same experience without using Jupyter? I tried the VSCode plugin [1] that tries to make things the same as a Juptyer Notebook, but it's no where near as smooth an experience and feels clunky. [1] https://marketplace.visualstudio.com/items?itemName=donjayamanne.jupyter https://marketplace.visualstudio.com/items?itemName=donjayam...

tempay 8y ago
I get a very similar experience using IPython from a terminal, sometimes copying code over from my main text editor (where I have linting, multiple cursors and other niceties). As I do a lot of plotting I use itermplot[1] to display inline graphics. When I'm done I copy everything using %history XX to a permanent home in libraries or standalone scripts. [1] https://github.com/daleroberts/itermplot https://github.com/daleroberts/itermplot
source99 8y ago
I’ve spent a fair amount of time architecting my software to be able to set a flag and instead of processing a full typical run lasting several hours of compute it processes a subset in minutes or seconds. It’s not as clean as a Jupiter notebook but it’s much more repeatable than working in a notebook. I have a hard time maintaining discipline in a notebook and therefore don’t find them very useful
molteanu 8y ago

skybrian 8y ago

It seems weird that notebooks don't have built-in unit testing. All you'd have to do is compare the current output to the saved "known good" output and show the diffs. It could be a checkbox next to each item in the notebook.

DarkCrusader2 8y ago
nbval does something similar to what you want. https://github.com/computationalmodelling/nbval https://github.com/computationalmodelling/nbval

tarasmatsyk 8y ago

TL;DR Did not get any insights here except links to books. What I prefer to do is to use Jupyter for all R&D tasks then `productize` modules/parts that proved to be working. When wrapping solutions as a product we go through typical engineering process: TDD, CI/CD, Docker, code review etc. In any case, we end up with having a bunch of machine instructions and there is no point in breaking best practices of engineering craft, they have been established for years. No need to invent another wheel.

fabatka 8y ago

My first thought when I saw the title: why would you want to do that? I feel like the only justification it gave (faster feedback) is way too insignificant compared to the downsides of developing inside a notebook. Joel Grus has a presentation that he gave at the 2018 JupyterCon about this which I found very enlightening and convincing. It is about why not to use notebooks apart from the very beginning of any serious project. The slides are available here: https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit#slide=id.g3d168d2fd3_0_255 https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh... Of course I still use notebooks for experimenting (usually it is open beside my IDE), I just think that developing a whole software architecture inside notebooks is not something notebooks are for.

ptd 8y ago
If you don't mind elaborating a bit, how deep do you have to get into a project before you get past the point of experimenting? Is there value in moving away from notebooks all together while you are still learning? Do these comments apply to platforms such as MyBinder or Jupyterhub? Finally,(other than Grus, he is excellent) are there any resources, people, or websites you would recommend for maintaining a sterile workflow or for general knowledge? Thanks.
- fabatka 8y ago
  Sorry, I'm afraid I'm much less experienced than you think I am. My comment's value is entirely in sharing the slides, I only put some of my thoughts there so that my comment isn't just a link to a google drive file.
murphy214 8y ago
Slide 21 is to real.

breatheoften 8y ago

I made a module to offer some more options for some of these problems recently - particularly useful when you have code that you _want_ to keep organized in a notebook for continued interactive experimentation and/or for explaining a prototype process — but you also want to reuse that code in an external script or program. Let’s you call any notebook as a function. Keyword arguments can be passed and there’s a simple way to receive these values within the notebook to allow the notebook to be parametrizable when invoked externally. https://github.com/breathe/NotebookScripter https://github.com/breathe/NotebookScripter

davecap1 8y ago
Is this similar to papermill? https://github.com/nteract/papermill https://github.com/nteract/papermill
- breatheoften 8y ago
  I hadn’t come across papermill before - looks quite full featured. Seems to be oriented a bit towards “scripting Notebooks” with the ability to generate output notebook files with populated output cells. I think it must have to emulate some of the logic of the Notebook ui to do that. NotebookScripter is much more minimal and doesn’t support generating/updating .ipynb files. It has just one api allowing one to treat a notebook file as a function that can be called from another python program. I’m not entirely sure whether papermill can run the target notebook in the calling process — it looks like it spins up a Jupyter kernel and communicates with it via message passing. NotebookScripter creates an ipython interpreter context in process and execs the notebook code in process.

Myrmornis 8y ago

I’m a backend python software engineer in my day job, working on production code in a standard git, text editor, .py files workflow. I occasionally use Jupyter notebooks (e.g. yesterday for the AdventOfCode challenges). I am ashamed to say that despite the beautiful front-end that works so well, I absolutely hate using Jupyter notebooks. - The mutable state with global variables from all cells in scope drives me crazy. I just want to run an ephemeral script repeatedly so I can be sure what the state is during execution. - The process of starting the server and moving code from version-controlled .py files into notebook cells to become part of Untitled(25).ipynb, which can’t be sanely version controlled, drives me crazy. - Not being able to use my normal text editor drives me crazy. Instead of building up lines of well-tested python functions in a disciplined line-by-line fashion, periodically committed to git with readable diffs, I end up with a chaotic mess of code fragments in various cells with no confidence regarding what will happen when some subset of the cells are executed, and none of it in git let alone with a sane commit history. I’ve tried so many times over the last 10 years, and I feel bad because it’s such an amazing project, but I really dislike the experience of using Jupyter instead of standard python development tools. The reason I do it is for graphics. This isn’t a Jupyter gripe other than the diverted attention, but why the fuck can’t I just import matplotlib in a normal python file? (under macOS it throws something about “frameworks” which no-one cares to understand. I think there’s some incantation that makes it work but seriously, this is ridiculous). And maybe draw graphics to a GUI graphics widget from the (i)python shell like R does. (No need to reply "Because you haven't written it"! This is deliberately a rant; I contribute to open source projects.)

rafiki6 8y ago
Fair enough. Using Jupyter Notebooks for any of the intended purposes you mention above is really out of scope of what Jupyter offers tbh. It's meant to be a Notebook. Run calculations, create visualizations, experiment with code. It's not a software engineering tool so much as it's an analysis tool closer to an advanced excel.
mlevental 8y ago
pycharm scientific mode let's you do exactly what you're describing re matpotlib

TuringTest 8y ago

Web notebooks are the new system shell, it's just that most developers haven't realized it yet. The original shell was an evolution over the teletype. Developers would type their code on a typewriter machine connected to the CPU over a wire (maybe involving punched tape or cards); and the CPU spouted back the results of the commands executed. Later, when video screens were adopted, they emulated the same linear approach of the terminal on the new device, because developers tend to be an extremely conservative bunch when it comes to the approach taken by their tools of the trade. It took several decades to exploit the advantages of interactive sessions that the new media allowed, creating a new family of IDEs with live debugging, property inspectors and Intellisense/contextual help. We are at a similar turning point with respect to classic development tools. The notebook is still used mostly as an improved version of the bash shell "with graphics", but there's a lot of research to do on how to enhance the model with new tools to take advantage of the new medium (including literate programming, web-based collaboration, or instant feedback of command execution on persistent objects). People are exploring the possibilities of this approach, with Bret Victor possibly being the most influential. I'm sure the personal experiences of thousands of developers with their home-made workflows, involving lots of heterogeneous tools used in creative ways, could be studied to design a more general environment that will change how large software projects are built.

solomatov 8y ago
I think https://datalore.io/ https://datalore.io/ is further away than Jupyter in respect to being a better shell for developers. It has interactive recomputation, collaboration built-in, integrated version control, etc.

ontouchstart 8y ago

Unlike most IDEs which are desktop based, Jupyter Notebook system is fundamentally network based with a standard HTML5 front end sending messages back and forth to the kernel. So fundamentally Jupyter is not a document based application that most of us software developers grow up with. You have to think in terms of messaging and the cells are just the input/output terminals. The rendering of cells is determined by the runtime container that parse the .ipynb (JSON) string. Here is an contrived example to explain it: https://nbviewer.jupyter.org/gist/ontouchstart/ea1631f69e507a81a9d9ec56b79e4d11 https://nbviewer.jupyter.org/gist/ontouchstart/ea1631f69e507... There is a lot going on depends on the container.

TuringTest 8y ago
That's good insight. Since the 50's, coding has been understood as creating text objects that contain the entire specification of a program in a programming language. Input data, parameters & configurations are considered external to the code object. It's the same with other activities like debugging, exploratory programming, debugging and building tests, which are considered peripheral to the "true coding" activity of altering the source code that will run on the production environment. The single successful exception to that model is the spreadsheet, were data and code are indissolubly merged in the same document; and, despite its success as a development tool for non-programmers, professional developers usually despise it. Could it be that Jupyter and notebooks are making developers comfortable with a new approach, where the distinction between "coding" and "running the code" is blurred, and where partial execution and mixing code with data are the norm? Coders are accustomed to using the REPL for this approach, but in the end only the code that is "commited" by copying it to the source file is considered the "final" version. Yet, with notebooks, code that has been written within an interactive session can be stored as the official definition for a part of a program, without having to transfer it to a separate object.
- ontouchstart 8y ago
  At semantic level, coding is still about creating objects and managing their relationships with the environment. An .ipynb file itself just an object tree. Since Jupyter people made the wise decision of using easy to parse JSON format instead of more efficient binary format, the platform is opened up to a big ecosystem that people can make tools for specific need. Like the document centric desktop application revolution in the 80s that gave us MS Office and Adobe Photoshop, the open, data centric web application revolution we are seeing in this decade will fundamentally change how we consume and produce software. It is a natural evolution of computing system production from hardware panels to punch card, to text code instructions to object graph management in network environments. (The buzzword is “the cloud”.)

InGodsName 8y ago

I no longer code anything. I've been working remote at a big company now. I write requirements of a function then i send it to my subcontractors on freelancer website. While i am waiting, i play with my kids, kiss my wife and take her out for shopping. I get back the functions, i paste them in the codebase... change a thing or two. I am writing 3000 lines of code with this approach daily. I used to write 200 lines of code alone. One key thing, my passion in programming is killed entirely because always I've to work according to what plans management devise. Management changes direction, randomly kills project, changes requirements - this makes me feel like a slave who has to dance to statisfy the masters, no matter how many landmines they place on the dance floor. So, now i just create outline and let cheap labor fill the blanks. My value proposition isn't code but because i can understand the problems at hand and create plans to tackle the problem. Choice of problem isn't in my hand tho area of problem is definitely in my hand. I am never going back to open office plan, sebum/ear wax filled noise cancelling headphones.

clueless123 8y ago
Doing about the same thing 50% of my day, except I have to spend another 50% on meetings explaining higher-up's the why's of every and each architectural decision. Then the third 50% of my time I spend on HR issues, like filling forms and compliance training.. They do pay me by the hour.. so no complaints here :)
BeetleB 8y ago
Are you an employee or a contractor? If the former, are you not violating IP by subcontracting?
- clueless123 8y ago
  one price, one person All-tasks Contractor.. I farm out/purchase all basic graphics, use tons of open source libraries and farm out most tedious stuff. My employers know, as long as we deliver they do not care.
PurpleRamen 8y ago
> I am writing 3000 lines of code with this approach daily. Don't you mean you "collect" ~3000 lines of code per day? And how much does that even cost you? How do you ensure quality? Do you pay per hour or result?

rafiki6 8y ago

I argue that this misses the mark. Jupyter Notebooks is much more an evolution of Excel then it is of other development tools.

GChevalier 8y ago

Author here. Whoa, thank you all for the nice comments, I didn't expect to make such a buzz here nor today. I'm glad to see the reactions - even the bad ones, it seems aligned with what I thought. Yes, notebooks are very useful for the faster coding cycle, but they become easily heavy (I'd love to see a better multiline edit and a better autocompletion in Jupiter). Seems like I already posted my article 2 months ago, but renamed the GitHub repo since then, which may explain why someone else (jedwhite) could submit my article again: https://news.ycombinator.com/item?id=18339703 https://news.ycombinator.com/item?id=18339703 I didn't submit it twice to HN. Well, nice to see that in a parallel world my post did the 1st page on HN! :-) But could this mean that my HN account is like "shadow banned" or something? Strange to see that all my own submissions on HN haven't got much attention for months. Or maybe it's just the random factor... Well, thanks!