3 ms·

Ok but I’m still not sold on the whole NBDev ecosystem. Is it worth building up lost of custom infra just so we can build in notebooks? Why should DS software d

Ok but I’m still not sold on the whole NBDev ecosystem. Is it worth building up lost of custom infra just so we can build in notebooks? Why should DS software dev be in a different special set of tools distinct from the other software it interacts with? And a host of other reasons https://gist.github.com/softwaredoug/d527a18643f29832b0f41af309f9ab98 https://gist.github.com/softwaredoug/d527a18643f29832b0f41af...

tsss 3y ago
Why shouldn't they use nbdev if it works for them? Plain Git obviously does not work and not only for data scientists. Git, with its inability to deal with non-textual artifacts, is completely unsuited to a modern software development workflow. Modern software development contains interactive graphs, audio and video assets, editable diagrams, vast amounts of binary data for reproduction of experiments, etc. etc. So much custom tooling has been built to somehow shoehorn this stuff into git and it never works well. I'm sick of diagramming software and IaC tools with horrible user experience just to keep a plaintext representation that no human dares to read anyway.
- softwaredoug 3y ago
  I'm not against notebooks. I'm against the idea of doing all of your software development in notebooks. There's a sane way to use notebooks. For me, nbdev is a step too far, as it really pushes notebooks as the primary dev artifact. (All my opinions, people can do whatever they want)
  - taeric 3y ago
    I am assuming you've seen this? https://www.youtube.com/watch?v=7jiPeIFXb6U https://www.youtube.com/watch?v=7jiPeIFXb6U :D
    - softwaredoug 3y ago
      Yes I'm a big fan :) I also like the "Why I like notebooks" counterpoint talk. I'm just not personally convinced. While they make some great arguments about where notebooks can be really powerful for software development, I don't think the speaker makes entirely valid counterpoints to the original "I don't like notebooks" talk and most of the problems of notebooks still stand. See this gist I posted above https://gist.github.com/softwaredoug/d527a18643f29832b0f41af309f9ab98 https://gist.github.com/softwaredoug/d527a18643f29832b0f41af...
jph00 3y ago
The premise behind nbdev isn’t that DS software should be a different set of tools - it’s that we should all consider using a different set of tools, based on literate and exploratory programming principles. I build all my projects, including a SQL lib, an EC2 interface, http and fastcgi frameworks, various web apps, and a mail merge system in notebooks, in nbdev and it’s made me much more productive. Amongst folks that have used nbdev for 1+ years that I’ve spoken to, all report a 3+ multiple of improvement in productivity (based on non rigorous self assessment). This could however be biased because these people also report enjoying coding much more, so it’s possible some of that effect is the time just doesn’t seem as long. Personally, after coding previously for over 20 years in various IDEs and editors (and for instance being prolific enough in vim I often gave talks about it) I wouldn’t ever want to go back to that old pre-notebooks time. But it’s a huge investment and requires a lot of relearning - to get the most out of it you have to rethink just about everything. So it might be better for those earlier in their careers that haven’t got as many habits to change.
- softwaredoug 3y ago
  Switching entirely to notebooks for development just feels like requiring everyone to use your favorite IDE for software development. As it binds the underlying representation to ipynb files, which will become inscrutable to a lot of developers, and can create an additional barrier to entry. I like the ideas of literate programming, but I think there should be a way to do it independent of the "editor" being used.
  - khqc 3y ago
    org mode ftw
sjkoelle 3y ago
ive written two projects with nbdev - its great, but after a while we moved off of it - too limiting

tsss 3y ago

Why shouldn't they use nbdev if it works for them? Plain Git obviously does not work and not only for data scientists. Git, with its inability to deal with non-textual artifacts, is completely unsuited to a modern software development workflow. Modern software development contains interactive graphs, audio and video assets, editable diagrams, vast amounts of binary data for reproduction of experiments, etc. etc. So much custom tooling has been built to somehow shoehorn this stuff into git and it never works well. I'm sick of diagramming software and IaC tools with horrible user experience just to keep a plaintext representation that no human dares to read anyway.

softwaredoug 3y ago
I'm not against notebooks. I'm against the idea of doing all of your software development in notebooks. There's a sane way to use notebooks. For me, nbdev is a step too far, as it really pushes notebooks as the primary dev artifact. (All my opinions, people can do whatever they want)
- taeric 3y ago
  I am assuming you've seen this? https://www.youtube.com/watch?v=7jiPeIFXb6U https://www.youtube.com/watch?v=7jiPeIFXb6U :D
  - softwaredoug 3y ago
    Yes I'm a big fan :) I also like the "Why I like notebooks" counterpoint talk. I'm just not personally convinced. While they make some great arguments about where notebooks can be really powerful for software development, I don't think the speaker makes entirely valid counterpoints to the original "I don't like notebooks" talk and most of the problems of notebooks still stand. See this gist I posted above https://gist.github.com/softwaredoug/d527a18643f29832b0f41af309f9ab98 https://gist.github.com/softwaredoug/d527a18643f29832b0f41af...

jph00 3y ago

The premise behind nbdev isn’t that DS software should be a different set of tools - it’s that we should all consider using a different set of tools, based on literate and exploratory programming principles. I build all my projects, including a SQL lib, an EC2 interface, http and fastcgi frameworks, various web apps, and a mail merge system in notebooks, in nbdev and it’s made me much more productive. Amongst folks that have used nbdev for 1+ years that I’ve spoken to, all report a 3+ multiple of improvement in productivity (based on non rigorous self assessment). This could however be biased because these people also report enjoying coding much more, so it’s possible some of that effect is the time just doesn’t seem as long. Personally, after coding previously for over 20 years in various IDEs and editors (and for instance being prolific enough in vim I often gave talks about it) I wouldn’t ever want to go back to that old pre-notebooks time. But it’s a huge investment and requires a lot of relearning - to get the most out of it you have to rethink just about everything. So it might be better for those earlier in their careers that haven’t got as many habits to change.

softwaredoug 3y ago
Switching entirely to notebooks for development just feels like requiring everyone to use your favorite IDE for software development. As it binds the underlying representation to ipynb files, which will become inscrutable to a lot of developers, and can create an additional barrier to entry. I like the ideas of literate programming, but I think there should be a way to do it independent of the "editor" being used.
- khqc 3y ago
  org mode ftw

sjkoelle 3y ago

ive written two projects with nbdev - its great, but after a while we moved off of it - too limiting