6 ms·

Yet another hardcore programmy type discovers that usability is infinity more important that how many clock-cycles you save. Programming languages aren't for c

by jrm4 5y ago

Yet another hardcore programmy type discovers that usability is infinity more important that how many clock-cycles you save. Programming languages aren't for computers, they're for people.

da39a3ee 5y ago
matplotlib doesn't score highly on usability. jupyter notebooks encourage disorganized, unprincipled programming; chaotic re-running of cells in the face of global mutable state; and prevent budding programmers from learning to use version control because the JSON format was designed without version control in mind.
- mistrial9 5y ago
  .. and get more new users each year than four generations of PCs combined
- monkeybutton 5y ago
  Being able to hack out code to explore and experiment with data while not having to reload and reprocess data (thanks to that global mutable state!) saves a hell of a lot of time in the long run.
- wiz21c 5y ago
  For Jupyter, it depends on the workflow. Especially with data sciences. In data science, you spend a lot of time playing with the data, testing things, drawing charts, computing, etc. When you do that, the cost of starting a python interpreter, loading the imports, loading the (usually big) data becomes a real pain 'cos you iterate like hell. Working in a REPL becomes really important. But even more, working with Jupyter allows you to work out a very detailed explanation of your thought process, describing your ideas, your experiments. Being able to mix code and explanations is really important (and reminiscent of literate programming). You got the same kind of flow with R. As data scientist, I'm concerned about data, statistics, maths, understanding the problem (instead of the solution). I don't care about code. Once I get my data understanding right then comes the time of turning all of that into a software that can be used. Before that, Jupyter really gives a productivity boost. For the code part, yep, you need other principles where Jupyter may not be suitable.
  - patrick451 5y ago
    It's interesting, I never feel like I get these exploratory benefits from jupyter notebooks. I just end up feeling like one hand and half my brain is tied behind my back. I'm most productive iterating in a similar way to what you describe, but in an ipython terminal, running a script and libraries that I'm iterating on in a real editor. If there are expensive computations that I want' to check point, I just save and load them as pickle files.
    - wiz21c 5y ago
      really interesting. I may have overlooked IPython a bit (I just thought Jupyter was its improved version). For the moment, maybe like you, I prerpocess the data (which takes minutes) into numpy array which then take seconds to load. But once I add imports, everything takes about 5 or 6 seconds to load everything I need. So Jupyter remains a good idea. Moreover, I love (and actually need) to mix math and text, so markdown+latex maths is really a great combo. I dont' know if one can do that in IPython, I'll sure look!
    - kzrdude 5y ago
      I have to say I think a jupyter notebook format is a 10x improvement in productivity over ipython. It's just so much easier to work with - and a step more reproducible too, my scribbles are all there saved in the notebook, at least!
- patrick451 5y ago
  > matplotlib doesn't score highly on usability. The one place matplotlib sucks is any kind of interactivity. But other than that, matplotlib has the best, most intuitive interface of all the python plotting libraries I've tried. It's also one of the few libraries that doesn't rely on generating html for a webbrowser, which makes for a miserable workflow. I still think Matlab's plotting is untouched by open source options.
  - taeric 5y ago
    The charts that R produces are typically better looking. But...R. :(
  - cinntaile 5y ago
    Matplotlib hides a lot of complexity if you ask me. As soon as you do something in a different way than intended you're off searching stackoverflow for a post that did something similar to what you want. Then you tweak it a little and hope it works.
- analog31 5y ago
  I've programmed in a number of languages over the past 40+ years, starting with BASIC, and every one of them encourages sloppy coding. The good discipline always has to be taught, learned, and willingly practiced. The closest I came to a language designed for teaching good practices was Pascal. I find it easier to read and understand bad code written in Python, than good code written in the C family languages.
  - da39a3ee 5y ago
    Yes, what I was saying was that writing Python code in files is a better and more educational way to program than writing Python code in a Jupyter Notebook. It wasn't a criticism of Python.
    - analog31 5y ago
      I use Jupyter a lot, but have a personal rule to do "restart kernel and run all cells" once in a while, to scare up any kind of hidden state or out-of-order execution problems. For instance, if I'm about to leave a notebook for a while, I'll make sure it runs without error from top to bottom. In that sense, I'm making it work like Python code in a file. The advantage of code in files is that I can use all of the slick code analysis tools that will warn me about my mistakes. I wish there were something that would let those tools go through the code in a Python notebook from top to bottom.
- kzrdude 5y ago
  papermill is good and ploomber is a thing to watch. Ploomber makes it systematic - store notebooks as .py (py:percent files for example), parameterize them with papermill and execute as a batch job. One can view the resulting jupyter notebooks as .ipynb later and produce reports as html if wanted. It's really good already, and better if ploomber gets more development. The whole reason it works is because it's easy to open the .py notebook and work on it, interactively, in jupyter. The main idea - jupytext for .py notebooks and papermill for parameters & execution - that's already "stable" and easy for anyone to use for their own purposes.
  - edublancas 5y ago
    (ploomber maintainer here) Any feedback for us? What can we do to improve Ploomber?
    - kzrdude 5y ago
      Maybe I haven't come far enough with my ploomber use to tell yet! It works nicely but I know I'll learn more and open my eyes more as I go. As a first impression, I eventually found meta.extract_upstream = False which I think is important. Reason: The code for each step should be a lego piece, a black box with inputs and outputs. That code should not itself hardcode what its predecessor in the pipeline is - you connect the pieces in pipeline.yaml. (extract_upstream = False is not by itself enough to solve this, since you also need to be able to rename inputs/outputs for a notebook to be fully reusable as a lego piece, but it's good enough for now.) I also for my own sanity need to know more about how the jupyter extension part works, how it decides to load injected-parameters or not. But maybe I could learn that somehow from docs. In general I want components that are easy to understand and plug together and less magic (but the whole jupyter ecosystem's source code feels this way to me unfortunately, lots of hard to follow abstractions passing things around). But it's developing rapidly and already very useful, thank you so much!
      - edublancas 5y ago
        This is great feedback, thanks a lot! I'll ensure we display the "extract_upstream" more prominently in the docs, we've been getting this feedback a couple times now :) Re: the Jupyter extension injects the cell when the file you're opening is declared in the pipeline.yaml file. You can turn off the extension if you prefer. Feel free to join our community, this feedback helps us make Ploomber better! https://ploomber.io/community https://ploomber.io/community
rewq4321 5y ago
Not infinity, but yeah it's worth more than people generally think. But in the end you don't really lose many clock cycles anyway because everything actually runs in C/CUDA/etc. behind the scenes
deleted 5y ago
[deleted]
Yoric 5y ago
I personally find numpy irritating, surprising, misleading and definitely a wrong definition of "usability". YMMV.
- Scene_Cast2 5y ago
  What are your thoughts on Pandas? As someone who uses numpy almost daily, I think that numpy is "overextended" beyond its core niche, sure. So - making it work with things outside that niche (e.g. streaming, non-rectangular data, non-uniform data, nonhomogeneous data, etc) is painful. However, 1) there's Pandas for that, and 2) I disagree with "misleading" and "surprising". What makes you think that?
  - m_mueller 5y ago
    Not GP, but I’m using pandas daily to build up a BI platform within a financial institution. Compared to Matlab and even Fortran it has some issues IMO: * why distinguish between Series and DataFrame? just give me an interface for m x n matrices or even higher dimensions. * pure vs. in-place operations. not such a big fan of having multiple versions of the same function, e.g. a more pythonic df[“my_col”] = series vs. a more functional df.assign({“my_col”: series}) ; I’d rather have everything like the latter to be able to more easily have best practices in place. That brings me to another point: if we keep everything purely functional, then python’s syntax is making things a bit awkward. Where in something like JS you could just put every function call with its dot on a new line without the need to assign, in Python this requires putting line break characters or wrapping it in round brackets. This is one place where a language with explicit assignment terminators (semicolons) are a bit cleaner to work with. All that being said scipy is still a great choice to have both system programming and numerical business logic in one language.
  - J253 5y ago
    In my opinion and experience, I think you’re right about “there’s Pandas for that” and “that” can be almost anything. It can do almost anything but making it do almost anything requires constant reference to the docs. And I find maintainability difficult. It seems like there’s 50 kwargs for every method. Sometimes things happen in place by default, other times they don’t. Compound indexes still confuse me. But I’m not a data scientist so I don’t do much ad-hoc analysis that seems typical with pandas users.
matsemann 5y ago
I wonder how many days have been wasted on non-programmers trying to get their Conda environment up and running or similar. Half the data science stuff isn't reproducible, not because of the science, but because getting the notebooks running with its dependencies is almost impossible.
- lysecret 5y ago
  have you heard of docker?
- irrational 5y ago
  Which begs the question, why has nobody built anything better?
  - taeric 5y ago
    I'd argue that they probably have. It just isn't free.
  - belval 5y ago
    Because it's a hard problem and people love hating on Python because it doesn't come with a way to handle all the compiled dependencies that work for every OS. Data science has a ton of moving library parts, it is genuinely difficult to distribute precompiled libraries for everyone when you have 2-3 actively maintained CUDA version with 2 cuDNN version for accelerators that change every 2 years. Most team fail to standardize on an environment (say Python 3.8, Ubuntu 20.04, CUDA 11.1, and cuDNN 8) and then get hung up on a dependency not building as if it's Python's fault that it does not have control of your entire OS.
    - matsemann 5y ago
      But why is it such a big problem in Python compared to other stacks? Why does all python projects end up depending on you having those exact tools of things installed locally and the planets aligned a certain way, when other stacks do not?
taeric 5y ago
This feels misguided, too. It begs the question that python has good usability. Anyone that has tried managing dependencies in it will know that is mostly a lie. What python had, was that it was preinstalled on many computers and then had a large cohort of users that are insisting that others use it. And mostly force proclaiming that it is easy and readable. I'll not claim that it is hard, per se. More that it is not intrinsically easier than any other dynamic language. For evidence, the main packages that are popular are often clones of packages from other environments that were not widely installed. Jupyter can be seen as free version of many scientific applications. Matlab, Mathematica, etc. Matplotlib is rather direct in it's copy. Pretty sure there are more examples.
- iainctduncan 5y ago
  Is managing deps in Python a pain? sure is!. Is it a pain in the other contenders of easily available dynamic languages? Yup. So that's a wash. Managing deps in dynamic languages is not a simple problem, I can't say I've tried one that did it super well yet.
  - taeric 5y ago
    I'm... Not sure insulting other languages is a path to victory. Npm, as much as it annoys me, is light-years ahead of anything in python. Quicklisp is rather pleasant, now. Ruby has had gems for a long time. I grant that it is a hard problem. I am not griping that it is not solved. More that the option community has largely failed to even pick a direction. The most used dependency methods are, in the modern spirit of python, deprecated already.
    - iainctduncan 5y ago
      I'm not insulting other languages - I'm pointing out they have the same situation! Where in that post is there an insult??? I have had many dep fights with Ruby and Node too, and I would disagree that their solutions are in any way better than pip and virtualenv.
dang 5y ago
"Don't be snarky." "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- jrm4 5y ago
  Harsh isn't the same as shallow. I don't believe snark is ALWAYS uncalled for, only that you deploy it when it's reasonably necessary. And this is a thing that people, especially HERE, get wrong a LOT. That it's unpleasant or hits people where it hurts doesn't mean it's not important, and I think it is.
  - dang 5y ago
    The issue is the effects that it has on the ecosystem. It dumbs down threads and makes people nastier. If someone is wrong and you know more than they do, you can share correct information without putting them down.

by jrm4 5y ago

Yet another hardcore programmy type discovers that usability is infinity more important that how many clock-cycles you save. Programming languages aren't for computers, they're for people.

da39a3ee 5y ago

matplotlib doesn't score highly on usability. jupyter notebooks encourage disorganized, unprincipled programming; chaotic re-running of cells in the face of global mutable state; and prevent budding programmers from learning to use version control because the JSON format was designed without version control in mind.

mistrial9 5y ago
.. and get more new users each year than four generations of PCs combined
monkeybutton 5y ago
Being able to hack out code to explore and experiment with data while not having to reload and reprocess data (thanks to that global mutable state!) saves a hell of a lot of time in the long run.
wiz21c 5y ago
For Jupyter, it depends on the workflow. Especially with data sciences. In data science, you spend a lot of time playing with the data, testing things, drawing charts, computing, etc. When you do that, the cost of starting a python interpreter, loading the imports, loading the (usually big) data becomes a real pain 'cos you iterate like hell. Working in a REPL becomes really important. But even more, working with Jupyter allows you to work out a very detailed explanation of your thought process, describing your ideas, your experiments. Being able to mix code and explanations is really important (and reminiscent of literate programming). You got the same kind of flow with R. As data scientist, I'm concerned about data, statistics, maths, understanding the problem (instead of the solution). I don't care about code. Once I get my data understanding right then comes the time of turning all of that into a software that can be used. Before that, Jupyter really gives a productivity boost. For the code part, yep, you need other principles where Jupyter may not be suitable.
- patrick451 5y ago
  It's interesting, I never feel like I get these exploratory benefits from jupyter notebooks. I just end up feeling like one hand and half my brain is tied behind my back. I'm most productive iterating in a similar way to what you describe, but in an ipython terminal, running a script and libraries that I'm iterating on in a real editor. If there are expensive computations that I want' to check point, I just save and load them as pickle files.
  - wiz21c 5y ago
    really interesting. I may have overlooked IPython a bit (I just thought Jupyter was its improved version). For the moment, maybe like you, I prerpocess the data (which takes minutes) into numpy array which then take seconds to load. But once I add imports, everything takes about 5 or 6 seconds to load everything I need. So Jupyter remains a good idea. Moreover, I love (and actually need) to mix math and text, so markdown+latex maths is really a great combo. I dont' know if one can do that in IPython, I'll sure look!
  - kzrdude 5y ago
    I have to say I think a jupyter notebook format is a 10x improvement in productivity over ipython. It's just so much easier to work with - and a step more reproducible too, my scribbles are all there saved in the notebook, at least!
patrick451 5y ago
> matplotlib doesn't score highly on usability. The one place matplotlib sucks is any kind of interactivity. But other than that, matplotlib has the best, most intuitive interface of all the python plotting libraries I've tried. It's also one of the few libraries that doesn't rely on generating html for a webbrowser, which makes for a miserable workflow. I still think Matlab's plotting is untouched by open source options.
- taeric 5y ago
  The charts that R produces are typically better looking. But...R. :(
- cinntaile 5y ago
  Matplotlib hides a lot of complexity if you ask me. As soon as you do something in a different way than intended you're off searching stackoverflow for a post that did something similar to what you want. Then you tweak it a little and hope it works.
analog31 5y ago
I've programmed in a number of languages over the past 40+ years, starting with BASIC, and every one of them encourages sloppy coding. The good discipline always has to be taught, learned, and willingly practiced. The closest I came to a language designed for teaching good practices was Pascal. I find it easier to read and understand bad code written in Python, than good code written in the C family languages.
- da39a3ee 5y ago
  Yes, what I was saying was that writing Python code in files is a better and more educational way to program than writing Python code in a Jupyter Notebook. It wasn't a criticism of Python.
  - analog31 5y ago
    I use Jupyter a lot, but have a personal rule to do "restart kernel and run all cells" once in a while, to scare up any kind of hidden state or out-of-order execution problems. For instance, if I'm about to leave a notebook for a while, I'll make sure it runs without error from top to bottom. In that sense, I'm making it work like Python code in a file. The advantage of code in files is that I can use all of the slick code analysis tools that will warn me about my mistakes. I wish there were something that would let those tools go through the code in a Python notebook from top to bottom.
kzrdude 5y ago
papermill is good and ploomber is a thing to watch. Ploomber makes it systematic - store notebooks as .py (py:percent files for example), parameterize them with papermill and execute as a batch job. One can view the resulting jupyter notebooks as .ipynb later and produce reports as html if wanted. It's really good already, and better if ploomber gets more development. The whole reason it works is because it's easy to open the .py notebook and work on it, interactively, in jupyter. The main idea - jupytext for .py notebooks and papermill for parameters & execution - that's already "stable" and easy for anyone to use for their own purposes.
- edublancas 5y ago
  (ploomber maintainer here) Any feedback for us? What can we do to improve Ploomber?
  - kzrdude 5y ago
    Maybe I haven't come far enough with my ploomber use to tell yet! It works nicely but I know I'll learn more and open my eyes more as I go. As a first impression, I eventually found meta.extract_upstream = False which I think is important. Reason: The code for each step should be a lego piece, a black box with inputs and outputs. That code should not itself hardcode what its predecessor in the pipeline is - you connect the pieces in pipeline.yaml. (extract_upstream = False is not by itself enough to solve this, since you also need to be able to rename inputs/outputs for a notebook to be fully reusable as a lego piece, but it's good enough for now.) I also for my own sanity need to know more about how the jupyter extension part works, how it decides to load injected-parameters or not. But maybe I could learn that somehow from docs. In general I want components that are easy to understand and plug together and less magic (but the whole jupyter ecosystem's source code feels this way to me unfortunately, lots of hard to follow abstractions passing things around). But it's developing rapidly and already very useful, thank you so much!
    - edublancas 5y ago
      This is great feedback, thanks a lot! I'll ensure we display the "extract_upstream" more prominently in the docs, we've been getting this feedback a couple times now :) Re: the Jupyter extension injects the cell when the file you're opening is declared in the pipeline.yaml file. You can turn off the extension if you prefer. Feel free to join our community, this feedback helps us make Ploomber better! https://ploomber.io/community https://ploomber.io/community

rewq4321 5y ago

Not infinity, but yeah it's worth more than people generally think. But in the end you don't really lose many clock cycles anyway because everything actually runs in C/CUDA/etc. behind the scenes

deleted 5y ago

[deleted]

Yoric 5y ago

I personally find numpy irritating, surprising, misleading and definitely a wrong definition of "usability". YMMV.

Scene_Cast2 5y ago
What are your thoughts on Pandas? As someone who uses numpy almost daily, I think that numpy is "overextended" beyond its core niche, sure. So - making it work with things outside that niche (e.g. streaming, non-rectangular data, non-uniform data, nonhomogeneous data, etc) is painful. However, 1) there's Pandas for that, and 2) I disagree with "misleading" and "surprising". What makes you think that?
- m_mueller 5y ago
  Not GP, but I’m using pandas daily to build up a BI platform within a financial institution. Compared to Matlab and even Fortran it has some issues IMO: * why distinguish between Series and DataFrame? just give me an interface for m x n matrices or even higher dimensions. * pure vs. in-place operations. not such a big fan of having multiple versions of the same function, e.g. a more pythonic df[“my_col”] = series vs. a more functional df.assign({“my_col”: series}) ; I’d rather have everything like the latter to be able to more easily have best practices in place. That brings me to another point: if we keep everything purely functional, then python’s syntax is making things a bit awkward. Where in something like JS you could just put every function call with its dot on a new line without the need to assign, in Python this requires putting line break characters or wrapping it in round brackets. This is one place where a language with explicit assignment terminators (semicolons) are a bit cleaner to work with. All that being said scipy is still a great choice to have both system programming and numerical business logic in one language.
- J253 5y ago
  In my opinion and experience, I think you’re right about “there’s Pandas for that” and “that” can be almost anything. It can do almost anything but making it do almost anything requires constant reference to the docs. And I find maintainability difficult. It seems like there’s 50 kwargs for every method. Sometimes things happen in place by default, other times they don’t. Compound indexes still confuse me. But I’m not a data scientist so I don’t do much ad-hoc analysis that seems typical with pandas users.

matsemann 5y ago

I wonder how many days have been wasted on non-programmers trying to get their Conda environment up and running or similar. Half the data science stuff isn't reproducible, not because of the science, but because getting the notebooks running with its dependencies is almost impossible.

lysecret 5y ago
have you heard of docker?
irrational 5y ago
Which begs the question, why has nobody built anything better?
- taeric 5y ago
  I'd argue that they probably have. It just isn't free.
- belval 5y ago
  Because it's a hard problem and people love hating on Python because it doesn't come with a way to handle all the compiled dependencies that work for every OS. Data science has a ton of moving library parts, it is genuinely difficult to distribute precompiled libraries for everyone when you have 2-3 actively maintained CUDA version with 2 cuDNN version for accelerators that change every 2 years. Most team fail to standardize on an environment (say Python 3.8, Ubuntu 20.04, CUDA 11.1, and cuDNN 8) and then get hung up on a dependency not building as if it's Python's fault that it does not have control of your entire OS.
  - matsemann 5y ago
    But why is it such a big problem in Python compared to other stacks? Why does all python projects end up depending on you having those exact tools of things installed locally and the planets aligned a certain way, when other stacks do not?

taeric 5y ago

This feels misguided, too. It begs the question that python has good usability. Anyone that has tried managing dependencies in it will know that is mostly a lie. What python had, was that it was preinstalled on many computers and then had a large cohort of users that are insisting that others use it. And mostly force proclaiming that it is easy and readable. I'll not claim that it is hard, per se. More that it is not intrinsically easier than any other dynamic language. For evidence, the main packages that are popular are often clones of packages from other environments that were not widely installed. Jupyter can be seen as free version of many scientific applications. Matlab, Mathematica, etc. Matplotlib is rather direct in it's copy. Pretty sure there are more examples.

iainctduncan 5y ago
Is managing deps in Python a pain? sure is!. Is it a pain in the other contenders of easily available dynamic languages? Yup. So that's a wash. Managing deps in dynamic languages is not a simple problem, I can't say I've tried one that did it super well yet.
- taeric 5y ago
  I'm... Not sure insulting other languages is a path to victory. Npm, as much as it annoys me, is light-years ahead of anything in python. Quicklisp is rather pleasant, now. Ruby has had gems for a long time. I grant that it is a hard problem. I am not griping that it is not solved. More that the option community has largely failed to even pick a direction. The most used dependency methods are, in the modern spirit of python, deprecated already.
  - iainctduncan 5y ago
    I'm not insulting other languages - I'm pointing out they have the same situation! Where in that post is there an insult??? I have had many dep fights with Ruby and Node too, and I would disagree that their solutions are in any way better than pip and virtualenv.

dang 5y ago

"Don't be snarky." "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

jrm4 5y ago
Harsh isn't the same as shallow. I don't believe snark is ALWAYS uncalled for, only that you deploy it when it's reasonably necessary. And this is a thing that people, especially HERE, get wrong a LOT. That it's unpleasant or hits people where it hurts doesn't mean it's not important, and I think it is.
- dang 5y ago
  The issue is the effects that it has on the ecosystem. It dumbs down threads and makes people nastier. If someone is wrong and you know more than they do, you can share correct information without putting them down.