7 ms·

From Python to NumPy (2017)

145 points by rahimnathwani 4y ago

boberoni 4y ago
This book was published in 2017. The author, Nicolas Rougier, also has a few other books online books on data visualization[0] and the reproducibility problem in research[1]. [0] https://github.com/rougier/scientific-visualization-book https://github.com/rougier/scientific-visualization-book [1] https://rr-france.github.io/bookrr https://rr-france.github.io/bookrr
DigiDigiorno 4y ago
I'm totally a fan of educational work aimed at a subset of programmers. And I myself went from Python to NumPy. This would have been great if it was around when I was working with SciPy stuff. That said, I think a NumPy to Python, or really any big package or framework to Python (e.g. Django to Python) is the normal progression these days. People seem to learn the language through a package and end up with some misleading inferences. Again, I don't intend to diminish the article, I just see the opposite progression so much I figured I'd comment on it.
- dragonwriter 4y ago
  > That said, I think a NumPy to Python, or really any big package or framework to Python (e.g. Django to Python) is the normal progression these days. It always has been common from the time Python had popular domain-focussed libraries that could be used without deep knowledge of the rest of the language, but once you get there you also tend to move out to other specialized packages.
dang 4y ago
Discussed at the time: From Python to Numpy - https://news.ycombinator.com/item?id=13355034 https://news.ycombinator.com/item?id=13355034 - Jan 2017 (48 comments)
- rahimnathwani 4y ago
  Thanks. I clicked on 'past' immediately after posting, but all I saw were Algolia search results for the keywords in the title. If I'd seen it had been posted before, I probably would have deleted my post. ISTR 'past' used to show matches for the URL, and not the title. Perhaps I'm mistaken.
  - SpectralTheory 4y ago
    I wouldn't treat these as warnings about posting (you're allowed to repost after some time). These are more about sharing relevant older conversations for the interested.
dhgaiK 4y ago
- l33t2328 4y ago
  How is prothletising a free tool an MLM scheme?
  - z9znz 4y ago
    I'm guessing the parent poster really meant something like "hyped snake oil". There is absolutely nothing about Python which makes it especially suited to scientific programming or any other kind of programming. In fact, it is poorly designed for most software engineering goals, short of just simple one-file, few line scripts. But those could just as well have been written in many other languages. Python just happened to be installed by default on most Linux systems, and it was more powerful than *sh. Just maybe one could argue that it was more accessible than perl because it had fewer $#!@$ symbols. If Python had not tried to offer OOP features (entirely built with duct tape, hotglue, and post-it notes) maybe Python projects wouldn't be such a mess. Simple procedural code would be better than the @decorator littered mess which is required to make Python have OO features approximating (but not fully supporting) other languages.
    - DigiDigiorno 4y ago
      Referring to Python hype as "snake oil" is hilarious; I recommend anyone arguing the underlying thought use it.
      - z9znz 4y ago
        I use Python all day every day. It's horrible compared to many other languages I have used. There are many versions of hype promoted for Python. Coal powered steam locomotives were great in their day, before there were better alternatives. But we don't irrationally hold the mentality that there is nothing better and why would you need anything else anyway? Tell me any library which can only be built for Python? Tell me any feature of Python which does not exist in one or more other languages? Then I'll tell you at least some features which exist in several other languages but which do not in Python. Python is inferior, and frankly anyone who is infatuated with it is ignorant of the other, better options.
z9znz 4y ago
Numpy is a great thing, and also a terrible thing (because it is built on Python). Python is a terrible language compared to many others, not the least of which is Ruby (and I would include Java, Clojure, Elixir, and even C++ and probably C# and almost certainly F# if I knew them well). To make matters worse, there's a common mentality amongst Pythonistas of being aggressively complacent. A typical response to a question about a missing capability is, "Why would you ever need that?" I have a lengthy collection of "why Python sucks" notes. However, I would argue that anyone who disagrees that Python is bad (excepting the practical capabilities afforded by libraries like Numpy which could just as well have been build for other languages) has simply not spent enough time other other, better languages. Python is Blub, from the Paul Graham essay Beating the Averages. http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html
- boredemployee 4y ago
  Sorry, that discussion is so overrated and elitist. I'll never understand people complaining about one technology/language/tool that made it possible to dozens of people to join to the - once restrict - small club which was computer science and which is now making it possible to a lot of people to get a job with an easy to learn and less verbose language. One can argue about speed/memory/better languages out there to make data science (hello Julia) but that's just how the world out there works. Python has a great/dope community that I didn't find when I worked with C back in the day, for example.
  - boberoni 4y ago
    > Python has a great/dope community Python's community is one of the greatest and worst parts of the language. Python might not be the best language for anything, but it's the second best language for everything. I've dabbled in many programming languages, a lot of which are backed by a single large company or consultancy: - Java (Oracle, at least initially) - TypeScript, C# (Microsoft) - Golang, Dart (Google) - Elixir (Nubank) - Clojure (Cognitect) - Kotlin (JetBrains) - Objective-C, Swift (Apple) - Julia (Julia Computing) Then, there are languages like Python, Ruby, JavaScript, PHP, Rust, etc. that are more diversified and distributed in terms of core contributors, maintainers, and evangelists. Meanwhile, there's a long list of other lesser known languages that have failed to grow a community as large as Python's. As I get older and older, I have realized that our programming languages are not static entities, but they change and evolve over time. They are influenced by users, companies, and researchers. I hope that one day, there's a humanities subject that studies programming languages just like we study natural languages like in linguistics. The closest thing right now seems to be history of science, but it's quite a broad field.
wodenokoto 4y ago
I feel like I never use directly numpy anymore. All my data is in Pandas, and I mostly import numpy for the odd `df.transform(np.somefunction)` I remember it being quite fun to build these vectorized numpy functions (in the crosswords and puzzles are fun, kind of way) Figuring out how to make a transformation in Pandas is always frustrating, when it isn't straightforward.
- CapmCrackaWaka 4y ago
  That's funny, I went the opposite direction. I only use pandas if I need to group by and aggregate, or to use in plotting functions. When doing array operations, I pretty much always use numpy, figuring out how to broadcast and vectorize operations is one of the most fun parts of my job in my opinion.
  - hcrad 4y ago
    I have had the same experience and instead of Pandas have been using numpy-groupies to handle aggregate/groupby operations. It's quite performant and feels a bit cleaner to use than importing pandas for a couple operations. https://github.com/ml31415/numpy-groupies https://github.com/ml31415/numpy-groupies
kwertzzz 4y ago
I think the readability issue of vectorized numpy if often ignored (while readability is one of the main strength of python in my opinion). I could not resists to write the example of section 2.2 in Julia: using Random, BenchmarkTools seq = rand(0:2,10_000); sub = rand(0:2,4); # translation of the "readable" but slow python code function_1(seq, sub) = [i for i in 1:(length(seq)-length(sub)) if view(seq,i:i+length(sub)-1) == sub]; @btime function_1(seq, sub); # 93.858 μs (5 allocations: 1.98 KiB) Which is more than twice as fast as the vectorized (and quite unreadable) python %timeit function_2(seq, sub) 215 µs ± 1.94 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) Having a JIT (like julia, numba...) has really a bit potential to improve code readability and reduce the need of vectorized numpy operations. In some cases, it can even be faster due to be better memory utilization and by computing more directly what you need.
- mjburgess 4y ago
  Blame python for being off in the weeds adding syntax for software engineering (eg., walrus operator), and not for data analysis.
  - deleted 4y ago
    [deleted]
  - deshpand 4y ago
    I think it's helpful to keep in mind that Python is general purpose and used in many domains in addition to data analysis (no matter which side of the walrus operator you are on)
    - mjburgess 4y ago
      I think it was. It isnt now. It should develop its syntax in response to its actual users, not now, merely the subset it has been serving for 20yr
      - p1esk 4y ago
mbeex 4y ago
I use numpy sporadically and always forget things about it. I always liked this decidedly shorter presentation of the library to reacquaint myself with the essential basic concepts: https://betterprogramming.pub/numpy-illustrated-the-visual-guide-to-numpy-3b1d4976de1d https://betterprogramming.pub/numpy-illustrated-the-visual-g...
ramblingsnake 4y ago
The problem is deeper in that, because NumPy is bloated too (see https://numpy.org/doc/stable/reference/generated/numpy.mean.html https://numpy.org/doc/stable/reference/generated/numpy.mean.... versus https://numpy.org/doc/stable/reference/generated/numpy.nanmean.html https://numpy.org/doc/stable/reference/generated/numpy.nanme...). Perhaps even further, the shift now is for "Python" to be written in neither vanilla Python nor NumPy directly, but at times in PyTorch or TensorFlow where each library provides its own workarounds.

145 points by rahimnathwani 4y ago

This book was published in 2017. The author, Nicolas Rougier, also has a few other books online books on data visualization[0] and the reproducibility problem in research[1]. [0] https://github.com/rougier/scientific-visualization-book https://github.com/rougier/scientific-visualization-book [1] https://rr-france.github.io/bookrr https://rr-france.github.io/bookrr

DigiDigiorno 4y ago

I'm totally a fan of educational work aimed at a subset of programmers. And I myself went from Python to NumPy. This would have been great if it was around when I was working with SciPy stuff. That said, I think a NumPy to Python, or really any big package or framework to Python (e.g. Django to Python) is the normal progression these days. People seem to learn the language through a package and end up with some misleading inferences. Again, I don't intend to diminish the article, I just see the opposite progression so much I figured I'd comment on it.

dragonwriter 4y ago
> That said, I think a NumPy to Python, or really any big package or framework to Python (e.g. Django to Python) is the normal progression these days. It always has been common from the time Python had popular domain-focussed libraries that could be used without deep knowledge of the rest of the language, but once you get there you also tend to move out to other specialized packages.

dang 4y ago

Discussed at the time: From Python to Numpy - https://news.ycombinator.com/item?id=13355034 https://news.ycombinator.com/item?id=13355034 - Jan 2017 (48 comments)

rahimnathwani 4y ago
Thanks. I clicked on 'past' immediately after posting, but all I saw were Algolia search results for the keywords in the title. If I'd seen it had been posted before, I probably would have deleted my post. ISTR 'past' used to show matches for the URL, and not the title. Perhaps I'm mistaken.
- SpectralTheory 4y ago
  I wouldn't treat these as warnings about posting (you're allowed to repost after some time). These are more about sharing relevant older conversations for the interested.

dhgaiK 4y ago

l33t2328 4y ago
How is prothletising a free tool an MLM scheme?
- z9znz 4y ago
  I'm guessing the parent poster really meant something like "hyped snake oil". There is absolutely nothing about Python which makes it especially suited to scientific programming or any other kind of programming. In fact, it is poorly designed for most software engineering goals, short of just simple one-file, few line scripts. But those could just as well have been written in many other languages. Python just happened to be installed by default on most Linux systems, and it was more powerful than *sh. Just maybe one could argue that it was more accessible than perl because it had fewer $#!@$ symbols. If Python had not tried to offer OOP features (entirely built with duct tape, hotglue, and post-it notes) maybe Python projects wouldn't be such a mess. Simple procedural code would be better than the @decorator littered mess which is required to make Python have OO features approximating (but not fully supporting) other languages.
  - DigiDigiorno 4y ago
    Referring to Python hype as "snake oil" is hilarious; I recommend anyone arguing the underlying thought use it.
    - z9znz 4y ago
      I use Python all day every day. It's horrible compared to many other languages I have used. There are many versions of hype promoted for Python. Coal powered steam locomotives were great in their day, before there were better alternatives. But we don't irrationally hold the mentality that there is nothing better and why would you need anything else anyway? Tell me any library which can only be built for Python? Tell me any feature of Python which does not exist in one or more other languages? Then I'll tell you at least some features which exist in several other languages but which do not in Python. Python is inferior, and frankly anyone who is infatuated with it is ignorant of the other, better options.

z9znz 4y ago

Numpy is a great thing, and also a terrible thing (because it is built on Python). Python is a terrible language compared to many others, not the least of which is Ruby (and I would include Java, Clojure, Elixir, and even C++ and probably C# and almost certainly F# if I knew them well). To make matters worse, there's a common mentality amongst Pythonistas of being aggressively complacent. A typical response to a question about a missing capability is, "Why would you ever need that?" I have a lengthy collection of "why Python sucks" notes. However, I would argue that anyone who disagrees that Python is bad (excepting the practical capabilities afforded by libraries like Numpy which could just as well have been build for other languages) has simply not spent enough time other other, better languages. Python is Blub, from the Paul Graham essay Beating the Averages. http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html

boredemployee 4y ago
Sorry, that discussion is so overrated and elitist. I'll never understand people complaining about one technology/language/tool that made it possible to dozens of people to join to the - once restrict - small club which was computer science and which is now making it possible to a lot of people to get a job with an easy to learn and less verbose language. One can argue about speed/memory/better languages out there to make data science (hello Julia) but that's just how the world out there works. Python has a great/dope community that I didn't find when I worked with C back in the day, for example.
- boberoni 4y ago
  > Python has a great/dope community Python's community is one of the greatest and worst parts of the language. Python might not be the best language for anything, but it's the second best language for everything. I've dabbled in many programming languages, a lot of which are backed by a single large company or consultancy: - Java (Oracle, at least initially) - TypeScript, C# (Microsoft) - Golang, Dart (Google) - Elixir (Nubank) - Clojure (Cognitect) - Kotlin (JetBrains) - Objective-C, Swift (Apple) - Julia (Julia Computing) Then, there are languages like Python, Ruby, JavaScript, PHP, Rust, etc. that are more diversified and distributed in terms of core contributors, maintainers, and evangelists. Meanwhile, there's a long list of other lesser known languages that have failed to grow a community as large as Python's. As I get older and older, I have realized that our programming languages are not static entities, but they change and evolve over time. They are influenced by users, companies, and researchers. I hope that one day, there's a humanities subject that studies programming languages just like we study natural languages like in linguistics. The closest thing right now seems to be history of science, but it's quite a broad field.

wodenokoto 4y ago

I feel like I never use directly numpy anymore. All my data is in Pandas, and I mostly import numpy for the odd `df.transform(np.somefunction)` I remember it being quite fun to build these vectorized numpy functions (in the crosswords and puzzles are fun, kind of way) Figuring out how to make a transformation in Pandas is always frustrating, when it isn't straightforward.

CapmCrackaWaka 4y ago
That's funny, I went the opposite direction. I only use pandas if I need to group by and aggregate, or to use in plotting functions. When doing array operations, I pretty much always use numpy, figuring out how to broadcast and vectorize operations is one of the most fun parts of my job in my opinion.
- hcrad 4y ago
  I have had the same experience and instead of Pandas have been using numpy-groupies to handle aggregate/groupby operations. It's quite performant and feels a bit cleaner to use than importing pandas for a couple operations. https://github.com/ml31415/numpy-groupies https://github.com/ml31415/numpy-groupies

kwertzzz 4y ago

I think the readability issue of vectorized numpy if often ignored (while readability is one of the main strength of python in my opinion). I could not resists to write the example of section 2.2 in Julia: using Random, BenchmarkTools seq = rand(0:2,10_000); sub = rand(0:2,4); # translation of the "readable" but slow python code function_1(seq, sub) = [i for i in 1:(length(seq)-length(sub)) if view(seq,i:i+length(sub)-1) == sub]; @btime function_1(seq, sub); # 93.858 μs (5 allocations: 1.98 KiB) Which is more than twice as fast as the vectorized (and quite unreadable) python %timeit function_2(seq, sub) 215 µs ± 1.94 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) Having a JIT (like julia, numba...) has really a bit potential to improve code readability and reduce the need of vectorized numpy operations. In some cases, it can even be faster due to be better memory utilization and by computing more directly what you need.

mjburgess 4y ago
Blame python for being off in the weeds adding syntax for software engineering (eg., walrus operator), and not for data analysis.
- deleted 4y ago
  [deleted]
- deshpand 4y ago
  I think it's helpful to keep in mind that Python is general purpose and used in many domains in addition to data analysis (no matter which side of the walrus operator you are on)
  - mjburgess 4y ago
    I think it was. It isnt now. It should develop its syntax in response to its actual users, not now, merely the subset it has been serving for 20yr
    - p1esk 4y ago

mbeex 4y ago

I use numpy sporadically and always forget things about it. I always liked this decidedly shorter presentation of the library to reacquaint myself with the essential basic concepts: https://betterprogramming.pub/numpy-illustrated-the-visual-guide-to-numpy-3b1d4976de1d https://betterprogramming.pub/numpy-illustrated-the-visual-g...

ramblingsnake 4y ago

The problem is deeper in that, because NumPy is bloated too (see https://numpy.org/doc/stable/reference/generated/numpy.mean.html https://numpy.org/doc/stable/reference/generated/numpy.mean.... versus https://numpy.org/doc/stable/reference/generated/numpy.nanmean.html https://numpy.org/doc/stable/reference/generated/numpy.nanme...). Perhaps even further, the shift now is for "Python" to be written in neither vanilla Python nor NumPy directly, but at times in PyTorch or TensorFlow where each library provides its own workarounds.