4 ms·

For anything involving data analysis and scientific computing Python really shines - though I'm often surprised the extent to which web-oriented programmers are

For anything involving data analysis and scientific computing Python really shines - though I'm often surprised the extent to which web-oriented programmers are unaware of this, I read things like "you can write better web services in Go, therefore Python is obsolete". Python in science is kinda like the One Ring the Binds them - it is the glue that holds together the HPC code at CERN, ESA, NASA, and more, it crunches climate models and delivers your daily weather forcasts. Some really great projects like Cython, Theano, and Numba boost the performance of numeric code to native speeds with a fraction of the hassle of hand-written C. Mountains of machine learning research is done in Python, as well as statistics and analytics. And of course Numpy remains the de-facto standard of N-dimensional array libraries - if you look at numeric computing libraries in many other languages they will be framed as "Numpy-for-X".

raverbashing 11y ago
Yeah, I don't know While Numpy/Scipy is great with a capital G, it seems the underlying infrastructure is flaky. I'm talking about the messy process of building and installing it. And it seems everything is held with duct-tape Also it seems that "inside Scipy' it is a different world, and that to really "talk to the outside world" there seems to be a data-translation step needed.
- kmike84 11y ago
  at least on OS X installing numpy and scipy is very easy - pip installs prebuilt wheels by default, no need to compile anything
- ZenoArrow 11y ago
  > "I'm talking about the messy process of building and installing it." If you're using Python for scientific computing, you can use the Anaconda distribution of Python, it makes it painless to work with those packages. https://www.continuum.io https://www.continuum.io > "Also it seems that "inside Scipy' it is a different world, and that to really "talk to the outside world" there seems to be a data-translation step needed." Python has excellent ORMs, I'm not sure what you're getting at.
- dr_zoidberg 11y ago
  Working in Windows, my biggest issue was "the 64 bit situation", but that has been greatly solved by Microsoft's MSVS for Python (lovely move by Redmond) and the Python Wheels. Now it has truly become `pip install something` or `pip install [wheel file]` for more complex stuff. That and Cristoph Gohlke[0]'s repository of pre-compiled modules. That guy saved me more than once. Should I ever meet him, I'll sure buy him a beer! [0] http://www.lfd.uci.edu/~gohlke/pythonlibs/ http://www.lfd.uci.edu/~gohlke/pythonlibs/
- gh02t 11y ago
  The complexity of building NumPy or SciPy is pretty much par for the course in HPC. Most of the libraries it is based on are battle-hardened and ridiculously well tested to a degree not seen in many places, but they date from a different era, including libraries where originally "building" it involved a hole puncher and a bucket of coffee. Getting all those libraries to play nicely together at all is a genuine miracle. Most any significant scientific package I've ever had to build (which is quite a few) had a complex build system that felt like it was built on top of shaky glass pillars. It's just the nature of what's involved and it's a fundamentally hard problem; it's difficult to replace those libraries because the extreme amount of verification/validation and optimization that have gone into something like LAPACK makes it economically undesirable. I'd say also that this is part of the appeal of NumPy and SciPy, which is they glue together a huge number of tools that are otherwise disjoint. You don't have to worry (most of the time) about matrix ordering, calling conventions, those mysterious "workspace" variables in LAPACK etc.
- mangecoeur 11y ago
  It's a really complicated piece of kit, that has also been around since long before modern build systems and conventions existed. That's why you have excellent scientific packages like Anaconda or Canopy, while on linux you can also install your distro's package. Situation is also becoming easier with pre-built python wheels.
- mturmon 11y ago
  Years ago, it used to be harder to install numpy, but this has changed. On MacOS, it's just % pip install numpy For more (e.g., other OS'es, build-from-source), see: http://docs.scipy.org/doc/numpy-1.10.1/user/install.html http://docs.scipy.org/doc/numpy-1.10.1/user/install.html
kale 11y ago
Two days ago I took a proof-of-concept project written in Python, using the GMPY2 library (Gnu Multiple Precision library), and ported it to C. My C code was very complicated in order to be fast. Lots of pointer swapping, bit-shifting, etc. The code was about 4% faster in C, which was more lines of code, much harder to read (and write), compiled with every imaginable flag that could offer a speedup. Switching operating systems, this code was about 8% faster in C on Linux than Python in Windows 10. C in Linux would take about 9.5 days to complete, Python in Windows 10 would be about 10.5 days. In addition, Python has the "pickle" module so that I can easily make the algorithm save state to pause and restore it any time I want. What's more, I can probably use mmap to share memory easily without making copies and switch to multiprocessing for an even greater speedup (my first attempt using Pool was a little slower despite using 3 cores, but that's the slowest way to multithread).
- marmaduke 11y ago
  All this because you didn't profile your code to see that 95% of the time was spent in GMP routines.
  - kale 11y ago
    Yeah. I wasn't sure how GMPY was compiled, so I used C with my own compilation of GMP. I bet the flag differences made up the few percentage points. Plus, with each variable being ~10Mb, I was concerned about memory copying (I studied mechanical engineering, I missed a lot of this stuff in school), and with C I could make memory copying vs passing pointers explicit. All of that was unnecessary. It was another reinforcement of the idea: "Build it first, then optimize".

raverbashing 11y ago

Yeah, I don't know While Numpy/Scipy is great with a capital G, it seems the underlying infrastructure is flaky. I'm talking about the messy process of building and installing it. And it seems everything is held with duct-tape Also it seems that "inside Scipy' it is a different world, and that to really "talk to the outside world" there seems to be a data-translation step needed.

kmike84 11y ago
at least on OS X installing numpy and scipy is very easy - pip installs prebuilt wheels by default, no need to compile anything
ZenoArrow 11y ago
> "I'm talking about the messy process of building and installing it." If you're using Python for scientific computing, you can use the Anaconda distribution of Python, it makes it painless to work with those packages. https://www.continuum.io https://www.continuum.io > "Also it seems that "inside Scipy' it is a different world, and that to really "talk to the outside world" there seems to be a data-translation step needed." Python has excellent ORMs, I'm not sure what you're getting at.
dr_zoidberg 11y ago
Working in Windows, my biggest issue was "the 64 bit situation", but that has been greatly solved by Microsoft's MSVS for Python (lovely move by Redmond) and the Python Wheels. Now it has truly become `pip install something` or `pip install [wheel file]` for more complex stuff. That and Cristoph Gohlke[0]'s repository of pre-compiled modules. That guy saved me more than once. Should I ever meet him, I'll sure buy him a beer! [0] http://www.lfd.uci.edu/~gohlke/pythonlibs/ http://www.lfd.uci.edu/~gohlke/pythonlibs/
gh02t 11y ago
The complexity of building NumPy or SciPy is pretty much par for the course in HPC. Most of the libraries it is based on are battle-hardened and ridiculously well tested to a degree not seen in many places, but they date from a different era, including libraries where originally "building" it involved a hole puncher and a bucket of coffee. Getting all those libraries to play nicely together at all is a genuine miracle. Most any significant scientific package I've ever had to build (which is quite a few) had a complex build system that felt like it was built on top of shaky glass pillars. It's just the nature of what's involved and it's a fundamentally hard problem; it's difficult to replace those libraries because the extreme amount of verification/validation and optimization that have gone into something like LAPACK makes it economically undesirable. I'd say also that this is part of the appeal of NumPy and SciPy, which is they glue together a huge number of tools that are otherwise disjoint. You don't have to worry (most of the time) about matrix ordering, calling conventions, those mysterious "workspace" variables in LAPACK etc.
mangecoeur 11y ago
It's a really complicated piece of kit, that has also been around since long before modern build systems and conventions existed. That's why you have excellent scientific packages like Anaconda or Canopy, while on linux you can also install your distro's package. Situation is also becoming easier with pre-built python wheels.
mturmon 11y ago
Years ago, it used to be harder to install numpy, but this has changed. On MacOS, it's just % pip install numpy For more (e.g., other OS'es, build-from-source), see: http://docs.scipy.org/doc/numpy-1.10.1/user/install.html http://docs.scipy.org/doc/numpy-1.10.1/user/install.html

kale 11y ago

Two days ago I took a proof-of-concept project written in Python, using the GMPY2 library (Gnu Multiple Precision library), and ported it to C. My C code was very complicated in order to be fast. Lots of pointer swapping, bit-shifting, etc. The code was about 4% faster in C, which was more lines of code, much harder to read (and write), compiled with every imaginable flag that could offer a speedup. Switching operating systems, this code was about 8% faster in C on Linux than Python in Windows 10. C in Linux would take about 9.5 days to complete, Python in Windows 10 would be about 10.5 days. In addition, Python has the "pickle" module so that I can easily make the algorithm save state to pause and restore it any time I want. What's more, I can probably use mmap to share memory easily without making copies and switch to multiprocessing for an even greater speedup (my first attempt using Pool was a little slower despite using 3 cores, but that's the slowest way to multithread).

marmaduke 11y ago
All this because you didn't profile your code to see that 95% of the time was spent in GMP routines.
- kale 11y ago
  Yeah. I wasn't sure how GMPY was compiled, so I used C with my own compilation of GMP. I bet the flag differences made up the few percentage points. Plus, with each variable being ~10Mb, I was concerned about memory copying (I studied mechanical engineering, I missed a lot of this stuff in school), and with C I could make memory copying vs passing pointers explicit. All of that was unnecessary. It was another reinforcement of the idea: "Build it first, then optimize".