5 ms·

If you're doing complex data analysis in python, the actual number crunching isn't happening in python it's happening in a C extension. Using numpy or tensor fl

If you're doing complex data analysis in python, the actual number crunching isn't happening in python it's happening in a C extension. Using numpy or tensor flow, you're really just calling C code.

grumpyprole 5y ago
And Fortran code (many parts of numpy). C doesn't have a complete monopoly on high-performance code.
- Qi_ 5y ago
  What is the advantage of Fortran over only C in numpy’s case specifically?
  - staticassertion 5y ago
    Fortran has the more optimized libraries (blas) and is also capable of optimizations that C isn't (guarantees that pointers dont alias).
    - qsort 5y ago
      Modern C has the restrict keyword for that. There isn't any competitive advantage left for Fortran over C or C++, the only reason why Fortran is still part of the modern numeric stack is that BLAS, LAPACK, QUADPACK and friends run the freaking world and nobody is ever going to rewrite them to C without a compelling reason to do so.
      - staticassertion 5y ago
        I'm aware of the restrict keyword, though its usage is not typical. It enables an optimization that fortran has by default, that was my point. And yes, I already said that packages like blas have had massive amounts of work put into them and have lasting power.
        qsort 5y ago
        I didn't downvote you, if that's what you were hinting at, however your phrasing was more likely to be interpreted as "C can't do that" as opposed to "C doesn't default to that". It seems we agree, after all.
        staticassertion 5y ago
        I believe we do.
      - mumblemumble 5y ago
        Fortran's primary competitive advantage over C and C++ is that it's a conceptually simpler language that's easier for humans to use safely and effectively for numerical computing tasks. It's not just being tied to BLAS and friends. Fortran lives on in academia, for example, because academics don't necessarily want to put a lot of effort into chasing wild pointers or grokking the standard template library. For my part, I'm watching LFortran with interest because, when I run up against limitations on what I can do with numpy, I'd much rather turn to Fortran than C, C++, or Rust if at all possible, because Fortran would let me get the job done in less time, and the code would likely be quite a bit more readable.
      - adgjlsfhk1 5y ago
        Note that while BLAS and friends aren't getting rewritten in C, there is an effort underway to write replacements in Julia. The basic reason is that metaprogramming and better optimization frameworks are making it possible to write these at a higher level where you basically specify a cost model, and generate an optimal method based on that. The big advantage is that this works for more than just standard matrix multiplies. The same framework can give you complex matrices, Integer and boolean matrices, matrices with different algebras (eg max-plus).
        qsort 5y ago
        That's a cool idea. I don't know how realistic it is to achieve performance parity, but the "generic" functionality is definitely intriguing.
        adgjlsfhk1 5y ago
        The initial results are that libraries like LoopVectorization can already generate optimal micro-kernels, and is competitive with MKL (for square matrix-matrix multiplication) up to around size 512. With help on macro-kernel side from Octavian, Julia is able to outperform MKL for sizes up to to 1000 or so (and is about 20% slower for bigger sizes). https://github.com/JuliaLinearAlgebra/Octavian.jl https://github.com/JuliaLinearAlgebra/Octavian.jl.
      - pjmlp 5y ago
        Except "modern" C does not prevent one to misuse restrict and doing so is UB.
    - brodouevencode 5y ago
      Also, when Travis Oliphant was writing numpy he liked the Fortran implementations of a lot of mathematical functions, so it was an "easy" borrow from there instead of reimplementing.
sparsely 5y ago
I think the OP's point was that it forces you to do all the number crunching via those extensions, and if what you want to do doesn't match up with their APIs it's a bit of a pain. Granted, they have fairly complete and flexible APIs but I can still imagine it being very annoying sometimes.
- xscott 5y ago
  > I think the OP's point was that it forces you to do all the number crunching via those extensions [...] Almost. If you're using C extensions for performance, those extensions can release the GIL, and then you get parallelism in CPython. Although if you've got a bunch of small steps, Amdahl's law is going to bite you as each step needs to re-acquire the the GIL to proceed. Note that for Numpy, I don't know which functions/methods actually bother to release the GIL, but it's possible in theory.
  - qsort 5y ago
    Yes, that's exactly what's frustrating about Python. It's not the performance per se, that doesn't really matter for Python's use cases, it's that the performance charateristics of your program are so erratic it doesn't feel like the pieces are fitting together. Denotationally equivalent operations having drastically diverging operational interpretations means that changing anything is a nightmare. If you have an API call that does what you want, Python is basically C++. If you don't, you're paying a 100x penalty. Most of the times performance doesn't matter, and when it doesn't, life is great. When it does matter, the situation feels worse than it actually is because optimizing Python is such a fight.
    - mdomans 5y ago
      In general optimising Python is: * understanding that pure Python is not a language to write CPU-heavy computational code in it and was never intended as such * understanding that Python lets you write 3-4 solutions to a problem easy and 1 will be slow, two will be OK and one will be optimal - often that involves using generators, collections, sets and is a bit nuaced * I'd say that optimising Python, compared to something like C or C++ is far easier - I can set up a decent testing and profiling environment for most of my problems easy with pip and ipython which is borderline simple compared to valgrind
mumblemumble 5y ago
spaCy might be a good example. It's a natural language processing library that's written in Cython, so yeah, a lot of the hard computation work is essentially just calling into C code. But it's usually being driven from a Python script that's in charge of loading the data and feeding it into spaCy's processing pipeline. That code is subject to the GIL, and, since it cannot be effectively multithreaded, it tends to become the program's bottleneck. So what you typically end up doing instead is using multiprocessing to parallelize. But that comes with its own costs. You end up wasting a lot of computrons on inter-process communication.

If you're doing complex data analysis in python, the actual number crunching isn't happening in python it's happening in a C extension. Using numpy or tensor flow, you're really just calling C code.

grumpyprole 5y ago

And Fortran code (many parts of numpy). C doesn't have a complete monopoly on high-performance code.

Qi_ 5y ago
What is the advantage of Fortran over only C in numpy’s case specifically?
- staticassertion 5y ago
  Fortran has the more optimized libraries (blas) and is also capable of optimizations that C isn't (guarantees that pointers dont alias).
  - qsort 5y ago
    Modern C has the restrict keyword for that. There isn't any competitive advantage left for Fortran over C or C++, the only reason why Fortran is still part of the modern numeric stack is that BLAS, LAPACK, QUADPACK and friends run the freaking world and nobody is ever going to rewrite them to C without a compelling reason to do so.
    - staticassertion 5y ago
      I'm aware of the restrict keyword, though its usage is not typical. It enables an optimization that fortran has by default, that was my point. And yes, I already said that packages like blas have had massive amounts of work put into them and have lasting power.
      - qsort 5y ago
        I didn't downvote you, if that's what you were hinting at, however your phrasing was more likely to be interpreted as "C can't do that" as opposed to "C doesn't default to that". It seems we agree, after all.
        staticassertion 5y ago
        I believe we do.
    - mumblemumble 5y ago
      Fortran's primary competitive advantage over C and C++ is that it's a conceptually simpler language that's easier for humans to use safely and effectively for numerical computing tasks. It's not just being tied to BLAS and friends. Fortran lives on in academia, for example, because academics don't necessarily want to put a lot of effort into chasing wild pointers or grokking the standard template library. For my part, I'm watching LFortran with interest because, when I run up against limitations on what I can do with numpy, I'd much rather turn to Fortran than C, C++, or Rust if at all possible, because Fortran would let me get the job done in less time, and the code would likely be quite a bit more readable.
    - adgjlsfhk1 5y ago
      Note that while BLAS and friends aren't getting rewritten in C, there is an effort underway to write replacements in Julia. The basic reason is that metaprogramming and better optimization frameworks are making it possible to write these at a higher level where you basically specify a cost model, and generate an optimal method based on that. The big advantage is that this works for more than just standard matrix multiplies. The same framework can give you complex matrices, Integer and boolean matrices, matrices with different algebras (eg max-plus).
      - qsort 5y ago
        That's a cool idea. I don't know how realistic it is to achieve performance parity, but the "generic" functionality is definitely intriguing.
        adgjlsfhk1 5y ago
        The initial results are that libraries like LoopVectorization can already generate optimal micro-kernels, and is competitive with MKL (for square matrix-matrix multiplication) up to around size 512. With help on macro-kernel side from Octavian, Julia is able to outperform MKL for sizes up to to 1000 or so (and is about 20% slower for bigger sizes). https://github.com/JuliaLinearAlgebra/Octavian.jl https://github.com/JuliaLinearAlgebra/Octavian.jl.
    - pjmlp 5y ago
      Except "modern" C does not prevent one to misuse restrict and doing so is UB.
  - brodouevencode 5y ago
    Also, when Travis Oliphant was writing numpy he liked the Fortran implementations of a lot of mathematical functions, so it was an "easy" borrow from there instead of reimplementing.

sparsely 5y ago

I think the OP's point was that it forces you to do all the number crunching via those extensions, and if what you want to do doesn't match up with their APIs it's a bit of a pain. Granted, they have fairly complete and flexible APIs but I can still imagine it being very annoying sometimes.

xscott 5y ago
> I think the OP's point was that it forces you to do all the number crunching via those extensions [...] Almost. If you're using C extensions for performance, those extensions can release the GIL, and then you get parallelism in CPython. Although if you've got a bunch of small steps, Amdahl's law is going to bite you as each step needs to re-acquire the the GIL to proceed. Note that for Numpy, I don't know which functions/methods actually bother to release the GIL, but it's possible in theory.
- qsort 5y ago
  Yes, that's exactly what's frustrating about Python. It's not the performance per se, that doesn't really matter for Python's use cases, it's that the performance charateristics of your program are so erratic it doesn't feel like the pieces are fitting together. Denotationally equivalent operations having drastically diverging operational interpretations means that changing anything is a nightmare. If you have an API call that does what you want, Python is basically C++. If you don't, you're paying a 100x penalty. Most of the times performance doesn't matter, and when it doesn't, life is great. When it does matter, the situation feels worse than it actually is because optimizing Python is such a fight.
  - mdomans 5y ago
    In general optimising Python is: * understanding that pure Python is not a language to write CPU-heavy computational code in it and was never intended as such * understanding that Python lets you write 3-4 solutions to a problem easy and 1 will be slow, two will be OK and one will be optimal - often that involves using generators, collections, sets and is a bit nuaced * I'd say that optimising Python, compared to something like C or C++ is far easier - I can set up a decent testing and profiling environment for most of my problems easy with pip and ipython which is borderline simple compared to valgrind

mumblemumble 5y ago

spaCy might be a good example. It's a natural language processing library that's written in Cython, so yeah, a lot of the hard computation work is essentially just calling into C code. But it's usually being driven from a Python script that's in charge of loading the data and feeding it into spaCy's processing pipeline. That code is subject to the GIL, and, since it cannot be effectively multithreaded, it tends to become the program's bottleneck. So what you typically end up doing instead is using multiprocessing to parallelize. But that comes with its own costs. You end up wasting a lot of computrons on inter-process communication.