6 ms·

Python without C extension compatibility would be Python without Numpy, Scipy, scikit-learn, PIL/Pillow, JSON, StringIO Would we really miss them if Python was

by moe 12y ago

Python without C extension compatibility would be Python without Numpy, Scipy, scikit-learn, PIL/Pillow, JSON, StringIO Would we really miss them if Python was then fast enough to do these things natively?

aardvark179 12y ago
The code backing things like Numpy and Scipy is not small, has been well optimised over many years, and would be extremely difficult to reimplement at speed even with a faster Python. The freedom high level languages give tends to prevent many low level optimisations that are really useful for big maths and data routines. For the language I work on we were able to write macros and functions to allow our C libraries to continue to work when we changed our VM, but we've spent considerable effort optimising things so the C has to reach back into objects as little as possible, and we could only do this because recompilation and refactoring was okay.
- ris 12y ago
  "The code backing things like Numpy and Scipy is not small, has been well optimised over many years, and would be extremely difficult to reimplement at speed even with a faster Python." I assume you're unaware of the numpypy project then, which is doing exactly that, with reasonable results. I also think you overestimate the performance of numpy/scipy when compared to, for instance, C++ template-based libraries which are able to make large performance gains from compiling specializations for the specific array shapes being used. JITs can potentially go further, being able to compile specializations in places where array shapes are not known at compile-time.
  - gaius 12y ago
    They are rewriting BLAS in Python? Seriously? Links please. You may not be aware that NumPy does it's heavy lifting by offloading the work to C and FORTRAN, and that's where the speed comes from. You set up your data in Python, use the nice high level language to parse files, etc, then you pass chunks of memory containing matrices off to the FORTRAN to crunch, then use Python again to say write it back to a file.
    - masklinn 12y ago
      > They are rewriting BLAS in Python? Nah. Note that these are optional even in the original numpy.
    - ris 12y ago
      It is yet to be seen whether they're going to defer to BLAS or try rewriting the routines in python for the linear algebra.
    - gnufx 12y ago
      Yes to the basic point, but note that (specifically) if you use BLAS in Fortran, you're typically losing a factor of several in speed, I'm sorry to say. Competitive serial numerical kernels typically still end up needing assembler (e.g. OpenBLAS, FFTW, ELPA). I'm not sure why Fortran wouldn't be considered a high level language, if that's the implication (and it hasn't been SHOUTED since the 70s).
nostrademons 12y ago
Somebody has to actually write the new functionality, and then many more people have to learn the new functionality. Anyway, this thought experiment has basically been done. "Ditch the backwards compatibility, start with a new language, make it fast by design, and rewrite all the packages that make it super popular." It's called Julia. And it's popular among some people, but for a lot of others, the maturity of the Python ecosystem still makes it a lot more attractive despite the performance & multi-language issues.
- rdc12 12y ago
  But Julia is a really young language, drawing that comparison is a bit premature, be more interesting to look at that in 2-4 years time
  - ForHackernews 12y ago
    Julia is incredible, but I worry about its future because, unlike some other promising-but-not-quite-mature languages like Go or Rust, Julia doesn't have any kind of corporate or organizational patron promoting it and investing big money in it.
    - rdc12 12y ago
      And yet it has created a remarkable amount of buzz around it already
    - Derbasti 12y ago
      Isn't it being developed by the MIT? I'm not sure about "big" money, but it sure is being backed.
      - ForHackernews 12y ago
        AFAIK, it doesn't have any kind of official institutional backing from MIT. Some of the creators are MIT faculty, but I don't think that makes it "An MIT Project" in the way that Go is a Google Project.
Derbasti 12y ago
Re-implementing matrix math, image processing, signal processing, statistics, scientific mathematics and a much much more is a daunting task. Not impossible, but a lot of work.

by moe 12y ago

aardvark179 12y ago

The code backing things like Numpy and Scipy is not small, has been well optimised over many years, and would be extremely difficult to reimplement at speed even with a faster Python. The freedom high level languages give tends to prevent many low level optimisations that are really useful for big maths and data routines. For the language I work on we were able to write macros and functions to allow our C libraries to continue to work when we changed our VM, but we've spent considerable effort optimising things so the C has to reach back into objects as little as possible, and we could only do this because recompilation and refactoring was okay.

ris 12y ago
"The code backing things like Numpy and Scipy is not small, has been well optimised over many years, and would be extremely difficult to reimplement at speed even with a faster Python." I assume you're unaware of the numpypy project then, which is doing exactly that, with reasonable results. I also think you overestimate the performance of numpy/scipy when compared to, for instance, C++ template-based libraries which are able to make large performance gains from compiling specializations for the specific array shapes being used. JITs can potentially go further, being able to compile specializations in places where array shapes are not known at compile-time.
- gaius 12y ago
  They are rewriting BLAS in Python? Seriously? Links please. You may not be aware that NumPy does it's heavy lifting by offloading the work to C and FORTRAN, and that's where the speed comes from. You set up your data in Python, use the nice high level language to parse files, etc, then you pass chunks of memory containing matrices off to the FORTRAN to crunch, then use Python again to say write it back to a file.
  - masklinn 12y ago
    > They are rewriting BLAS in Python? Nah. Note that these are optional even in the original numpy.
  - ris 12y ago
    It is yet to be seen whether they're going to defer to BLAS or try rewriting the routines in python for the linear algebra.
  - gnufx 12y ago
    Yes to the basic point, but note that (specifically) if you use BLAS in Fortran, you're typically losing a factor of several in speed, I'm sorry to say. Competitive serial numerical kernels typically still end up needing assembler (e.g. OpenBLAS, FFTW, ELPA). I'm not sure why Fortran wouldn't be considered a high level language, if that's the implication (and it hasn't been SHOUTED since the 70s).

nostrademons 12y ago

Somebody has to actually write the new functionality, and then many more people have to learn the new functionality. Anyway, this thought experiment has basically been done. "Ditch the backwards compatibility, start with a new language, make it fast by design, and rewrite all the packages that make it super popular." It's called Julia. And it's popular among some people, but for a lot of others, the maturity of the Python ecosystem still makes it a lot more attractive despite the performance & multi-language issues.

rdc12 12y ago
But Julia is a really young language, drawing that comparison is a bit premature, be more interesting to look at that in 2-4 years time
- ForHackernews 12y ago
  Julia is incredible, but I worry about its future because, unlike some other promising-but-not-quite-mature languages like Go or Rust, Julia doesn't have any kind of corporate or organizational patron promoting it and investing big money in it.
  - rdc12 12y ago
    And yet it has created a remarkable amount of buzz around it already
  - Derbasti 12y ago
    Isn't it being developed by the MIT? I'm not sure about "big" money, but it sure is being backed.
    - ForHackernews 12y ago
      AFAIK, it doesn't have any kind of official institutional backing from MIT. Some of the creators are MIT faculty, but I don't think that makes it "An MIT Project" in the way that Go is a Google Project.

Derbasti 12y ago

Re-implementing matrix math, image processing, signal processing, statistics, scientific mathematics and a much much more is a daunting task. Not impossible, but a lot of work.