4 ms·

Always interested in replies to this kind of comment, which basically boil down to "Python is so slow that we have to write any important code in C. And this i

Always interested in replies to this kind of comment, which basically boil down to "Python is so slow that we have to write any important code in C. And this is somehow a good thing." I mean, it's great that you can write some of your code in C. But wouldn't it be great if you could just write your libraries in Python and have them still be really fast?

dagw 3y ago
But wouldn't it be great if you could just write your libraries in Python Everybody obviously wants that. The question is are you willing to lose what you have in order to hopefully, eventually, get there. If Python 3 development stopped and Python 4 came out tomorrow and was 5x faster than python 3 and a promise of being 50-100x faster in the future, but you have to rewrite all the libraries that use the C API, it would probably be DOA and kill python. People who want a faster 'almost python' already have several options to choose from, none of which are popular. Or they use Julia.
- AndrewDucker 3y ago
  Why are you assuming that they'd have to rewrite all of their libraries? I don't see anything in the article that says that.
  - dagw 3y ago
    The reason this approach is so much slower than some of the other 'fast' pythons out there that have come before is that they are making sure you don't have to rewrite a bunch of existing libraries. That is the problem with all the fast python implementations that have come before. Yes, they're faster than 'normal' python in many benchmarks, but they don't support the entire current ecosystem. For example Instagram's python implementation is blazing fast for doing exactly what Instagram is using python for, but is probably completely useless for what I'm using python for.
    - AndrewDucker 3y ago
      Aaah, so it's not this approach that you're saying is an issue, it's the ones that significantly change Python. Gotcha, that makes sense. Thank you.
bdd8f1df777b 3y ago
Yes, but not so good when the JIT-ed Python can no longer reference those fast C code others have written. Every Python JIT project so far has suffered from incompatibility with some C-base Python extension, and users just go back to the slow interpreter in those cases.
- AndrewDucker 3y ago
  "not so good when the JIT-ed Python can no longer reference those fast C code others have written" I don't see an indication in the article that that's the case. Am I missing something?
  - kragen 3y ago
    this was a big obstacle for pypy specifically https://www.pypy.org/posts/2011/05/numpy-follow-up-6928627691060102514.html https://www.pypy.org/posts/2011/05/numpy-follow-up-692862769... https://doc.pypy.org/en/latest/faq.html#what-about-numpy-numpypy-micronumpy https://doc.pypy.org/en/latest/faq.html#what-about-numpy-num... i'm not sure what version they gave up at
JodieBenitez 3y ago
Between writing C code and writing Python code, there is also Cython. But sure, I'm all for removing build steps and avoiding yet another layer.
dannymi 3y ago
>I mean, it's great that you can write some of your code in C. But wouldn't it be great if you could just write your libraries in Python and have them still be really fast? That really depends. To make the issue clear, let's think about a similar situation: bash is nice because you can plug together inputs and outputs of different sub-executables (like grep, sed and so on) and have a big "functional" pipeline deliver the final result. Your idea would be "wouldn't it be great if you could just write your libraries in bash and have them still be really fast?". Not if you make bash into C, tanking productivity. And definitely not if that new bash can't run the old grep anymore (which is what usually is implied by the proposal in the case of Python). Also, I'm fine with not writing my search engines, databases and matrix multiplication algorithm implementations in bash, really. So are most other people, I suspect. Also, many proposals would weaken Python-the-language so it's not as expressive anymore. But I want it to stay as dynamic as it is. It's nice as a scripting language about 30 levels above bash. As always, there are tradeoffs. Also with this proposal there will be tradeoffs. Are the tradeoffs worth it or not? For the record, rewriting BLAS in Python (or anything else), even if the result was faster (!), would be a phenomenally bad idea. It would just introduce bugs, waste everyone's time, essentially be a fork of BLAS. There's no upside I can see that justifies it.
el_oni 3y ago
it depends what speed is most important to you. When i was a scientist, speed was getting the code written during my break, and if it took all afternoon to run that's fine because i was in the lab anyway. Even as i moved more into the software engineer direction, and started profiling code more, most of the bottlenecks come from things like "creating objects on every incovation rather than pooling them", "blocking IO", "using a bad algorithm" or "using the wrong datasctructure for the task". problems that exist in every language, though "bad algorithm" or "using the wrong datasctructure" might matter less in a faster language you're still leaving performance on the table. > "Python is so slow that we have to write any important code in C. And this is somehow a good thing." The good thing is that python has a very vibrant ecosystem filled with great libraries, so we don't have to write it in C, because somebody else has. We can just benefit from that when the situation calls for it
pdpi 3y ago
languages don’t need to all be good at the same thing. Python currently excels as a glue language you use to write drivers for modules written in lower-level languages, which is a niche that (afaik) nobody else seems to fill right now. While I’m all for making Python itself faster, it would be a shame to lose the glue language par excellence.
hot_gril 3y ago
Pure JS libs are more portable. In Python, portability doesn't matter as much.
yowlingcat 3y ago
> basically boil down to "Python is so slow that we have to write any important code in C. And this is somehow a good thing." I think that's a pretty ignorant interpretation. Python has been built to have a giant ecosystem of useful, feature-complete, stable, well built code that has been used for decades and for which there is no need to reinvent the wheel. If that already describes the universe of libraries that you /need/ to be extremely fast and the rest of your code is IO limited and not CPU limited, why reinvent the wheel? That makes your comment even more inaccurate because you likely don't need to write any "important" (which you are stretching to mean "fast") code in C -- you utilize existing off the shelf fast libraries that are written in Fortran, CUDA, C, Rust or any other language a pre-existing ecosystem was built in. Try and think of a language that has mature capabilities for domains as far away as what Django solves for, what pandas solves for, what pytorch solves for, and still has fantastic tooling like jupyter and streamlit. I can't think of any other language that has the combined off the shelf breadth and depth of Python. I don't want to have to write fast code in any language unless forced to, because the vast majority of the time I can customize a great off the shelf package and only write the remaining 1% of glue. I can't see why a professional engineer would 99% of the time would need to take a remotely different approach.

dagw 3y ago

But wouldn't it be great if you could just write your libraries in Python Everybody obviously wants that. The question is are you willing to lose what you have in order to hopefully, eventually, get there. If Python 3 development stopped and Python 4 came out tomorrow and was 5x faster than python 3 and a promise of being 50-100x faster in the future, but you have to rewrite all the libraries that use the C API, it would probably be DOA and kill python. People who want a faster 'almost python' already have several options to choose from, none of which are popular. Or they use Julia.

AndrewDucker 3y ago
Why are you assuming that they'd have to rewrite all of their libraries? I don't see anything in the article that says that.
- dagw 3y ago
  The reason this approach is so much slower than some of the other 'fast' pythons out there that have come before is that they are making sure you don't have to rewrite a bunch of existing libraries. That is the problem with all the fast python implementations that have come before. Yes, they're faster than 'normal' python in many benchmarks, but they don't support the entire current ecosystem. For example Instagram's python implementation is blazing fast for doing exactly what Instagram is using python for, but is probably completely useless for what I'm using python for.
  - AndrewDucker 3y ago
    Aaah, so it's not this approach that you're saying is an issue, it's the ones that significantly change Python. Gotcha, that makes sense. Thank you.

bdd8f1df777b 3y ago

Yes, but not so good when the JIT-ed Python can no longer reference those fast C code others have written. Every Python JIT project so far has suffered from incompatibility with some C-base Python extension, and users just go back to the slow interpreter in those cases.

AndrewDucker 3y ago
"not so good when the JIT-ed Python can no longer reference those fast C code others have written" I don't see an indication in the article that that's the case. Am I missing something?
- kragen 3y ago
  this was a big obstacle for pypy specifically https://www.pypy.org/posts/2011/05/numpy-follow-up-6928627691060102514.html https://www.pypy.org/posts/2011/05/numpy-follow-up-692862769... https://doc.pypy.org/en/latest/faq.html#what-about-numpy-numpypy-micronumpy https://doc.pypy.org/en/latest/faq.html#what-about-numpy-num... i'm not sure what version they gave up at

JodieBenitez 3y ago

Between writing C code and writing Python code, there is also Cython. But sure, I'm all for removing build steps and avoiding yet another layer.

dannymi 3y ago

>I mean, it's great that you can write some of your code in C. But wouldn't it be great if you could just write your libraries in Python and have them still be really fast? That really depends. To make the issue clear, let's think about a similar situation: bash is nice because you can plug together inputs and outputs of different sub-executables (like grep, sed and so on) and have a big "functional" pipeline deliver the final result. Your idea would be "wouldn't it be great if you could just write your libraries in bash and have them still be really fast?". Not if you make bash into C, tanking productivity. And definitely not if that new bash can't run the old grep anymore (which is what usually is implied by the proposal in the case of Python). Also, I'm fine with not writing my search engines, databases and matrix multiplication algorithm implementations in bash, really. So are most other people, I suspect. Also, many proposals would weaken Python-the-language so it's not as expressive anymore. But I want it to stay as dynamic as it is. It's nice as a scripting language about 30 levels above bash. As always, there are tradeoffs. Also with this proposal there will be tradeoffs. Are the tradeoffs worth it or not? For the record, rewriting BLAS in Python (or anything else), even if the result was faster (!), would be a phenomenally bad idea. It would just introduce bugs, waste everyone's time, essentially be a fork of BLAS. There's no upside I can see that justifies it.

el_oni 3y ago

it depends what speed is most important to you. When i was a scientist, speed was getting the code written during my break, and if it took all afternoon to run that's fine because i was in the lab anyway. Even as i moved more into the software engineer direction, and started profiling code more, most of the bottlenecks come from things like "creating objects on every incovation rather than pooling them", "blocking IO", "using a bad algorithm" or "using the wrong datasctructure for the task". problems that exist in every language, though "bad algorithm" or "using the wrong datasctructure" might matter less in a faster language you're still leaving performance on the table. > "Python is so slow that we have to write any important code in C. And this is somehow a good thing." The good thing is that python has a very vibrant ecosystem filled with great libraries, so we don't have to write it in C, because somebody else has. We can just benefit from that when the situation calls for it

pdpi 3y ago

languages don’t need to all be good at the same thing. Python currently excels as a glue language you use to write drivers for modules written in lower-level languages, which is a niche that (afaik) nobody else seems to fill right now. While I’m all for making Python itself faster, it would be a shame to lose the glue language par excellence.

hot_gril 3y ago

Pure JS libs are more portable. In Python, portability doesn't matter as much.

yowlingcat 3y ago

> basically boil down to "Python is so slow that we have to write any important code in C. And this is somehow a good thing." I think that's a pretty ignorant interpretation. Python has been built to have a giant ecosystem of useful, feature-complete, stable, well built code that has been used for decades and for which there is no need to reinvent the wheel. If that already describes the universe of libraries that you /need/ to be extremely fast and the rest of your code is IO limited and not CPU limited, why reinvent the wheel? That makes your comment even more inaccurate because you likely don't need to write any "important" (which you are stretching to mean "fast") code in C -- you utilize existing off the shelf fast libraries that are written in Fortran, CUDA, C, Rust or any other language a pre-existing ecosystem was built in. Try and think of a language that has mature capabilities for domains as far away as what Django solves for, what pandas solves for, what pytorch solves for, and still has fantastic tooling like jupyter and streamlit. I can't think of any other language that has the combined off the shelf breadth and depth of Python. I don't want to have to write fast code in any language unless forced to, because the vast majority of the time I can customize a great off the shelf package and only write the remaining 1% of glue. I can't see why a professional engineer would 99% of the time would need to take a remotely different approach.