5 ms·

This could be headed for another dead end imo. Context: I use python for data processing and webdev. When doing data processing, Python is merely glue for libr

This could be headed for another dead end imo. Context: I use python for data processing and webdev. When doing data processing, Python is merely glue for libraries in compiled languages. When doing webdev, I mostly use python itself. First, any numbers regarding benchmarks need to be treated with contempt. JITs are unbenchmarkable. No matter what you do, someone says you do it wrong. Warmed up the JIT? You did it wrong. Didn't warm up the JIT? You did it wrong. Warmed up and didn't warm up the JIT? Wrong. You lose predictable performance characteristics due to the above. It's difficult to describe the importance of this to the people who look at Python as glue for their compiled code. Next, on the face of it, it doesn't look like it will compose well with subinterpreters. If each subinterpreter does it's own tracing, it's going to be harder to hit the 10k watermark of jitting hot code. This uses LLVM's JIT which is particularly slow and heavy (16MB added to the binary size) last time I tried to use it. So this limits Python attractiveness in being an embedded language. While this remains experimental - and hence strictly optional, packaging this in distributions that use gcc would now apparently need llvm tools installed to build python. Expect feedback from distribution packagers. Idea for improvement: when I do data processing, I'm using python to glue bits of C, Rust, and other compiled languages so this is not very useful. When I'm doing webdev, it could be useful - and I am deploying using Docker. So why not make this AOT so I can add it to a docker build step and get all the benefit without the complications of tracing jits.

robertlagrant 3y ago
> So why not make this AOT so I can add it to a docker build step and get all the benefit without the complications of tracing jits. I've also thought this. If you're pre-baking your software, why not do this step as well?
oblio 3y ago
JS V8, Java (any JVM), .NET use JITs. If those aren't solid proving grounds for JITs, nothing is.
- guappa 3y ago
  And nobody writes short lived commands in those because of startup time.
  - oblio 3y ago
    Do a quick test with the latest JVM. Print Hello, world. That's it. Time it. Do the same with Python. The difference is minuscule. People don't build CLIs with them due to cultural reasons (fashion). Developers are creatures of fashion.
    - guappa 3y ago
      > Do a quick test with the latest JVM. java: 031s python: 013s it's 3x slower to start up… it means that any bash script with a loop calling a java command will run 3x slower. 3x slower isn't my definition of minuscule.
      - oblio 3y ago
        > java: 031s python: 013s I'm confused, what kind of time measurement system is this? Are those 31 seconds versus 13 seconds? 0.31s vs 0.13s? If it's the first case, something is wrong with that machine, Hello World should not take that long. If it's the second, are we talking about 0.18s? See below. > it's 3x slower to start up… it means that any bash script with a loop calling a java command will run 3x slower. I don't know how you write your scripts, but in practice most scripts hang waiting for IO (especially network) or waiting for a specific command to process something... > 3x slower isn't my definition of minuscule. Nope, but based on what you've presented so far, unless I'm misunderstanding, it's my definition of "premature optimization".
        guappa 3y ago
        double click selection issue. 0m0,031s vs 0m0,013s. > I don't know how you write your scripts, You know how… I put print hello world in them… that was the test you asked for? > premature optimization Not using something that has 3x startup time, for a short lived command is just "common sense". I'm starting to think that the issue here is that you know java but don't know python or C, and you are unconsciously trying to get water to your own watermill.
mort96 3y ago
IMO, Python's performance can be a huge problem even when using it for purposes it's supposedly good for, like data processing. If you're doing NLP, you can use NLTK's built-in tokenisers and get pretty good performance. But if your situation calls for a slightly different tokeniser? You're gonna either write that in Python and absolutely tank your performance, or have a miserable time writing your own tokeniser as a C extension. Or you can use PyPy of course, but that's a compatibility nightmare exactly when you're gluing together a bunch of C libraries with Python interfaces. And in principle, JIT doesn't necessarily mean non-embeddable; LuaJIT is pretty good from an embedding perspective. Besides, I would assume they make JIT a build-time option so that people whose use case makes it problematic can use only the bytecode interpreter instead. That said, your concerns about performance variability are warranted, and there are probably specific criticisms to be made about the choice of LLVM. LLVM is a gigantic dependency and I hope the performance wins of choosing LLVM rather than, say, crankshaft are worth it.
FartyMcFarter 3y ago
> JITs are unbenchmarkable. No matter what you do, someone says you do it wrong. Why does that matter? Presumably if you care about Python performance, you have a real world scenario that you can benchmark. Just make a benchmark that is as representative as possible and check if the JIT helps. If it doesn't help, it surely can be disabled via an option.
- fnord123 3y ago
  > Why does it matter? Because you need to have an idea of how many machines you need. And you need an idea of whether 'it will take as long as it takes' or if there's an issue in your code. Sometimes Java decides to JIT using aes instructions. Sometimes it doesn't. Even if you run the same benchmark suite twice in a row it just does its own thing and gives wildly different results. My learning was that one shouldn't depend on JIT performance and prefer using JNI libraries if you want consistency. Hence my comment being negative on JIT.
  - FartyMcFarter 3y ago
    I don't know a lot about the Java JIT. But my guess is that the scenario you observed is the result of profile-guided optimisations done at runtime. Hopefully this is a feature that can be disabled when you want deterministic behaviour. There's no reason to make it mandatory in a well-engineered VM.
lifthrasiir 3y ago
> This uses LLVM's JIT which is particularly slow and heavy (16MB added to the binary size) last time I tried to use it. Incorrect, LLVM is only used in the compile time. See my other comments for details.
- fnord123 3y ago
  I looked into PR and you're correct. That's at least better than expectations.
yxhuvud 3y ago
More glaring about the claimed performance gain is that they didn't mention what code they ran (as far as I could see). Was it a selected small piece of code or was it a big Django monolith? As I didn't see any mention of what code it was I'm going to assume the former, and that the latter will actually get worse performance. Because that is what typically happens when someone build a new JIT compiler. It takes a couple of iterations of optimizations for them to become useful on nontrivial code.

robertlagrant 3y ago

> So why not make this AOT so I can add it to a docker build step and get all the benefit without the complications of tracing jits. I've also thought this. If you're pre-baking your software, why not do this step as well?

oblio 3y ago

JS V8, Java (any JVM), .NET use JITs. If those aren't solid proving grounds for JITs, nothing is.

guappa 3y ago
And nobody writes short lived commands in those because of startup time.
- oblio 3y ago
  Do a quick test with the latest JVM. Print Hello, world. That's it. Time it. Do the same with Python. The difference is minuscule. People don't build CLIs with them due to cultural reasons (fashion). Developers are creatures of fashion.
  - guappa 3y ago
    > Do a quick test with the latest JVM. java: 031s python: 013s it's 3x slower to start up… it means that any bash script with a loop calling a java command will run 3x slower. 3x slower isn't my definition of minuscule.
    - oblio 3y ago
      > java: 031s python: 013s I'm confused, what kind of time measurement system is this? Are those 31 seconds versus 13 seconds? 0.31s vs 0.13s? If it's the first case, something is wrong with that machine, Hello World should not take that long. If it's the second, are we talking about 0.18s? See below. > it's 3x slower to start up… it means that any bash script with a loop calling a java command will run 3x slower. I don't know how you write your scripts, but in practice most scripts hang waiting for IO (especially network) or waiting for a specific command to process something... > 3x slower isn't my definition of minuscule. Nope, but based on what you've presented so far, unless I'm misunderstanding, it's my definition of "premature optimization".
      - guappa 3y ago
        double click selection issue. 0m0,031s vs 0m0,013s. > I don't know how you write your scripts, You know how… I put print hello world in them… that was the test you asked for? > premature optimization Not using something that has 3x startup time, for a short lived command is just "common sense". I'm starting to think that the issue here is that you know java but don't know python or C, and you are unconsciously trying to get water to your own watermill.

mort96 3y ago

IMO, Python's performance can be a huge problem even when using it for purposes it's supposedly good for, like data processing. If you're doing NLP, you can use NLTK's built-in tokenisers and get pretty good performance. But if your situation calls for a slightly different tokeniser? You're gonna either write that in Python and absolutely tank your performance, or have a miserable time writing your own tokeniser as a C extension. Or you can use PyPy of course, but that's a compatibility nightmare exactly when you're gluing together a bunch of C libraries with Python interfaces. And in principle, JIT doesn't necessarily mean non-embeddable; LuaJIT is pretty good from an embedding perspective. Besides, I would assume they make JIT a build-time option so that people whose use case makes it problematic can use only the bytecode interpreter instead. That said, your concerns about performance variability are warranted, and there are probably specific criticisms to be made about the choice of LLVM. LLVM is a gigantic dependency and I hope the performance wins of choosing LLVM rather than, say, crankshaft are worth it.

FartyMcFarter 3y ago

> JITs are unbenchmarkable. No matter what you do, someone says you do it wrong. Why does that matter? Presumably if you care about Python performance, you have a real world scenario that you can benchmark. Just make a benchmark that is as representative as possible and check if the JIT helps. If it doesn't help, it surely can be disabled via an option.

fnord123 3y ago
> Why does it matter? Because you need to have an idea of how many machines you need. And you need an idea of whether 'it will take as long as it takes' or if there's an issue in your code. Sometimes Java decides to JIT using aes instructions. Sometimes it doesn't. Even if you run the same benchmark suite twice in a row it just does its own thing and gives wildly different results. My learning was that one shouldn't depend on JIT performance and prefer using JNI libraries if you want consistency. Hence my comment being negative on JIT.
- FartyMcFarter 3y ago
  I don't know a lot about the Java JIT. But my guess is that the scenario you observed is the result of profile-guided optimisations done at runtime. Hopefully this is a feature that can be disabled when you want deterministic behaviour. There's no reason to make it mandatory in a well-engineered VM.

lifthrasiir 3y ago

> This uses LLVM's JIT which is particularly slow and heavy (16MB added to the binary size) last time I tried to use it. Incorrect, LLVM is only used in the compile time. See my other comments for details.

fnord123 3y ago
I looked into PR and you're correct. That's at least better than expectations.

yxhuvud 3y ago

More glaring about the claimed performance gain is that they didn't mention what code they ran (as far as I could see). Was it a selected small piece of code or was it a big Django monolith? As I didn't see any mention of what code it was I'm going to assume the former, and that the latter will actually get worse performance. Because that is what typically happens when someone build a new JIT compiler. It takes a couple of iterations of optimizations for them to become useful on nontrivial code.