4 ms·

That kind of tit-for-tat in benchmarks seems like it's counter to the goal of benchmarks: what kind of performance could I expect to see using technology $FOO?

That kind of tit-for-tat in benchmarks seems like it's counter to the goal of benchmarks: what kind of performance could I expect to see using technology $FOO? Crucially, that question depends on how someone will realistically implement $FOO. I like PyPy as an example: on the surface, implementing a Python runtime in Python and expecting performance gains seems crazy. PyPy manages to outperform CPython because although a C implementation should theoretically be faster, realistically the increased expressiveness of Python lets the PyPy devs opt into optimizations the CPython devs find out of reach. I don't know C or Rust well enough to comment on these specific scenarios, but if two technologies can be fast, and one makes that speed accessible while the other encourages implementors to leave performance on the table, that's much more useful information to me than seeing a back-and-forth where folks compete to implement optimizations I'll never make in my own projects.

vitus 6y ago
You raise a good point. I've always been fascinating by PyPy's performance, personally -- anecdotally, I've achieved ~10x speedups from just running a script with `pypy` instead of `python`. I always attributed that to better performance of the JIT, but I could be wrong. I have nothing against Rust personally, but it's ultimately not an apples-to-apples comparison if they're not implementing the same algorithm, or even if they're not using the same mechanisms (e.g. Rust explicitly using SSE intrinsics, which are certainly available to C in just as idiomatic a fashion).
- gameswithgo 6y ago
  comparing apples to apples is pointless, they are both apples. and you can always compare “equivalent” algorithms as some languages may not be able to efficiently express the same algorithm as another. i know what you are after, but trying to have some benchmark that is “fair” according to some spec that is important to your needs will just be seen as pointless to others. benchmark game at least lets us see what the performance ceiling is of various languages, and anytime you object to a language having a worse implementation you may go fix it. i think what anyone with a clue understands is that C C++ and Rust all have roughly equivalent performance ceilings. Nobody really thinks Rust is faster than C now.
  - sli 6y ago
    I admit I'm destroying the metaphor here but there are a lot of useful comparisons to be made between different cultivars of apple and those comparisons have implications on their applications.
    - zeroimpl 6y ago
      That's the whole point of the metaphor - comparing apples to apples is good, apples to oranges is bad.
      - gameswithgo 6y ago
        and its a bad metaphor. there are many reasonable comparisons to be made between any two things even when both are not fruits
        ghostwriter 6y ago
        Technically speaking, there are limits and you cannot compare incomparable. You can find distinctions between two phenomena, a.k.a. inherent traits that make them different-by-definition from one another, but you cannot compare (liken, collate, contrast) them in any meaningful way. Like, what's the outcome of comparing "soft" with "warm"? And hence, comparing apples to apples are not pointless, quite contrary - they are comparable items and it makes total sense to compare them in the context of superiority by desirable value axes.
        igouy 6y ago
        > Like, what's the outcome of comparing "soft" with "warm"? You can have "soft" or you can have "warm" — which do you want? You can have an apple or you can have an orange — which do you want?
      - jcelerier 6y ago
        I'm comparing apples to oranges every time I'm trying to decide which fruit to buy at the supermarket though.
FartyMcFarter 6y ago
> I like PyPy as an example: on the surface, implementing a Python runtime in Python and expecting performance gains seems crazy. Not if it includes a JIT compiler, as PyPy does. I don't know much about PyPy, but if most of the time is spent in JITted machine code, the fact that it's written in Python may not affect performance much.
- pjmlp 6y ago
  Kind of, PyPy is a metacircular JIT compiler, it uses RPython a Python subset.
moonchild 6y ago
> PyPy manages to outperform CPython because although a C implementation should theoretically be faster, realistically the increased expressiveness of Python lets the PyPy devs opt into optimizations the CPython devs find out of reach. Pypy outperforms cpython for the simple reason that pypy is a jit where cpython is a basic bytecode interpreter. Anything else is icing on the cake. The only reason python seems slow as an implementation language is because it was traditionally slow; the reason it was traditionally slow is that cpython, the only major implementation, is slow. Common lisp, for instance, is similarly dynamic (moreso, in fact), also is also generally natively compiled by itself, and is quite performant.
- spiffytech 6y ago
  PyPy has a JIT because using Python as the implementation language made it simple to build a runtime generator instead of a single runtime implementation, so RPython (the underlying PyPy tech) can generate a JIT and non-JIT runtime from the same source code. Conversely, the CPython team doesn't have the resources to make a JIT runtime the bespoke way that's more typical for these projects. https://morepypy.blogspot.com/2011/04/tutorial-part-2-adding-jit.html https://morepypy.blogspot.com/2011/04/tutorial-part-2-adding...
  - naniwaduni 6y ago
    CPython isn't a JIT because its goals as a portable reference implementation would be much harder to achieve with a JIT. To put it another way, there's a reason PyPy trails CPython language features on the scale of literal years. This is not explained by the affordances of the language they're written in! They're just projects with very different requirements which, on some platforms, happen to run a lot of similar-looking code.
    - ghostwriter 6y ago
      > there's a reason PyPy trails CPython language features on the scale of literal years. the reason is under-funding, and the lack of developers to the cause as a consequence, not the portability complexities of jit.
cygx 6y ago
realistically the increased expressiveness of Python lets the PyPy devs opt into optimizations the CPython devs find out of reach Not really: It's the compilation strategy (that whole meta-tracing JIT compiler thing, compared to a simple bytecode interpreter) that makes the difference, not the surface syntax of the implementation language - which is actually not 'real' Python, but a restricted, less dynamic subset known as RPython. Also note that the CPython is deliberately kept 'dumb'.
mkl 6y ago
> That kind of tit-for-tat in benchmarks seems like it's counter to the goal of benchmarks Yep. There are good reasons why this set of benchmarks is called The Computer Language Benchmarks Game.
- igouy 6y ago
  https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#name-game https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

vitus 6y ago

You raise a good point. I've always been fascinating by PyPy's performance, personally -- anecdotally, I've achieved ~10x speedups from just running a script with `pypy` instead of `python`. I always attributed that to better performance of the JIT, but I could be wrong. I have nothing against Rust personally, but it's ultimately not an apples-to-apples comparison if they're not implementing the same algorithm, or even if they're not using the same mechanisms (e.g. Rust explicitly using SSE intrinsics, which are certainly available to C in just as idiomatic a fashion).

gameswithgo 6y ago
comparing apples to apples is pointless, they are both apples. and you can always compare “equivalent” algorithms as some languages may not be able to efficiently express the same algorithm as another. i know what you are after, but trying to have some benchmark that is “fair” according to some spec that is important to your needs will just be seen as pointless to others. benchmark game at least lets us see what the performance ceiling is of various languages, and anytime you object to a language having a worse implementation you may go fix it. i think what anyone with a clue understands is that C C++ and Rust all have roughly equivalent performance ceilings. Nobody really thinks Rust is faster than C now.
- sli 6y ago
  I admit I'm destroying the metaphor here but there are a lot of useful comparisons to be made between different cultivars of apple and those comparisons have implications on their applications.
  - zeroimpl 6y ago
    That's the whole point of the metaphor - comparing apples to apples is good, apples to oranges is bad.
    - gameswithgo 6y ago
      and its a bad metaphor. there are many reasonable comparisons to be made between any two things even when both are not fruits
      - ghostwriter 6y ago
        Technically speaking, there are limits and you cannot compare incomparable. You can find distinctions between two phenomena, a.k.a. inherent traits that make them different-by-definition from one another, but you cannot compare (liken, collate, contrast) them in any meaningful way. Like, what's the outcome of comparing "soft" with "warm"? And hence, comparing apples to apples are not pointless, quite contrary - they are comparable items and it makes total sense to compare them in the context of superiority by desirable value axes.
        igouy 6y ago
        > Like, what's the outcome of comparing "soft" with "warm"? You can have "soft" or you can have "warm" — which do you want? You can have an apple or you can have an orange — which do you want?
    - jcelerier 6y ago
      I'm comparing apples to oranges every time I'm trying to decide which fruit to buy at the supermarket though.

FartyMcFarter 6y ago

> I like PyPy as an example: on the surface, implementing a Python runtime in Python and expecting performance gains seems crazy. Not if it includes a JIT compiler, as PyPy does. I don't know much about PyPy, but if most of the time is spent in JITted machine code, the fact that it's written in Python may not affect performance much.

pjmlp 6y ago
Kind of, PyPy is a metacircular JIT compiler, it uses RPython a Python subset.

moonchild 6y ago

> PyPy manages to outperform CPython because although a C implementation should theoretically be faster, realistically the increased expressiveness of Python lets the PyPy devs opt into optimizations the CPython devs find out of reach. Pypy outperforms cpython for the simple reason that pypy is a jit where cpython is a basic bytecode interpreter. Anything else is icing on the cake. The only reason python seems slow as an implementation language is because it was traditionally slow; the reason it was traditionally slow is that cpython, the only major implementation, is slow. Common lisp, for instance, is similarly dynamic (moreso, in fact), also is also generally natively compiled by itself, and is quite performant.

spiffytech 6y ago
PyPy has a JIT because using Python as the implementation language made it simple to build a runtime generator instead of a single runtime implementation, so RPython (the underlying PyPy tech) can generate a JIT and non-JIT runtime from the same source code. Conversely, the CPython team doesn't have the resources to make a JIT runtime the bespoke way that's more typical for these projects. https://morepypy.blogspot.com/2011/04/tutorial-part-2-adding-jit.html https://morepypy.blogspot.com/2011/04/tutorial-part-2-adding...
- naniwaduni 6y ago
  CPython isn't a JIT because its goals as a portable reference implementation would be much harder to achieve with a JIT. To put it another way, there's a reason PyPy trails CPython language features on the scale of literal years. This is not explained by the affordances of the language they're written in! They're just projects with very different requirements which, on some platforms, happen to run a lot of similar-looking code.
  - ghostwriter 6y ago
    > there's a reason PyPy trails CPython language features on the scale of literal years. the reason is under-funding, and the lack of developers to the cause as a consequence, not the portability complexities of jit.

cygx 6y ago

realistically the increased expressiveness of Python lets the PyPy devs opt into optimizations the CPython devs find out of reach Not really: It's the compilation strategy (that whole meta-tracing JIT compiler thing, compared to a simple bytecode interpreter) that makes the difference, not the surface syntax of the implementation language - which is actually not 'real' Python, but a restricted, less dynamic subset known as RPython. Also note that the CPython is deliberately kept 'dumb'.

mkl 6y ago

> That kind of tit-for-tat in benchmarks seems like it's counter to the goal of benchmarks Yep. There are good reasons why this set of benchmarks is called The Computer Language Benchmarks Game.

igouy 6y ago
https://benchmarksgame-team.pages.debian.net/benchmarksgame/sometimes-people-just-make-up-stuff.html#name-game https://benchmarksgame-team.pages.debian.net/benchmarksgame/...