3 ms·

Would be interested to see a similar comparison for a problem where the Python solution is unable to offload a significant amount of the computation onto its st

Would be interested to see a similar comparison for a problem where the Python solution is unable to offload a significant amount of the computation onto its standard-library. Also, nice to see Forth included. I strongly suspect that if the Python solution were unable to leverage its rich and well optimised standard-library, it would be easily outpaced by Forth. Also, the Forth solution used Gforth, which is far from the fastest Forth engine around. [0] It would likely have performed close to C if they had used a native-code-compiling Forth engine (SwiftForth, VFX Forth, or iForth, all of which unfortunately are proprietary payware). [0] https://github.com/ForthHub/discussion/issues/88#issuecomment-796955928 https://github.com/ForthHub/discussion/issues/88#issuecommen...

oefrha 6y ago
> if the Python solution were unable to leverage its rich and well optimised standard-library collections.Counter is a very straightforward collection backed by dict. The code isn't specially optimized, you can write the same code yourself with dict and even avoid (probably a negligible amount of) overhead. Now, dict is optimized, but that's a builtin type. You don't need PSL. https://github.com/python/cpython/blob/2fe408497e6838b6fb761e42e8f8ffc70bd6f3b1/Lib/collections/__init__.py#L649-L681 https://github.com/python/cpython/blob/2fe408497e6838b6fb761...
- MaxBarraclough 6y ago
  The article provides profiling data of the optimised Python solution. If I'm reading it right, 90% of the time (1148 milliseconds out of a total of 1280) is spent executing the _collections._count_elements and str.split methods, both of which are Python standard-library methods presumably implemented in optimized C. Of course, they're right to optimize this way, and it's to Python's credit that it's possible to leverage this approach so successfully using only the standard-library, but I think my earlier point stands.
  - oefrha 6y ago
    I practically linked to _collections._count_elements already (linked Counter.update instead because I thought linking to the call site is clearer), but here it is: https://github.com/python/cpython/blob/2fe408497e6838b6fb761e42e8f8ffc70bd6f3b1/Lib/collections/__init__.py#L519-L523 https://github.com/python/cpython/blob/2fe408497e6838b6fb761... Again, very straightforward, based on dict. Python itself does not function at all without the dict implementation since it's the underpinning of the object model. str is a builtin type. Calling str.split() a standard library method is... okay.
    - MaxBarraclough 6y ago
      Thanks for the link, interesting that _count_elements is written in Python. At a glance it looks like the kind of thing that might benefit from being written in C, but presumably that's not the case. Perhaps because of the dynamic typing of both arguments? > Python itself does not function at all without the dict implementation since it's the underpinning of the object model. Sure, but I don't see the point here. > Calling str.split() a standard library method is... okay. For our purposes there's no reason to draw a distinction between standard-library functionality closely integrated into the language, and standard-library functionality that isn't. The relevant point is that computational work is being handed off to optimised C code. Again, if you're optimising Python, the smart move is indeed to make maximal use of the optimised machinery in the standard-library (or for that matter some other dependable library). My point still stands: it would be interesting to look at a problem that isn't amenable to this approach, where the performance of the Python interpreter itself would be brought to the fore by necessity. How easy it is to find such a problem will be a function of how good a job Python does of providing applicable optimised functionality in its standard-library. Perhaps something like computing matrix determinants? I don't know enough Python to say.
      - heavyset_go 6y ago
        > Perhaps because of the dynamic typing of both arguments? It wouldn't matter, you'd be dealing with PyObject pointers in C anyway.
musingsole 6y ago
I often wonder how fast I'd run a race if I shot my foot too.

oefrha 6y ago

> if the Python solution were unable to leverage its rich and well optimised standard-library collections.Counter is a very straightforward collection backed by dict. The code isn't specially optimized, you can write the same code yourself with dict and even avoid (probably a negligible amount of) overhead. Now, dict is optimized, but that's a builtin type. You don't need PSL. https://github.com/python/cpython/blob/2fe408497e6838b6fb761e42e8f8ffc70bd6f3b1/Lib/collections/__init__.py#L649-L681 https://github.com/python/cpython/blob/2fe408497e6838b6fb761...

MaxBarraclough 6y ago
The article provides profiling data of the optimised Python solution. If I'm reading it right, 90% of the time (1148 milliseconds out of a total of 1280) is spent executing the _collections._count_elements and str.split methods, both of which are Python standard-library methods presumably implemented in optimized C. Of course, they're right to optimize this way, and it's to Python's credit that it's possible to leverage this approach so successfully using only the standard-library, but I think my earlier point stands.
- oefrha 6y ago
  I practically linked to _collections._count_elements already (linked Counter.update instead because I thought linking to the call site is clearer), but here it is: https://github.com/python/cpython/blob/2fe408497e6838b6fb761e42e8f8ffc70bd6f3b1/Lib/collections/__init__.py#L519-L523 https://github.com/python/cpython/blob/2fe408497e6838b6fb761... Again, very straightforward, based on dict. Python itself does not function at all without the dict implementation since it's the underpinning of the object model. str is a builtin type. Calling str.split() a standard library method is... okay.
  - MaxBarraclough 6y ago
    Thanks for the link, interesting that _count_elements is written in Python. At a glance it looks like the kind of thing that might benefit from being written in C, but presumably that's not the case. Perhaps because of the dynamic typing of both arguments? > Python itself does not function at all without the dict implementation since it's the underpinning of the object model. Sure, but I don't see the point here. > Calling str.split() a standard library method is... okay. For our purposes there's no reason to draw a distinction between standard-library functionality closely integrated into the language, and standard-library functionality that isn't. The relevant point is that computational work is being handed off to optimised C code. Again, if you're optimising Python, the smart move is indeed to make maximal use of the optimised machinery in the standard-library (or for that matter some other dependable library). My point still stands: it would be interesting to look at a problem that isn't amenable to this approach, where the performance of the Python interpreter itself would be brought to the fore by necessity. How easy it is to find such a problem will be a function of how good a job Python does of providing applicable optimised functionality in its standard-library. Perhaps something like computing matrix determinants? I don't know enough Python to say.
    - heavyset_go 6y ago
      > Perhaps because of the dynamic typing of both arguments? It wouldn't matter, you'd be dealing with PyObject pointers in C anyway.

musingsole 6y ago

I often wonder how fast I'd run a race if I shot my foot too.