5 ms·

Specifically Python is just a very problematic language to re-implement, since CPython has leaked internals literally everywhere. Last time I looked into Pypy (

by dom0 9y ago

Specifically Python is just a very problematic language to re-implement, since CPython has leaked internals literally everywhere. Last time I looked into Pypy (to be fair, that's now a few years ago) that was the main pain point. Implementing a sane subset of Python is not so difficult, because, well, d'uh, it's sane. Implementing all the leaky details, the insane data model (slightly less insane in Python 3), all the invasive interpreter APIs which are frequently used by frameworks and extensions, is a whole different story. More importantly, if you implement all those, there's not much to gain in performance any more. Pypy has made tremendous progress there, but I'd expect that it still suffers from this — bottom line is, there is no way to efficiently implement what CPython does, and I'd be surprised if Pypy is significantly faster than CPython in code that uses all this shit. I expect Pypy produces a lot of gain as soon as code doesn't use that.

orf 9y ago
Can you elaborate on what internals you've seen leaked?
- __s 9y ago
  They're moving towards standardizing the ordered dict layout https://docs.python.org/3/library/inspect.html https://docs.python.org/3/library/inspect.html PyPy has a glue layer to implement the C API. The C API is designed around reference counting, which PyPy doesn't do. This also shows up in Python code assuming files will be immediatly closed in lines like 'x = open(fname).read()' Method resolution order. __slots__. globals(). id
  - orf 9y ago
    Ok, fair enough, thanks for providing some examples, but I'm not sure how many are valid though (Don't get me wrong, there are a bunch, the main one being reliance on the GC/ref count semantics). > They're moving towards standardizing the ordered dict layout Can you elaborate? Not sure what you mean. > inspect library Yes, a fair few functions in that module are CPython specific (or assume a CPython-ish runtime). > C API Isn't the C API CPython specific? It is not a Python-language feature. PyPy implemented it to drive up adoption. How would the C API be expected to work with Jython, or IronPython, or a JS implementation? > Method resolution order This is not a CPython leaking, it's a specified thing. Python uses the C3 algorithm. > __slots__ Again, this is a language (or object model) feature? > globals() Again, specified by the language? Not a CPython specific thing? > id ID returns a unique identifier per object. That's all, in CPython it returns the memory address. Is that what you mean? --- There are also only 5 'CPython implementation detail' warnings in the language data model docs[1] 1. https://docs.python.org/3.6/reference/datamodel.html https://docs.python.org/3.6/reference/datamodel.html
    - chrisseaton 9y ago
      > Isn't the C API CPython specific? It is not a Python-language feature. But the Python ecosystem depends on extensions written using the C API. If you are trying to write a new Python implementation and they tell you they need to be able to run some C extension you can tell them 'actually that's a CPython feature' all you want, but they'll go away and not use your implementation. > How would the C API be expected to work with Jython, or IronPython, or a JS implementation? Well you'd implement the same API using JNI, PlatformInvoke, or the V8 C extension API, or whatever else the platform you are building on has, respectively. But it's hard!
    - __s 9y ago
      Calling __slots__ a language/object feature is a crutch. It was made a language feature based on how CPython implements objects with a __dict__ field showing the backing dictionary. Also there's the 'is' operator which has unspecified behavior with integers/strings due to integer-cache & string interning. If you used unboxed integers then 'is' would likely work as equals for all integers that weren't bignums. Python's tried to define a standard outside CPython's details, but CPython is essentially de facto, so for serious implementations it's details are important to follow. Similar thing happened with clang needing to implement many GCC C extensions Ordered dict layout: https://mail.python.org/pipermail/python-dev/2016-September/146489.html https://mail.python.org/pipermail/python-dev/2016-September/...
- dom0 9y ago
  Instead of typing a long list, I'll just refer you to this talk: https://www.youtube.com/watch?v=qCGofLIzX6g https://www.youtube.com/watch?v=qCGofLIzX6g Some of that has been fixed / got make up applied to play pretend.
  - orf 9y ago
    Thank you!
v3ss0n 9y ago
PyPy today is nothing like pypy years ago. It had improved in many areas including cext and python 3.5 is fully supported now. complex suite like odoo ca run on it effectively. Their next goal is to run entire scipy suite on it without modification. We are running production systems in pypy and performance improvement is significant. Average 40percent faster than node in many areas.

by dom0 9y ago

orf 9y ago

Can you elaborate on what internals you've seen leaked?

__s 9y ago
They're moving towards standardizing the ordered dict layout https://docs.python.org/3/library/inspect.html https://docs.python.org/3/library/inspect.html PyPy has a glue layer to implement the C API. The C API is designed around reference counting, which PyPy doesn't do. This also shows up in Python code assuming files will be immediatly closed in lines like 'x = open(fname).read()' Method resolution order. __slots__. globals(). id
- orf 9y ago
  Ok, fair enough, thanks for providing some examples, but I'm not sure how many are valid though (Don't get me wrong, there are a bunch, the main one being reliance on the GC/ref count semantics). > They're moving towards standardizing the ordered dict layout Can you elaborate? Not sure what you mean. > inspect library Yes, a fair few functions in that module are CPython specific (or assume a CPython-ish runtime). > C API Isn't the C API CPython specific? It is not a Python-language feature. PyPy implemented it to drive up adoption. How would the C API be expected to work with Jython, or IronPython, or a JS implementation? > Method resolution order This is not a CPython leaking, it's a specified thing. Python uses the C3 algorithm. > __slots__ Again, this is a language (or object model) feature? > globals() Again, specified by the language? Not a CPython specific thing? > id ID returns a unique identifier per object. That's all, in CPython it returns the memory address. Is that what you mean? --- There are also only 5 'CPython implementation detail' warnings in the language data model docs[1] 1. https://docs.python.org/3.6/reference/datamodel.html https://docs.python.org/3.6/reference/datamodel.html
  - chrisseaton 9y ago
    > Isn't the C API CPython specific? It is not a Python-language feature. But the Python ecosystem depends on extensions written using the C API. If you are trying to write a new Python implementation and they tell you they need to be able to run some C extension you can tell them 'actually that's a CPython feature' all you want, but they'll go away and not use your implementation. > How would the C API be expected to work with Jython, or IronPython, or a JS implementation? Well you'd implement the same API using JNI, PlatformInvoke, or the V8 C extension API, or whatever else the platform you are building on has, respectively. But it's hard!
  - __s 9y ago
    Calling __slots__ a language/object feature is a crutch. It was made a language feature based on how CPython implements objects with a __dict__ field showing the backing dictionary. Also there's the 'is' operator which has unspecified behavior with integers/strings due to integer-cache & string interning. If you used unboxed integers then 'is' would likely work as equals for all integers that weren't bignums. Python's tried to define a standard outside CPython's details, but CPython is essentially de facto, so for serious implementations it's details are important to follow. Similar thing happened with clang needing to implement many GCC C extensions Ordered dict layout: https://mail.python.org/pipermail/python-dev/2016-September/146489.html https://mail.python.org/pipermail/python-dev/2016-September/...
dom0 9y ago
Instead of typing a long list, I'll just refer you to this talk: https://www.youtube.com/watch?v=qCGofLIzX6g https://www.youtube.com/watch?v=qCGofLIzX6g Some of that has been fixed / got make up applied to play pretend.
- orf 9y ago
  Thank you!

v3ss0n 9y ago

PyPy today is nothing like pypy years ago. It had improved in many areas including cext and python 3.5 is fully supported now. complex suite like odoo ca run on it effectively. Their next goal is to run entire scipy suite on it without modification. We are running production systems in pypy and performance improvement is significant. Average 40percent faster than node in many areas.