4 ms·
Balm in GILead: Fast string construction for CPython extensions
- chrismorgan 3y agoAn explanation of the title, for those not familiar with it: Gilead was a region situated in modern-day Jordan, and was apparently renowned for its medicinal balm, mentioned a few times in the Bible. Further reading: https://en.wikipedia.org/wiki/Balm_of_Gilead https://en.wikipedia.org/wiki/Balm_of_Gilead.
- svat 3y agoMost familiar to me from Edgar Allan Poe's The Raven, where the deranged narrator asks this to the bird: “Prophet!” said I, “thing of evil!—prophet still, if bird or devil!— Whether Tempter sent, or whether tempest tossed thee here ashore, Desolate yet all undaunted, on this desert land enchanted— On this home by Horror haunted—tell me truly, I implore— Is there—*is* there balm in Gilead?—tell me—tell me, I implore!” Quoth the Raven “Nevermore.”
- ashvardanian 3y agoThis looks exceptionally interesting and potentially useful to some of our projects. Anyone else using similar methods? Would love to know if there are other caveats not covered by the article before I start refactoring 10K LOC :)
- haberman 3y agoI think this technique would not work with the limited API: https://docs.python.org/3/c-api/stable.html#limited-c-api https://docs.python.org/3/c-api/stable.html#limited-c-api The Limited API has a nice benefit of letting you build artifacts that are compatible with a wide range of Python versions. The technique described in this article tightly couples the extension code to the interpreter internals.
- nickelpro 3y agoIt doesn't work with any version of the public API, Limited, Stable, or Unstable, because this is not a part of the API. It's more of an application of Hyrum's Law[1]. That said, assuming the structures themselves exist on the versions of Python you're targeting in a format compatible with whatever hacking you're doing on them, it's very easy to compile for lots of Python versions using cibuildwheel[2] and the rest of the PyPA ecosystem. I don't think the Limited API is very useful, as a practical matter for the common distribution methods you need the wheel to be built with the target Python version. [1]: https://www.hyrumslaw.com/ https://www.hyrumslaw.com/ [2]: https://github.com/pypa/cibuildwheel https://github.com/pypa/cibuildwheel
- haberman 3y ago> I don't think the Limited API is very useful, as a practical matter for the common distribution methods you need the wheel to be built with the target Python version. The limited API provides ABI stability, which allows building a single artifact that will work across Python versions. We use this for the Protobuf PyPI package. A single artifact like protobuf-4.25.1-cp37-abi3-manylinux2014_x86_64.whl can be used with any Python version >= 3.7.
- nickelpro 3y agoYa, but so what? It's zero engineering effort to build the wheels for all the compatible versions. You don't win anything with the limited API.