4 ms·

> In the last roughly 10 years, Google, Instagram, Dropbox, Facebook, and reddit all invested a lot of time into trying to make Python fast and was pushed away

> In the last roughly 10 years, Google, Instagram, Dropbox, Facebook, and reddit all invested a lot of time into trying to make Python fast and was pushed away with harsh no's. There are dozens of forks and branches with basic optimizations that were rejected. Is that true? The talk you link was about a very secretive fork of Python that Instagram was very adamant about not open sourcing until very recently (in fact it's not clear that what's open sourced is their full project). Other than that until recently all I've ever seen is either breaking the C-API or no clear commitment from the maintainers that this was a sustained effort to support and not just a drive-by complicated patch. In more recent times we have cinder, pyjion, pyston, and others. And CPython Devs are up-streaming either their work or their general approach.

hitekker 5y ago
I can believe it. The PSF and its hanger-ons have historically fulminated against improving CPython performance at the cost of complexity in the implementation language. They've caused flamewars on HN in the past, though Guido's recent about-face seems to have settled the field.
bjourne 5y ago
Yes, it is true. Many years ago someone showed up on python-dev with a very ambitious patch for improved string handling and especially slicing. He had some impressive benchmark numbers to back up his claims. The patch was rejected because it was large and would have made the string.c file double in size. What goes in CPython depends on whether Guido and his seconds likes it or not. And if it is their ideas and their implementations it is treated differently than if it is from outsiders. Then sometimes they suddenly change their minds and something that previously weren't ever going to happen gets implemented. JITting which, apparently, now is on their road map is one such example. Works for them and Python continues to be a great language, but frustrating for contributors.
- oblvious-earth 5y ago
  Do you have the link? I would be fascinated to read. The way you describe it sounds like it's part of the "drive-by complicated patch" category though. I think we've seen recent languages that Strings are basically one of the hardest things to "get right". Just look at Swift and Rusts multiple attempts to implement strings even though the authors all had many years of experience of working with string implementations in other mature languages. Python doesn't have the luxury of messing up strings, it was the number 1 reason that migrating from Python 2 to 3 took a decade. Unlike lower level compiled languages it can't stick in multiple string attempts and tell the users to pick one. So Core Devs are on the hook for maintaining any implementation for the life time of Python. But maybe this example really was a concerted effort by someone willing to maintain Python strings for as long as needed.
  - Jasper_ 5y ago
    Not to revive the Unicode wars again, but a big portion of Python 3's disaster transition to Unicode wasn't caused by "strings being surprisingly hard", but more rather the Python 3 team having chosen an unworkable text model which multiple people (including me, way back in 2005) all said wouldn't work in practice. Our hard-earned experience was ignored, and it took a long time for future Python editions to eat crow and roll back their mistakes. A large part of why Python 3.6 is a better port target than 3.0 is because they slowly and silently stepped away from their very strong opinions about "what is text", and "what is bytes", and what each of those things meant. Even back when the model was debated, it was clear that the operations the core maintainers cared about (manipulation of individual codepoints [0]) was a dead-end, as the Unicode Standards committee was just about realizing that codepoints weren't an ideal unit, and grapheme clusters were added to the spec. (my dates might be a bit off here, it's all a blur I've mostly forgotten) Python 3.0 was released as basically a completely nonfunctional piece of software. You basically couldn't write anything that touched a binary file in it. The built-in zipfile module, I remember, crashed if you tried to store or load a binary file stored inside a .zip, the only tests at the time stored .txt files, and the bug wasn't fixed for a long time. I remember others having troubles with the email and mime modules, though I didn't have any code that worked with those, personally. I had left the community by that point, but I have heard tales of the mountains Armin Ronacher had to move in order to add the u'' prefix back to Python 3; the core team was against it, instead believing that if 2to3 wasn't working for people, it should be fixed. Alice Bevan–McGregor spent years making a version of the WSGI specification that worked with Python 3's text model. We honestly could have shaved 5-6 years off of that disaster if they had just listened to the community; pretty much every opinion we had shared eventually came true, and the Python 3 text model these days is in a very different, and much healthier, place. [0] One of their major strongholds was that they really needed O(1) indexing of codepoints (it wasn't clear why this was a priority, and I remember arguing that wanting this is a sign of poorly written code). I think in Python 3.8 they finally caved on this, with a PEP and an implementation that can use UTF-16 or UTF-8 internally.

hitekker 5y ago

I can believe it. The PSF and its hanger-ons have historically fulminated against improving CPython performance at the cost of complexity in the implementation language. They've caused flamewars on HN in the past, though Guido's recent about-face seems to have settled the field.

bjourne 5y ago

Yes, it is true. Many years ago someone showed up on python-dev with a very ambitious patch for improved string handling and especially slicing. He had some impressive benchmark numbers to back up his claims. The patch was rejected because it was large and would have made the string.c file double in size. What goes in CPython depends on whether Guido and his seconds likes it or not. And if it is their ideas and their implementations it is treated differently than if it is from outsiders. Then sometimes they suddenly change their minds and something that previously weren't ever going to happen gets implemented. JITting which, apparently, now is on their road map is one such example. Works for them and Python continues to be a great language, but frustrating for contributors.

oblvious-earth 5y ago
Do you have the link? I would be fascinated to read. The way you describe it sounds like it's part of the "drive-by complicated patch" category though. I think we've seen recent languages that Strings are basically one of the hardest things to "get right". Just look at Swift and Rusts multiple attempts to implement strings even though the authors all had many years of experience of working with string implementations in other mature languages. Python doesn't have the luxury of messing up strings, it was the number 1 reason that migrating from Python 2 to 3 took a decade. Unlike lower level compiled languages it can't stick in multiple string attempts and tell the users to pick one. So Core Devs are on the hook for maintaining any implementation for the life time of Python. But maybe this example really was a concerted effort by someone willing to maintain Python strings for as long as needed.
- Jasper_ 5y ago
  Not to revive the Unicode wars again, but a big portion of Python 3's disaster transition to Unicode wasn't caused by "strings being surprisingly hard", but more rather the Python 3 team having chosen an unworkable text model which multiple people (including me, way back in 2005) all said wouldn't work in practice. Our hard-earned experience was ignored, and it took a long time for future Python editions to eat crow and roll back their mistakes. A large part of why Python 3.6 is a better port target than 3.0 is because they slowly and silently stepped away from their very strong opinions about "what is text", and "what is bytes", and what each of those things meant. Even back when the model was debated, it was clear that the operations the core maintainers cared about (manipulation of individual codepoints [0]) was a dead-end, as the Unicode Standards committee was just about realizing that codepoints weren't an ideal unit, and grapheme clusters were added to the spec. (my dates might be a bit off here, it's all a blur I've mostly forgotten) Python 3.0 was released as basically a completely nonfunctional piece of software. You basically couldn't write anything that touched a binary file in it. The built-in zipfile module, I remember, crashed if you tried to store or load a binary file stored inside a .zip, the only tests at the time stored .txt files, and the bug wasn't fixed for a long time. I remember others having troubles with the email and mime modules, though I didn't have any code that worked with those, personally. I had left the community by that point, but I have heard tales of the mountains Armin Ronacher had to move in order to add the u'' prefix back to Python 3; the core team was against it, instead believing that if 2to3 wasn't working for people, it should be fixed. Alice Bevan–McGregor spent years making a version of the WSGI specification that worked with Python 3's text model. We honestly could have shaved 5-6 years off of that disaster if they had just listened to the community; pretty much every opinion we had shared eventually came true, and the Python 3 text model these days is in a very different, and much healthier, place. [0] One of their major strongholds was that they really needed O(1) indexing of codepoints (it wasn't clear why this was a priority, and I remember arguing that wanting this is a sign of poorly written code). I think in Python 3.8 they finally caved on this, with a PEP and an implementation that can use UTF-16 or UTF-8 internally.