6 ms·
I don't think it was a mistake. There were some intractably difficult, "rip-the-band-aid-off" types of changes that had to happen at some point. The PSF decided
by kstrauser 7y ago
I don't think it was a mistake. There were some intractably difficult, "rip-the-band-aid-off" types of changes that had to happen at some point. The PSF decided that 3.0 was the right time to make those one-time changes so that future migrations could be as small as possible.
- cft 7y agoThen why didn't they remove GIL? Given the end of Moore's law and Python's performance, this alone would have justified the discontinuity.
- fiedzia 7y agoBecause GIL is useful, preventing you from shooting yourself terribly in the foot among other benefits. Languages with better concurrency support (main reason for removing GIL) were designed from scratch for that purpose. You can't remove GIL and keep Python being Python everyone knows. And if you don't mind using another language, there are many choices.
- laurencerowe 7y agoI believe Jython offers the same level of protection against shooting yourself in the foot as CPython, yet it allows allows for parallel execution by relying on finer grained locking. The concurrency issues with threads are still present in CPython with the GIL as the interpreter may switch threads between any instruction.
- kstrauser 7y agoThere've been various attempts at this over the years, but it's really orthogonal to changes to the language itself. Also, check out the SharedMemory objects[1] in 3.8. For my own use cases, this goes a long way toward making me no longer care about the GIL. I'd rather work on multiprocessing code than multithreaded code any time, and if you can easily share objects across processes, then the GIL isn't relevant to anything I'm working on. [1] https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory https://docs.python.org/3/library/multiprocessing.shared_mem...
- dickeytk 7y agoI would argue the GIL is a feature put in place to make it easier to write parallel code. If you want to do this in a more performant way you should be looking at a different language. People talk about the GIL as if it’s just some drawback that could be removed and everything would be faster. There is a very good reason why it’s there.
- masklinn 7y ago> I would argue the GIL is a feature put in place to make it easier to write parallel code. It doesn't do that though. The GIL protects interpreter internals. It doesn't make Python code thread-safe, although it does make various operations atomic: * native calls (which doesn't release the GIL) * individual bytecode instructions (which don't call into more python code) The latter can make it seem like it magically protects your code, but even a simple method call is at least 3 instructions (LOAD_FAST, LOAD_METHOD and CALL_METHOD). Each instruction is atomic and the method call will be if it's a native call (e.g. dict.get) but that's it, the entire call itself is not atomic and you could thread-switch between LOAD_FAST and LOAD_METHOD, or LOAD_METHOD and CALL_METHOD.
- dickeytk 7y agoYou just explained a bunch of implementation details but at the end of the day my point remains: having the GIL makes it easier to write parallel code. I didn't say it magically makes everything threadsafe.
- masklinn 7y ago> having the GIL makes it easier to write parallel code. It does not. What it does is create a bigger trap, because the code looks to be working in more testing situations, and will be more difficult to debug when it starts failing.
- pavanky 7y agoIsnt the GIL part of the implementation rather than the language?
- dashwav 7y agoBecause the GIL is only a problem in specific sets of problems that the majority of python users won't run into. With the addition of asyncio etc, the negative of GIL is usually pretty restricted to CPU-heavy, blocking services which the core team decided was a very small subset of users. So removing the GIL would require an extremely heavy rewrite of the way python works at a fundamental basis, and it would break a lot of the "promises" that python affords in regards to memory management and safety. As an aside, there is currently a PEP that will allow for better concurrency __without__ removing the GIL, [PEP554](https://www.python.org/dev/peps/pep-0554/ https://www.python.org/dev/peps/pep-0554/) and a larger project overall for multi-core support, which is (hopefully) being included in 3.9 [Multi-core](https://github.com/ericsnowcurrently/multi-core-python https://github.com/ericsnowcurrently/multi-core-python)
- deleted 7y ago[deleted]
- Ralfp 7y agoRemoving GIL will break ecosystem in worse way than Python 2->3 ever did, as both interpreter and compiled libraries rely on it for thread safety. There are other paths for improving performance that Python is embrancing: asyncio provides considerable performance gains to IO bound apps, and memory-sharing between processes allows for relatively cheap sidestep for those who find threads too slow for their use-case.
- the_why_of_y 7y agoIn addition to the points other people raised, "simply" removing the GIL by changing the current reference-counting GC to do atomic reference updates will also substantially reduce single threaded performance; there was a PyCon talk about that. https://www.youtube.com/watch?v=P3AyI_u66Bw https://www.youtube.com/watch?v=P3AyI_u66Bw See also Swift for another language whose implementation has performance issues caused by ubiquitous atomic reference counting.
- kec 7y agoAside from Unicode strings, what changes actually required breaking backwards compatibility. For that matter, why did the switch to unicode strings require breaking backwards compatibility? I don't understand why we couldn't just move to assuming all strings are utf8 unless specified otherwise. Keep [] byte oriented, add a .at() method to index by code point.
- takeda 7y agoThe reason for it were bad decisions made in older version of python. The problems were: - conflating text with bytes, python had no way to tell whether given string is a text or bytes, because in 2.7 was the same thing. - introducing unicode as an unicode type, this essentially made the problem worse because in addition to mixing text with strings, they added extra type to represent text, which was optional and some people used unicode some didn't. - a cherry on top was implicit conversion, so if you passed unicode type where str was expected python implicitly converted it, and vice versa This basically resulted in people writing a broken code in python 2, code that worked fine with us-ascii, but randomly blew up if there ware some non standard characters processed. In python 3 instead applying another fix, they decided to do it correctly from the start. So str (Guido actually regretted he didn't called it text) is representing text and bytes are representing, well bytes. Python 3 also does not do implicit conversion as well. I actually think they could perhaps help with migration, by adding to 2.7 one more import to __futures__ that disables implicit conversion and then treat bytes type as a distinctly different type than str. That could help people fix their code in 2.7 before migrating it, but anyway it's already too late to do it, also there is a hack that you could disable implicit conversion in python 2, but it showed that stdlib also relied on it heavily, so fixing that maybe was not worth it.
- excessive 7y ago> There were some intractably difficult, "rip-the-band-aid-off" types of changes that had to happen at some point. I suspect you're referring to Unicode here. In that case, I think they could've just added a flag to Python 2's str type to indicate "is UTF-8" and deprecated the old unicode object. Then add some functions to extract code points or grapheme clusters or whatever else you need from the old school str object. I might be in the minority, but I really like that Python 2's str could hold arbitrary binary data, of which UTF-8 is just one possibility. It had good interop with C, which I think is fundamental to a glue language like Python. I'd rather have fewer string types instead of more (one of my complaints about Rust too). If you meant the print function, there were other ways to solve that too. The simplest might be to create a new name for the function and deprecate the print statement. So old code uses the statement "print 123" while new code is encouraged to call the function "echo(123)" or "ouput(123)". Bikeshed the actual name... Note when I say "deprecate", I mean provide a timeline over several releases where it continues to work. Then issue deprecation warnings which can be silenced. All of the newer features in Python 3 (@ operator, async/await, type annotations, etc...) could've been added in a mostly backwards compatible way. (Note: adding async wasn't really backwards compatible even in Python 3). Anyways, hindsight is 20/20, but I really do think the path for Python 3 was a poor choice in comparison to other options.
- kevin_thibedeau 7y agoYour proposal only works well for US ASCII users. What if I want to manage multiple ISO-8859 encodings in conjunction with 7-bit ASCII? Maybe I also have some EUC-JP multi-byte text to deal with. It becomes an intractable mess without explicit encoding management. Someone will absolutely end up misinterpreting encoded text as bytes and cause all manner of compatibility and security issues. Having a Unicode string type forces this to be dealt with even if it is inconvenient when taking in data from outside the Python environment.
- excessive 7y ago> Your proposal only works well for US ASCII users. No, and I explicitly mentioned UTF-8. My suggestion is that str holds arbitrary immutable binary data and that you have a method which can interrogate whether that binary data is valid UTF-8. Yes, real world text is messy and there are lots of encodings, compression schemes, and exceptions (UTF-8 with byte order marks, overlong encodings, or surrogate pairs, as examples). If your main task is converting text between outdated or broken encodings, I don't have any problem saying you need a separate library and shouldn't burden the rest of the user base. Despite it's flaws, the majority of the world has settled on Unicode with a UTF-8 encoding. "Special cases aren't special enough to break the rules."