3 ms·
these examples of "what one would assume to be atomic" did not seem useful to me, they looked like things that are obviously not threadsafe. a more interesting
by zzzeek 3y ago
these examples of "what one would assume to be atomic" did not seem useful to me, they looked like things that are obviously not threadsafe.
a more interesting example is something like this:
# setup
l = []
# thread A
l.extend([1, 2, 3])
# thread B
l.extend([4, 5, 6])
is the resulting list always within the set of [1,2,3,4,5,6] or [4,5,6,1,2,3] ? or are the two sets of numbers randomly interleaved in the list? or if the GIL is removed does the interpreter segfault (I'm pretty sure this latter will not be the case for GIL removal but I don't understand the gil remove plan very much yet).
Edit: before people jump in and correct how the above is a bad idea anyway, it's not like I'd ever do the above and expect anything but disaster. This is more of a thought experiment to understand what GIL removal is going to do.
- ngoldbaum 3y agoIt’ll be guarded by a fine-grained mutex, so it won’t seg fault. They’re using a performant mutex based on webkit’s wtf::lock.
- OskarS 3y agoI'm guessing interleaving is probably not possible because the `extend` call is passed the full list, and it holds the GIL until it's finished. But you'd have to look at the bytecode to be sure, I suppose. If there were three append calls instead of a single extend, I would assume any interleaving would be valid. > or if the GIL is removed does the interpreter segfault (I'm pretty sure this latter will not be the case for GIL removal but I don't understand the gil remove plan very much yet). I haven't looked into the plan in detail either, but presumably not, that would be nuts. My understanding is that they're going to replace the GIL with locks on the objects themselves (your list `l` in this case). This is why in all the tests single-threaded performance suffer, you have to take and release more locks if you don't have a GIL, and the objects themselves grow larger as well.
- Gabrys1 3y agoLet's say you're passing 1 million ints to each extend. One may start wondering if those operations are then split into chunks...
- dfox 3y agoYou can probably trigger the behavior of random interleaving even with GIL if you use something other than list or tuple as an argument to list.extend(). CPython special cases list and tuples and copies the contents directly (list_extend_fast() in listobject.c) while it uses iterator for other types (list_extend_iter()). Because the iterator can be pretty much arbitrary Python code it seems to be a bad idea to guarantee extend() to be atomic, as you don't want to hold the list mutex while calling out into user code.
- nerdponx 3y agoAll the more reason to wonder. What's going to be special-cased to become atomic and thread-safe, and what won't be? A surprising amount of the CPython stdlib is just pure Python code.
- colesbury 3y agoYou will always get either [1,2,3,4,5,6] or [4,5,6,1,2,3] in the upcoming `--disable-gil` builds of CPython 3.13 and the nogil forks. Most operations on mutable collections hold a per-object lock. Part of the integration work will be to better document the thread-safety guarantees, but there is still a lot of work to do before we get there.
- HDThoreaun 3y agoEvery mutable collection needs to lock before write? Sounds slow af
- quietbritishjim 3y agoLocks are generally very cheap if they're not contended. Of course it's all relative though!
- gnulinux 3y agoThey're cheap but they're not free. If you do it at literally every single rw operation at runtime, it's going to add up.
- fbdab103 3y agoLast I looked, the nogil implementation was some 5-10% slower than the current, owing to all of the extra locking.