3 ms·

indeed not sure there's much they can do about this, other than protecting all the built-in data structure operations with mutexes, like java's original data s

indeed not sure there's much they can do about this, other than protecting all the built-in data structure operations with mutexes, like java's original data structures (Hashtable, Vector, etc) (but then how do you get a non-synchronized [] if you want one?)

Znafon 3y ago
I think you can make a non thread-safe list by just making the same object without the lock described at https://peps.python.org/pep-0703/#container-thread-safety https://peps.python.org/pep-0703/#container-thread-safety if you really want the maximum performance of single threads. Maybe it could be part of a non_threadsafe_containers module on Pypi.
insanitybit 3y ago
You could fast path it by checking if the reference count of the list is `1` and avoid taking the lock in that case, I think.
- deleted 3y ago
  [deleted]
- blibble 3y ago
  that would work on x86 but not on an arch that can re-order loads (e.g. arm)
  - insanitybit 3y ago
    I'm assuming it would be a fenced operation
    - blibble 3y ago
      but then there's still an advantage to using the fenceless version :) (admittedly it's python so it's so slow it's probably not even measurable)
      - insanitybit 3y ago
        Yeah, for Python I feel like the difference between fenced vs unfenced doesn't matter. The primary cost is around your L3 cache getting slammed with contentious atomics but your L3 is already absolutely fucked if you're using Python.
- smaddox 3y ago
  Could you? What enforces only a single thread having access to a given reference? What about global variables?
  - insanitybit 3y ago
    If there's a refcount of 1 you can mutate the value safely because no other thread could be trying to read/ write to it. And the only thread that can give it to others would be the one that's doing that mutation, so it can't suddenly change. I'd assume that importing any sort of module level variable would imply an increment of the counter, but unsure.
OskarS 3y ago
My impression was that that was exactly what they were going to do: replace the GIL with fine-grained locking on the objects themselves. I can't imagine they'd let multiple python threads manipulate python data-structures concurrently, the interpreter would segfault immediately. > (but then how do you get a non-synchronized [] if you want one?) You don't. This is one of the reasons why using the GIL is higher performance for single-threaded use-cases: stuff like lists and dicts can be non-synchronized
dwaite 3y ago
> not sure there's much they can do about this, other than protecting all the built-in data structure operations with mutexes, like java's original data structures (Hashtable, Vector, etc) However - this is fundamentally the incorrect approach, because Vector and Hashtable aren't protected from read-then-write race conditions. Such internal locking guarantees that the collection stays structurally sound, but not that code accessing it is dealing with a single consistent state until it finishes.

Znafon 3y ago

I think you can make a non thread-safe list by just making the same object without the lock described at https://peps.python.org/pep-0703/#container-thread-safety https://peps.python.org/pep-0703/#container-thread-safety if you really want the maximum performance of single threads. Maybe it could be part of a non_threadsafe_containers module on Pypi.

insanitybit 3y ago

You could fast path it by checking if the reference count of the list is `1` and avoid taking the lock in that case, I think.

deleted 3y ago
[deleted]
blibble 3y ago
that would work on x86 but not on an arch that can re-order loads (e.g. arm)
- insanitybit 3y ago
  I'm assuming it would be a fenced operation
  - blibble 3y ago
    but then there's still an advantage to using the fenceless version :) (admittedly it's python so it's so slow it's probably not even measurable)
    - insanitybit 3y ago
      Yeah, for Python I feel like the difference between fenced vs unfenced doesn't matter. The primary cost is around your L3 cache getting slammed with contentious atomics but your L3 is already absolutely fucked if you're using Python.
smaddox 3y ago
Could you? What enforces only a single thread having access to a given reference? What about global variables?
- insanitybit 3y ago
  If there's a refcount of 1 you can mutate the value safely because no other thread could be trying to read/ write to it. And the only thread that can give it to others would be the one that's doing that mutation, so it can't suddenly change. I'd assume that importing any sort of module level variable would imply an increment of the counter, but unsure.

OskarS 3y ago

My impression was that that was exactly what they were going to do: replace the GIL with fine-grained locking on the objects themselves. I can't imagine they'd let multiple python threads manipulate python data-structures concurrently, the interpreter would segfault immediately. > (but then how do you get a non-synchronized [] if you want one?) You don't. This is one of the reasons why using the GIL is higher performance for single-threaded use-cases: stuff like lists and dicts can be non-synchronized

dwaite 3y ago

> not sure there's much they can do about this, other than protecting all the built-in data structure operations with mutexes, like java's original data structures (Hashtable, Vector, etc) However - this is fundamentally the incorrect approach, because Vector and Hashtable aren't protected from read-then-write race conditions. Such internal locking guarantees that the collection stays structurally sound, but not that code accessing it is dealing with a single consistent state until it finishes.