4 ms·

This prompted me to ask this question on stackoverflow: http://stackoverflow.com/questions/1914605/what-does-pythons-gil-have-to-do-with-the-garbage-collector

by j_baker 17y ago

This prompted me to ask this question on stackoverflow: http://stackoverflow.com/questions/1914605/what-does-pythons-gil-have-to-do-with-the-garbage-collector http://stackoverflow.com/questions/1914605/what-does-pythons... I'd like it if someone could show me how the garbage collector is related to removing the GIL.

Tuna-Fish 17y ago
Glad to be of service. ;)
silentbicycle 17y ago
Python's garbage collector itself isn't "thread safe", so removing the GIL without replacing the GC would cause major problems.
ig1 17y ago
I'm no expert in the GIL, but pretty much every widely adopted Garbage Collection algorithm requires a "stop-the-world" phase where object references can't be changed. Every VM has some concept of "stop points" where all user code is suspended, but Python's GIL is much more wide-ranging than that found in say the JVM or .NET.
- Tuna-Fish 17y ago
  Python doesn't really have real GC, it refcounts. See the stackoverflow answer.
  - jemfinch 17y ago
    Python refcounts and it has a real mark+sweep collector for collecting cycles. It's not a dichotomy, you know.
    - silentbicycle 17y ago
      The reference counting causes the problem here. When multithreading means that incrementing/decrementing a reference count is no longer deterministic, you need locks, or all hell breaks loose. Adding a mark&sweep GC isn't going to fix that. While I'm not familiar with the specifics of Python's GC, a mark&sweep phase is usually added to reference counting so that if there's garbage which contains references to itself but has no external references, it will eventually be collected. (_Garbage Collection_ by Richard Jones and Rafael Lins is an excellent resource on GC details, btw. There's also a decent overview in the O'Reilly OCaml book (http://caml.inria.fr/pub/docs/oreilly-book/html/book-ora082.html http://caml.inria.fr/pub/docs/oreilly-book/html/book-ora082.... )). In other words, it plugs the worst memory leaks caused by reference counting. How to do multiprocessor / multithread GC well is still an area of active research. In the mean time, one simpler solution is to have several independent VM states, each running in their own thread (or process), and communicating via message passing. Lua makes this easy, but its VM is considerably lighter than Python's.
    - kingkilr 17y ago
      The mark and sweep collector is a little more complicated than that. It doesn't use a single "seen" bit the way a normal GC header would, instead it uses the refcount itself in very, very clever ways
  - deleted 17y ago
    [deleted]
  - bioweek 17y ago
    How does a real GC work? I can't picture how you can do GC without ref counts. (sorry if that's a dumb question)
    - coliveira 17y ago
      A GC looks at the content of memory and checks if each piece of memory is accessible (there are sophisticated algorithms to do this) - no need for a ref count is necessary with GC.
      - bioweek 17y ago
        why do you mean by accessible? Wouldn't that still mean reference counting, i.e., it's not accessible if it has 0 references?
        jeffcoat 17y ago
        Here's a (very naive) algorithm that doesn't count references: 1. Start at some well-known root object. Mark this object reachable. 2. For each object that is reachable 2a. For each object that this reachable object has a reference to 2b. Mark this object reachable if it's not already. 3. If any object was marked reachable in 2b, repeat step 2. 4. Garbage collect every object not marked reachable. To see an important difference between this kind of strategy ("mark and sweep") and reference counting, consider the classic example of two objects, each one of which has a link to the other, but are not visible to anything else in the system.
    - j_baker 17y ago
      I would disagree with saying that refcounting isn't "real" garbage collection. But with that said, you can get all the gory details from wikipedia: http://en.wikipedia.org/wiki/Garbage_collection_(computer_science)#Reachability_of_an_object http://en.wikipedia.org/wiki/Garbage_collection_(computer_sc...
- cemerick 17y ago
  The JVM (and I presume the CLR as well) has a couple of different garbage collectors that support concurrent (i.e. not stop-the-world) operation: http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning... I'm no GC expert, but these get you good, fast, and mostly concurrent GC (in very high-pressure situations where one is running out of memory, then I think both fall back to a serialized stop-the-world last-ditch-effort).
  - j_baker 17y ago
    In fact, I believe that the JVM and the CLR both run GC in a different thread altogether.
  - ig1 17y ago
    Luckily as it happens I am an expert in JVM Garbage Collectors :-). All of Sun Java's Garbage Collectors have a stop-the-world phase, the concurrent ones minimize the length of that stop-the-world period but it still has one. If you read the section on the concurrent collector on the link you posted you can see it says "During each major collection cycle, the concurrent collector will pause all the application threads for a brief period at the beginning of the collection and again toward the middle of the collection. The second pause tends to be the longer of the two pauses and multiple threads are used to do the collection work during that pause"

by j_baker 17y ago

Tuna-Fish 17y ago

Glad to be of service. ;)

silentbicycle 17y ago

Python's garbage collector itself isn't "thread safe", so removing the GIL without replacing the GC would cause major problems.

ig1 17y ago

I'm no expert in the GIL, but pretty much every widely adopted Garbage Collection algorithm requires a "stop-the-world" phase where object references can't be changed. Every VM has some concept of "stop points" where all user code is suspended, but Python's GIL is much more wide-ranging than that found in say the JVM or .NET.

Tuna-Fish 17y ago
Python doesn't really have real GC, it refcounts. See the stackoverflow answer.
- jemfinch 17y ago
  Python refcounts and it has a real mark+sweep collector for collecting cycles. It's not a dichotomy, you know.
  - silentbicycle 17y ago
    The reference counting causes the problem here. When multithreading means that incrementing/decrementing a reference count is no longer deterministic, you need locks, or all hell breaks loose. Adding a mark&sweep GC isn't going to fix that. While I'm not familiar with the specifics of Python's GC, a mark&sweep phase is usually added to reference counting so that if there's garbage which contains references to itself but has no external references, it will eventually be collected. (_Garbage Collection_ by Richard Jones and Rafael Lins is an excellent resource on GC details, btw. There's also a decent overview in the O'Reilly OCaml book (http://caml.inria.fr/pub/docs/oreilly-book/html/book-ora082.html http://caml.inria.fr/pub/docs/oreilly-book/html/book-ora082.... )). In other words, it plugs the worst memory leaks caused by reference counting. How to do multiprocessor / multithread GC well is still an area of active research. In the mean time, one simpler solution is to have several independent VM states, each running in their own thread (or process), and communicating via message passing. Lua makes this easy, but its VM is considerably lighter than Python's.
  - kingkilr 17y ago
    The mark and sweep collector is a little more complicated than that. It doesn't use a single "seen" bit the way a normal GC header would, instead it uses the refcount itself in very, very clever ways
- deleted 17y ago
  [deleted]
- bioweek 17y ago
  How does a real GC work? I can't picture how you can do GC without ref counts. (sorry if that's a dumb question)
  - coliveira 17y ago
    A GC looks at the content of memory and checks if each piece of memory is accessible (there are sophisticated algorithms to do this) - no need for a ref count is necessary with GC.
    - bioweek 17y ago
      why do you mean by accessible? Wouldn't that still mean reference counting, i.e., it's not accessible if it has 0 references?
      - jeffcoat 17y ago
        Here's a (very naive) algorithm that doesn't count references: 1. Start at some well-known root object. Mark this object reachable. 2. For each object that is reachable 2a. For each object that this reachable object has a reference to 2b. Mark this object reachable if it's not already. 3. If any object was marked reachable in 2b, repeat step 2. 4. Garbage collect every object not marked reachable. To see an important difference between this kind of strategy ("mark and sweep") and reference counting, consider the classic example of two objects, each one of which has a link to the other, but are not visible to anything else in the system.
  - j_baker 17y ago
    I would disagree with saying that refcounting isn't "real" garbage collection. But with that said, you can get all the gory details from wikipedia: http://en.wikipedia.org/wiki/Garbage_collection_(computer_science)#Reachability_of_an_object http://en.wikipedia.org/wiki/Garbage_collection_(computer_sc...
cemerick 17y ago
The JVM (and I presume the CLR as well) has a couple of different garbage collectors that support concurrent (i.e. not stop-the-world) operation: http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning... I'm no GC expert, but these get you good, fast, and mostly concurrent GC (in very high-pressure situations where one is running out of memory, then I think both fall back to a serialized stop-the-world last-ditch-effort).
- j_baker 17y ago
  In fact, I believe that the JVM and the CLR both run GC in a different thread altogether.
- ig1 17y ago
  Luckily as it happens I am an expert in JVM Garbage Collectors :-). All of Sun Java's Garbage Collectors have a stop-the-world phase, the concurrent ones minimize the length of that stop-the-world period but it still has one. If you read the section on the concurrent collector on the link you posted you can see it says "During each major collection cycle, the concurrent collector will pause all the application threads for a brief period at the beginning of the collection and again toward the middle of the collection. The second pause tends to be the longer of the two pauses and multiple threads are used to do the collection work during that pause"