8 ms·

Copy-on-write friendly Python garbage collection

196 points by happy-go-lucky 9y ago

vosper 9y ago
When exactly is this new GC behavior useful? When forking processes? Is it something we'd see used in the multiprocessing or concurrent modules?
- bpicolo 9y ago
  Memory is typically the scaling bottleneck for Python workers for web servers. One way to cut back drastically on this is to load as much as possible in startup and then fork for request serving processes. The problem is that python refcounting causes a lot of copy-on-writes for data that's really just used as read-only data. This change allows you to run more workers with less memory. Instagram has had a few good posts about how they've approached this problem, here's another: https://engineering.instagram.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172 https://engineering.instagram.com/dismissing-python-garbage-...
  - sametmax 9y ago
    > Memory is typically the scaling bottleneck for Python workers for web servers. Wut ? When I have performances problems, my RAM is full with varnish cache, redis stored objects and postgres buffer. Python is like, low, low on the list. Again, most people are NOT instagram or Google. They have a very atypical load.
    - bpicolo 9y ago
      That's latency, not bandwidth. This is to help scale out concurrent request counts per box, rather than time to serve a single request (though obviously with request queuing, failure to serve requests fast enough makes concurrency a latency bottleneck). And sure, most people aren't instagram. This is a python-at-big-scale problem, when the cost benefits of fitting more requests onto fewer servers actually matter. One of the biggest reasons this actually makes a difference is that, in SoA, internal API scaling against your monolith can become expensive.
    - ComputerGuru 9y ago
      > Again, most people are NOT Instagram or Google And, again, can we stop with these pointless comparisons? Past 1 front end server the only cost that matters is how many requests your one node can handle and how much that one node costs. If you’re not a VPS and you have your frontend http cache correctly configured, then it doesn’t matter how much smaller than Google you are, comparisons are valid (although, of course, the fewer servers you have the more you can afford to spend on them; though you probably aren’t making as much money as IG/FB/Google either...).
      - sametmax 9y ago
        The only metrics is economic. The question is how much does those performance constrains cost you ?
- ot 9y ago
  Yes, when you fork all the pages in the address space are referenced, and copied when modified. However if you have a GC it will go around touching objects in the child process, and there's a good chance it will end up touching a few bytes in each page, nullifying the benefits of COW.
  - rurban 9y ago
    Exactly. That's why you normally don't call refcounting a GC, as it writes to objects on referencing, and thereby destroys COW. With Perl it's even more stupid, they write to the const strings also, not just the refcount. And still call it COW. I call it write on assign, WOA. A normal GC leaves it alone and does not force fresh memory maps for these.
    - ot 9y ago
      It's not just refcounting, GCs also normally have some sort of bookkeeping that is updated in the background, for example a mark&sweep GC will need to mark the objects. See the comment about the object header in the OP.
      - rurban 9y ago
        This GC bookkeeping cannot be compared to the refcounting madness. A GC needs 1-2 bits during the GC, but does not disturb COW pages at all during run-time. It also has no word overhead, these 2 bits easily fit into every object. Also a GC compacts the live heap, which makes all upcoming accesses much faster. Well, not Mark & Sweep, but a compacting GC.
- masklinn 9y ago
  > When exactly is this new GC behavior useful? When forking processes? When forking a process and performing lots of allocations in the parent process (e.g. lots of imports & objects creation before forking) and doing so on a CoW-forking OS (but I'm guessing that's most modern POSIX systems).
IgorPartola 9y ago
A more general question: why is it so typical for a Python we worker to have its memory grow infinitely with each request processed? I don’t expect the kind of performance I can get out of C with its ability to only allocate memory on the stack or to pre-allocate all the buffers I’d need. And I have found a number of memory leaks in Python libraries and especially poorly written Python C modules over the years. But still I would expect something like a simple ideomatic Django application to not have its memory footprint grow idefinitely. Is this just life now? Are there any good tools out there for figuring out what part of the application keeps requesting objects that don’t get destroyed? Are corporations people?
- sametmax 9y ago
  > why is it so typical for a Python we worker to have its memory grow infinitely with each request processed? It's not. The cases where it happened to me during the last 10 years and dozens of web projects spanning on multiple frameworks were generally me doing something stupid: - using a mutable object as default value or in a class attribute - doing something in __del__ - keeping DEBUG=True for Django (which is very well known for causing memory leaks) > Is this just life now? Nope. I have big Django apps running (like 500k users/day streaming video sites) and it doesn't happen. > Are there any good tools out there for figuring out what part of the application keeps requesting objects that don’t get destroyed? Yes for course: http://tech.labs.oliverwyman.com/blog/2008/11/14/tracing-python-memory-leaks/ http://tech.labs.oliverwyman.com/blog/2008/11/14/tracing-pyt...
  - ubernostrum 9y ago
    It's hard to call DEBUG in Django a "memory leak" when the "leak" is documented behavior -- when DEBUG=True, Django keeps a running per-connection log in memory of database queries issued. For, y'know, debugging :) This does cause memory use to increase with the number of queries issued, but a debug log is not really what I'd call a leak (and genuinely leaking memory in Python takes some effort).
    - ComputerGuru 9y ago
sigmonsays 9y ago
One web request should not be served by one process at a time. This is a terrible software architecture which is just the fault of python. I love python, but it is not suitable for backend work at scale. Its simply too costly in terms of resources required. While its interesting to see improvements in the GC, it also makes equal if not more sense to migrate away from python. Just my 0.02$
- sametmax 9y ago
  > One web request should not be served by one process at a time It's doesn't have to be. You can use asyncio and deal with many requests at a time by one process. > I love python, but it is not suitable for backend work at scale. What's at scale ? Most projects I work on IRL, including in banks and administrations, never reach the level of scale this even remotely a problem. Too many people think they are Google.
  - ComputerGuru 9y ago
    Do you mean intranet bank apps? Or public facing sites?
    - sametmax 9y ago
      It doesn't matter. In France, there are 70 millions people. Remove old and young ones, and the ones that are not clients in your bank. Then spread the usage on the week, then on 24h, divide by the number of requests that actually hit your Python backend (so no cache, no static files, etc). You got what ? 1000 requests/s top ? That's nothing. Bank sites are not youtube. Besides, operations on your bank account don't even hit the Python backend, but a dedicated system. Usually some COBOL dinosaure they froze, wrapped into a Java service and exposed through a RESTish API so that the rest of their system can use it without ever having to touch it again.
      - figgis 9y ago
        And if you were really wanting to get as much performance as you wanted out of python there's plenty of options like epoll...
ComputerGuru 9y ago
Not directly pertaining to the article but in relation to the comments here, I feel it’s necesary to remind that the real problem with GC isn’t average processing time/max simultaneous requests but rather the large standard deviation resulting from GC pauses (and not to mention the poor cache friendliness, memory evictions, and more as a result - but those are taken into account in the the average run times so we can ignore them in this context). As a related metric, sometimes it’s not your average response times and scale that matters but how much headroom you’ve got for a sudden influx of traffic. If memory/pagefile thrashing and hard faults bring your system screeching to a halt, you won’t be happy no matter your scale when you can’t benefit from the eyeballs and pageclicks coming your way.
- jacobolus 9y ago
  Moreover, in a distributed systems context where many machines are involved in each task some 'rare' issues can end up happening a large proportion of tasks, and solitary problems can cascade and cause positive feedback loops that bring your whole system down.
- rbehrends 9y ago
  Eh, incremental garbage collection is technology that has been around since the 1980s/1990s. For purely sequential systems (which is what we're talking about here), it is pretty easy to make GC soft real-time [1]. We are talking about maximum pause times well below just the network latency between the client and the web server. Cache friendliness is a double-edged sword. There are plenty of use cases where a GC can be more cache-friendly than manual memory management, in particular if you have a generational GC with a bump allocator. [1] What makes life (considerably) harder is when you have multiple threads sharing a heap, but that's not at issue here.
  - CoolGuySteve 9y ago
    If it’s pretty easy, where are all these soft real-time garbage collectors?
    - 9y ago
kasabali 9y ago
I've recently read their previous post, now this one and I really enjoyed reading both of them. I appreciate their practical approach and solving their issues with minimal changes. On the other hand, I am disappointed that this change goes into upstream Python instead of properly solving the problem by making reference count implementation really CoW friendly so that all applications would benefit from it without the need for a careful use of a special function.
eeks 9y ago
Managing large memory objects, shared or not, is a very commmon problem with garbage-collected languages. The solution of “hiding” the region from the GC has been around for quite some time for languages such as Java, C#, and even OCaml. That being said, it’s a very enjoyable little write up.
lathiat 9y ago
Aaron Patterson recently talked about doing similar work for Ruby, with good technical details - he's quite good at explaning these kinds of issues for people less familiar with them. Explains how and why you want to be copy-on-write friendly and what that means for Ruby's GC - and the impacts of his work. I watched the RubyConf AU version: https://www.youtube.com/watch?v=nAEt36XNtAE&t=2482s https://www.youtube.com/watch?v=nAEt36XNtAE&t=2482s But looks like there was a version at Rubyconf as well that may or may not be better/different: https://www.youtube.com/watch?v=8Q7M513vewk https://www.youtube.com/watch?v=8Q7M513vewk

When exactly is this new GC behavior useful? When forking processes? Is it something we'd see used in the multiprocessing or concurrent modules?

bpicolo 9y ago
Memory is typically the scaling bottleneck for Python workers for web servers. One way to cut back drastically on this is to load as much as possible in startup and then fork for request serving processes. The problem is that python refcounting causes a lot of copy-on-writes for data that's really just used as read-only data. This change allows you to run more workers with less memory. Instagram has had a few good posts about how they've approached this problem, here's another: https://engineering.instagram.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172 https://engineering.instagram.com/dismissing-python-garbage-...
- sametmax 9y ago
  > Memory is typically the scaling bottleneck for Python workers for web servers. Wut ? When I have performances problems, my RAM is full with varnish cache, redis stored objects and postgres buffer. Python is like, low, low on the list. Again, most people are NOT instagram or Google. They have a very atypical load.
  - bpicolo 9y ago
    That's latency, not bandwidth. This is to help scale out concurrent request counts per box, rather than time to serve a single request (though obviously with request queuing, failure to serve requests fast enough makes concurrency a latency bottleneck). And sure, most people aren't instagram. This is a python-at-big-scale problem, when the cost benefits of fitting more requests onto fewer servers actually matter. One of the biggest reasons this actually makes a difference is that, in SoA, internal API scaling against your monolith can become expensive.
  - ComputerGuru 9y ago
    > Again, most people are NOT Instagram or Google And, again, can we stop with these pointless comparisons? Past 1 front end server the only cost that matters is how many requests your one node can handle and how much that one node costs. If you’re not a VPS and you have your frontend http cache correctly configured, then it doesn’t matter how much smaller than Google you are, comparisons are valid (although, of course, the fewer servers you have the more you can afford to spend on them; though you probably aren’t making as much money as IG/FB/Google either...).
    - sametmax 9y ago
      The only metrics is economic. The question is how much does those performance constrains cost you ?
ot 9y ago
Yes, when you fork all the pages in the address space are referenced, and copied when modified. However if you have a GC it will go around touching objects in the child process, and there's a good chance it will end up touching a few bytes in each page, nullifying the benefits of COW.
- rurban 9y ago
  Exactly. That's why you normally don't call refcounting a GC, as it writes to objects on referencing, and thereby destroys COW. With Perl it's even more stupid, they write to the const strings also, not just the refcount. And still call it COW. I call it write on assign, WOA. A normal GC leaves it alone and does not force fresh memory maps for these.
  - ot 9y ago
    It's not just refcounting, GCs also normally have some sort of bookkeeping that is updated in the background, for example a mark&sweep GC will need to mark the objects. See the comment about the object header in the OP.
    - rurban 9y ago
      This GC bookkeeping cannot be compared to the refcounting madness. A GC needs 1-2 bits during the GC, but does not disturb COW pages at all during run-time. It also has no word overhead, these 2 bits easily fit into every object. Also a GC compacts the live heap, which makes all upcoming accesses much faster. Well, not Mark & Sweep, but a compacting GC.
masklinn 9y ago
> When exactly is this new GC behavior useful? When forking processes? When forking a process and performing lots of allocations in the parent process (e.g. lots of imports & objects creation before forking) and doing so on a CoW-forking OS (but I'm guessing that's most modern POSIX systems).

IgorPartola 9y ago

A more general question: why is it so typical for a Python we worker to have its memory grow infinitely with each request processed? I don’t expect the kind of performance I can get out of C with its ability to only allocate memory on the stack or to pre-allocate all the buffers I’d need. And I have found a number of memory leaks in Python libraries and especially poorly written Python C modules over the years. But still I would expect something like a simple ideomatic Django application to not have its memory footprint grow idefinitely. Is this just life now? Are there any good tools out there for figuring out what part of the application keeps requesting objects that don’t get destroyed? Are corporations people?

sametmax 9y ago
> why is it so typical for a Python we worker to have its memory grow infinitely with each request processed? It's not. The cases where it happened to me during the last 10 years and dozens of web projects spanning on multiple frameworks were generally me doing something stupid: - using a mutable object as default value or in a class attribute - doing something in __del__ - keeping DEBUG=True for Django (which is very well known for causing memory leaks) > Is this just life now? Nope. I have big Django apps running (like 500k users/day streaming video sites) and it doesn't happen. > Are there any good tools out there for figuring out what part of the application keeps requesting objects that don’t get destroyed? Yes for course: http://tech.labs.oliverwyman.com/blog/2008/11/14/tracing-python-memory-leaks/ http://tech.labs.oliverwyman.com/blog/2008/11/14/tracing-pyt...
- ubernostrum 9y ago
  It's hard to call DEBUG in Django a "memory leak" when the "leak" is documented behavior -- when DEBUG=True, Django keeps a running per-connection log in memory of database queries issued. For, y'know, debugging :) This does cause memory use to increase with the number of queries issued, but a debug log is not really what I'd call a leak (and genuinely leaking memory in Python takes some effort).
  - ComputerGuru 9y ago

sigmonsays 9y ago

One web request should not be served by one process at a time. This is a terrible software architecture which is just the fault of python. I love python, but it is not suitable for backend work at scale. Its simply too costly in terms of resources required. While its interesting to see improvements in the GC, it also makes equal if not more sense to migrate away from python. Just my 0.02$

sametmax 9y ago
> One web request should not be served by one process at a time It's doesn't have to be. You can use asyncio and deal with many requests at a time by one process. > I love python, but it is not suitable for backend work at scale. What's at scale ? Most projects I work on IRL, including in banks and administrations, never reach the level of scale this even remotely a problem. Too many people think they are Google.
- ComputerGuru 9y ago
  Do you mean intranet bank apps? Or public facing sites?
  - sametmax 9y ago
    It doesn't matter. In France, there are 70 millions people. Remove old and young ones, and the ones that are not clients in your bank. Then spread the usage on the week, then on 24h, divide by the number of requests that actually hit your Python backend (so no cache, no static files, etc). You got what ? 1000 requests/s top ? That's nothing. Bank sites are not youtube. Besides, operations on your bank account don't even hit the Python backend, but a dedicated system. Usually some COBOL dinosaure they froze, wrapped into a Java service and exposed through a RESTish API so that the rest of their system can use it without ever having to touch it again.
    - figgis 9y ago
      And if you were really wanting to get as much performance as you wanted out of python there's plenty of options like epoll...

ComputerGuru 9y ago

Not directly pertaining to the article but in relation to the comments here, I feel it’s necesary to remind that the real problem with GC isn’t average processing time/max simultaneous requests but rather the large standard deviation resulting from GC pauses (and not to mention the poor cache friendliness, memory evictions, and more as a result - but those are taken into account in the the average run times so we can ignore them in this context). As a related metric, sometimes it’s not your average response times and scale that matters but how much headroom you’ve got for a sudden influx of traffic. If memory/pagefile thrashing and hard faults bring your system screeching to a halt, you won’t be happy no matter your scale when you can’t benefit from the eyeballs and pageclicks coming your way.

jacobolus 9y ago
Moreover, in a distributed systems context where many machines are involved in each task some 'rare' issues can end up happening a large proportion of tasks, and solitary problems can cascade and cause positive feedback loops that bring your whole system down.
rbehrends 9y ago
Eh, incremental garbage collection is technology that has been around since the 1980s/1990s. For purely sequential systems (which is what we're talking about here), it is pretty easy to make GC soft real-time [1]. We are talking about maximum pause times well below just the network latency between the client and the web server. Cache friendliness is a double-edged sword. There are plenty of use cases where a GC can be more cache-friendly than manual memory management, in particular if you have a generational GC with a bump allocator. [1] What makes life (considerably) harder is when you have multiple threads sharing a heap, but that's not at issue here.
- CoolGuySteve 9y ago
  If it’s pretty easy, where are all these soft real-time garbage collectors?
  - 9y ago

kasabali 9y ago

I've recently read their previous post, now this one and I really enjoyed reading both of them. I appreciate their practical approach and solving their issues with minimal changes. On the other hand, I am disappointed that this change goes into upstream Python instead of properly solving the problem by making reference count implementation really CoW friendly so that all applications would benefit from it without the need for a careful use of a special function.

eeks 9y ago

Managing large memory objects, shared or not, is a very commmon problem with garbage-collected languages. The solution of “hiding” the region from the GC has been around for quite some time for languages such as Java, C#, and even OCaml. That being said, it’s a very enjoyable little write up.

lathiat 9y ago

Aaron Patterson recently talked about doing similar work for Ruby, with good technical details - he's quite good at explaning these kinds of issues for people less familiar with them. Explains how and why you want to be copy-on-write friendly and what that means for Ruby's GC - and the impacts of his work. I watched the RubyConf AU version: https://www.youtube.com/watch?v=nAEt36XNtAE&t=2482s https://www.youtube.com/watch?v=nAEt36XNtAE&t=2482s But looks like there was a version at Rubyconf as well that may or may not be better/different: https://www.youtube.com/watch?v=8Q7M513vewk https://www.youtube.com/watch?v=8Q7M513vewk