8 ms·

Memory efficiency of parallel IO operations in Python

168 points by underyx 9y ago

blattimwind 9y ago
tl;dr If you want to parallelize[1] network I/O, use async. Otherwise, don't. [1] Technically: not parallelize, but overlap.
- deleted 9y ago
  [deleted]
- nullp0tr 9y ago
  Concurrent is the technical term:)
  - blattimwind 9y ago
    Overlapping is a technical term, too, just not from the Unix world.
- icegreentea2 9y ago
  Well, it's use async if you need the ~40% decrease in ram usage for your concurrent I/O access.
- xapata 9y ago
  Very few of my networking projects risk running out of RAM. I'm usually fine with spawning a new thread per connection.
konschubert 9y ago
Honest question. Can somebody explain to me why async IO is so important and why it is better than using the operating system scheduler? If process A is blocked because of IO, then then thing that needs to be done will need to wait for the IO anyways. Of course, in a server context, process A cannot handle new server requests while it is blocked. But luckily we can run more than one process, so process B will be free to pick it up. I will need to run a few more worker processes than there are CPU cores, but is there a problem with that? EDIT: I'm thinking now the problem is maybe that running more workers than there are cores will mean that the server accepts more concurrent connections than it can handle? If I use async code and run exacly as many workers as I have cores, the workers will never blocked. But then, I have the scenario where multiple async callbacks resolve in short sequence, but cannot be picked up by a worker because all workers are busy. So, in both scenarios (no async but more workers than cores VS async with as many workers as cores) it can happen that the server puts too much on its plate and accepts more than it can handle. I have a feeling that this is a fundamental problem that manifests itself differently in both paradigms, but exists notheless?
- QasimK 9y ago
  If we continue looking at memory usage as is the focus of the article, then I suspect that this multi-process model will have the highest memory usage of all the approaches. I’m under the impression that processes are quite heavyweight in resources compared to using threads or async. Also, you may only need the concurrency for a small portion of your code, and the other approaches may be simpler to code and maintain (I’m looking at async).
- marcosdumay 9y ago
  A thread requires at minimum a new stack segment. That is 1kb or something like that, but if you plan on accepting a lot of connections and keep them on hold, it adds up fast and quickly becomes the bottleneck of your server. A new process requires a new stack and a new heap segment, that later one is usually on the MB range. Besides, starting and stopping threads take some time. If you are serving small files, forking at their start means that your process will spend most of its time forking. And if you don't use an easy architecture that forks on every connection, it isn't much more difficult to go all the way and make your server fully asynchronous.
- lmm 9y ago
  If you flip the perspective and look at it from an async-first point of view, then OS-level processes are just a task implementation where every operation is a (potential) yield point and the state passed between them is the C stack. There are two flaws to this, one arguable, one less so: 1. Yield points are implicit rather than explicit, so their interaction with other effects is unpredictable. https://glyph.twistedmatrix.com/2014/02/unyielding.html https://glyph.twistedmatrix.com/2014/02/unyielding.html has a description of the problem; more generally think about the reputation that thread safety bugs have. (There are those who argue that the advantages of implicit pervasive yielding outweigh the disadvantages). 2. At every yield the processor has to swap out the full C stack (usually 4K or 8K). This is a slow operation ("context switch") and inefficient when often the only information that actually needed to be passed from one task to the next was a single integer (e.g. a socket ID, user ID, SQL query ID, etc.) or something similarly small. Whereas with userspace scheduling, a task switch only has to pass the actual task state that's needed for the task in question.
deleted 9y ago
[deleted]
scott_s 9y ago
> In my case, there was a ~40% speed increase compared to sequential processing. Once a code runs in parallel, the difference in speed performance between the parallel methods is very low. But they didn't actually present the total processing time for all of the methods - I assume all of the parallel methods were about 17 seconds? (Compared to the sequential baseline of 29 seconds.) And how were the threaded frameworks configured? How many threads were they told to use (or just the default?), how many threads can they use, and what kind of parallel hardware did they run on? This blog post presents the decision as one-dimensional; it claims all parallelization methods are the same, so the only dimension to choose on is memory efficiency. But I'm skeptical that all parallelization methods are the same, and the experimental design gives me no information on that front.
- nickcw 9y ago
  I had a squint at the ThreadPoolExecutor code on github and it uses the default parameters. Exactly how many threads you get depends on which version of python you are running: > Changed in version 3.5: If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor. Not knowing how many things are happening in parallel means it is difficult to draw conclusions from it.
  - ku3o 9y ago
    Thanks for pointing this out. I updated the article.
- ku3o 9y ago
  > it claims all parallelization methods are the same, so the only dimension to choose on is memory efficiency From a speed (overall script duration), they are very similar and the differences between them might be as well caused by a different state of a network. My main interest was how the methods perform regarding memory usage. That's why I focused on one dimension only.
trevman 9y ago
If you have a lot of RAM I highly recommend you look at Plasma + PyArrow
barrkel 9y ago
Title should refer to concurrent IO, not parallel IO.
- nine_k 9y ago
  On the interpreter side, most of the I/O read time is waiting for an OS buffer to fill with the right amount of data. Waiting happens to occur "in parallel" when using async I/O. I suppose that sending is similar: you pass the OS a buffer, and wait for completion. Sending of the data occurs while you wait. So, if you can parallelize waits, the OS could be doing strictly parallel I/O for you (e.g. via two network interfaces, or network and disk), even though your code is concurrent but not parallel, and you don't run two sync OS I/O calls in parallel.
  - zzzeek 9y ago
    > Waiting happens to occur "in parallel" when using async I/O. > and you don't run two sync OS I/O calls in parallel. you can run sync IO calls and wait for each in separate threads. the GIL is released for IO. the waiting is "in parallel" just as much with a threaded / blocking approach as with a non-blocking.
  - barrkel 9y ago
    Waiting for multiple concurrent things to complete isn't parallelism, in the on-topic jargon meaning of the term. Parallelism is things physically being executed simultaneously, increasing throughput via physically multiple resources. The multiple IOs you're waiting on might be executing in parallel if you have e.g. a RAID array or SAN in the loop, but it's an implementation detail and isn't what the article is about.
- zzzeek 9y ago
  if I write a web crawler and send off requests to 100 web servers, then handle results as they come in, those 100 web servers are working in parallel to get me my data back. So while I can't process the IO in parallel I am parallelizing my workload. Without parallelism of some form, your program would take the entire time the non-concurrent form takes.
wenc 9y ago
There is also a drop-in replacement for asyncio called uvloop [0]. It claims to be faster than asyncio, gevent, node.js, etc. and comparable to golang. [0] https://magic.io/blog/uvloop-blazing-fast-python-networking/ https://magic.io/blog/uvloop-blazing-fast-python-networking/
- jwandborg 9y ago
  More precisely, uvloop is an "asyncio.AbstractEventLoop" implementation using libuv. It can be used instead of the default asyncio.SelectorEventLoop[0] on Lunix(? not Windows at least: https://github.com/MagicStack/uvloop/issues/14 https://github.com/MagicStack/uvloop/issues/14). You will still, for better or worse, use the asyncio standard library when writing code running on uvloop. [0]: https://docs.python.org/3/library/asyncio-eventloops.html#available-event-loops https://docs.python.org/3/library/asyncio-eventloops.html#av...
azylman 9y ago
Something seems off here - they mention using 100 workers (a worker for every request). I would expect that to perform way more than 40% faster unless there's a ton of overhead in creating those workers.
xstartup 9y ago
I use gevent + gunicorn, it improved by API's latency and throughput by 10x.
yahyaheee 9y ago
Sounds like a good job for Go :)

168 points by underyx 9y ago

tl;dr If you want to parallelize[1] network I/O, use async. Otherwise, don't. [1] Technically: not parallelize, but overlap.

deleted 9y ago
[deleted]
nullp0tr 9y ago
Concurrent is the technical term:)
- blattimwind 9y ago
  Overlapping is a technical term, too, just not from the Unix world.
icegreentea2 9y ago
Well, it's use async if you need the ~40% decrease in ram usage for your concurrent I/O access.
xapata 9y ago
Very few of my networking projects risk running out of RAM. I'm usually fine with spawning a new thread per connection.

konschubert 9y ago

Honest question. Can somebody explain to me why async IO is so important and why it is better than using the operating system scheduler? If process A is blocked because of IO, then then thing that needs to be done will need to wait for the IO anyways. Of course, in a server context, process A cannot handle new server requests while it is blocked. But luckily we can run more than one process, so process B will be free to pick it up. I will need to run a few more worker processes than there are CPU cores, but is there a problem with that? EDIT: I'm thinking now the problem is maybe that running more workers than there are cores will mean that the server accepts more concurrent connections than it can handle? If I use async code and run exacly as many workers as I have cores, the workers will never blocked. But then, I have the scenario where multiple async callbacks resolve in short sequence, but cannot be picked up by a worker because all workers are busy. So, in both scenarios (no async but more workers than cores VS async with as many workers as cores) it can happen that the server puts too much on its plate and accepts more than it can handle. I have a feeling that this is a fundamental problem that manifests itself differently in both paradigms, but exists notheless?

QasimK 9y ago
If we continue looking at memory usage as is the focus of the article, then I suspect that this multi-process model will have the highest memory usage of all the approaches. I’m under the impression that processes are quite heavyweight in resources compared to using threads or async. Also, you may only need the concurrency for a small portion of your code, and the other approaches may be simpler to code and maintain (I’m looking at async).
marcosdumay 9y ago
A thread requires at minimum a new stack segment. That is 1kb or something like that, but if you plan on accepting a lot of connections and keep them on hold, it adds up fast and quickly becomes the bottleneck of your server. A new process requires a new stack and a new heap segment, that later one is usually on the MB range. Besides, starting and stopping threads take some time. If you are serving small files, forking at their start means that your process will spend most of its time forking. And if you don't use an easy architecture that forks on every connection, it isn't much more difficult to go all the way and make your server fully asynchronous.
lmm 9y ago
If you flip the perspective and look at it from an async-first point of view, then OS-level processes are just a task implementation where every operation is a (potential) yield point and the state passed between them is the C stack. There are two flaws to this, one arguable, one less so: 1. Yield points are implicit rather than explicit, so their interaction with other effects is unpredictable. https://glyph.twistedmatrix.com/2014/02/unyielding.html https://glyph.twistedmatrix.com/2014/02/unyielding.html has a description of the problem; more generally think about the reputation that thread safety bugs have. (There are those who argue that the advantages of implicit pervasive yielding outweigh the disadvantages). 2. At every yield the processor has to swap out the full C stack (usually 4K or 8K). This is a slow operation ("context switch") and inefficient when often the only information that actually needed to be passed from one task to the next was a single integer (e.g. a socket ID, user ID, SQL query ID, etc.) or something similarly small. Whereas with userspace scheduling, a task switch only has to pass the actual task state that's needed for the task in question.

deleted 9y ago

[deleted]

scott_s 9y ago

> In my case, there was a ~40% speed increase compared to sequential processing. Once a code runs in parallel, the difference in speed performance between the parallel methods is very low. But they didn't actually present the total processing time for all of the methods - I assume all of the parallel methods were about 17 seconds? (Compared to the sequential baseline of 29 seconds.) And how were the threaded frameworks configured? How many threads were they told to use (or just the default?), how many threads can they use, and what kind of parallel hardware did they run on? This blog post presents the decision as one-dimensional; it claims all parallelization methods are the same, so the only dimension to choose on is memory efficiency. But I'm skeptical that all parallelization methods are the same, and the experimental design gives me no information on that front.

nickcw 9y ago
I had a squint at the ThreadPoolExecutor code on github and it uses the default parameters. Exactly how many threads you get depends on which version of python you are running: > Changed in version 3.5: If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor. Not knowing how many things are happening in parallel means it is difficult to draw conclusions from it.
- ku3o 9y ago
  Thanks for pointing this out. I updated the article.
ku3o 9y ago
> it claims all parallelization methods are the same, so the only dimension to choose on is memory efficiency From a speed (overall script duration), they are very similar and the differences between them might be as well caused by a different state of a network. My main interest was how the methods perform regarding memory usage. That's why I focused on one dimension only.

trevman 9y ago

If you have a lot of RAM I highly recommend you look at Plasma + PyArrow

barrkel 9y ago

Title should refer to concurrent IO, not parallel IO.

nine_k 9y ago
On the interpreter side, most of the I/O read time is waiting for an OS buffer to fill with the right amount of data. Waiting happens to occur "in parallel" when using async I/O. I suppose that sending is similar: you pass the OS a buffer, and wait for completion. Sending of the data occurs while you wait. So, if you can parallelize waits, the OS could be doing strictly parallel I/O for you (e.g. via two network interfaces, or network and disk), even though your code is concurrent but not parallel, and you don't run two sync OS I/O calls in parallel.
- zzzeek 9y ago
  > Waiting happens to occur "in parallel" when using async I/O. > and you don't run two sync OS I/O calls in parallel. you can run sync IO calls and wait for each in separate threads. the GIL is released for IO. the waiting is "in parallel" just as much with a threaded / blocking approach as with a non-blocking.
- barrkel 9y ago
  Waiting for multiple concurrent things to complete isn't parallelism, in the on-topic jargon meaning of the term. Parallelism is things physically being executed simultaneously, increasing throughput via physically multiple resources. The multiple IOs you're waiting on might be executing in parallel if you have e.g. a RAID array or SAN in the loop, but it's an implementation detail and isn't what the article is about.
zzzeek 9y ago
if I write a web crawler and send off requests to 100 web servers, then handle results as they come in, those 100 web servers are working in parallel to get me my data back. So while I can't process the IO in parallel I am parallelizing my workload. Without parallelism of some form, your program would take the entire time the non-concurrent form takes.

wenc 9y ago

There is also a drop-in replacement for asyncio called uvloop [0]. It claims to be faster than asyncio, gevent, node.js, etc. and comparable to golang. [0] https://magic.io/blog/uvloop-blazing-fast-python-networking/ https://magic.io/blog/uvloop-blazing-fast-python-networking/

jwandborg 9y ago
More precisely, uvloop is an "asyncio.AbstractEventLoop" implementation using libuv. It can be used instead of the default asyncio.SelectorEventLoop[0] on Lunix(? not Windows at least: https://github.com/MagicStack/uvloop/issues/14 https://github.com/MagicStack/uvloop/issues/14). You will still, for better or worse, use the asyncio standard library when writing code running on uvloop. [0]: https://docs.python.org/3/library/asyncio-eventloops.html#available-event-loops https://docs.python.org/3/library/asyncio-eventloops.html#av...

azylman 9y ago

Something seems off here - they mention using 100 workers (a worker for every request). I would expect that to perform way more than 40% faster unless there's a ton of overhead in creating those workers.

xstartup 9y ago

I use gevent + gunicorn, it improved by API's latency and throughput by 10x.

yahyaheee 9y ago

Sounds like a good job for Go :)