33 ms·

How much memory do you need to run 1M concurrent tasks?

253 points by pkolaczk 3y ago

guenthert 3y ago
If those 'tasks' can be expressed as state machines, then I'd think log2(<number of states>) * 1e6 bits would be the lower limit.
- deleted 3y ago
  [deleted]
mkoubaa 3y ago
Is memory really the right resource bottleneck for this many tasks?
- phamilton 3y ago
  For a workload like websockets, where things are generally idle, this is a decent benchmark.
lostmsu 3y ago
In C# program you'd generally want to call .ConfigureAwait(false) on the task you are about to await if the continuation does not require to be completed on a special thread (like UI). P.S. the statement above is imprecise: https://devblogs.microsoft.com/dotnet/configureawait-faq/ https://devblogs.microsoft.com/dotnet/configureawait-faq/
brunoborges 3y ago
The Java tests (Loom in particular), and possibly other langauge runtimes too in this benchmark, can't be compared without looking at the runtime settings. JVM for example has a default heap size of 25% to 50%, depending on how much memory is available in the system. Given that the code is using ArrayList, there will be more need for bigger heap, due to array expansion. To eliminate that noise, the code should be changed to a pure Thread[] array with a fixed size based on numTasks. Once that is done, you may as well run up to 10_000 tasks (virtual threads) within a JVM that consumes no more than 32 MB of RAM (compared to 78MB as originally reported by OP). These were the best results I could find after playing a bit: # Run single task docker run --memory 26m --rm java-virtual-threads 1 # Run 10 tasks docker run --memory 26m --rm java-virtual-threads 10 # Run 100 tasks docker run --memory 26m --rm java-virtual-threads 100 # Run 1_000 tasks docker run --memory 28m --rm java-virtual-threads 1000 # Run 10_000 tasks docker run --memory 32m -e JAVA_TOOL_OPTIONS=-Xmx25m --rm java-virtual-threads 10000 # Run 100_000 tasks docker run --memory 200m -e JAVA_TOOL_OPTIONS=-Xmx175m --rm java-virtual-threads 100000 # Run 1_000_000 tasks docker run --memory 950m -e JAVA_TOOL_OPTIONS=-Xmx750m --rm java-virtual-threads 1000000
nilsmagnus 3y ago
I get very different results on the 1 mill tasks than the author here, so it would be interesting to know how the consumed memory is measured. My results is 565Mb for the 1 mill tasks, vs the authors 2.6Gb. My method is described in [1], feel free to point out what I am missing here. [1] https://gist.github.com/nilsmagnus/38b1b5f63237a2a63179428497031a0b https://gist.github.com/nilsmagnus/38b1b5f63237a2a6317942849...
znpy 3y ago
The go memory usage can probably be explained by the fact that go uses fairly large stacks for goroutines (according to https://go.dev/doc/go1.2#stack_size https://go.dev/doc/go1.2#stack_size it’s 8kb per goroutine, even though that’s old)
- pkolaczk 3y ago
  If it was true, the results would be much worse. The average from my measurements is a bit above 2 kB. This website says it is 2 kB: https://tpaschalis.me/goroutines-size/ https://tpaschalis.me/goroutines-size/
- amscanne 3y ago
  Stacks changed a lot in Go subsequently. I believe that in 1.2, segmented stacks were still being used. (They now are contiguous and copied on upsize.)
- ilyt 3y ago
  It also runs actually in parallel, unlike the Rust and JS ones
  - tialaramex 3y ago
    The Rust threaded solution obviously produces actual threads, which will run "actually in parallel" if that is technically possible on your system. The Rust tokio solution will use tokio's runtime which automatically spawns an appropriate amount of worker threads to parallelize your async tasks (by default it asks the OS how many CPUs there are and creates one such thread for each CPU). The async-std implementation is roughly similar.
  - kibwen 3y ago
    The Rust version is running in parallel. Tokio's runtime is parallel by default.
elteto 3y ago
I’m surprised at Elixir failing with 1M tasks. I’ve definitely done this sort of test with Erlang without an issue. Although I can’t really remember right now if I pushed it to 1M tasks.
- crazydoggers 3y ago
  Almost certainly a system cofiguration issue. I’m pretty sure the default process limit on BEAM is something like 200K. Additionally I don’t see mention of updating file handles or anything else on the operating system. Maybe I’m cynical but this is what happens if you rely on chatGPT to give you programming answers without understanding the fundamentals first.
- sneako 3y ago
  “The maximum number of simultaneously alive Erlang processes is by default 262,144. This limit can be configured at startup. For more information, see the +P command-line flag in the erl(1) manual page in ERTS.” https://www.erlang.org/doc/efficiency_guide/advanced.html#:~:text=The%20maximum%20number%20of%20simultaneously%20alive%20Erlang%20processes%20is%20by%20default%20262%2C144 https://www.erlang.org/doc/efficiency_guide/advanced.html#:~....
- fwlr 3y ago
  Unless I’m missing something in the writeup, it doesn’t mention how much RAM is on the system. A little odd to mention the CPU and the OS but not the RAM, for a benchmark that doesn’t care about execution speed but does care about memory usage. It makes figuring out that Elixir error of “system limit reached” a little harder to figure out. edit: oh apparently it’s got nothing to do with memory, 1M just happens to be higher than the default process limit for Erlang and thus Elixir.
- conradfr 3y ago
  Also someone needs to tell ChatGPT that its Elixir code has a bug ;)
bsenftner 3y ago
Why no C++ version?
- cynicalsecurity 3y ago
  Because C/C++ would kick ass, exposing the rest of the hipster technologies as lame.
  - Cyph0n 3y ago
    They would also kick memory safety in the ass, unless you’re the kind of dev who doesn’t make those kinds of silly mistakes
    - imran-iq 3y ago
      Don't be that guy. https://github.com/search?q=repo%3Atokio-rs%2Ftokio%20unsafe&type=code https://github.com/search?q=repo%3Atokio-rs%2Ftokio%20unsafe...
      - WinstonSmith84 3y ago
        Are you trying to say that you can write from scratch a code just as safe or better than one of the most peer reviewed crate of Rust? Congrats, the world has been waiting for you, but nonetheless, that's why commoners prefer to stick to what's safer for them
      - kouteiheika 3y ago
        `unsafe` doesn't mean that it's not safe, just that the compiler can't prove that it's safe so the human has to do it manually.
  - josephg 3y ago
    Pfft doubtful. Rust, C and C++ all have very similar performance. And apparently a javascript framework is currently beating all of those languages in the composite score on techempower, though that seems a bit weird to me: https://www.techempower.com/benchmarks/#section=data-r21&test=composite https://www.techempower.com/benchmarks/#section=data-r21&tes... C doesn't have a built in, optimized async execution engine of any sort. C with threads would lose to any of the async runtimes here (including javascript). C++ has async now. By all means, write up an async C++ example and compare it. I suspect a good C++ async executor would perform about as well as tokio, but it'd be nice to know for sure. I'd also be interested to see how hard it is to write that C++ code. One nice thing about Rust and Go is that they perform very well with naively written code. (Ie, the sort of thing chatgpt, or most of your coworkers would write.) Naive C would use kernel threads, so if we're comparing average C code to average Go code at this test, the average go code will end up faster. (This isn't always true of rust. I've seen plenty of rust code which uses Box everywhere that runs crazy slow as a result.)
neonsunset 3y ago
As of today (with .NET 7 and 8 preview) 120MB base footprint memory usage by .NET is indeed surprising. A few questions: - How are you exactly launching the workload? - How much memory does your system have? - Could you try running it with .NET 7 instead? The last two are especially relevant because in the recent versions each release got extra work to reduce memory footprint on Linux (mostly because it's an important metric for container deployments). Overall, 120MB sounds about right if you are launching a .NET 6 workload with some extra instrumentation attached while the runtime picks Server GC which is much more enthusiastic about allocating large segments of memory, especially if the system has large amount of RAM. As for Tasks - C# tasks are in many ways similar to Rust's futures (both are yielding-to-the-caller state machines) and take very little memory, which scales directly with the amount of variables that persist across "yield points" (which you have none).
- conradfr 3y ago
  I noticed he didn't use the latest version of each language. Maybe that's just what you get on Ubuntu 22.04 LTS?
  - neonsunset 3y ago
    Using .NET 6 is perfectly fine! It is the latest LTS after all, it's just that on machines with a lot of RAM the GC can indeed pre-allocate quite a bit of memory if, based on hardware configuration, it chooses to use Server GC. This, however, is being tuned with each subsequent release hence I'm curious to see the numbers on 7 on the author's hardware.
- DaiPlusPlus 3y ago
  > Overall, 120MB sounds about right if you are launching a .NET 6 workload with some extra instrumentation attached while the runtime picks Server GC which is much more enthusiastic about allocating large segments of memory, especially if the system has large amount of RAM. Also if you have lots of CPU cores then the CLR will (pre-)allocate up-to ~10MB/core for per-hardware-thread GC heaps. So my 16-core (32-thread) desktop will run every .NET program with a starting (OS-level) memory usage of ~200MB for a Hello, World program - though actual total GC usage will be < 1 MB.
deleted 3y ago
[deleted]
wwarner 3y ago
Would be nice to see throughput tested at this level. There is may be a good reason for Go to allocate more per thread if each thread is really doing something. Hat tip to Rust and C# tho!
- bsaul 3y ago
  I suspect 1 million concurrent task very likely becomes unmanageable in any real world scenario anyway. 1KB per task giving 1GB of total memory consumption, which means 100KB would be enough to raise memory limits on most systems. There may be scenarios such as network connexion handling where those connexion are kept idle 99% of the time, but you still need something to handle the very rare case of receiving something. However, is 1 million network connexion manageable from a kernel point of view anyway ? Wouldn't you reach system limits before that ?
  - toast0 3y ago
    > However, is 1 million network connexion manageable from a kernel point of view anyway ? Wouldn't you reach system limits before that ? You can raise all those defaults as long as you have sufficient memory. Yes, you need to be working with fairly idle connections, otherwise you're likely to run out of CPU. Real world examples include Chat, IMAP, push messaging, that sort of thing where having a live connection means a significant reduction in latency vs periodically opening a connection to poll, but at the same time, there's not a whole lot going on. I was at WhatsApp when we accidentally hit 2.8M connections on one machine [1] I can't remember if we ever got to 3M, but we ended up doing a lot more work per connection so didn't keep those connection counts for long. Kernel wise, as long as you're mostly inbound connections, everything is well oiled; I ran into problems initiating many thousands of outbound connections/second in an HAProxy use case, although I think FreeBSD 13 may have improved that bottleneck. I hear Linux also works, but I've not scaled it nearly as far. [1] http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf http://www.erlang-factory.com/upload/presentations/558/efsf2...
    - bsaul 3y ago
      Thanks for sharing. What’s your take on elixir consuming an outstanding amount of memory per task in OP benchmark ?
lordnacho 3y ago
I'm a big Rust fan, so I'm happy Tokio holds up so well. I find it to be one of the simplest and quickest ways to write task based programs. Gotta wonder whether you can challenge those numbers with c++ though. I suspect you can but with a lot of work, where Rust is pretty effective written naively. Of course the big issue with these comparisons is it's hard to be an expert in so many different languages. How do you know if what you're doing is the best way in each language?
- jeroenhd 3y ago
  I find the opposite to be true, Tokio is a fine library but it makes the whole program very complex. Perhaps it's because I generally don't write code for many short lived workloads, but I find using real threads to be a lot easier to work with (plus it saves time during compilation). Tokio is great at what it does, but I have to wonder how relevant these types of threads are when "fearless concurrency" is one of the token advantages of the language. I think this article would've been a lot better if it did some actual calculations in the spawned threads, rather than simply waiting. A smart enough compiler would see that the results of the wait calls, which usually don't come with any other side effects, are never used and optimize them out entirely.
jasonwatkinspdx 3y ago
> "All programs were launched using the release mode if available. Other options were left default." So it's likely these numbers simply reflect choices in the default configuration, not the ultimate limit with tuning. I'm starting to dislike these low effort ChatGPT articles. Getting an answer from it is not the same thing as actually learning what's going on, but readers will walk away with the answers uncritically.
- vasco 3y ago
  > I'm starting to dislike these low effort ChatGPT articles. The next few decades are going to be tough for you, I think. Brace yourself!
- dceddia 3y ago
  Lately I’m seeing a lot of this baked-in assumption that ChatGPT writes good/idiomatic/performant code, as if the code it generates is ~The Answer~. I guess it’s the modern version of copy/pasting from Stack Overflow. The code I’ve had it write isn’t universally terrible but it almost always has some low-hanging performance fruit (in JS for instance, it seems to love spreading arrays). I’m not sure how much I trust it to write good benchmark code!
  - kubb 3y ago
    Does spreading arrays come with a performance penalty?
    - spicebox 3y ago
      Spreading an array requires looping over all the array elements which is O(n) compared to something like Array.push() which is just 0(1), although spreading has other benefits like copying rather than mutating
    - dexwiz 3y ago
      Spreading allocates a new array. Most things you can do with spread you can do with normal functions that either just directly access an entry or mutate an array directly. [first, …rest] is nice, but will always be slower than direct access.
withinboredom 3y ago
Haha, I had to try a PHP build with the parallel extension[1] with some rather simple code[2]: - single job: 27.8 mb (respectable) - 10k: 8.3 GB! At this point, I gave up. I'm 99% sure trying to go to 1 million will crash my computer. [1]: https://www.php.net/manual/en/intro.parallel.php https://www.php.net/manual/en/intro.parallel.php [2]: https://gist.github.com/withinboredom/8256055709c0e6a8272b3a20e93cc140 https://gist.github.com/withinboredom/8256055709c0e6a8272b3a...
- withinboredom 3y ago
  I should add that the good ole' fork approach[1] only uses 40.7mb for 10k, 40.7mb for 100k, and 40.7mb for 1 million. Got to love COW memory. :) [1]: https://gist.github.com/withinboredom/c811f3c433904235a5f19602dccbbf01 https://gist.github.com/withinboredom/c811f3c433904235a5f196...
Traubenfuchs 3y ago
The Java tests should be optimized by making the ArrayList a fixed size. All this resizing of the huge array backing it and the potential final very high amount of empty backing array slots probably makes the Java result worse than it could be. I wonder if the parallel stream API with a virtual thread pool would make it better or worse.
- tialaramex 3y ago
  The ArrayList is just a growable array and is presumably using an exponential growth strategy so it's equivalent to what several of the others are doing. You could indeed tell Java how big the ArrayList will grow in advance, but you could for example likewise do that with the Rust Vec by asking for Vec::with_capacity(num_tasks) instead of Vec::new(). Since this is amortized exponential growth it's likely making negligible difference, although it is good style to tell data structures what you know about their size early. The Go approach doesn't appear to pay this penalty, it's just incrementing a (presumably atomic?) counter, although I have no idea where these tasks actually "live" until they decrement the counter again - but most of the others are likewise using a growable array.
  - JB_Dev 3y ago
    It looks like most of the languages are doing the same thing w.r.t using a 'List'-style implementation that re-sizes/copies it's internal buffer when it runs out of space. One thing I wonder though is whether the managed languages pay a bigger penalty on the memory usage than the unmanaged languages due to this. In the unmanaged languages, those intermediate buffers should be freed immediately during the re-size operation. But in the managed languages, they might stick around until the next GC. So the peak memory would be higher. In this example the runtime overhead is going to represent more of the memory usage anyway but it quickly isn't negligible. E.g. a list of size 1M is going to be ~8MB (64 bit addresses). So even if the array is re-sized from 1->2->4->8->...->~500K, the end result is not going to be worse than 2X the size.
- tester756 3y ago
  C#'s list works the same way
deleted 3y ago
[deleted]
themerone 3y ago
This is not a comparison of 1 million concurrent tasks. For a bunch of these tests, it is measuring 1 million tasks multiplexed onto a smaller number of os threads. In many implementations the cost of concurrency is going to be small enough that this test is really a measure of the stack size of a million functions.
- creatonez 3y ago
  There is sometimes a distinction made between concurrency and parallelism, where "concurrency" just means the programming interface looks parallel but is allowed to use out-of-order task-switching under the hood, and "parallelism" is real-deal usage of CPU cores. Both ways of looking at things are useful in different situations.
  - themerone 3y ago
    That is a good point, but parallelism degrades to concurrency when you run out of CPU cores.
wongarsu 3y ago
I like the general idea of the test. Things I would have liked to see (apart from basic system tweaks like allowing more file handles) are a performance comparison (how much CPU time is spent) and maybe a comparison between operating systems (this might highlight a lot of windows-linux differences, like Windows not committing memory - better have a large swap file to handle stacks of 100k real threads - or windows having a very different scheduler)
kevingadd 3y ago
It would be interesting to see how much the C# memory usage would go down if you allocated a single Task.Delay at the start and had all the tasks wait for it. It would probably result in it finishing in 10sec instead of climbing up to 12, while still running 1 million tasks. I'm not sure how you would do this in other runtimes, and there is still a risk of new problems cropping up. 1 million Task.Delays means 1 million queued waits in the sleep machinery, which could run into some bad scaling and overhead. But then 1 million tasks waiting on a single awaitable probably hits overhead in other subsystems. My guess would be that 1 million waits is much cheaper than 1 million sleeps, because sleeps have to be sorted by due time while the list of tasks waiting for completion just needs to be sequential. Entries in the sleep queue are also going to be bigger (at minimum, 4-8 bytes for due time + 4-8 bytes per completion callback) while entries in a completion queue would just be 4-8 bytes for the completion callback. Task.WhenAll also has potentially worse behavior than the other apps if you care about memory, since it will need to allocate a big array of 1 million Tasks in order to pass it in. Many of the other programs instead loop over the tasks, which eliminates that overhead.
- JB_Dev 3y ago
  The other thing I noticed is that they are wrapping every Task.Delay(…) inside a Task.Run(…) call. That should not be necessary. The inner task will yield back to the called when it hits the delay call anyway. There is no need to queue it with an immediate yield as if it is background or cpu-bound work. The Run() call might be creating a more complex state machine and using more memory than necessary (2 tasks vs just 1 for each delay). The memory usage might be even lower if the Run() call is dropped.
  - dodyg 3y ago
    Yup - this would do the job faster and better. Task task = Task.Delay(TimeSpan.FromSeconds(1)); tasks.Add(task);
    - JB_Dev 3y ago
      I was sufficiently curious and just went and tested this using BenchmarkDotNet. The example code is here (https://github.com/J-Bax/CSharpBenchmarkExamples https://github.com/J-Bax/CSharpBenchmarkExamples). The difference is quite significant. (1) With the authors code (using Task.Run), I get ~428MB of allocations. (2) Dropping the unnecessary Task.Run(...), I get ~183MB of allocations. (3) Doing (2) and waiting N times on the same delay, I get ~39MB of allocations. This was all using .NET 6 too. .NET 7 or the 8 preview might be even better since they are working hard on performance in recent releases. So even looking at just (2), that puts .NET on par with the rust library.
dceddia 3y ago
Tokio seems oddly good compared to the others. Is that because it’s using a thread pool instead of creating something new for every task? I wonder how it would fare if the tasks were more intense, and how the time-to-completion compares between these approaches.
- bombela 3y ago
  async Rust functions are compiled down to a finite state machine. A stackless coroutine I think it can be called. I suspect that C#/.NET does something similar though I am just speculating here. The difference between async-std and tokio would then be explained by a lower overhead runtime per task in tokio.
- pornel 3y ago
  All the async and green thread runtimes are using thread pools by definition. Avoiding spawning as many OS threads as program's tasks is their main job. This benchmark wasn't very rigorous, so maybe it has overlooked something, but the result seems plausible. Tokio is a good, mature runtime. Tokio-based frameworks are near top of TechEmpower benchmarks. Rust's Futures were designed from the start with efficiency in mind. Unlike most async runtimes where chaining of multiple async calls creates new heap allocations, Rust flattens the entire call tree and all await points into a single struct representing a state machine. This allows the runtime to use exactly one heap allocation per task, even if the task is pretty complex.
cyberax 3y ago
It's not really fair to compare async/await systems with Go and Java. Go and Java with lightweight threads provide a full-blown system, without any limitations. You don't have to "color" your code into async and blocking functions, everything "just works". Go can even interrupt tight inner loops with async pre-emption. The downside is that Go needs around 2k of stack minimum for each goroutine. Java is similar, but has a lower per-thread constant. The upside is that Go can be _much_ faster due to these contiguous stacks. With async/await each yield point is basically isomorphic to a segmented stack segment. I did a quick benchmark to show this: https://blog.alex.net/benchmarking-go-and-c-async-await https://blog.alex.net/benchmarking-go-and-c-async-await
- jimsimmons 3y ago
  None of what you say is a gotcha. An ignorant programmer stands only to benefit from Go
- djsavvy 3y ago
  In your Go benchmark code, how is the calculation actually being done across threads? I don't see the `go` keyword anywhere in the source code.
  - h4x0rr 3y ago
    I'm wondering that as well. The c# code seems really unrealistic though, using a lot of tasks for CPU bound work. A fair comparison would at least need to involve some degree of IO. Does go maybe automatically parallelize everythinh it can? That would be one potential answer to the initial question
  - SmooL 3y ago
    Yeah this isn't using goroutines at all? I don't see how this is a good comparison of goroutines vs coroutines.
  - cyberax 3y ago
    But that's the point I'm trying to show. In Go code all calls are the same. There is no sync/async distinction. In contrast, in C# (or any other similar system) async calls are _expensive_ compared to regular function calls.
asabil 3y ago
The maximum number of simultaneously alive Erlang processes is by default 262,144. This needs to be raised to avoid the `system_limit` error.
whateveracct 3y ago
The title is asking a very quantifiable question. And I was hoping the article would be a summary of expertise and deep dives the author took into various language's runtimes. But then I read > With a little help from ChatGPT And sure enough the results & prose end up pretty much worthless. Because the author didn't do any actual critical thinking or technical work. (For instance, there's nothing more to "Go's memory usage ballooned at 1M tasks" beyond the observation. Why not? Just dig in there it's all documented you just have to read some stuff.)
- lexandstuff 3y ago
  Do you mind sharing why Go ballooned to 1M?
  - pkolaczk 3y ago
    Because goroutine stacks start at minimum 2 kB. But it is not obvious if this is the only factor that contributes to memory usage so I believe experimenting to verify theory is still quite valuable.
- faizshah 3y ago
  It’s also really strange using python and node in an article about concurrency without mentioning the GIL (python) or node single threading.
  - whateveracct 3y ago
    That's what happens when you write a blog about a technical topic you don't know much of anything about I guess
  - Too 3y ago
    The whole idea of async functions is to run tasks concurrently and with simple syntax on a single thread. Hiding the actual concurrency under the runtime, only when it is waiting for IO. It doesn’t matter that node is single threaded, nor does the GIL have any impact on single thread. It’s once you start doing cpu heavy work inside the tasks this will hit you.
malkia 3y ago
What's the point of running 1M concurrent tasks? I don't see a good reason tbh.
- pestatije 3y ago
  website with that many requests
  - pornel 3y ago
    Not for a typical website. Even if you're getting a massive amount of traffic, like a million requests per second, you'd want the request to finish as quickly as possible, so they never pile up to be 1M concurrent tasks. If the tasks have real work to do and can't finish quickly, the having 1M in-flight tasks compete for orders of magnitude fewer CPU cores may not be a good idea — it will waste lots of memory for their state and it will be hard to keep latencies reasonably even. You'd probably use a simpler queue and/or load-balance to more machines. A scenario with this many tasks spawned is more realistic if you have mostly idle connections. For example, a massive chat server with tons of websockets open.
    - Seattle3503 3y ago
      You might not have a choice. How should a load balancer be implemented? It has to keep those connections alive while the underlying services fulfill the request.
  - malkia 3y ago
    This is bound to problems - slighest change in your stack - language, runtime, OS would affect that so much that it's completely pointless to rely on measures taken and apply them later (that to be said, this is perfectly valid choice when you are locked in - like in the embedded or game console world, but rarely that's the case outside) For example - https://go.dev/doc/go1.3#stacks https://go.dev/doc/go1.3#stacks https://agis.io/post/contiguous-stacks-golang/ https://agis.io/post/contiguous-stacks-golang/ So your measurements in go 1.2 would probably differed in go 1.3 (and this is just one example, in only one of the possible axises that have changed). It's not like you own the data structure there to control it.
d0mine 3y ago
The results can be ignored. You can't just copy-paste ChatGPT without even a sanity check e.g., Python code is wrong: await sleep(10) sleeps 10+ seconds and returns None but the intent is to pass a coroutine to create_task() (await should be removed at the very least)
- pkolaczk 3y ago
  Ok, this one is funny. Actually ChatGPT got it right and there was a create_task before, but right before committing I "inlined it" and introduced a bug. Thanks for letting me know. Will double check if it didn't messed up the results. And BTW - I'm very far from copy pasting from chat gpt. ChatGPT helped with this code, but often required many iterations and also some manual cleanup.
- mrj 3y ago
  Yeah Python got done dirty here. I tried a million tasks and I saw a peak of about 166MB. I'm still getting used to async in Python, but this seems to work fine though? import asyncio async def do_task(): await asyncio.sleep(10) async def main(num_tasks: int): tasks = [] for task_id in range(num_tasks): task = asyncio.create_task(do_task()) tasks.append(task) print("spawned tasks") await asyncio.gather(*tasks) if __name__ == "__main__": asyncio.run(main(1_000_000))
  - pkolaczk 3y ago
    Thank you! I fixed the code and updated the benchmarks. However, your 166 MB doesn't match my testing (I'm getting about 240M at 100k tasks) and 10x more at 1M. Are you sure you haven't misread your RSS?
    - mrj 3y ago
      Nope, that's what it is for me but you're using an older Python. 3.11 has a lot of improvements and not sure why you wouldn't use the latest?
throwawaaarrgh 3y ago
I see C is missing. If you're going to do a language comparison, might as well include the one language we know will have the best performance and lowest memory
- gigel82 3y ago
  There is no light-weight threading concept in standard C/C++, and the cost of one OS thread is at least 1Mb (stack) + 4Kb (tls) reserved memory (at least by default, on modern Windows), so for 1m threads that will be 1Tb of RAM :)
  - shrimp_emoji 3y ago
    How is your compiler addressing bits like this?
  - tialaramex 3y ago
    C++ 20 has async/await (but of course named co_async and co_await, because C++ isn't the sort of language where you're allowed nice things, the beatings will continue until morale improves) However AIUI C++ 20 doesn't actually supply an executor out of the box, so you would need to choose what to do here as for Rust, where they picked both tokio and async-std and you can see they have different performance. C does only have threads, but you could presumably pull off the same trick in C that Rust does, to get Linux to give you the bare minimum actual resident memory for your thread and you needn't care about the notional "real" size of the thread since we're 64-bit and address space is basically free.
    - foderking 3y ago
      > (but of course named co_async and co_await, because C++ isn't the sort of language where you're allowed nice things, the beatings will continue until morale improves) LMAO
      - gpderetta 3y ago
        co_<whatever> has become a meme in the C++ community.
matklad 3y ago
Benchmarks for threads might slightly underestimate memory usage. For threads, some amount of memory is spent by the kernel (eg, memory maps themselves) and won’t show as maxrss. > a high number of concurrent tasks can consume a significant amount of memory, So Go needs 3gigs for a million tasks. Relatively speaking, for an app that does a million concurrent somethings, 3gb is probably peanuts. It seems that memory isn’t the limiting factor here.
beders 3y ago
Another benchmark of the "I don't know what I'm actually measuring" flavor. I don't even know where to start: from the lack of env settings, system settings, compiler settings to the fact that no concurrent work was actually done, this is one giant flawed benchmark. The only lesson here should be: Don't take any of this seriously.
- tester756 3y ago
  >from the lack of env settings, system settings, compiler settings compiler settings?? I bet he used defaults everywhere
- pkolaczk 3y ago
  Just read the text. It lists versions and states it was using defaults in the release mode. Most of those compilers don't have any fancy options actually. E.g go is just `go build`, Python is just `python3 ...`. Rust options are in the project definition. Anyway, I'll add those command line invocations if it causes so much controversy.
deleted 3y ago
[deleted]
thethirdone 3y ago
Surely a Sufficiently Smart Compiler could optimize all of these to only take the a constant amount of memory (and less time). I wonder if there are any correctness constraints that would actually prevent such optimizations. For example, the compiler might be required to execute the exact same system calls as an unoptimized program. It seems quite reasonable that the programs using system threads could not be optimized significantly under such a constraint. I would expect the larger limit that prevents compilers from optimizing this is the large amount of code that defines the semantics of the lightweight threads. For example, I would expect the observable / required behaviors of tokio to not prevent a SCC from optimizing it, but the massive amount of Rust code that defines the lightweight thread semantics cannot be reasonably optimized. Which programming language / framework is the closest to actually being optimized to like < 4 byte / task? I would make an uneducated guess of Elixir or Go.
- sb8244 3y ago
  Each BEAM process comes with its own stack / heap space. This uses a minimum of 326 words of memory (https://www.erlang.org/doc/efficiency_guide/processes.html#creating-an-erlang-process https://www.erlang.org/doc/efficiency_guide/processes.html#c...)
bastawhiz 3y ago
Understanding full well that these are pretty artificial results, I'm surprised at just how well Node and Python faired against Go and Java. For all the default assumptions around languages that folks have, they've certainly held their own here.
- josephg 3y ago
  As others have mentioned, the results come from the difference between async concurrency vs "lightweight threads" (in Go and Java). In the nodejs test, there's very little javascript code being actually executed here. Whats really being tested is nodejs's event queue (libuv), which is written in C. And its being compared to Go's coroutine system - which is also in C. But, if I recall correctly, Go's threading model includes a preemptive scheduler (which is very complex) and it allocates a stack per goroutine. This doesn't test if Javascript or Go code is faster / more efficient. There's almost no javascript or go code being executed here at all. This is testing the execution engines of both languages. Go's execution engine has more features, and as we can see those features have a cost at runtime.
davidgrenier 3y ago
For the sake of pitching an analog to go's concurrency model. Hopac is a concurrency library for F# (partly written in C#) delivering an informal implementation of CSP (similar to Go's goroutine and Clojure Core.async). Since the job does nothing but wait for 10 seconds, the code is: seq{1..1000000} |> Seq.Con.iterJobIgnore (fun _ -> timeOutMillis 10000) |> run I'm not sure how to calculate total memory usage but instrumenting the following way: GC.Collect() GC.Collect() GC.WaitForPendingFinalizers() GC.TryStartNoGCRegion 3000000000L let before = GC.GetTotalMemory false seq{1..1000000} |> Seq.Con.iterJobIgnore (fun _ -> timeOutMillis 10000) |> run GC.GetTotalMemory false - before |> printfn "%d" GC.EndNoGCRegion() Gives the following output, I'm reading 144MB for 1M lightweight thread (in the REPL) unless I did something dumb: 144449024 Real: 00:00:10.902, CPU: 00:00:18.570, GC gen0: 3, gen1: 3, gen2: 3 val before: int64 = 53902280L
- davidgrenier 3y ago
  This is on .net core 7.0.5, Debian 11, Server GC (which is expected by Hopac). I removed all GC code and ran it as an executable using @compumike's `ps ax -o pid,rss,args' command. I think the only one that triggered a GC was the execution with 10M concurrent jobs. N Real(s) User(s) RSS(kiB) RSS(MiB) ================================================== 1 10.08 00.04 42660 41 1000 10.07 00.06 40736 39 10000 10.09 00.20 44176 43 100000 10.15 01.85 58532 57 1000000 10.88 18.18 182024 177 10000000 22.28 4m04.84 1452224 1418
deleted 3y ago
[deleted]
Seattle3503 3y ago
Rust is going to do well on toy examples. This is because in a naive implementation, for each thread, Rust statically allocates enough space for the largest possible stack the thread will need. On a toy example this will be quite small. In real code, some branches will have large futures that will bloat the size of every thread, even if the thread doesn't take the bloated branch. You can mitigate this issue by heap allocating large futures on cold paths, but this requires insight and tuning. Some runtimes can handle this for you, but they come with other performance tradeoffs.
- pkolaczk 3y ago
  In real code I saw Rust beating Go and Java based code by 10x-50x in memory use. But too complex and too many factors to analyze, so not a material for a blog post. Actually this tiny benchmark favors GC based runtimes because it just sleeps - so doesnt actually allocate much on the heap.
augusto-moura 3y ago
The nodejs code doesn't actually work in parallel. The code being tested is only creating new timeouts but it is working on a single V8 thread. It has a clock thread on the C++ part of the engine and when it timeouts it pushes the continuations on the event loop, and that is it
- h1fra 3y ago
  Came here to say that, not sure if it actually would improve the test but at least it would be correct. Also, Node 12 was released 4 years ago.
deleted 3y ago
[deleted]
sakras 3y ago
> Notably, at 1 million tasks, I observed that the overhead of launching tasks became evident, and most programs required more than 12 seconds to complete. I didn’t quite get this part. Even if your runtime has no overhead, the fastest way you can execute 1 million 10-second tasks is 10 * 1 million / num_cores, right? Why would these million tasks only take 12 sec?
- josephg 3y ago
  Because the tasks are all running concurrently. (They're all running at the same time).
- hamburglar 3y ago
  You can “run” more threads than you have cores when they are all inside sleep()
pdimitar 3y ago
To the author: Elixir (well, Erlang's BEAM VM actually) can be configured to allow more than the default 256k tasks (262_144). I'd do that and post the numbers for 1M tasks.
compumike 3y ago
Crystal 1.8.2 https://crystal-lang.org/ https://crystal-lang.org/: n = ARGV.first.to_i32 finished = Channel(Nil).new n.times do spawn do sleep(10.seconds) finished.send(nil) end end n.times { finished.receive } Memory usage determined with `ps ax -o pid,rss,args`. Runtime measured with `/usr/bin/time -v`. (Had to run `sudo sysctl vm.max_map_count=2500000` to get the final two rows.) Results: N Runtime(s) RSS(kiB) RSS(MiB) ======================================= 1 10.01 1792 2 1000 10.01 6016 6 10000 10.11 45184 44 100000 11.00 435840 426 1000000 20.25 4336768 4235 "System time" rather than "user time" is the majority (7.94s system time, 2.83s user time in the 20.25s wall time run). Is this pointing to memory allocations? Crystal Fiber docs https://crystal-lang.org/api/1.8.2/Fiber.html https://crystal-lang.org/api/1.8.2/Fiber.html says "A Fiber has a stack size of 8 MiB which is usually also assigned to an operating system thread. But only 4KiB are actually allocated at first so the memory footprint is very small." -- and perhaps unsurprisingly, 4KiB times 1000000 is approximately 4 GiB. Nice when the math works out like that :) To me this is useful as a baseline for something like a websocket server, with some number of idle connections, each Websocket connection mapped to a Fiber that is waiting for I/O activity but mostly idle.
benatkin 3y ago
World's largest compact car. Yawn.
Aardwolf 3y ago
> popular languages like Rust, Go, Java, C#, Python, Node.js and Elixir. Oof, has C/C++ become that unpopular? It would be a useful comparison as low level system language anyway.
lbriner 3y ago
Lots of pedants like to dump on these sorts of articles but there are enough interesting things to see, even in a relatively trivial comparison like this. It is interesting that some systems allocate more memory up-front but this doesn't grow much per-task. It is interesting that in the form the app was written the Go program doesn't appear to scale as well. Sure there are possibly reasons for it so anyone with more knowledge of Go could attempt to write a more "correct" program and ask to rerun it. It is interesting that with the defaults, things behave as they do, again, someone who cares more about "it's just that the defaults are different" is welcome to investigate that and ask "why" some program defaults might lead to better performance. It is interesting that as-in most comparisons, the languages/frameworks don't neatly fit into Stereotypes. It is interesting that most of them probably perform well enough for 99% of applications we might write. It seems like most people still don't appreicate that YMMV and that trade-offs are always made. Just add it to your stash of brain juice that helps you become a better engineer and stop complaining.
hdjjhhvvhga 3y ago
It's nice to see but frankly speaking with current memory prices all I care is speed, not so much memory consumption, unless I have very specific requirements.
Alifatisk 3y ago
I wonder how Ruby performs in this benchmark
Patrickmi 3y ago
Let’s understand that Go’s runtime and scheduler does a lot, and there’s not enough information to justify such benchmarks
sigmonsays 3y ago
this is a terrible benchmark. You're literally doing nothing per task. There is so much left up to the implementation details that you are not benching anything.
fulafel 3y ago
Would be interesting to hear what problems was hit trying to run 100k threads. I remember testing out a million threads on an by now older desktop and just had to do a couple of sysctl setting changes that were easy to find. Not sure if this is what I used but the settings sound familiar: https://www.baeldung.com/linux/max-threads-per-process https://www.baeldung.com/linux/max-threads-per-process Also would be good to specify how the memory as measured (eg to be able to verify they are actually measuring physical memory usage).
pajko 3y ago
Using thread pools/connection pools could help a lot both in performance (less time to create threads) and memory use (less fragmentation + less memory to garbage collect after short-lived allocations). There's a guide somewhere about using connection pools, multiprocessing and async I/O vs. pthreads, showing that the most obvious solution to the concurrency implementation does not necessarily has the best performance, but couldn't find it right now. EDIT: polls->pools typo
Sammi 3y ago
Some of these tests are doing concurrency. Some are doing parallelism. Concurrent meaning multiple tasks run within a given stretch of time. Parallel meaning multiple tasks run at a given point of time. These are not the same. So the tests are not testing the same thing.

253 points by pkolaczk 3y ago

If those 'tasks' can be expressed as state machines, then I'd think log2(<number of states>) * 1e6 bits would be the lower limit.

deleted 3y ago
[deleted]

mkoubaa 3y ago

Is memory really the right resource bottleneck for this many tasks?

phamilton 3y ago
For a workload like websockets, where things are generally idle, this is a decent benchmark.

lostmsu 3y ago

In C# program you'd generally want to call .ConfigureAwait(false) on the task you are about to await if the continuation does not require to be completed on a special thread (like UI). P.S. the statement above is imprecise: https://devblogs.microsoft.com/dotnet/configureawait-faq/ https://devblogs.microsoft.com/dotnet/configureawait-faq/

brunoborges 3y ago

The Java tests (Loom in particular), and possibly other langauge runtimes too in this benchmark, can't be compared without looking at the runtime settings. JVM for example has a default heap size of 25% to 50%, depending on how much memory is available in the system. Given that the code is using ArrayList, there will be more need for bigger heap, due to array expansion. To eliminate that noise, the code should be changed to a pure Thread[] array with a fixed size based on numTasks. Once that is done, you may as well run up to 10_000 tasks (virtual threads) within a JVM that consumes no more than 32 MB of RAM (compared to 78MB as originally reported by OP). These were the best results I could find after playing a bit: # Run single task docker run --memory 26m --rm java-virtual-threads 1 # Run 10 tasks docker run --memory 26m --rm java-virtual-threads 10 # Run 100 tasks docker run --memory 26m --rm java-virtual-threads 100 # Run 1_000 tasks docker run --memory 28m --rm java-virtual-threads 1000 # Run 10_000 tasks docker run --memory 32m -e JAVA_TOOL_OPTIONS=-Xmx25m --rm java-virtual-threads 10000 # Run 100_000 tasks docker run --memory 200m -e JAVA_TOOL_OPTIONS=-Xmx175m --rm java-virtual-threads 100000 # Run 1_000_000 tasks docker run --memory 950m -e JAVA_TOOL_OPTIONS=-Xmx750m --rm java-virtual-threads 1000000

nilsmagnus 3y ago

I get very different results on the 1 mill tasks than the author here, so it would be interesting to know how the consumed memory is measured. My results is 565Mb for the 1 mill tasks, vs the authors 2.6Gb. My method is described in [1], feel free to point out what I am missing here. [1] https://gist.github.com/nilsmagnus/38b1b5f63237a2a63179428497031a0b https://gist.github.com/nilsmagnus/38b1b5f63237a2a6317942849...

znpy 3y ago

The go memory usage can probably be explained by the fact that go uses fairly large stacks for goroutines (according to https://go.dev/doc/go1.2#stack_size https://go.dev/doc/go1.2#stack_size it’s 8kb per goroutine, even though that’s old)

pkolaczk 3y ago
If it was true, the results would be much worse. The average from my measurements is a bit above 2 kB. This website says it is 2 kB: https://tpaschalis.me/goroutines-size/ https://tpaschalis.me/goroutines-size/
amscanne 3y ago
Stacks changed a lot in Go subsequently. I believe that in 1.2, segmented stacks were still being used. (They now are contiguous and copied on upsize.)
ilyt 3y ago
It also runs actually in parallel, unlike the Rust and JS ones
- tialaramex 3y ago
  The Rust threaded solution obviously produces actual threads, which will run "actually in parallel" if that is technically possible on your system. The Rust tokio solution will use tokio's runtime which automatically spawns an appropriate amount of worker threads to parallelize your async tasks (by default it asks the OS how many CPUs there are and creates one such thread for each CPU). The async-std implementation is roughly similar.
- kibwen 3y ago
  The Rust version is running in parallel. Tokio's runtime is parallel by default.

elteto 3y ago

I’m surprised at Elixir failing with 1M tasks. I’ve definitely done this sort of test with Erlang without an issue. Although I can’t really remember right now if I pushed it to 1M tasks.

crazydoggers 3y ago
Almost certainly a system cofiguration issue. I’m pretty sure the default process limit on BEAM is something like 200K. Additionally I don’t see mention of updating file handles or anything else on the operating system. Maybe I’m cynical but this is what happens if you rely on chatGPT to give you programming answers without understanding the fundamentals first.
sneako 3y ago
“The maximum number of simultaneously alive Erlang processes is by default 262,144. This limit can be configured at startup. For more information, see the +P command-line flag in the erl(1) manual page in ERTS.” https://www.erlang.org/doc/efficiency_guide/advanced.html#:~:text=The%20maximum%20number%20of%20simultaneously%20alive%20Erlang%20processes%20is%20by%20default%20262%2C144 https://www.erlang.org/doc/efficiency_guide/advanced.html#:~....
fwlr 3y ago
Unless I’m missing something in the writeup, it doesn’t mention how much RAM is on the system. A little odd to mention the CPU and the OS but not the RAM, for a benchmark that doesn’t care about execution speed but does care about memory usage. It makes figuring out that Elixir error of “system limit reached” a little harder to figure out. edit: oh apparently it’s got nothing to do with memory, 1M just happens to be higher than the default process limit for Erlang and thus Elixir.
conradfr 3y ago
Also someone needs to tell ChatGPT that its Elixir code has a bug ;)

bsenftner 3y ago

Why no C++ version?

cynicalsecurity 3y ago
Because C/C++ would kick ass, exposing the rest of the hipster technologies as lame.
- Cyph0n 3y ago
  They would also kick memory safety in the ass, unless you’re the kind of dev who doesn’t make those kinds of silly mistakes
  - imran-iq 3y ago
    Don't be that guy. https://github.com/search?q=repo%3Atokio-rs%2Ftokio%20unsafe&type=code https://github.com/search?q=repo%3Atokio-rs%2Ftokio%20unsafe...
    - WinstonSmith84 3y ago
      Are you trying to say that you can write from scratch a code just as safe or better than one of the most peer reviewed crate of Rust? Congrats, the world has been waiting for you, but nonetheless, that's why commoners prefer to stick to what's safer for them
    - kouteiheika 3y ago
      `unsafe` doesn't mean that it's not safe, just that the compiler can't prove that it's safe so the human has to do it manually.
- josephg 3y ago
  Pfft doubtful. Rust, C and C++ all have very similar performance. And apparently a javascript framework is currently beating all of those languages in the composite score on techempower, though that seems a bit weird to me: https://www.techempower.com/benchmarks/#section=data-r21&test=composite https://www.techempower.com/benchmarks/#section=data-r21&tes... C doesn't have a built in, optimized async execution engine of any sort. C with threads would lose to any of the async runtimes here (including javascript). C++ has async now. By all means, write up an async C++ example and compare it. I suspect a good C++ async executor would perform about as well as tokio, but it'd be nice to know for sure. I'd also be interested to see how hard it is to write that C++ code. One nice thing about Rust and Go is that they perform very well with naively written code. (Ie, the sort of thing chatgpt, or most of your coworkers would write.) Naive C would use kernel threads, so if we're comparing average C code to average Go code at this test, the average go code will end up faster. (This isn't always true of rust. I've seen plenty of rust code which uses Box everywhere that runs crazy slow as a result.)

neonsunset 3y ago

As of today (with .NET 7 and 8 preview) 120MB base footprint memory usage by .NET is indeed surprising. A few questions: - How are you exactly launching the workload? - How much memory does your system have? - Could you try running it with .NET 7 instead? The last two are especially relevant because in the recent versions each release got extra work to reduce memory footprint on Linux (mostly because it's an important metric for container deployments). Overall, 120MB sounds about right if you are launching a .NET 6 workload with some extra instrumentation attached while the runtime picks Server GC which is much more enthusiastic about allocating large segments of memory, especially if the system has large amount of RAM. As for Tasks - C# tasks are in many ways similar to Rust's futures (both are yielding-to-the-caller state machines) and take very little memory, which scales directly with the amount of variables that persist across "yield points" (which you have none).

conradfr 3y ago
I noticed he didn't use the latest version of each language. Maybe that's just what you get on Ubuntu 22.04 LTS?
- neonsunset 3y ago
  Using .NET 6 is perfectly fine! It is the latest LTS after all, it's just that on machines with a lot of RAM the GC can indeed pre-allocate quite a bit of memory if, based on hardware configuration, it chooses to use Server GC. This, however, is being tuned with each subsequent release hence I'm curious to see the numbers on 7 on the author's hardware.
DaiPlusPlus 3y ago
> Overall, 120MB sounds about right if you are launching a .NET 6 workload with some extra instrumentation attached while the runtime picks Server GC which is much more enthusiastic about allocating large segments of memory, especially if the system has large amount of RAM. Also if you have lots of CPU cores then the CLR will (pre-)allocate up-to ~10MB/core for per-hardware-thread GC heaps. So my 16-core (32-thread) desktop will run every .NET program with a starting (OS-level) memory usage of ~200MB for a Hello, World program - though actual total GC usage will be < 1 MB.

deleted 3y ago

[deleted]

wwarner 3y ago

Would be nice to see throughput tested at this level. There is may be a good reason for Go to allocate more per thread if each thread is really doing something. Hat tip to Rust and C# tho!

bsaul 3y ago
I suspect 1 million concurrent task very likely becomes unmanageable in any real world scenario anyway. 1KB per task giving 1GB of total memory consumption, which means 100KB would be enough to raise memory limits on most systems. There may be scenarios such as network connexion handling where those connexion are kept idle 99% of the time, but you still need something to handle the very rare case of receiving something. However, is 1 million network connexion manageable from a kernel point of view anyway ? Wouldn't you reach system limits before that ?
- toast0 3y ago
  > However, is 1 million network connexion manageable from a kernel point of view anyway ? Wouldn't you reach system limits before that ? You can raise all those defaults as long as you have sufficient memory. Yes, you need to be working with fairly idle connections, otherwise you're likely to run out of CPU. Real world examples include Chat, IMAP, push messaging, that sort of thing where having a live connection means a significant reduction in latency vs periodically opening a connection to poll, but at the same time, there's not a whole lot going on. I was at WhatsApp when we accidentally hit 2.8M connections on one machine [1] I can't remember if we ever got to 3M, but we ended up doing a lot more work per connection so didn't keep those connection counts for long. Kernel wise, as long as you're mostly inbound connections, everything is well oiled; I ran into problems initiating many thousands of outbound connections/second in an HAProxy use case, although I think FreeBSD 13 may have improved that bottleneck. I hear Linux also works, but I've not scaled it nearly as far. [1] http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf http://www.erlang-factory.com/upload/presentations/558/efsf2...
  - bsaul 3y ago
    Thanks for sharing. What’s your take on elixir consuming an outstanding amount of memory per task in OP benchmark ?

lordnacho 3y ago

I'm a big Rust fan, so I'm happy Tokio holds up so well. I find it to be one of the simplest and quickest ways to write task based programs. Gotta wonder whether you can challenge those numbers with c++ though. I suspect you can but with a lot of work, where Rust is pretty effective written naively. Of course the big issue with these comparisons is it's hard to be an expert in so many different languages. How do you know if what you're doing is the best way in each language?

jeroenhd 3y ago
I find the opposite to be true, Tokio is a fine library but it makes the whole program very complex. Perhaps it's because I generally don't write code for many short lived workloads, but I find using real threads to be a lot easier to work with (plus it saves time during compilation). Tokio is great at what it does, but I have to wonder how relevant these types of threads are when "fearless concurrency" is one of the token advantages of the language. I think this article would've been a lot better if it did some actual calculations in the spawned threads, rather than simply waiting. A smart enough compiler would see that the results of the wait calls, which usually don't come with any other side effects, are never used and optimize them out entirely.

jasonwatkinspdx 3y ago

> "All programs were launched using the release mode if available. Other options were left default." So it's likely these numbers simply reflect choices in the default configuration, not the ultimate limit with tuning. I'm starting to dislike these low effort ChatGPT articles. Getting an answer from it is not the same thing as actually learning what's going on, but readers will walk away with the answers uncritically.

vasco 3y ago
> I'm starting to dislike these low effort ChatGPT articles. The next few decades are going to be tough for you, I think. Brace yourself!
dceddia 3y ago
Lately I’m seeing a lot of this baked-in assumption that ChatGPT writes good/idiomatic/performant code, as if the code it generates is ~The Answer~. I guess it’s the modern version of copy/pasting from Stack Overflow. The code I’ve had it write isn’t universally terrible but it almost always has some low-hanging performance fruit (in JS for instance, it seems to love spreading arrays). I’m not sure how much I trust it to write good benchmark code!
- kubb 3y ago
  Does spreading arrays come with a performance penalty?
  - spicebox 3y ago
    Spreading an array requires looping over all the array elements which is O(n) compared to something like Array.push() which is just 0(1), although spreading has other benefits like copying rather than mutating
  - dexwiz 3y ago
    Spreading allocates a new array. Most things you can do with spread you can do with normal functions that either just directly access an entry or mutate an array directly. [first, …rest] is nice, but will always be slower than direct access.

withinboredom 3y ago

Haha, I had to try a PHP build with the parallel extension[1] with some rather simple code[2]: - single job: 27.8 mb (respectable) - 10k: 8.3 GB! At this point, I gave up. I'm 99% sure trying to go to 1 million will crash my computer. [1]: https://www.php.net/manual/en/intro.parallel.php https://www.php.net/manual/en/intro.parallel.php [2]: https://gist.github.com/withinboredom/8256055709c0e6a8272b3a20e93cc140 https://gist.github.com/withinboredom/8256055709c0e6a8272b3a...

withinboredom 3y ago
I should add that the good ole' fork approach[1] only uses 40.7mb for 10k, 40.7mb for 100k, and 40.7mb for 1 million. Got to love COW memory. :) [1]: https://gist.github.com/withinboredom/c811f3c433904235a5f19602dccbbf01 https://gist.github.com/withinboredom/c811f3c433904235a5f196...

Traubenfuchs 3y ago

The Java tests should be optimized by making the ArrayList a fixed size. All this resizing of the huge array backing it and the potential final very high amount of empty backing array slots probably makes the Java result worse than it could be. I wonder if the parallel stream API with a virtual thread pool would make it better or worse.

tialaramex 3y ago
The ArrayList is just a growable array and is presumably using an exponential growth strategy so it's equivalent to what several of the others are doing. You could indeed tell Java how big the ArrayList will grow in advance, but you could for example likewise do that with the Rust Vec by asking for Vec::with_capacity(num_tasks) instead of Vec::new(). Since this is amortized exponential growth it's likely making negligible difference, although it is good style to tell data structures what you know about their size early. The Go approach doesn't appear to pay this penalty, it's just incrementing a (presumably atomic?) counter, although I have no idea where these tasks actually "live" until they decrement the counter again - but most of the others are likewise using a growable array.
- JB_Dev 3y ago
  It looks like most of the languages are doing the same thing w.r.t using a 'List'-style implementation that re-sizes/copies it's internal buffer when it runs out of space. One thing I wonder though is whether the managed languages pay a bigger penalty on the memory usage than the unmanaged languages due to this. In the unmanaged languages, those intermediate buffers should be freed immediately during the re-size operation. But in the managed languages, they might stick around until the next GC. So the peak memory would be higher. In this example the runtime overhead is going to represent more of the memory usage anyway but it quickly isn't negligible. E.g. a list of size 1M is going to be ~8MB (64 bit addresses). So even if the array is re-sized from 1->2->4->8->...->~500K, the end result is not going to be worse than 2X the size.
tester756 3y ago
C#'s list works the same way

deleted 3y ago

[deleted]

themerone 3y ago

This is not a comparison of 1 million concurrent tasks. For a bunch of these tests, it is measuring 1 million tasks multiplexed onto a smaller number of os threads. In many implementations the cost of concurrency is going to be small enough that this test is really a measure of the stack size of a million functions.

creatonez 3y ago
There is sometimes a distinction made between concurrency and parallelism, where "concurrency" just means the programming interface looks parallel but is allowed to use out-of-order task-switching under the hood, and "parallelism" is real-deal usage of CPU cores. Both ways of looking at things are useful in different situations.
- themerone 3y ago
  That is a good point, but parallelism degrades to concurrency when you run out of CPU cores.

wongarsu 3y ago

I like the general idea of the test. Things I would have liked to see (apart from basic system tweaks like allowing more file handles) are a performance comparison (how much CPU time is spent) and maybe a comparison between operating systems (this might highlight a lot of windows-linux differences, like Windows not committing memory - better have a large swap file to handle stacks of 100k real threads - or windows having a very different scheduler)

kevingadd 3y ago

It would be interesting to see how much the C# memory usage would go down if you allocated a single Task.Delay at the start and had all the tasks wait for it. It would probably result in it finishing in 10sec instead of climbing up to 12, while still running 1 million tasks. I'm not sure how you would do this in other runtimes, and there is still a risk of new problems cropping up. 1 million Task.Delays means 1 million queued waits in the sleep machinery, which could run into some bad scaling and overhead. But then 1 million tasks waiting on a single awaitable probably hits overhead in other subsystems. My guess would be that 1 million waits is much cheaper than 1 million sleeps, because sleeps have to be sorted by due time while the list of tasks waiting for completion just needs to be sequential. Entries in the sleep queue are also going to be bigger (at minimum, 4-8 bytes for due time + 4-8 bytes per completion callback) while entries in a completion queue would just be 4-8 bytes for the completion callback. Task.WhenAll also has potentially worse behavior than the other apps if you care about memory, since it will need to allocate a big array of 1 million Tasks in order to pass it in. Many of the other programs instead loop over the tasks, which eliminates that overhead.

JB_Dev 3y ago
The other thing I noticed is that they are wrapping every Task.Delay(…) inside a Task.Run(…) call. That should not be necessary. The inner task will yield back to the called when it hits the delay call anyway. There is no need to queue it with an immediate yield as if it is background or cpu-bound work. The Run() call might be creating a more complex state machine and using more memory than necessary (2 tasks vs just 1 for each delay). The memory usage might be even lower if the Run() call is dropped.
- dodyg 3y ago
  Yup - this would do the job faster and better. Task task = Task.Delay(TimeSpan.FromSeconds(1)); tasks.Add(task);
  - JB_Dev 3y ago
    I was sufficiently curious and just went and tested this using BenchmarkDotNet. The example code is here (https://github.com/J-Bax/CSharpBenchmarkExamples https://github.com/J-Bax/CSharpBenchmarkExamples). The difference is quite significant. (1) With the authors code (using Task.Run), I get ~428MB of allocations. (2) Dropping the unnecessary Task.Run(...), I get ~183MB of allocations. (3) Doing (2) and waiting N times on the same delay, I get ~39MB of allocations. This was all using .NET 6 too. .NET 7 or the 8 preview might be even better since they are working hard on performance in recent releases. So even looking at just (2), that puts .NET on par with the rust library.

dceddia 3y ago

Tokio seems oddly good compared to the others. Is that because it’s using a thread pool instead of creating something new for every task? I wonder how it would fare if the tasks were more intense, and how the time-to-completion compares between these approaches.

bombela 3y ago
async Rust functions are compiled down to a finite state machine. A stackless coroutine I think it can be called. I suspect that C#/.NET does something similar though I am just speculating here. The difference between async-std and tokio would then be explained by a lower overhead runtime per task in tokio.
pornel 3y ago
All the async and green thread runtimes are using thread pools by definition. Avoiding spawning as many OS threads as program's tasks is their main job. This benchmark wasn't very rigorous, so maybe it has overlooked something, but the result seems plausible. Tokio is a good, mature runtime. Tokio-based frameworks are near top of TechEmpower benchmarks. Rust's Futures were designed from the start with efficiency in mind. Unlike most async runtimes where chaining of multiple async calls creates new heap allocations, Rust flattens the entire call tree and all await points into a single struct representing a state machine. This allows the runtime to use exactly one heap allocation per task, even if the task is pretty complex.

cyberax 3y ago

It's not really fair to compare async/await systems with Go and Java. Go and Java with lightweight threads provide a full-blown system, without any limitations. You don't have to "color" your code into async and blocking functions, everything "just works". Go can even interrupt tight inner loops with async pre-emption. The downside is that Go needs around 2k of stack minimum for each goroutine. Java is similar, but has a lower per-thread constant. The upside is that Go can be _much_ faster due to these contiguous stacks. With async/await each yield point is basically isomorphic to a segmented stack segment. I did a quick benchmark to show this: https://blog.alex.net/benchmarking-go-and-c-async-await https://blog.alex.net/benchmarking-go-and-c-async-await

jimsimmons 3y ago
None of what you say is a gotcha. An ignorant programmer stands only to benefit from Go
djsavvy 3y ago
In your Go benchmark code, how is the calculation actually being done across threads? I don't see the `go` keyword anywhere in the source code.
- h4x0rr 3y ago
  I'm wondering that as well. The c# code seems really unrealistic though, using a lot of tasks for CPU bound work. A fair comparison would at least need to involve some degree of IO. Does go maybe automatically parallelize everythinh it can? That would be one potential answer to the initial question
- SmooL 3y ago
  Yeah this isn't using goroutines at all? I don't see how this is a good comparison of goroutines vs coroutines.
- cyberax 3y ago
  But that's the point I'm trying to show. In Go code all calls are the same. There is no sync/async distinction. In contrast, in C# (or any other similar system) async calls are _expensive_ compared to regular function calls.

asabil 3y ago

The maximum number of simultaneously alive Erlang processes is by default 262,144. This needs to be raised to avoid the `system_limit` error.

whateveracct 3y ago

The title is asking a very quantifiable question. And I was hoping the article would be a summary of expertise and deep dives the author took into various language's runtimes. But then I read > With a little help from ChatGPT And sure enough the results & prose end up pretty much worthless. Because the author didn't do any actual critical thinking or technical work. (For instance, there's nothing more to "Go's memory usage ballooned at 1M tasks" beyond the observation. Why not? Just dig in there it's all documented you just have to read some stuff.)

lexandstuff 3y ago
Do you mind sharing why Go ballooned to 1M?
- pkolaczk 3y ago
  Because goroutine stacks start at minimum 2 kB. But it is not obvious if this is the only factor that contributes to memory usage so I believe experimenting to verify theory is still quite valuable.
faizshah 3y ago
It’s also really strange using python and node in an article about concurrency without mentioning the GIL (python) or node single threading.
- whateveracct 3y ago
  That's what happens when you write a blog about a technical topic you don't know much of anything about I guess
- Too 3y ago
  The whole idea of async functions is to run tasks concurrently and with simple syntax on a single thread. Hiding the actual concurrency under the runtime, only when it is waiting for IO. It doesn’t matter that node is single threaded, nor does the GIL have any impact on single thread. It’s once you start doing cpu heavy work inside the tasks this will hit you.

malkia 3y ago

What's the point of running 1M concurrent tasks? I don't see a good reason tbh.

pestatije 3y ago
website with that many requests
- pornel 3y ago
  Not for a typical website. Even if you're getting a massive amount of traffic, like a million requests per second, you'd want the request to finish as quickly as possible, so they never pile up to be 1M concurrent tasks. If the tasks have real work to do and can't finish quickly, the having 1M in-flight tasks compete for orders of magnitude fewer CPU cores may not be a good idea — it will waste lots of memory for their state and it will be hard to keep latencies reasonably even. You'd probably use a simpler queue and/or load-balance to more machines. A scenario with this many tasks spawned is more realistic if you have mostly idle connections. For example, a massive chat server with tons of websockets open.
  - Seattle3503 3y ago
    You might not have a choice. How should a load balancer be implemented? It has to keep those connections alive while the underlying services fulfill the request.
- malkia 3y ago
  This is bound to problems - slighest change in your stack - language, runtime, OS would affect that so much that it's completely pointless to rely on measures taken and apply them later (that to be said, this is perfectly valid choice when you are locked in - like in the embedded or game console world, but rarely that's the case outside) For example - https://go.dev/doc/go1.3#stacks https://go.dev/doc/go1.3#stacks https://agis.io/post/contiguous-stacks-golang/ https://agis.io/post/contiguous-stacks-golang/ So your measurements in go 1.2 would probably differed in go 1.3 (and this is just one example, in only one of the possible axises that have changed). It's not like you own the data structure there to control it.

d0mine 3y ago

The results can be ignored. You can't just copy-paste ChatGPT without even a sanity check e.g., Python code is wrong: await sleep(10) sleeps 10+ seconds and returns None but the intent is to pass a coroutine to create_task() (await should be removed at the very least)

pkolaczk 3y ago
Ok, this one is funny. Actually ChatGPT got it right and there was a create_task before, but right before committing I "inlined it" and introduced a bug. Thanks for letting me know. Will double check if it didn't messed up the results. And BTW - I'm very far from copy pasting from chat gpt. ChatGPT helped with this code, but often required many iterations and also some manual cleanup.
mrj 3y ago
Yeah Python got done dirty here. I tried a million tasks and I saw a peak of about 166MB. I'm still getting used to async in Python, but this seems to work fine though? import asyncio async def do_task(): await asyncio.sleep(10) async def main(num_tasks: int): tasks = [] for task_id in range(num_tasks): task = asyncio.create_task(do_task()) tasks.append(task) print("spawned tasks") await asyncio.gather(*tasks) if __name__ == "__main__": asyncio.run(main(1_000_000))
- pkolaczk 3y ago
  Thank you! I fixed the code and updated the benchmarks. However, your 166 MB doesn't match my testing (I'm getting about 240M at 100k tasks) and 10x more at 1M. Are you sure you haven't misread your RSS?
  - mrj 3y ago
    Nope, that's what it is for me but you're using an older Python. 3.11 has a lot of improvements and not sure why you wouldn't use the latest?

throwawaaarrgh 3y ago

I see C is missing. If you're going to do a language comparison, might as well include the one language we know will have the best performance and lowest memory

gigel82 3y ago
There is no light-weight threading concept in standard C/C++, and the cost of one OS thread is at least 1Mb (stack) + 4Kb (tls) reserved memory (at least by default, on modern Windows), so for 1m threads that will be 1Tb of RAM :)
- shrimp_emoji 3y ago
  How is your compiler addressing bits like this?
- tialaramex 3y ago
  C++ 20 has async/await (but of course named co_async and co_await, because C++ isn't the sort of language where you're allowed nice things, the beatings will continue until morale improves) However AIUI C++ 20 doesn't actually supply an executor out of the box, so you would need to choose what to do here as for Rust, where they picked both tokio and async-std and you can see they have different performance. C does only have threads, but you could presumably pull off the same trick in C that Rust does, to get Linux to give you the bare minimum actual resident memory for your thread and you needn't care about the notional "real" size of the thread since we're 64-bit and address space is basically free.
  - foderking 3y ago
    > (but of course named co_async and co_await, because C++ isn't the sort of language where you're allowed nice things, the beatings will continue until morale improves) LMAO
    - gpderetta 3y ago
      co_<whatever> has become a meme in the C++ community.

matklad 3y ago

Benchmarks for threads might slightly underestimate memory usage. For threads, some amount of memory is spent by the kernel (eg, memory maps themselves) and won’t show as maxrss. > a high number of concurrent tasks can consume a significant amount of memory, So Go needs 3gigs for a million tasks. Relatively speaking, for an app that does a million concurrent somethings, 3gb is probably peanuts. It seems that memory isn’t the limiting factor here.

beders 3y ago

Another benchmark of the "I don't know what I'm actually measuring" flavor. I don't even know where to start: from the lack of env settings, system settings, compiler settings to the fact that no concurrent work was actually done, this is one giant flawed benchmark. The only lesson here should be: Don't take any of this seriously.

tester756 3y ago
>from the lack of env settings, system settings, compiler settings compiler settings?? I bet he used defaults everywhere
pkolaczk 3y ago
Just read the text. It lists versions and states it was using defaults in the release mode. Most of those compilers don't have any fancy options actually. E.g go is just `go build`, Python is just `python3 ...`. Rust options are in the project definition. Anyway, I'll add those command line invocations if it causes so much controversy.

deleted 3y ago

[deleted]

thethirdone 3y ago

Surely a Sufficiently Smart Compiler could optimize all of these to only take the a constant amount of memory (and less time). I wonder if there are any correctness constraints that would actually prevent such optimizations. For example, the compiler might be required to execute the exact same system calls as an unoptimized program. It seems quite reasonable that the programs using system threads could not be optimized significantly under such a constraint. I would expect the larger limit that prevents compilers from optimizing this is the large amount of code that defines the semantics of the lightweight threads. For example, I would expect the observable / required behaviors of tokio to not prevent a SCC from optimizing it, but the massive amount of Rust code that defines the lightweight thread semantics cannot be reasonably optimized. Which programming language / framework is the closest to actually being optimized to like < 4 byte / task? I would make an uneducated guess of Elixir or Go.

sb8244 3y ago
Each BEAM process comes with its own stack / heap space. This uses a minimum of 326 words of memory (https://www.erlang.org/doc/efficiency_guide/processes.html#creating-an-erlang-process https://www.erlang.org/doc/efficiency_guide/processes.html#c...)

bastawhiz 3y ago

Understanding full well that these are pretty artificial results, I'm surprised at just how well Node and Python faired against Go and Java. For all the default assumptions around languages that folks have, they've certainly held their own here.

josephg 3y ago
As others have mentioned, the results come from the difference between async concurrency vs "lightweight threads" (in Go and Java). In the nodejs test, there's very little javascript code being actually executed here. Whats really being tested is nodejs's event queue (libuv), which is written in C. And its being compared to Go's coroutine system - which is also in C. But, if I recall correctly, Go's threading model includes a preemptive scheduler (which is very complex) and it allocates a stack per goroutine. This doesn't test if Javascript or Go code is faster / more efficient. There's almost no javascript or go code being executed here at all. This is testing the execution engines of both languages. Go's execution engine has more features, and as we can see those features have a cost at runtime.

davidgrenier 3y ago

For the sake of pitching an analog to go's concurrency model. Hopac is a concurrency library for F# (partly written in C#) delivering an informal implementation of CSP (similar to Go's goroutine and Clojure Core.async). Since the job does nothing but wait for 10 seconds, the code is: seq{1..1000000} |> Seq.Con.iterJobIgnore (fun _ -> timeOutMillis 10000) |> run I'm not sure how to calculate total memory usage but instrumenting the following way: GC.Collect() GC.Collect() GC.WaitForPendingFinalizers() GC.TryStartNoGCRegion 3000000000L let before = GC.GetTotalMemory false seq{1..1000000} |> Seq.Con.iterJobIgnore (fun _ -> timeOutMillis 10000) |> run GC.GetTotalMemory false - before |> printfn "%d" GC.EndNoGCRegion() Gives the following output, I'm reading 144MB for 1M lightweight thread (in the REPL) unless I did something dumb: 144449024 Real: 00:00:10.902, CPU: 00:00:18.570, GC gen0: 3, gen1: 3, gen2: 3 val before: int64 = 53902280L

davidgrenier 3y ago
This is on .net core 7.0.5, Debian 11, Server GC (which is expected by Hopac). I removed all GC code and ran it as an executable using @compumike's `ps ax -o pid,rss,args' command. I think the only one that triggered a GC was the execution with 10M concurrent jobs. N Real(s) User(s) RSS(kiB) RSS(MiB) ================================================== 1 10.08 00.04 42660 41 1000 10.07 00.06 40736 39 10000 10.09 00.20 44176 43 100000 10.15 01.85 58532 57 1000000 10.88 18.18 182024 177 10000000 22.28 4m04.84 1452224 1418

deleted 3y ago

[deleted]

Seattle3503 3y ago

Rust is going to do well on toy examples. This is because in a naive implementation, for each thread, Rust statically allocates enough space for the largest possible stack the thread will need. On a toy example this will be quite small. In real code, some branches will have large futures that will bloat the size of every thread, even if the thread doesn't take the bloated branch. You can mitigate this issue by heap allocating large futures on cold paths, but this requires insight and tuning. Some runtimes can handle this for you, but they come with other performance tradeoffs.

pkolaczk 3y ago
In real code I saw Rust beating Go and Java based code by 10x-50x in memory use. But too complex and too many factors to analyze, so not a material for a blog post. Actually this tiny benchmark favors GC based runtimes because it just sleeps - so doesnt actually allocate much on the heap.

augusto-moura 3y ago

The nodejs code doesn't actually work in parallel. The code being tested is only creating new timeouts but it is working on a single V8 thread. It has a clock thread on the C++ part of the engine and when it timeouts it pushes the continuations on the event loop, and that is it

h1fra 3y ago
Came here to say that, not sure if it actually would improve the test but at least it would be correct. Also, Node 12 was released 4 years ago.

deleted 3y ago

[deleted]

sakras 3y ago

> Notably, at 1 million tasks, I observed that the overhead of launching tasks became evident, and most programs required more than 12 seconds to complete. I didn’t quite get this part. Even if your runtime has no overhead, the fastest way you can execute 1 million 10-second tasks is 10 * 1 million / num_cores, right? Why would these million tasks only take 12 sec?

josephg 3y ago
Because the tasks are all running concurrently. (They're all running at the same time).
hamburglar 3y ago
You can “run” more threads than you have cores when they are all inside sleep()

pdimitar 3y ago

To the author: Elixir (well, Erlang's BEAM VM actually) can be configured to allow more than the default 256k tasks (262_144). I'd do that and post the numbers for 1M tasks.

compumike 3y ago

Crystal 1.8.2 https://crystal-lang.org/ https://crystal-lang.org/: n = ARGV.first.to_i32 finished = Channel(Nil).new n.times do spawn do sleep(10.seconds) finished.send(nil) end end n.times { finished.receive } Memory usage determined with `ps ax -o pid,rss,args`. Runtime measured with `/usr/bin/time -v`. (Had to run `sudo sysctl vm.max_map_count=2500000` to get the final two rows.) Results: N Runtime(s) RSS(kiB) RSS(MiB) ======================================= 1 10.01 1792 2 1000 10.01 6016 6 10000 10.11 45184 44 100000 11.00 435840 426 1000000 20.25 4336768 4235 "System time" rather than "user time" is the majority (7.94s system time, 2.83s user time in the 20.25s wall time run). Is this pointing to memory allocations? Crystal Fiber docs https://crystal-lang.org/api/1.8.2/Fiber.html https://crystal-lang.org/api/1.8.2/Fiber.html says "A Fiber has a stack size of 8 MiB which is usually also assigned to an operating system thread. But only 4KiB are actually allocated at first so the memory footprint is very small." -- and perhaps unsurprisingly, 4KiB times 1000000 is approximately 4 GiB. Nice when the math works out like that :) To me this is useful as a baseline for something like a websocket server, with some number of idle connections, each Websocket connection mapped to a Fiber that is waiting for I/O activity but mostly idle.

benatkin 3y ago

World's largest compact car. Yawn.

Aardwolf 3y ago

> popular languages like Rust, Go, Java, C#, Python, Node.js and Elixir. Oof, has C/C++ become that unpopular? It would be a useful comparison as low level system language anyway.

lbriner 3y ago

Lots of pedants like to dump on these sorts of articles but there are enough interesting things to see, even in a relatively trivial comparison like this. It is interesting that some systems allocate more memory up-front but this doesn't grow much per-task. It is interesting that in the form the app was written the Go program doesn't appear to scale as well. Sure there are possibly reasons for it so anyone with more knowledge of Go could attempt to write a more "correct" program and ask to rerun it. It is interesting that with the defaults, things behave as they do, again, someone who cares more about "it's just that the defaults are different" is welcome to investigate that and ask "why" some program defaults might lead to better performance. It is interesting that as-in most comparisons, the languages/frameworks don't neatly fit into Stereotypes. It is interesting that most of them probably perform well enough for 99% of applications we might write. It seems like most people still don't appreicate that YMMV and that trade-offs are always made. Just add it to your stash of brain juice that helps you become a better engineer and stop complaining.

hdjjhhvvhga 3y ago

It's nice to see but frankly speaking with current memory prices all I care is speed, not so much memory consumption, unless I have very specific requirements.

Alifatisk 3y ago

I wonder how Ruby performs in this benchmark

Patrickmi 3y ago

Let’s understand that Go’s runtime and scheduler does a lot, and there’s not enough information to justify such benchmarks

sigmonsays 3y ago

this is a terrible benchmark. You're literally doing nothing per task. There is so much left up to the implementation details that you are not benching anything.

fulafel 3y ago

Would be interesting to hear what problems was hit trying to run 100k threads. I remember testing out a million threads on an by now older desktop and just had to do a couple of sysctl setting changes that were easy to find. Not sure if this is what I used but the settings sound familiar: https://www.baeldung.com/linux/max-threads-per-process https://www.baeldung.com/linux/max-threads-per-process Also would be good to specify how the memory as measured (eg to be able to verify they are actually measuring physical memory usage).

pajko 3y ago

Using thread pools/connection pools could help a lot both in performance (less time to create threads) and memory use (less fragmentation + less memory to garbage collect after short-lived allocations). There's a guide somewhere about using connection pools, multiprocessing and async I/O vs. pthreads, showing that the most obvious solution to the concurrency implementation does not necessarily has the best performance, but couldn't find it right now. EDIT: polls->pools typo

Sammi 3y ago

Some of these tests are doing concurrency. Some are doing parallelism. Concurrent meaning multiple tasks run within a given stretch of time. Parallel meaning multiple tasks run at a given point of time. These are not the same. So the tests are not testing the same thing.