11 ms·

How much memory do you need in 2024 to run 1M concurrent tasks?

• fortran77 2y ago
  He should have looked at Erlang.
  • neonsunset 2y ago
    Erlang is likely going to have about the same or greater starting overhead as Go here from what I measured[0] with Elixir. Each Erlang process carries its own independent GC which allows it to isolate allocation impact, contributing to the robustness of its implementation. I assume this is where the cost comes from. If you do measure Erlang - please post the numbers. Processes in Erlang, Goroutines in Go and Virtual Threads in Java do not fully replace lightweight asynchronous state machines - many small highly granular concurrent operations is their strength. [0]: https://gist.github.com/neon-sunset/8fcc31d6853ebcde3b45dc7a2af32127#tldr https://gist.github.com/neon-sunset/8fcc31d6853ebcde3b45dc7a... (disclaimer: as pointed out in a sibling comment it uses Elixir's Task abstraction which adds overhead on top of the processes)
• amazingamazing 2y ago
  Yet again nodejs surpasses my pre-read expectations 3rd best (generalized) for a million? Wow. I must be missing something - isn’t Go supposed to be memory efficient? Perhaps promises and goroutines aren’t comparable?
  • fuzzybear3965 2y ago
    I'm not sure what "memory efficient" means. But, Go sprung as a competitor to Java (portability, language stability, corporate language support/development) and C++ (faster compile times). Can't beat C++ in terms of memory management (performance, guys, not safety) much. But, you can fare well against the JVM, I'm guessing.
    • jpgvm 2y ago
      In this benchmark actually no, Go doesn't fare well. There is actually higher static overhead per goroutine than JVM VirtualThread. I presume this is because of a larger initial stack size though/ This probably doesn't matter in the real world as you will actually use the tasks to do some real work which should really dwarf the static overhead is almost all cases.
  • brabel 2y ago
    If you want to run 1 million coroutines that just sleep in your app, yeah nodejs looks very efficient. The problem is that when each coroutine needs to allocate memory, which I would suppose anything real would do, the 2Kb Go pre-allocates will be an advantage - as it will probably be required except for the most trivial workloads (like in this benchmark) - and then because Go actually runs them in parallel, unlike nodejs, you would likely see a huge improvement in both performance and memory usage with Go or Rust.
• rwaksmunski 2y ago
  Be sure to read the Appendix, Rust's state machine async implementation is indeed very efficient.
• davidatbu 2y ago
  I write (async) Rust regularly, and I don't understand how the version in the appendix doesn't take 10x1,000,000 seconds to complete. In other words, I'd have expected no concurrency to take place. Am I wrong? UPDATE: From the replies below, it looks like I was right about "no concurrency takes place", but I was wrong about how long it takes, because `tokio::time::sleep()` keeps track of when the future was created, (ie when `sleep()` was called) instead of when the future is first `.await`ed (which was my unsaid assumption).
  • sour-taste 2y ago
    Tokyo::sleep is async
    • davidatbu 2y ago
      I think the points people made in other replies make sense, but "Tokio::sleep is async" by itself is not enough of an explanation. If it were the case that `Tokio::sleep()` tracked the moment `.await` was called as it's start time, I believe it would indeed take 10x1,000,000 seconds, _even if it's async_.
  • claytonwramsey 2y ago
    The implementation of `sleep` [1] decides the wake up time by when `sleep` is called, rather than when its future is polled. So the first task waits one second, then the remaining tasks see that they have already passed the wake-up time and so return instantly. [1]: https://docs.rs/tokio/latest/tokio/time/fn.sleep.html https://docs.rs/tokio/latest/tokio/time/fn.sleep.html
    • davidatbu 2y ago
      This makes total sense!
  • fuzzybear3965 2y ago
    Yeah, I think you're wrong. It should only take ~10s. tokio::time::sleep records the time it was called before returning the future [1]. So, all 1 million tasks should be stamped with +/- the same time (within a few milliseconds). [1]: https://docs.rs/tokio/1.41.1/src/tokio/time/sleep.rs.html#128-131 https://docs.rs/tokio/1.41.1/src/tokio/time/sleep.rs.html#12...
• theamk 2y ago
  > high number of concurrent tasks can consume a significant amount of memory note absolute numbers here: in the worst case, 1M tasks consumed 2.7 GB of RAM, with ~2700 bytes overhead per task. That'd still fit in the cheapest server with room to spare. My conclusion would be opposite: as long as per-task data is more than a few KB, the memory overhead of task scheduler is negligible.
  • pkulak 2y ago
    Except it’s more than that. Go and Java maintain a stack for every virtual thread. They are clever about it, but it’s very possible that doing anything more than a sleep would have blown up memory on those two systems.
    • bilbo0s 2y ago
      I have a sneaky suspicion if you do anything other than the sleep during these 1 million tasks, you'll blow up memory on all of these systems. That's kind of the Achille's Heel of the benchmark. Any business needing to spawn 1 million tasks, certainly wants to do something on them. It's the "do something on them" part that usually leads to difficulties for these things. Not really the "spawn a million tasks" part.
      • vlovich123 2y ago
        The “do something” OP is referring to is simple things like a deeply nested set of function calls and on stack data structures allocated and freed before you sleep. This increases the size of the stack that Go needs to save. By comparison stackless coroutines only save enough information for the continuation, no more no less. That’s going to be strictly smaller than saving the entire stack. The argument you seem to be making is that that could be the same size as the stack (eg heap allocations) but I think that’s being unreasonably optimistic. It should always end up being strictly smaller.
        • theamk 2y ago
          I have no doubt it's going to be strictly smaller, it's just the difference could be too small to care. Like say you are making a server, and each client has 16KB of state. Then memory usage would be 17KB in Node vs 19 KB in Go. Smaller? Yes. Smaller enough that you want to rewrite the whole app? Probably not.
• cperciva 2y ago
  This depends a lot on how you define "concurrent tasks", but the article provides a definition: Let's launch N concurrent tasks, where each task waits for 10 seconds and then the program exists after all tasks finish. The number of tasks is controlled by the command line argument. Leaving aside semantics like "since the tasks aren't specified as doing anything with side effects, the compiler can remove them as dead code", all you really need here is a timer and a continuation for each "task" -- i.e 24 bytes on most platforms. Allowing for allocation overhead and a data structure to manage all the timers efficiently, you might use as much as double that; with some tricks (e.g. function pointer compression) you could get it down to half that. Eyeballing the graph, it looks like the winner is around 200MB for 1M concurrent tasks, so about 4x worse than a reasonably efficient but not heavily optimized implementation would be. I have no idea what Go is doing to get 2500 bytes per task.
  • liveoneggs 2y ago
    https://tpaschalis.me/goroutines-size/ https://tpaschalis.me/goroutines-size/ https://github.com/golang/go/blob/master/src/runtime/stack.go#L75 https://github.com/golang/go/blob/master/src/runtime/stack.g...
  • masklinn 2y ago
    > I have no idea what Go is doing to get 2500 bytes per task. TFA creates a goroutine (green thread) for each task (using a waitgroup to synchronise them). IIRC goroutines default to 2k stacks, so that’s about right. One could argue it’s not fair and it should be timers which would be much lighter. There’s no “efficient wait” for them but that’s essentially the same as the appendix rust program.
    • jakewins 2y ago
      Fair or not, it’s a strange way to count - Go isn’t using that RAM. It’s preallocating it because any real world program will.
      • winternewt 2y ago
• ankit70 2y ago
  NodeJS is better at memory than go?
  • erik_seaberg 2y ago
    I'd expect that because Promises are small Javascript objects while goroutines each get a stack that grows from at least 2 KB.
    • dboreham 2y ago
      Otoh Go actually supports concurrency.
      • jpgvm 2y ago
        Well they are all concurrent. I think what you mean is Go is also parallel. As is C#, Rust and Java in this bench.
  • hu3 2y ago
    They are doing different things in this benchmark. NodeJS has one thread with a very tight loop. Go actually spawned 1M green threads. Honestly this benchmark is just noise. Not to say useless in most real world scenarios. Specially because each operation is doing nothing. It would be somewhat useful if they were doing some operation like a DB or HTTP call.
• AkshitGarg 2y ago
  I feel this benchmark compares apples to oranges in some cases. For example, for node, the author puts a million promises into the runtime event loop and uses `Promise.all` to wait for them all. This is very different from, say, the Go version where the author creates a million goroutines and puts `waitgroup.Done` as a defer call. While this might be the idiomatic way of concurrency in the respective languages, it does not account for how goroutines are fundamentally different from promises, and how the runtime does things differently. For JS, there's a single event loop. Counting the JS execution threads, the event loop thread and whatever else the runtime uses for async I/O, the execution model is fundamentally different from Go. Go (if not using `GOMAXPROCS`) spawns an OS thread for every physical thread that your machine has, and then uses a userspace scheduler to distribute goroutines to those threads. It may spawn more OS threads to account for OS threads sleeping on syscalls. Although I don't think the runtime will spawn extra threads in this case. It also depends on what the "concurrent tasks" (I know, concurrency != parallelism) are. Tasks such as reading a file or doing a network call are better done with something like promises, but CPU-bound tasks are better done with goroutines or Node worker_threads. It would be interesting to see how the memory usage changes when doing async I/O vs CPU-bound tasks concurrently in different languages.
  • SPascareli13 2y ago
    As far as I know there is no way to do Promise like async in go, you HAVE to create a goroutine for each concurrent async task. If this is really the case then I believe the submition is valid. But I do think that spawning a goroutine just to do a non-blocking task and get its return is kinda wasteful.
    • n2d4 2y ago
      You could in theory create your own event loop and then get the exact same behaviour as Promises in Go, but you probably shouldn't. Goroutines are the way to do this in Go, and it wouldn't be useful to benchmark code that would never be written in real life.
      • peterhon 2y ago
• lxe 2y ago
  NodeJS does what it was designed to do well.
  • ankit70 2y ago
    I wonder how compiled (using Deno or others) JS would perform.
• deleted 2y ago
  [deleted]
• pgAdmin4 2y ago
  Why C with pthreads missing in this benchmark ?
  • throwaway81523 2y ago
    I don't think 1M posix threads is a thing. 1K is no big deal though.
    • liontwist 2y ago
      ~100k is a thing on Linux.
      • fulafel 2y ago
        On Linux, 400k threads was a thing on even on 2002 hardware that was 32-bit and thus limited to 4GB: https://news.ycombinator.com/item?id=37621887 https://news.ycombinator.com/item?id=37621887 In the titular post there's a link to a previous comparison between approaches, and plain OS threads used from Rust fare quite well, even if the author doesn't up the OS limits to keep that in the running for the higher thread cases: https://pkolaczk.github.io/memory-consumption-of-async/ https://pkolaczk.github.io/memory-consumption-of-async/
  • thesnide 2y ago
    that. Or just using a C coroutine lib.
• liontwist 2y ago
  No baseline against UNIX processes?
  • masklinn 2y ago
    A million processes?
    • liontwist 2y ago
      I've seen 100k. What happens at a million? How many is unworkable?
      • Izkata 2y ago
        $ cat /proc/sys/kernel/pid_max 4194304 My computer can handle that many processes, after that no new processes can be spawned (see: forkbomb)
        • blitzar 2y ago
          I am left wondering what happens at 4194305
          • masklinn 2y ago
            You can fork bomb your system and observe.
            • liontwist 2y ago
              The fork syscall will fail at the hard limit. What’s missing here is that all these async/await, lightweight threads, etc features exist because presumably because OS processes and threads consume too many resources to feasibly serve a similar role. However nobody seems to have any hard numbers about the topic. How bad is a Linux context switch? How much memory does a process use? Suddenly everyone is a micro optimizer seeking to gain a small constant multiple. Since this is not ready at hand I suspect the rewards are much less clear. It’s more likely that the languages benefit from greater cross platform control and want to avoid cross platform inconsistencies.
• neonsunset 2y ago
  To add a data point for Elixir: https://gist.github.com/neon-sunset/8fcc31d6853ebcde3b45dc7a2af32127#tldr https://gist.github.com/neon-sunset/8fcc31d6853ebcde3b45dc7a... Note 1: The gist is in Ukrainian, and the blog post by Steve does a much better job, but hopefully you will find this useful. Feel free to replicate the results and post them. Note 2: The absolute numbers do not necessarily imply good/bad. Both Go and BEAM focus on userspace scheduling and its fairness. Stackful coroutines have their own advantages. I think where the blog post's data is most relevant is understanding the advantages of stackless coroutines when it comes to "highly granular" concurrency - dispatching concurrent requests, fanning out to process many small items at once, etc. In any case, I did not expect sibling comments to go onto praising Node.js, is it really that surprising for event loop based concurrency? :) Also, if you are an Elixir aficionado and were impressed by C#'s numbers - know that they translate ~1:1 to F# now that it has task CE, just sayin'. Here's how the program looks in F#: open System open System.Threading.Tasks let argv = Environment.GetCommandLineArgs() [1..int argv[1]] |> Seq.map (fun _ -> Task.Delay(TimeSpan.FromSeconds 10.0)) |> Task.WaitAll
  • eproxus 2y ago
    Note that the Task library in Elixir uses supervised processes so it adds a lot more overhead. It would be interesting to see the benchmark with just normal Erlang processes.
• 0xcoffee 2y ago
  The C# version will copy the list into an array during Task.WhenAll, it may save some memory to use an array directly. Souce: https://github.com/microsoft/referencesource/blob/master/mscorlib/system/threading/Tasks/Task.cs#L5998 https://github.com/microsoft/referencesource/blob/master/msc...
  • neonsunset 2y ago
    It doesn't take that much space, and not all languages have option to easily map an initial range onto an iterator that produces tasks. Most are dominated by the size of state machines/virtual threads. Please note that the link above leads to old code from .NET Framework. The up-to-date .NET implementation lives here: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs#L5933-L6489 https://github.com/dotnet/runtime/blob/main/src/libraries/Sy...
  • MarkSweep 2y ago
    The referencesource repository is only relevant if you are using the legacy .NET Framework. Modern .NET has a special case for passing a List<Task> and avoids the allocation: https://github.com/dotnet/runtime/blob/1f01bee2a41e0df97089fe083c09c59401437b49/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs#L5947 https://github.com/dotnet/runtime/blob/1f01bee2a41e0df97089f...
  • deleted 2y ago
    [deleted]
• promiseofbeans 2y ago
  It would be nice if the author also compared different runtimes (e.g. NodeJS vs Deno, or cpython vs pypy) and core language engines (e.g. v8 vs spider monkey vs JavaScript core)
• lowyek 2y ago
  depends on the tasks.
  • deleted 2y ago
    [deleted]
• bilbo-b-baggins 2y ago
  This benchmark is nonsense. Apart from the fact that Go has an average Goroutine overhead of a 4kB stack (meaning an average usage of 3.9GB for 1M tasks), the code written is also in a closure, and scheduling a 2nd Goroutine in the wg.Done(), so unlike some of the others it had at least 2M function calls on the event loop stack in addition to at least 1M closure references. So yeah, it’s a great example of bad code in any language.
  • neonsunset 2y ago
    Here's an implementation in C# that more faithfully matches what you have to do in Go: var cnt = int.Parse(args[0]); var evt = new CountdownEvent(cnt); for (var i = 0; i < cnt; i++) { async Task Execute() { await Task.Delay(TimeSpan.FromSeconds(10)); evt.Signal(); } _ = Execute(); } evt.Wait(); It ends up consuming roughly 264.5 MB on ARM64 macOS 15.1.1 (compiled with NativeAOT).
• deleted 2y ago
  [deleted]
• tzahifadida 2y ago
  This test is no real. These languages works differently. Garbage collected and manually allocating and deallocating. If you do not configure a garbage collected language correctly it will spin out of control in memory consumption because it will just not garbage collect. If you would have configured the garbage collection to be low for java and go then go would look like rust.
• afavour 2y ago
  Maybe I’m missing something here but surely Node isn’t doing anything concurrently? Promises don’t execute concurrently, they just tidy up async execution. The code as given will just sequentially resolve a million promises. No wonder it looks so good. You’d need to be using workers to actually do anything concurrently.
  • charlotte-fyi 2y ago
    That's not entirely true. There's a thread pool of workers underneath libuv. Tasks that would block do indeed execute concurrently.
    • afavour 2y ago
      Oh, I know. But the code used in this test doesn’t utilize that thread pool at all. It just uses setTimeout.
  • Izkata 2y ago
    You're thinking of parallelism. Concurrency doesn't require them to actually be running at the same time.
    • afavour 2y ago
      Fair point. Kind of makes the comparisons between languages a little unfair, though. Go and Rust would be executing these operations in parallel, Node would not. Would make a significant difference to real world performance!
      • Izkata 2y ago
        The measurement is memory, not performance. Paused/queued tasks in the sequential node version still count, and in theory could be worse since the Go and Rust ones would be consuming them in parallel and not building up the queue as much.
        • 2y ago
• Mawr 2y ago
  > Now Go loses by over 13 times to the winner. It also loses by over 2 times to Java, which contradicts the general perception of the JVM being a memory hog and Go being lightweight. Well, if it isn't the classic unwavering confidence that an artificial "hello world"-like benchmark is in any way representative of real world programs.
  • phillipcarter 2y ago
    Yes, but also, languages like Java and C# have caught up a great deal over the past 10 years and run incredibly smoothly. Most peoples' perception of them being slow is really just from legacy tech that they encountered a long time ago, or (oof) being exposed to some terrible piece of .NET Framework code that's still running on an underprovisioned IIS server.
  • fulafel 2y ago
    Also of course Java and C# overwhelmingly use threads and not async for this kind of thing.
    • neonsunset 2y ago
      Very few .NET projects rely on explicit threading. There is almost never a reason to do so when tasks can be used instead, like in this benchmark.
• aba_cz 2y ago
  Regarding Java I'm pretty sure that benchmark is broken at least a little bit and testing something else as not specifying initial size for ArrayList means list of size 10 which gets resized all the time when `add()` is called, leading to big amount of unused objects needing garbage collection.
  • brabel 2y ago
    Yeah that is a junior mistake... They should've pre-sized the ArrayList, or better, used an array because that's more memory efficient (and I would say would be what any decent dev would do when the size of tasks is known beforehand). > Some folks pointed out that in Rust (tokio) it can use a loop iterating over the Vec instead of join_all to avoid the resize to the list Right, but some folks also pointed out you should've used an array in Java in the previous blog post, 2 years ago, and you didn't do that. And folks also pointed out Elixir shouldn't have used Task in the previous benchmark (folk here being the creator of Elixir himself): https://github.com/pkolaczk/async-runtimes-benchmarks/pull/7 https://github.com/pkolaczk/async-runtimes-benchmarks/pull/7
    • sfn42 2y ago
      The difference between an arraylist with correct initial size and an array is almost nothing. Arraylist itself is just a wrapper around an array.
      • brabel 2y ago
        It can be a big difference if boxing is involved. Or if the list is very big, because all access to items in the list require casting at the bytecode level (due to type erasure).
  • bekantan 2y ago
    It would indeed be better to create appropriately sized storage. However, I don't think that underlying array is resized every time `add` is called. I'd expect that resize will happen less than 30 times for 1M adds (capacity grows geometrically with a=10 and r=1.5)
• abdellah123 2y ago
  Can we do something more real world at least? what's the cost (hetzner monthly) of maintaining 1M concurrent websocket connection where each make a query to a postgres db randomly every 1-4 seconds. The cost wouldn't be just Memory because the network card and CPU also enter the game.
• samsartor 2y ago
  Reminder that Rust does not automatically schedule anything. Unless you _explicitly_ call `tokio::spawn` or `async_std::spawn` you are still living entirely in state-machine land. Rust's `join_all` uses `FuturesUnordered` behind the scenes, which is pretty intelligent in terms of keeping track of which tasks are ready to make progress, but it does not use tokio/async_std for scheduling. AFAICT the only thing being measured about tokio/async_std is the heap size of their `sleep` implementations. I'd be very interesting in seeing how Tokio's actual scheduler performs. The two ways to do that are: - using https://docs.rs/tokio/latest/tokio/task/join_set/struct.JoinSet.html https://docs.rs/tokio/latest/tokio/task/join_set/struct.Join... to spawn all the futures and then await them - spawn each future in the global scheduler, and then await the JoinHandles using the for loop from the appendix As other commenters have noted, calling `sleep` only constructs a state machine. So the Appendix isn't actually concurrent. Again, you need to either put those state machines into the tokio/async_std schedulers with `spawn`, or combine the state machines with `FuturesUnordered`.
  • jhgg 2y ago
    I did this measurement, and using time -v, the maximum resident size in KB comes out to 440,424 kb for 1m tasks, 46,820 kb for 100k, and 7,156 kb for 10k.
• maxyurk 2y ago
  https://eli.thegreenplace.net/2018/measuring-context-switching-and-memory-overheads-for-linux-threads/ https://eli.thegreenplace.net/2018/measuring-context-switchi... it's not a good enough example of a "task" to reflect real world usage.
• citrin_ru 2y ago
  It would be more interesting to see a benchmark where a task will not be empty but would have an open network connection e.g. would make an HTTP request to a test server with 10 seconds response time. Network is a frequent reason real world applications spawn 1M tasks.
• davidelettieri 2y ago
  Do we know how memory consumption?
• pjmlp 2y ago
  While the idea in general is interesting, the whole site is unreadable in Firefox, text gets all over the place the graphics.
  • mrweasel 2y ago
    Seems fine to me.
    • pjmlp 2y ago
      Strange, killing Firefox and going again did it for me. I guess some FF bug then. Thanks.
• iforgotpassword 2y ago
  Can someone explain the node version to me? My js knowledge is from a decade ago. AFAIK, setTimeout creates a timer and returns a handle to it. What does promisify do? I'd assume it's a general wrapper that takes a function that returns X and wraps it so that it returns Promise<X>. So that code actually runs 10k tasks that each create a timer with a timeout of 10 seconds and return immediately.
  • throwitaway1123 2y ago
    Promisify converts a callback based function into a promise returning function [1]. If the function has a `promisify.custom` method, `promisify` will simply return the `promisify.custom` method instead of wrapping the original function. Calling `promisify` on `setTimeout` in Node is redundant because Node already ships a built in promisified version of `setTimeout`. So the following is true: setTimeout[promisify.custom] === require('node:timers/promises').setTimeout You could of course manually wrap `setTimeout` yourself as well: const sleep = n => new Promise(resolve => setTimeout(resolve, n)) [1] https://nodejs.org/docs/latest-v22.x/api/util.html#utilpromisifyoriginal https://nodejs.org/docs/latest-v22.x/api/util.html#utilpromi...
• m4r1k 2y ago
  For those interested in Java Native Image, I previously wrote about its advantages beyond just memory footprint reduction, such as the dramatic improvement in startup time -> https://medium.com/google-cloud/cut-container-startup-time-for-better-performance-and-costs-part2-3a30edf2b82c https://medium.com/google-cloud/cut-container-startup-time-f...
• arresin 2y ago
  123 MB of ram to run 1 million tasks in rust. That is mind blowing.
• martypitt 2y ago
  Seriously impressive results from C#. I'm a JVM guy by day, and long-time admirer of C# as a language, but always assumed the two were broadly comparable performance-wise. This is a sample of 1 usecase, (so questionable real-worldness) but the difference is really eye-opening. Congrats to the C# team!
  • ReptileMan 2y ago
    Just a minor nitpick - this is the .NET Runtime vs JVM. My personal observations are that CPU wise they are close, but for reasons unknown JVM has always been more memory hoggish. The JIT compiler that microsoft created has been nothing short of amazing.
• piterrro 2y ago
  I don't know what's a fair way to do this for all languages listed in the benchmark, but for Go vs Node the only fair way would be to use a single goroutine to schedule timers and another one to pick them up when they tick, this way we don't create a huge stack and it's much more comparable to what you're really doing in Node. Consider the following code: package main import ( "fmt" "os" "strconv" "time" ) func main() { numTimers, _ := strconv.Atoi(os.Args[1]) timerChan := make(chan struct{}) // Goroutine 1: Schedule timers go func() { for i := 0; i < numTimers; i++ { timer := time.NewTimer(10 * time.Second) go func(t *time.Timer) { <-t.C timerChan <- struct{}{} }(timer) } }() // Goroutine 2: Receive and process timer signals for i := 0; i < numTimers; i++ { <-timerChan } } Also for Node it's weird not to have Bun and Deno included. I suppose you can have other runtimes for other languages too. In the end I think this benchmark is comparing different things and not really useful for anything...
• blixt 2y ago
  While it’s nice to compare languages with simple idiomatic code I think it’s unfair to developers to show them the performance of an entirely empty function body and graphs with bars that focus on only one variable. It paints a picture that you can safely pick language X because it had the smaller bar. I urge anyone making decisions from looking at these graphs to run this benchmark themselves and add two things: - Add at least the most minimal real world task inside of these function bodies to get a better feel for how the languages use memory - Measure the duration in addition to the memory to get a feel for the difference in scheduling between the languages
  • tossandthrow 2y ago
    This urge is as old as statistics. And I dare to say that most people after reading the article in question are well prepared to use the results for what they are.
    • blixt 2y ago
      I can’t say I share your optimism. I’ve seen plenty of developers point to graphs like these as a reason for why they picked a language or framework for a problem. And it comes down to the benchmark how good of a proxy it actually is for such problems. I just hope that with enough feedback the author would consider making the benchmark more nuanced to paint a picture of why these differences in languages exist (as opposed to saying which languages “lose” or “win”).
    • sfn42 2y ago
      And by use the results for what they are you mean ignore them because they are completely useless?
• jillesvangurp 2y ago
  Did a similar benchmark in Kotlin using co-routines. import kotlin.time.Duration.Companion.milliseconds import kotlin.time.measureTime import kotlinx.coroutines.async import kotlinx.coroutines.awaitAll import kotlinx.coroutines.coroutineScope import kotlinx.coroutines.delay suspend fun main() { measureTime { coroutineScope { (0..1000000).map { async { delay(1.milliseconds) } }.awaitAll() } }.let { t -> println("Took $t") val runtime = Runtime.getRuntime() val maxHeapSize = runtime.maxMemory() val allocatedHeapSize = runtime.totalMemory() val freeHeapSize = runtime.freeMemory() println("Max Heap: ${maxHeapSize / 1024 / 1024} MB") println("Allocated Heap: ${allocatedHeapSize / 1024 / 1024} MB") println("Free Heap: ${freeHeapSize / 1024 / 1024} MB") } } This produces the following output: Took 1.597011084s Max Heap: 4096 MB Allocated Heap: 2238 MB Free Heap: 1548 MB So whatever is needed to load classes and a million co-routines with some heap state. Of course the whole thing isn't doing any work and this isn't much of a benchmark. And of course if I run it with kotlin-js it actually ends up using promises. So, it's not going to be any better there than on the JVM.
• rcarmo 2y ago
  No Erlang, though. That ought to be amazingly small for that kind of synthetic benchmark.
• win32_func 2y ago
  The JS version can quickly be improved to use less memory (~10%). async function main() { const numTasks = parseInt(process.argv[2], 10); const taskDuration = 10000; // 10 seconds const tasks = Array.from({ length: numTasks }, () => new Promise(resolve => setTimeout(resolve, taskDuration)) ); await Promise.all(tasks); // Wait for all tasks to resolve console.log("All tasks completed."); } main().catch(err => { console.error("Error occurred:", err); });
• jeswin 2y ago
  Good to see NativeAOT getting positive press. Go won because it served a need felt by many programmers: a garbage-collected language which compiled to native code, with robust libraries supported by a large corp. With Native AOT, C# is walking into the same space. With arguably better library selection, equivalent performance, and native code compilation. And a much more powerful, well-thought-out language - at a slight complexity cost. If you're starting a project today (with the luxury of choosing a language), you should give C# + NativeAOT a consideration.
  • sfn42 2y ago
    C# is my daily driver and I'd use it for almost anything, great language. However I think "slight complexity cost" is an understatement. It's a very complex language by my standards, and they keep adding more stuff. A lot of it is just syntax sugar to do the same things in a different way, like primary constructors. It's nice to have that stuff when you know the language, but it does make the learning curve steeper and it can be a bit annoying when working in a team. Even after 4 years of using it professionally I still see code some times that uses obscure syntax I had no idea existed. I would describe C# as a language for experts. If you know what you're doing it's an amazing language, maybe actually the best current programming language. But learning and understanding everything is a monumental task, simpler languages like go or Java can be learned much faster.
• xargon7 2y ago
  There's a difference between "running a task that waits for 10 seconds" and "scheduling a wakeup in 10 seconds". The code for several of the languages that are low-memory usage that do the second while the high memory usage results do the first. For example, on my machine the article's go code uses 2.5GB of memory but the following code uses only 124MB. That difference is in-line with the rust results. package main import ( "os" "strconv" "sync" "time" ) func main() { numRoutines, _ := strconv.Atoi(os.Args[1]) var wg sync.WaitGroup for i := 0; i < numRoutines; i++ { wg.Add(1) time.AfterFunc(10*time.Second, wg.Done) } wg.Wait() }
  • neonsunset 2y ago
    Spawning a periodically waking up Task in .NET (say every 250ms) that performs work like sending out a network request would retain comparable memory usage (in terms of async overhead itself). Even at 100k tasks the bottleneck is going to be the network stack (sending outgoing 400k RPS takes a lot of CPU and syscall overhead, even with SocketAsyncEngine!). Doing so in Go would require either spawning Goroutines, or performing scheduling by hand or through some form of aggregation over channel readers. Something that Tasks make immediately available. The concurrency primitive overhead becomes more important if you want to quickly interleave multiple operations at once. In .NET you simply do not await them at callsite until you need their result later - this post showcases how low the overhead of doing so is.
  • mrighele 2y ago
    I agree with you. Even something as simple as a loop like (pseudocode) for (n=0;n<10;n++) { sleep(1 second); } Changes the results quite a bit: for some reasons java use a _lot_ more memory and takes longer (~20 seconds), C# uses more that 1GB of memory, while python struggles with just scheduling all those tasks and takes more than one minute (beside taking more memory). node.js seems unfazed by this change. I think this would be a more reasonable benchmark
    • neonsunset 2y ago
      Indeed, looping over a Task.Delay likely causes a lot of churn in timer queues - that's 10M timers allocated and scheduled! If it is replaced with 'PeriodicTimer', the end result becomes more reasonable. This (AOT-compiled) F# implementation peaks at 566 MB with WKS GC and 509 MB with SRV GC: open System open System.Threading open System.Threading.Tasks let argv = Environment.GetCommandLineArgs() [1..int argv[1]] |> Seq.map (fun _ -> task { let timer = PeriodicTimer(TimeSpan.FromSeconds 1.0) let mutable count = 10 while! timer.WaitForNextTickAsync() do count <- count - 1 if count = 0 then timer.Dispose() } :> Task) |> Task.WaitAll To Go's credit, it remains at consistent 2.53 GB and consumes quite a bit less CPU. We're really spoiled with choice these days in compiled languages. It takes 1M coroutines to push the runtime and even at 100k the impact is easily tolerable, which is far more than regular applications would see. At 100K .NET consumes ~57 MB and Go consumes ~264 MB (and wins at CPU by up to 2x).
• Comma2976 2y ago
  I'd be ashamed to publish such shoddy "work" using any real account or name
• ww520 2y ago
  How do they measure the memory usage?
• bob_alderman 2y ago
  good thing you didn't benchmark ruby because we probably couldn't see the other chart bars.
• jeffreygoesto 2y ago
  How would Erlang/Elixir do in this?
• datadeft 2y ago
  No Erlang/Elixir?? The author would be surprised.
  • skyake 2y ago
    Erlang/Elixir failed the 1M tasks benchmark that's why it was excluded.
    • SirGiggles 2y ago
      Most likely they did not adjust +P which bumps the max process limit
• JyB 2y ago
  I’m still baffled that some people are bold enough to voluntarily posts those kind of most-of-the-time useless “benchmark” that will inevitably be riddled with errors. I don’t know what pushes them. In the end you look like a clown more often than not.
  • enginoid 2y ago
    Trying things casually out of curiosity isn’t harmful. I expect people understand that these kinds of blog posts aren’t rigorous science to draw foundational conclusions from. And the errors are a feature — I learn the most from the errata!
  • wiseowise 2y ago
    The fastest way to learn truth is by posting wrong thing on the internet, or something.
• nolist_policy 2y ago
  Reminds me of the famous SCO: "thread creation is about a thousand times faster than on native Linux" linux kernel mailing list thread where Linus Torvalds replies Talk is cheap. Show me the code. https://lkml.org/lkml/2000/8/25/132 https://lkml.org/lkml/2000/8/25/132 https://lkml.org/lkml/2000/8/26/52 https://lkml.org/lkml/2000/8/26/52
• voodooEntity 2y ago
  I came here to rage (im just honest) because the go code example is bad and absolute not representative. Im coding go code for multiple years, especially alot of multithreading, and what is presented there as result is just wrong. Apart from no necessaty to use a waitgroup for threading, as many others here already have stated even with waitgroup you cacn reduce the memory significantly down to like 130mb for 1mio threads. Also some other languages seem to be missrepresented. Seems like someone had good intentions but no idea about the languages he tried to compare and the result is this article.
• indulona 2y ago
  Go's performance has been neglected in past 10y. I even posted about it https://news.ycombinator.com/item?id=41827288 https://news.ycombinator.com/item?id=41827288
• piokoch 2y ago
  It seems there are just two clubs: you go with bare metal (Rust, C# native AOT) or you use some higher level abstraction (virtual machine, garbage collector) and then there is no significant difference between Java, Node, Go or Python. For me Python worked surprisingly well, while Go was surprisingly high on memory consumption.
• jakobnissen 2y ago
  Just tried this in Julia: 16.0 GB of memory for 1M tasks! I believe each task in Julia has its own stack, so this makes sense. Still, it does mean you've got to take account of ~16 KB of memory per running task which is not great.
• signa11 2y ago
  fta, please consider reading this: https://hauleth.dev/post/beam-process-memory-usage/ https://hauleth.dev/post/beam-process-memory-usage/ as well to see how erlang (elixir) fares as well.
• anonymousDan 2y ago
  Would be interesting to see the numbers for elixir/Erlang.
• deleted 2y ago
  [deleted]
• joshka 2y ago
  RUST The rust code is really checking how big Tokio's structures that track timers are. Solving the problem in a fully degenerate manner, the following code runs correct correctly and uses only 35MB peak. 35 bytes per future seems pretty small. 1 billion futures was ~14GB and ran fine. #[tokio::main] async fn main() { let sleep = SleepUntil { end: Instant::now() + Duration::from_secs(10), }; let timers: Vec<_> = iter::repeat_n(sleep, 1_000_000_0).collect(); for sleep in timers { sleep.await; } } #[derive(Clone)] struct SleepUntil { end: Instant, } impl Future for SleepUntil { type Output = (); fn poll(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Self::Output> { if Instant::now() >= self.end { Poll::Ready(()) } else { cx.waker().wake_by_ref(); Poll::Pending } } } Note: I do understand why this isn't good code, and why it solves a subtly different problem than posed (the sleep is cloned, including the deadline, so every timer is the same). The point I'm making here is that synthetic benchmarks often measure something which doesn't help much. While the above is really degenerate, it shares the same problems as the article's code (it just leans into problems much harder).
• wiseowise 2y ago
  Remember, folks, none of it matters if nobody uses your software.
• vrnvu 2y ago
  Conclusion by author: > Now Go loses by over 13 times to the winner. It also loses by over 2 times to Java, which contradicts the general perception of the JVM being a memory hog and Go being lightweight. Note that Go and Java code are not doing the same! See xargon7 comment.
• octacat 2y ago
  Where is erlang? Sleeping is not running, by the way. If you just sleep, in Erlang you would use a hibernated process. I feel this is so misleading. For example, by default after spawning, Erlang would have some memory preallocated for each process, so they don't need to ask the operation system for new allocations (and if you want to shrink it, you call hibernate). Do something more real, like message passing with one million processes or websockets. Or 1M tcp connections. Because, the moment you send messages, here is when the magic happens (and memory would grow, the delay when each message is processed would be different in different languages). Oh, and btw, if you want to do THAT in erlang, use timer:apply_after(Time, Module, Function, Arguments). Which would not spawn an erlang process, just would put the task to the timer scheduling table. And Elixir was in the old article, and they implemented it all wrong. Sad.
• reverseblade2 2y ago
  It is odd that C# version doesn't increase list capacity: List<Task> tasks = new List<Task>(numTasks);
• renox 2y ago
  .Net AoT looks good but when you look at the compatibility matrix Android and iOS are "experimental".. https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/?tabs=windows%2Cnet8 https://learn.microsoft.com/en-us/dotnet/core/deploying/nati...
• polyrand 2y ago
  Out of curiosity, I checked if using uvloop[0] in Python changed the numbers. This is the code: # /// script # requires-python = ">=3.12" # dependencies = ["uvloop"] # /// import asyncio import sys import uvloop async def main(num_tasks): tasks = [] for task_id in range(num_tasks): tasks.append(asyncio.sleep(10)) await asyncio.gather(*tasks) if __name__ == "__main__": num_tasks = int(sys.argv[1]) # uvloop.run(main(num_tasks)) asyncio.run(main(num_tasks)) I ran it with 100k tasks: /usr/bin/time -l -p -h uv run async-memory.py 100000 On my M1 MacBook Pro, using asyncio reports (~170MB): 170835968 maximum resident set size Using uvloop (~204MB): 204259328 maximum resident set size I kept the `import uvloop` statement when just using asyncio so that both cases start in the same conditions. [0]: https://github.com/MagicStack/uvloop/ https://github.com/MagicStack/uvloop/
• deostroll 2y ago
  Did anyone do an analysis on what happens after the test? Like in nodejs if we keep the application running after the so called test and left it idle for 1 or more hours. Does the memory come down?