6 ms·

Oh, hey, didn't expect to see this here. Thanks for submitting! I tried a few days ago but didn't really get any traction.[0] Since I submitted it, though, I p

Oh, hey, didn't expect to see this here. Thanks for submitting! I tried a few days ago but didn't really get any traction.[0] Since I submitted it, though, I posted it to /r/rust and got a lot of feedback. At the time, rust and dotnet were comparable and at the top of the list, with ~10k req/sec. Now rust is far and away the most performant at ~20k req/sec! I also was able to improve Go's performance 30% or so. Still, I want to let the other communities chip in and see if I can improve them. I was actually just in the midst of exploring how to improve Elixir's results. I'm finding I can almost double my requests per second from simply switching the JSON encoder from Jason to Jiffy. That sort of surprised me since Jason is the de-facto standard, and I thought was super fast. [0] https://news.ycombinator.com/item?id=33291604 https://news.ycombinator.com/item?id=33291604

di4na 4y ago
Are you on OTP 25 ? With the JIT, in theory Jason is faster than Jiffy
- di4na 4y ago
  Another thing to add, do you log things in the other languages ? If not, what you are probably benchmarking there is how fast your console is in receiving the text (with a lock). Also use a release, it will help some things here, especially on "short" benchmark, and make sure that the JIT is run.
  - losvedir 4y ago
    Yep, Elixir 1.14, OTP 25. I set the log level to warn so Phoenix isn't logging anything that I see. Do you think that still impacts performance? I thought Logger compiled out log calls below the level threshold. Good call on running as a release! I'll try that next. edit: Ah well, good thoughts but didn't pan out. I updated the logger config to use `compile_time_purge_matching` just to make sure there wouldn't be any logging impact, and I ran the app as a release, but didn't really make any difference in the numbers that I saw.
bhaney 4y ago
> I'm finding I can almost double my requests per second from simply switching the JSON encoder from Jason to Jiffy Personal plug, but have you tried Jsonrs[0]? I typically get much better performance out of it (especially in lean mode, which seems to be the mode you'd want to use for this benchmark) than Jiffy for large JSON encoding workloads. [0] https://hexdocs.pm/jsonrs/readme.html https://hexdocs.pm/jsonrs/readme.html
- losvedir 4y ago
  This is great! Just pushed up a commit that uses it and updated the benchmarks[0]. I'm seeing a 1.6X - 2X improvement in overall performance. Not bad for a drop-in replacement. And since it's based on serde, I trust it, and I feel like trying out a different JSON library is within scope for me of not just "gaming the benchmarks", as this is actually something I'd now consider using at work. It's not quite as high as I was seeing with `jiffy` (3,800 req/sec here vs 4,000+ with jiffy), but I'm not confident that was a totally fair comparison. `jiffy` doesn't integrate as nicely with Phoenix, so I was just calling `:jiffy.encode(...)` in the controller and then doing a `text(...)` response. I need to double-check if `json(...)` is doing more work here. [0] https://github.com/losvedir/transit-lang-cmp/commit/140d693b62522c2ae839a3a944054ba04d2d3e37 https://github.com/losvedir/transit-lang-cmp/commit/140d693b...
  - bhaney 4y ago
    One major difference you'll run into here is support for encoding protocols. Jason, Poison, and Jsonrs (by default) will have protocol impls for Elixir structs that define special logic for how to serialize those structs into JSON strings. ie, you probably want a DateTime struct to serialize as a date string rather than a map containing the year, month, etc. Jiffy doesn't support any of that, so you end up with differing behavior between it and other libraries as soon as you start encoding structs that aren't meant to serialize as maps. Jsonrs supports encoding protocols by default, but lets you turn it off for a speed boost if you're okay with more Jiffy-like behavior. Plugging it in as Phoenix's JSON library won't turn off that protocol handling, so you're getting the slower operational mode of Jsonrs (which is fine and probably what you want in most cases for correctness, but will definitely eat a few ops/sec in a benchmark)
ad404b8a372f2b9 4y ago
Did you post on the elixir forum? Those results seem very low, I feel like there has to be something amiss.
- cklee 4y ago
  I posted to the Elixir Forum https://elixirforum.com/t/an-informal-comparison-of-several-programming-languages/51285 https://elixirforum.com/t/an-informal-comparison-of-several-... ---- It's not surprising that Elixir is not good at large json handling and complicated data structure manipulation. This reminds me of https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users https://discord.com/blog/using-rust-to-scale-elixir-for-11-m... At the same time I see that Elixir gave the most consistent response time. Max response time is less than 6x of median compared to ~x10 (Deno / C#) or x23 (Go / Rust) / x58 (Scala), probably thanks to preemtive scheduling of per-request process in Erlang VM
pg_bot 4y ago
The elixir implementation seems super slow. The first thing that jumps out to me is that we should be using streams instead of just reading the file. The second would be the `schedule_for_route` which should just be a GenServer lookup instead of reading from ets.
- losvedir 4y ago
  Yeah, I should benchmark streams. In my experience streams are rarely faster, though, and are instead something you should reach for if 1) you can't fit the file in memory or 2) you might stop the enumeration early. Regarding GenServer, I'm not so sure about that. I suppose I should benchmark it, but intuitively I expect ETS to be better here. There's more overhead in getting the data, but it allows you to concurrently read it from different processes. A GenServer, meanwhile, could respond to a given message faster, but now you're serializing the (e.g.) 50 concurrent virtual users through a single bottleneck. I could have multiple copies of the data in several GenServers, I suppose, at the expense of much more memory use.
  - pg_bot 4y ago
    I would guess that there's too much work being done at runtime. I would expect all of the data to be cached in such a way that it is a read operation without any transformation. It's doing multiple lookups and unnecessary maps when the data should just be formatted a single time and cached. You could even skip the JSON transformation by just doing it once if you want to get super fast.
  - Kabie 4y ago
    Using stream can be faster here. Because you don't have to rebuild the same huge list again and again.
- innocentoldguy 4y ago
  Agreed. I've seen significant performance improvements by using streaming with binary pattern-matching to parse files rather than loading them into memory and using String functions to get at the data. These Elixir numbers seem suspect to me, based on my company's extensive benchmarking on many of these same languages. Elixir is somewhat slow at some tasks, but serving up web requests isn't one of them.
int_19h 4y ago
Regarding C#: > The "billion dollar mistake" is important to me, and while C# has non-nullability sugar in its typesystem (i.e. with ? after a number of types), the type system wasn't as rigorous as I was maybe hoping. I'm genuinely curious if you have any examples? Nullability checks may be bolted on, but once you enable them, they should consistently prevent you from dealing with any null values that you haven't explicitly allowed. And other than that, the only hole in the type system that I can think of is array covariance, which was unfortunately common at the time (Java had it also) as a way to skimp on generics, but which doesn't occur often today because using generic collections is much more common. > At one point I had a bug because I did a stopWatch.Elapsed / 1000 by accident instead of stopWatch.ElapsedTicks / 1000. The former is a TimeSpan struct instead of a long like ElapsedTicks, so intuitively it feels like I shouldn't be able to divide it, though it did a best effort and did something to it, though I'm not quite sure what. This particular one doesn't have anything to do with the type system per se, it's just the way TimeSpan itself works. I'm not sure why "intuitively it feels like I shouldn't be able to divide it", since there are fairly obvious definitions of arithmetic operations on spans and numbers - if T is 10 seconds, then surely T*2 is 20 seconds, and T/2 is 5 seconds? Which is exactly what TimeSpan does by overloading the corresponding operators: https://learn.microsoft.com/en-us/dotnet/api/system.timespan#operators https://learn.microsoft.com/en-us/dotnet/api/system.timespan... Note that the type of (TimeSpan/double) is still TimeSpan, so the type system is still enforcing proper use - it wouldn't have let you assign it to an int or a double. But it looks like you were just printing it, which is legal for any type: Console.WriteLine($"loaded stop_times.txt in {watch.ElapsedMilliseconds} ms");
- Someone 4y ago
  > I'm genuinely curious if you have any examples? Nullability checks may be bolted on, but once you enable them, they should consistently prevent you from dealing with any null values that you haven't explicitly allowed. I’m not the one you asked, but there’s a difference between you no longer having to deal with null checks and your program no longer having to deal with null checks. The CLR will still do the null checks and fail your program fast when needed. Other languages will require you to properly handle them (or at least, make it obvious where you chose to bluntly abort the program when they occur) I also don’t think C# will prevent you from forgetting to do a if null check in cases where you expect values to sometimes be null.
chrismorgan 4y ago
For anyone interested about the “posted it to /r/rust and got a lot of feedback”: https://old.reddit.com/r/rust/comments/ya4xfw/why_is_cdotnet_outperforming_rust_in_my_simple/ https://old.reddit.com/r/rust/comments/ya4xfw/why_is_cdotnet...
zigzag312 4y ago
Few tips to improve C# performance: - enable source generation for JSON serialization https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/source-generation?pivots=dotnet-6-0#source-generation-support-in-aspnet-core https://learn.microsoft.com/en-us/dotnet/standard/serializat... - use `TryGetValue` to do single dictionary lookup instead of two. if (TripsIxByRoute.TryGetValue(route, out var tripIxs)) - specify List's capacity, if it's known when creating it, to avoid resizing operations while adding elements to the List var schedules = new List<StopTimeResponse>(stopTimeIxs.Count);

di4na 4y ago

Are you on OTP 25 ? With the JIT, in theory Jason is faster than Jiffy

di4na 4y ago
Another thing to add, do you log things in the other languages ? If not, what you are probably benchmarking there is how fast your console is in receiving the text (with a lock). Also use a release, it will help some things here, especially on "short" benchmark, and make sure that the JIT is run.
- losvedir 4y ago
  Yep, Elixir 1.14, OTP 25. I set the log level to warn so Phoenix isn't logging anything that I see. Do you think that still impacts performance? I thought Logger compiled out log calls below the level threshold. Good call on running as a release! I'll try that next. edit: Ah well, good thoughts but didn't pan out. I updated the logger config to use `compile_time_purge_matching` just to make sure there wouldn't be any logging impact, and I ran the app as a release, but didn't really make any difference in the numbers that I saw.

bhaney 4y ago

> I'm finding I can almost double my requests per second from simply switching the JSON encoder from Jason to Jiffy Personal plug, but have you tried Jsonrs[0]? I typically get much better performance out of it (especially in lean mode, which seems to be the mode you'd want to use for this benchmark) than Jiffy for large JSON encoding workloads. [0] https://hexdocs.pm/jsonrs/readme.html https://hexdocs.pm/jsonrs/readme.html

losvedir 4y ago
This is great! Just pushed up a commit that uses it and updated the benchmarks[0]. I'm seeing a 1.6X - 2X improvement in overall performance. Not bad for a drop-in replacement. And since it's based on serde, I trust it, and I feel like trying out a different JSON library is within scope for me of not just "gaming the benchmarks", as this is actually something I'd now consider using at work. It's not quite as high as I was seeing with `jiffy` (3,800 req/sec here vs 4,000+ with jiffy), but I'm not confident that was a totally fair comparison. `jiffy` doesn't integrate as nicely with Phoenix, so I was just calling `:jiffy.encode(...)` in the controller and then doing a `text(...)` response. I need to double-check if `json(...)` is doing more work here. [0] https://github.com/losvedir/transit-lang-cmp/commit/140d693b62522c2ae839a3a944054ba04d2d3e37 https://github.com/losvedir/transit-lang-cmp/commit/140d693b...
- bhaney 4y ago
  One major difference you'll run into here is support for encoding protocols. Jason, Poison, and Jsonrs (by default) will have protocol impls for Elixir structs that define special logic for how to serialize those structs into JSON strings. ie, you probably want a DateTime struct to serialize as a date string rather than a map containing the year, month, etc. Jiffy doesn't support any of that, so you end up with differing behavior between it and other libraries as soon as you start encoding structs that aren't meant to serialize as maps. Jsonrs supports encoding protocols by default, but lets you turn it off for a speed boost if you're okay with more Jiffy-like behavior. Plugging it in as Phoenix's JSON library won't turn off that protocol handling, so you're getting the slower operational mode of Jsonrs (which is fine and probably what you want in most cases for correctness, but will definitely eat a few ops/sec in a benchmark)

ad404b8a372f2b9 4y ago

Did you post on the elixir forum? Those results seem very low, I feel like there has to be something amiss.

cklee 4y ago
I posted to the Elixir Forum https://elixirforum.com/t/an-informal-comparison-of-several-programming-languages/51285 https://elixirforum.com/t/an-informal-comparison-of-several-... ---- It's not surprising that Elixir is not good at large json handling and complicated data structure manipulation. This reminds me of https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users https://discord.com/blog/using-rust-to-scale-elixir-for-11-m... At the same time I see that Elixir gave the most consistent response time. Max response time is less than 6x of median compared to ~x10 (Deno / C#) or x23 (Go / Rust) / x58 (Scala), probably thanks to preemtive scheduling of per-request process in Erlang VM

pg_bot 4y ago

The elixir implementation seems super slow. The first thing that jumps out to me is that we should be using streams instead of just reading the file. The second would be the `schedule_for_route` which should just be a GenServer lookup instead of reading from ets.

losvedir 4y ago
Yeah, I should benchmark streams. In my experience streams are rarely faster, though, and are instead something you should reach for if 1) you can't fit the file in memory or 2) you might stop the enumeration early. Regarding GenServer, I'm not so sure about that. I suppose I should benchmark it, but intuitively I expect ETS to be better here. There's more overhead in getting the data, but it allows you to concurrently read it from different processes. A GenServer, meanwhile, could respond to a given message faster, but now you're serializing the (e.g.) 50 concurrent virtual users through a single bottleneck. I could have multiple copies of the data in several GenServers, I suppose, at the expense of much more memory use.
- pg_bot 4y ago
  I would guess that there's too much work being done at runtime. I would expect all of the data to be cached in such a way that it is a read operation without any transformation. It's doing multiple lookups and unnecessary maps when the data should just be formatted a single time and cached. You could even skip the JSON transformation by just doing it once if you want to get super fast.
- Kabie 4y ago
  Using stream can be faster here. Because you don't have to rebuild the same huge list again and again.
innocentoldguy 4y ago
Agreed. I've seen significant performance improvements by using streaming with binary pattern-matching to parse files rather than loading them into memory and using String functions to get at the data. These Elixir numbers seem suspect to me, based on my company's extensive benchmarking on many of these same languages. Elixir is somewhat slow at some tasks, but serving up web requests isn't one of them.

int_19h 4y ago

Regarding C#: > The "billion dollar mistake" is important to me, and while C# has non-nullability sugar in its typesystem (i.e. with ? after a number of types), the type system wasn't as rigorous as I was maybe hoping. I'm genuinely curious if you have any examples? Nullability checks may be bolted on, but once you enable them, they should consistently prevent you from dealing with any null values that you haven't explicitly allowed. And other than that, the only hole in the type system that I can think of is array covariance, which was unfortunately common at the time (Java had it also) as a way to skimp on generics, but which doesn't occur often today because using generic collections is much more common. > At one point I had a bug because I did a stopWatch.Elapsed / 1000 by accident instead of stopWatch.ElapsedTicks / 1000. The former is a TimeSpan struct instead of a long like ElapsedTicks, so intuitively it feels like I shouldn't be able to divide it, though it did a best effort and did something to it, though I'm not quite sure what. This particular one doesn't have anything to do with the type system per se, it's just the way TimeSpan itself works. I'm not sure why "intuitively it feels like I shouldn't be able to divide it", since there are fairly obvious definitions of arithmetic operations on spans and numbers - if T is 10 seconds, then surely T*2 is 20 seconds, and T/2 is 5 seconds? Which is exactly what TimeSpan does by overloading the corresponding operators: https://learn.microsoft.com/en-us/dotnet/api/system.timespan#operators https://learn.microsoft.com/en-us/dotnet/api/system.timespan... Note that the type of (TimeSpan/double) is still TimeSpan, so the type system is still enforcing proper use - it wouldn't have let you assign it to an int or a double. But it looks like you were just printing it, which is legal for any type: Console.WriteLine($"loaded stop_times.txt in {watch.ElapsedMilliseconds} ms");

Someone 4y ago
> I'm genuinely curious if you have any examples? Nullability checks may be bolted on, but once you enable them, they should consistently prevent you from dealing with any null values that you haven't explicitly allowed. I’m not the one you asked, but there’s a difference between you no longer having to deal with null checks and your program no longer having to deal with null checks. The CLR will still do the null checks and fail your program fast when needed. Other languages will require you to properly handle them (or at least, make it obvious where you chose to bluntly abort the program when they occur) I also don’t think C# will prevent you from forgetting to do a if null check in cases where you expect values to sometimes be null.

chrismorgan 4y ago

For anyone interested about the “posted it to /r/rust and got a lot of feedback”: https://old.reddit.com/r/rust/comments/ya4xfw/why_is_cdotnet_outperforming_rust_in_my_simple/ https://old.reddit.com/r/rust/comments/ya4xfw/why_is_cdotnet...

zigzag312 4y ago

Few tips to improve C# performance: - enable source generation for JSON serialization https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/source-generation?pivots=dotnet-6-0#source-generation-support-in-aspnet-core https://learn.microsoft.com/en-us/dotnet/standard/serializat... - use `TryGetValue` to do single dictionary lookup instead of two. if (TripsIxByRoute.TryGetValue(route, out var tripIxs)) - specify List's capacity, if it's known when creating it, to avoid resizing operations while adding elements to the List var schedules = new List<StopTimeResponse>(stopTimeIxs.Count);