5 ms·

An Analysis of the Performance of WebSockets in Various Programming Languages (2021)

87 points by max0563 2y ago

5Qn8mNbc2FNCiVV 2y ago
Too bad that uWebsockets was used for Node because a lot of higher level libraries are built on top of https://www.npmjs.com/package/ws https://www.npmjs.com/package/ws
- windlep 2y ago
  I was able to make a uWebsockets adapter for NestJS pretty easily. It's a bit sensitive of a library to integrate though, a single write when the connection is gone and you get a segfault, which means a lot of checking before writing if you've yielded since you last checked. This was a few years ago, perhaps they fixed that.
paulgb 2y ago
The SSRN link doesn’t have a login-wall: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3778525 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3778525
- chrisweekly 2y ago
  Thanks! Here's the direct link to the ungated PDF: https://download.ssrn.com/21/02/03/ssrn_id3778525_code4568915.pdf?response-content-disposition=inline&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEDQaCXVzLWVhc3QtMSJHMEUCIBz9ASyrOZb7lgu4c7TPazOVmNtXFIfbhGR%2BPNEQRaLXAiEA8J7SZDs1dYpi7gpDiCI78zaJZIWZHkXIhbu99ui%2B3%2FEqxgUIzf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAEGgwzMDg0NzUzMDEyNTciDIv7tdkuoYsFpo6F7yqaBcYKhbXn5qs7JiWcTskh8279TVWylAjLpM2l9SEz2cHG8mE6XYRUCxg3r9mnm2c3ouVX8RheZ3i%2BapmEy12Nf6E1%2FYKsH9OpU9%2BAVjOn0n8gaHUAEitd1avpiSk662x0b86p09X8Dy7NV1uXI27%2FYfxEGUWvrvfYC%2BvYrLBSDFEaWOAxqPWFAoxVtz0wyDk07pVY%2FbDydStaAaEUGws9e6%2FcSOKv4tA3gr1DtS3Wc8UMsPMOlV6oUHK0jFVZyo1pL%2F6jT8KCZSRAiubk0XTk1TRfcJtI5htH9lZEJMusuky%2BNRaRcuOsLeXt0bRZGRMyQ7fm74yTc9QEckQB%2B%2FZ0cb7%2Fw3hd7C7Sq%2B2Wy9zyWGxoejoxHDMtvga%2BiBH6%2FyQGSltBG358vO3grSynJ3rpAxCwcCNlwY2Pb1yWjCzodVFYAsn6AUALL8gjg4oRQdjHSrZs%2F0I7RPdVYsrPevaU0pM9xIMDRANcdQtGoTsYoPgzRP%2FfPcQRnVXE4nJ8GB%2FlNtazu%2BLgoHTo6JHe4ds1y6GGGy7CHeDnBRyXdqveDq%2Brgjozivf0l96Lc8kqjY8PpAtJvUTYhmz%2FQsy%2B41Wu7bv1QRsaogIgapIAdHkY8Xv6FyXvZyfCKLqVAv4C%2FRRrGKJwycFQJ%2BXzUpJc8vl79Cbwn%2Fmf1gdKPFjhMB0s2skkSOQUcojbQnxfILJKVG1Pv904wH%2BapVRkPCohkRL4%2BafWioVD5d74%2BqbstnZIRoJw0dO0AocLPH0FP%2BdWXQRLB2qNVqUQOhdCYXxXX7iPpGXGhhj3hk7cvFmrgwDADm6DWHgXLX8d1vrKUMEjGJPxcB4sb6CzKs3bPh4Mb4%2BPMLczUMfei%2B6a8f5iY5NWJhnu2kIHVJI%2BoNXilTCUl4W6BjqxAfAUk4qD%2FPqQmUX4ZGPsZe3rEzgKdJr9RWN2ui5sbn6cp5Z6%2BAiT2VNCxd7VYCq0ZdhuDf7YDMkGcTEIBaCVKFWqk6TFNa7VTK4NnIh519yHZIcpStx5hUgQXrKfadCQ8PCb8a7BNAIq1rIXPdjeyc1XOzw1QDbcM3nARYhJY6UlHOYjEznllSKDjK%2BFuWOzdvg%2B9IlHDSisGa6sYJtqXRPYQtgcpXVC9EspXg0FqJZSlQ%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20241123T042500Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Credential=ASIAUPUUPRWE7KW6RUGY%2F20241123%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=8a05d683177c04b274cb176dc107cda22c22e6f4180f2827578f5e6772c45f51&abstractId=3778525 https://download.ssrn.com/21/02/03/ssrn_id3778525_code456891... TLDR; NodeJS is the clear winner, and Python far and away the worst of the bunch.
travisgriggs 2y ago
Thanks for the free access links. I did read through a bit. The title is misleading because exactly one implementation was chosen for each of the tested languages. They conclude “do not us e Python” because the Python websockets library performs pretty poorly. Each language is scored based on the library chosen. I have to believe there are more options for some of these languages. As someone who is implementing an Elixir LiveView app right now, I was particularly curious to see how Elixir performed given LiveViews reliance on websockets, but as Elixir didn’t make the cut.
- nelsonic 2y ago
  Was also surprised they omitted Elixir/Erlang from the list of languages. Crazy considering how many messaging apps use OTP on the backend.
- Terretta 2y ago
  > The title is misleading because exactly one implementation was chosen for each of the tested languages. They conclude “do not use Python” because the Python websockets library performs pretty poorly. On the contrary, they tried autobahn and aiohttp as well: For the Python websocket, a generic module is used which is simply named "websockets". ... This is most likely a module that offers the simplest of websocket functionality. Now, it was mentioned that this only partly explains the poor performance. While writing this report, it seemed unjust not to give Python a fighting chance. So, the websocket server has been rebuilt with the more trusted Autobahn library and the benchmark test has been rerun. This new server does lead to better results ... still unable to finish the benchmark test.... [T]he Python server is rebuilt one more time, this time with a library by the name of "aiohttp." At last, all 100 rounds of the benchmark are able to be completed, though not very well. Aiohttp still takes longer than Go, and becomes substantially unreliable after round 50, dropping anywhere from 30-50% of the messages. It can only be concluded that the reason for this dreadful performance is Python itself.
latch 2y ago
Their explanation for why Go performs badly didn't make any sense to me. I'm not sure if they don't understand how goroutines work, if I don't understand how goroutines work or if I just don't understand their explanation. Also, in the end, they didn't use the JSON payload. It would have been interesting if they had just written a static string. I'm curious how much of this is really measuring JSON [de]serialization performance. Finally, it's worth pointing out that WebSocket is a standard. It's possible that some of these implementations follow the standard better than others. For example, WebSocket requires that a text message be valid UTF8. Personally, I think that's a dumb requirement (and in my own websocket server implementation for Zig, I don't enforce this - if the application wants to, it can). But it's completely possible that some implementations enforce this and others don't, and that (along with every other check) could make a difference.
- vandot 2y ago
  They didn’t use goroutines, which is explains the poor perf. https://github.com/matttomasetti/Go-Gorilla_Websocket-Benchmark-Server/blob/master/go-gorilla_websocket-benchmark-server.go#L58 https://github.com/matttomasetti/Go-Gorilla_Websocket-Benchm... Also, this paper is from Feb 2021.
  - windlep 2y ago
    I was under the impression that the underlying net/http library uses a new goroutine for every connection, so each websocket gets its own goroutine. Or is there somewhere else you were expecting goroutines in addition to the one per connection?
    - donjoe 2y ago
      Which is perfectly fine. However, you will be able to process only a single message per connection at once. What you would do in go is: - either a new goroutine per message - or installing a worker pool with a predefined goroutine size accepting messages for processing
      - 2y ago
emmanueloga_ 2y ago
If the author is reading this, I think a single repository would be more appropriate than multiple repos [1]. It would be nice to set things up so we can simply git pull, docker run, and execute the benchmarks for each language sequentially. Something that stood out to me is the author’s conclusion that "Node.js wins." However, both the Node.js and C++ versions use the same library, uWebSockets! I suspect the actual takeaway is this: "uWebSockets wins, and the uWebSockets authors know their library well enough that even their JavaScript wrapper outperforms my own implementation in plain C++ using the same library!" :-p Makes me wonder if there’s something different that could be done in Go to achieve better performance. Alternatively, this may highlight which language/library makes it easier to do the right thing out of the box (for example, it seems easier to use uWebsockets in nodejs than in C++). TechEmpower controversies also come to mind, where "winning" implementations often don’t reflect how developers typically write code in a given language, framework, or library. -- 1: https://github.com/matttomasetti?tab=repositories&q=websocket-benchmark https://github.com/matttomasetti?tab=repositories&q=websocke...
fnordpiglet 2y ago
(2021) Was surprised it used a depreciated Rust crate until I noticed how out of date it is
simpaticoder 2y ago
Interesting that https://github.com/uNetworking/uWebSockets.js https://github.com/uNetworking/uWebSockets.js (which is C++ with node bindings) outperforms the raw C++ uWebSockets implementation. It's also interesting that https://github.com/websockets/ws https://github.com/websockets/ws does not appear in this study, given that in the node ecosystem it is ~3x more likely to be used (not a perfect measurement but ws has 28k github stars vs uWebSockets 8k stars)
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
zo1 2y ago
Was this published as-is to some sort of prominent CS journal? I honestly can't tell from the link. If that's the case, I'm very disappointed and would have a few choice words about the state of "academia".
- ndusart 2y ago
  Yes, that would be concerning indeed... The author couldn't tell why he didn't manage to make run the C or python program but figured it is probably the blame of the language for some obscure reasons. He also mentioned that he should have implemented multithreading in C++ to be comparable with Node, but meh that's probably also not of his concern, let compare them as is ^^` Also he doesn't mention the actual language of the library used, but that would have voided the interest of the article, so I quite may understand that omission :P But at the end, nothing can be learned from this and it is hard to believe it is what "research" can produce
  - josephg 2y ago
    Yeah it’s a rubbish paper. It’s just a comparison of some websocket implementations at some particular point in time. It tells you how fast some of the fastest WS implementations are in absolute terms, but there are no broad conclusions you can make other than the fact that there’s more room for optimisation in a few libraries. Whoopty doo. News at 11.
indulona 2y ago
The DX for websockets in Go(gorilla) is horrible. But i do not believe these numbers one bit.
wuschel 2y ago
Is this a peer reviewed paper? It does not seem to be. At a first glance, the researchgate URI and the way the title was formulated made me think it would be the case.
frizlab 2y ago
Not including Swift in such a research seems to be a big oversight to me.
cess11 2y ago
I'd like to know why Elixir and Erlang were excluded.
- cess11 2y ago
  Seems the author went silent after this, maybe he decided to run a café or something instead.
austin-cheney 2y ago
I have a home grown websocket library I wrote in TypeScript for node.js. When I measured it a couple of years ago here were my findings: * I was able to send a little under 11x faster than I could process the messages on the receiving end. I suspected this was due to accounting for processing of frame headers with consideration of the various forms of message fragmentation. I also ran both send and receive operations on the same machine which could have biased the numbers * I was able to send messages on my hardware at 280,000 messages per second. Bun claimed, at that time, a send rate of about 780,000 messages per second. My hardware is old with DDR3 memory. I suspect faster memory would increase those numbers more than anything else, but I never validated that * In real world practical use switching from HTTP for data messaging to WebSockets made my big application about 8x faster overall in test automation. Things I suspect, my other assumptions: * A WebSocket library can achieve superior performance if written in a strongly typed language that is statically compiled and without garbage collection. Bun achieved far superior numbers and is written in Zig. * I suspect that faster memory would lower the performance gap between sending and receiving when perf testing on a single machine
pier25 2y ago
I'm surprised at how well php is doing here. I'm guessing they are using fibers?
- alganet 2y ago
  It uses reactphp event-loop library: https://github.com/reactphp/event-loop https://github.com/reactphp/event-loop That library can use either select, libuv, libev or libevent if I'm not mistaken. Fibers are not used at this point, although other libraries have explored the idea (revoltphp). If we're assuming the paper author installed a typical PHP, then it's using select for async I/O. It's the slowest implementation of the event loop. Using something like swoole would extract even more performance out of PHP for async io scenarios.
fredtalty5 2y ago
[flagged]
timkofu 2y ago
It would be interesting to this repeated with Starlette and Granian on Python 13 (with GIL and JIT).
- aprilfoo 2y ago
  I had a quick run with Starlette/uvicorn: similar results than node/uws, a bit faster actually but not significant enough to be meaningful. So I would expect similar results with other modern/fast libraries. I also found that the "websockets" library is a bit slower (25% or so), all with the default settings of that benchmark. The issue with Python that the author faced takes 2 minutes to identify and fix: raise ulimits. Finally, one can question the value of such benchmark in real world applications, especially when the supporting article is so poorly researched as other already pointed.
fastaguy88 2y ago
A meta comment: This paper gives an example of a "teaser abstract". It says what was done, but does not say anything about the actual results. This style is relatively common, but I find it very annoying. There was certainly enough room in the abstract to provide a concise summary of the actual results, which would both inform the reader and perhaps encourage more people to read the entire paper.

87 points by max0563 2y ago

Too bad that uWebsockets was used for Node because a lot of higher level libraries are built on top of https://www.npmjs.com/package/ws https://www.npmjs.com/package/ws

windlep 2y ago
I was able to make a uWebsockets adapter for NestJS pretty easily. It's a bit sensitive of a library to integrate though, a single write when the connection is gone and you get a segfault, which means a lot of checking before writing if you've yielded since you last checked. This was a few years ago, perhaps they fixed that.

paulgb 2y ago

The SSRN link doesn’t have a login-wall: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3778525 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3778525

chrisweekly 2y ago
Thanks! Here's the direct link to the ungated PDF: https://download.ssrn.com/21/02/03/ssrn_id3778525_code4568915.pdf?response-content-disposition=inline&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEDQaCXVzLWVhc3QtMSJHMEUCIBz9ASyrOZb7lgu4c7TPazOVmNtXFIfbhGR%2BPNEQRaLXAiEA8J7SZDs1dYpi7gpDiCI78zaJZIWZHkXIhbu99ui%2B3%2FEqxgUIzf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAEGgwzMDg0NzUzMDEyNTciDIv7tdkuoYsFpo6F7yqaBcYKhbXn5qs7JiWcTskh8279TVWylAjLpM2l9SEz2cHG8mE6XYRUCxg3r9mnm2c3ouVX8RheZ3i%2BapmEy12Nf6E1%2FYKsH9OpU9%2BAVjOn0n8gaHUAEitd1avpiSk662x0b86p09X8Dy7NV1uXI27%2FYfxEGUWvrvfYC%2BvYrLBSDFEaWOAxqPWFAoxVtz0wyDk07pVY%2FbDydStaAaEUGws9e6%2FcSOKv4tA3gr1DtS3Wc8UMsPMOlV6oUHK0jFVZyo1pL%2F6jT8KCZSRAiubk0XTk1TRfcJtI5htH9lZEJMusuky%2BNRaRcuOsLeXt0bRZGRMyQ7fm74yTc9QEckQB%2B%2FZ0cb7%2Fw3hd7C7Sq%2B2Wy9zyWGxoejoxHDMtvga%2BiBH6%2FyQGSltBG358vO3grSynJ3rpAxCwcCNlwY2Pb1yWjCzodVFYAsn6AUALL8gjg4oRQdjHSrZs%2F0I7RPdVYsrPevaU0pM9xIMDRANcdQtGoTsYoPgzRP%2FfPcQRnVXE4nJ8GB%2FlNtazu%2BLgoHTo6JHe4ds1y6GGGy7CHeDnBRyXdqveDq%2Brgjozivf0l96Lc8kqjY8PpAtJvUTYhmz%2FQsy%2B41Wu7bv1QRsaogIgapIAdHkY8Xv6FyXvZyfCKLqVAv4C%2FRRrGKJwycFQJ%2BXzUpJc8vl79Cbwn%2Fmf1gdKPFjhMB0s2skkSOQUcojbQnxfILJKVG1Pv904wH%2BapVRkPCohkRL4%2BafWioVD5d74%2BqbstnZIRoJw0dO0AocLPH0FP%2BdWXQRLB2qNVqUQOhdCYXxXX7iPpGXGhhj3hk7cvFmrgwDADm6DWHgXLX8d1vrKUMEjGJPxcB4sb6CzKs3bPh4Mb4%2BPMLczUMfei%2B6a8f5iY5NWJhnu2kIHVJI%2BoNXilTCUl4W6BjqxAfAUk4qD%2FPqQmUX4ZGPsZe3rEzgKdJr9RWN2ui5sbn6cp5Z6%2BAiT2VNCxd7VYCq0ZdhuDf7YDMkGcTEIBaCVKFWqk6TFNa7VTK4NnIh519yHZIcpStx5hUgQXrKfadCQ8PCb8a7BNAIq1rIXPdjeyc1XOzw1QDbcM3nARYhJY6UlHOYjEznllSKDjK%2BFuWOzdvg%2B9IlHDSisGa6sYJtqXRPYQtgcpXVC9EspXg0FqJZSlQ%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20241123T042500Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Credential=ASIAUPUUPRWE7KW6RUGY%2F20241123%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=8a05d683177c04b274cb176dc107cda22c22e6f4180f2827578f5e6772c45f51&abstractId=3778525 https://download.ssrn.com/21/02/03/ssrn_id3778525_code456891... TLDR; NodeJS is the clear winner, and Python far and away the worst of the bunch.

travisgriggs 2y ago

Thanks for the free access links. I did read through a bit. The title is misleading because exactly one implementation was chosen for each of the tested languages. They conclude “do not us e Python” because the Python websockets library performs pretty poorly. Each language is scored based on the library chosen. I have to believe there are more options for some of these languages. As someone who is implementing an Elixir LiveView app right now, I was particularly curious to see how Elixir performed given LiveViews reliance on websockets, but as Elixir didn’t make the cut.

nelsonic 2y ago
Was also surprised they omitted Elixir/Erlang from the list of languages. Crazy considering how many messaging apps use OTP on the backend.
Terretta 2y ago
> The title is misleading because exactly one implementation was chosen for each of the tested languages. They conclude “do not use Python” because the Python websockets library performs pretty poorly. On the contrary, they tried autobahn and aiohttp as well: For the Python websocket, a generic module is used which is simply named "websockets". ... This is most likely a module that offers the simplest of websocket functionality. Now, it was mentioned that this only partly explains the poor performance. While writing this report, it seemed unjust not to give Python a fighting chance. So, the websocket server has been rebuilt with the more trusted Autobahn library and the benchmark test has been rerun. This new server does lead to better results ... still unable to finish the benchmark test.... [T]he Python server is rebuilt one more time, this time with a library by the name of "aiohttp." At last, all 100 rounds of the benchmark are able to be completed, though not very well. Aiohttp still takes longer than Go, and becomes substantially unreliable after round 50, dropping anywhere from 30-50% of the messages. It can only be concluded that the reason for this dreadful performance is Python itself.

latch 2y ago

Their explanation for why Go performs badly didn't make any sense to me. I'm not sure if they don't understand how goroutines work, if I don't understand how goroutines work or if I just don't understand their explanation. Also, in the end, they didn't use the JSON payload. It would have been interesting if they had just written a static string. I'm curious how much of this is really measuring JSON [de]serialization performance. Finally, it's worth pointing out that WebSocket is a standard. It's possible that some of these implementations follow the standard better than others. For example, WebSocket requires that a text message be valid UTF8. Personally, I think that's a dumb requirement (and in my own websocket server implementation for Zig, I don't enforce this - if the application wants to, it can). But it's completely possible that some implementations enforce this and others don't, and that (along with every other check) could make a difference.

vandot 2y ago
They didn’t use goroutines, which is explains the poor perf. https://github.com/matttomasetti/Go-Gorilla_Websocket-Benchmark-Server/blob/master/go-gorilla_websocket-benchmark-server.go#L58 https://github.com/matttomasetti/Go-Gorilla_Websocket-Benchm... Also, this paper is from Feb 2021.
- windlep 2y ago
  I was under the impression that the underlying net/http library uses a new goroutine for every connection, so each websocket gets its own goroutine. Or is there somewhere else you were expecting goroutines in addition to the one per connection?
  - donjoe 2y ago
    Which is perfectly fine. However, you will be able to process only a single message per connection at once. What you would do in go is: - either a new goroutine per message - or installing a worker pool with a predefined goroutine size accepting messages for processing
    - 2y ago

emmanueloga_ 2y ago

If the author is reading this, I think a single repository would be more appropriate than multiple repos [1]. It would be nice to set things up so we can simply git pull, docker run, and execute the benchmarks for each language sequentially. Something that stood out to me is the author’s conclusion that "Node.js wins." However, both the Node.js and C++ versions use the same library, uWebSockets! I suspect the actual takeaway is this: "uWebSockets wins, and the uWebSockets authors know their library well enough that even their JavaScript wrapper outperforms my own implementation in plain C++ using the same library!" :-p Makes me wonder if there’s something different that could be done in Go to achieve better performance. Alternatively, this may highlight which language/library makes it easier to do the right thing out of the box (for example, it seems easier to use uWebsockets in nodejs than in C++). TechEmpower controversies also come to mind, where "winning" implementations often don’t reflect how developers typically write code in a given language, framework, or library. -- 1: https://github.com/matttomasetti?tab=repositories&q=websocket-benchmark https://github.com/matttomasetti?tab=repositories&q=websocke...

fnordpiglet 2y ago

(2021) Was surprised it used a depreciated Rust crate until I noticed how out of date it is

simpaticoder 2y ago

Interesting that https://github.com/uNetworking/uWebSockets.js https://github.com/uNetworking/uWebSockets.js (which is C++ with node bindings) outperforms the raw C++ uWebSockets implementation. It's also interesting that https://github.com/websockets/ws https://github.com/websockets/ws does not appear in this study, given that in the node ecosystem it is ~3x more likely to be used (not a perfect measurement but ws has 28k github stars vs uWebSockets 8k stars)

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

zo1 2y ago

Was this published as-is to some sort of prominent CS journal? I honestly can't tell from the link. If that's the case, I'm very disappointed and would have a few choice words about the state of "academia".

ndusart 2y ago
Yes, that would be concerning indeed... The author couldn't tell why he didn't manage to make run the C or python program but figured it is probably the blame of the language for some obscure reasons. He also mentioned that he should have implemented multithreading in C++ to be comparable with Node, but meh that's probably also not of his concern, let compare them as is ^^` Also he doesn't mention the actual language of the library used, but that would have voided the interest of the article, so I quite may understand that omission :P But at the end, nothing can be learned from this and it is hard to believe it is what "research" can produce
- josephg 2y ago
  Yeah it’s a rubbish paper. It’s just a comparison of some websocket implementations at some particular point in time. It tells you how fast some of the fastest WS implementations are in absolute terms, but there are no broad conclusions you can make other than the fact that there’s more room for optimisation in a few libraries. Whoopty doo. News at 11.

indulona 2y ago

The DX for websockets in Go(gorilla) is horrible. But i do not believe these numbers one bit.

wuschel 2y ago

Is this a peer reviewed paper? It does not seem to be. At a first glance, the researchgate URI and the way the title was formulated made me think it would be the case.

frizlab 2y ago

Not including Swift in such a research seems to be a big oversight to me.

cess11 2y ago

I'd like to know why Elixir and Erlang were excluded.

cess11 2y ago
Seems the author went silent after this, maybe he decided to run a café or something instead.

austin-cheney 2y ago

I have a home grown websocket library I wrote in TypeScript for node.js. When I measured it a couple of years ago here were my findings: * I was able to send a little under 11x faster than I could process the messages on the receiving end. I suspected this was due to accounting for processing of frame headers with consideration of the various forms of message fragmentation. I also ran both send and receive operations on the same machine which could have biased the numbers * I was able to send messages on my hardware at 280,000 messages per second. Bun claimed, at that time, a send rate of about 780,000 messages per second. My hardware is old with DDR3 memory. I suspect faster memory would increase those numbers more than anything else, but I never validated that * In real world practical use switching from HTTP for data messaging to WebSockets made my big application about 8x faster overall in test automation. Things I suspect, my other assumptions: * A WebSocket library can achieve superior performance if written in a strongly typed language that is statically compiled and without garbage collection. Bun achieved far superior numbers and is written in Zig. * I suspect that faster memory would lower the performance gap between sending and receiving when perf testing on a single machine

pier25 2y ago

I'm surprised at how well php is doing here. I'm guessing they are using fibers?

alganet 2y ago
It uses reactphp event-loop library: https://github.com/reactphp/event-loop https://github.com/reactphp/event-loop That library can use either select, libuv, libev or libevent if I'm not mistaken. Fibers are not used at this point, although other libraries have explored the idea (revoltphp). If we're assuming the paper author installed a typical PHP, then it's using select for async I/O. It's the slowest implementation of the event loop. Using something like swoole would extract even more performance out of PHP for async io scenarios.

fredtalty5 2y ago

[flagged]

timkofu 2y ago

It would be interesting to this repeated with Starlette and Granian on Python 13 (with GIL and JIT).

aprilfoo 2y ago
I had a quick run with Starlette/uvicorn: similar results than node/uws, a bit faster actually but not significant enough to be meaningful. So I would expect similar results with other modern/fast libraries. I also found that the "websockets" library is a bit slower (25% or so), all with the default settings of that benchmark. The issue with Python that the author faced takes 2 minutes to identify and fix: raise ulimits. Finally, one can question the value of such benchmark in real world applications, especially when the supporting article is so poorly researched as other already pointed.

fastaguy88 2y ago

A meta comment: This paper gives an example of a "teaser abstract". It says what was done, but does not say anything about the actual results. This style is relatively common, but I find it very annoying. There was certainly enough room in the abstract to provide a concise summary of the actual results, which would both inform the reader and perhaps encourage more people to read the entire paper.