6 ms·
Go, D, Erlang and C in real life: MQTT broker implementation shootout (2013)
- hlieberman 11y ago"Since the Erlang unit tests are in the same files as the implementation, it’s hard to know exactly how many lines long it is. It gets worse since it implements most of MQTT, the D implementation essentially only implements what’s necessary to run the benchmarks." Benchmarking the entire spec versus only the minimal set is almost certainly part of the problem here. If you want to benchmark implementations against each other, you should probably make sure they implement the same thing!
- atilaneves 11y agoNo, not really. I understand why you'd think that but knowing the MQTT spec and the benchmark, it shouldn't make any difference at all.
- xixi77 11y agoAre you sure about that one? I am thinking about the situation where presence of alternative code paths that never actually get executed can lead to fairly large differences in timings, particularly for tight loops. (At least in computational code; I'd expect it to be much less common for protocol handling benchmarks like these...)
- atilaneves 11y agoPretty sure yeah, I looked at a lot of profiling logs.
- nailer 11y ago> pingtest (latency - bigger is better) This is an odd way to measure latency. Could someone explain further?
- atilaneves 11y agoAs someone pointed out to me later, it's not really latency but latency-constrained throughput.
- wehadfun 11y agoIs there any hardware that uses MQTT?
- playing_colours 11y agoThere is. For example, we use it for our hardware product: https://www.relayr.io/products-services/wunderbar/ https://www.relayr.io/products-services/wunderbar/
- baldeagle 11y agoMy now somewhat defunt ninjasphere uses it. I've also heard it is popular with openHab or other IoT devices.
- sborsje 11y agoKarma does: https://blog.yourkarma.com/how-we-use-mqtt-at-karma https://blog.yourkarma.com/how-we-use-mqtt-at-karma
- atilaneves 11y agoI'm the author but not the OP.
- throwawayaway 11y agoOut of morbid curiousity, how many lines of code did the C implementation clock in at?
- atilaneves 11y ago~20kSLOC IIRC. It's not a fair comparison since it does more, but let's face it, if these implementations did the exact same thing I doubt they'd pass the 5kSLOC mark.
- throwawayaway 11y agoI get about 1.5-2x for c vs c++. Genuinely surprised that erlang took so many lines of code.
- nanny 11y agoI really can't imagine that this benchmark is relevant anymore. Haven't Go, D, and Erlang had major changes since then? E.g., Go is now compiled by a compiler written in Go.
- atilaneves 11y agoI'm planning on updating it this year once I finally find the time to learn Rust. Developing an MQTT broker has become my go-to task for learning new languages.
- acqq 11y agoI'm really glad that you've done the D implementation. I think D gets less attention than it should, especially compared to Go. Maybe somebody already proficient in Rust, reading this, can contribute?
- nanny 11y agoNo big deal, it wasn't a slight against you, but rather the OP. It's not your fault he reposted your two year old benchmarks.
- pron 11y agoHow about Java, then? It's a little less esoteric than most of those languages.
- atilaneves 11y agoFunny you should ask: https://atilanevesoncode.wordpress.com/2014/01/08/adding-java-and-c-to-the-mqtt-benchmarks-or-how-i-learned-to-stop-worrying-and-love-the-garbage-collector/ https://atilanevesoncode.wordpress.com/2014/01/08/adding-jav...
- pron 11y agoSo Java is the pingtest winner, and I'm sure that with just a little more work it can be the loadtest winner as well.
- areusch 11y agoupdate from go 1.2? since there have been many performance improvements since then...
- tcfunk 11y agoIt seems this article was posted in 2013. Not sure where that falls in the Go release timeline, but I'm guessing the version wasn't was far behind then as it seems now. Edit: After a quick search, it looks like Go 1.2 was released only 4 days prior to this article being posted.
- atilaneves 11y agoI run Arch Linux, all the versions were whatever was newest at the time.
- rdtsc 11y ago> What about readability and ease of writing? I can’t read Erlang so I can’t comment on that. The Erlang code looks very nice. If you read this, great work Patrick! https://bitbucket.org/pvalsecc/ https://bitbucket.org/pvalsecc/ Nice use of gen_fsm + binary matching. Here is an example of the client code that takes only 200 lines: https://bitbucket.org/pvalsecc/erlangmqtt/src/f37505188c1f1c62aa2b4b94121330d779975d3d/src/mqtt_client_protocol.erl?at=default https://bitbucket.org/pvalsecc/erlangmqtt/src/f37505188c1f1c...
- playing_colours 11y agoYeah, it all depends on code authors. This is one more example of an mqtt broker in Erlang - VerneMQ; and it's very readable: https://github.com/erlio/vmq_server https://github.com/erlio/vmq_server We are currently estimating it.
- dergraf 11y agoThanks, we're working hard to improve our codebase even more! :) Umbrella Project can be found on https://github.com/erlio/vernemq https://github.com/erlio/vernemq as well as our website https://verne.mq https://verne.mq
- atilaneves 11y agoI'll let him know :)
- rdtsc 11y agoThanks and thanks for sharing the benchmarks!
- shepardrtc 11y agoWriting code in Erlang is such a pleasure. I really hope more people get exposed to it. The pattern matching alone is just so nice (and concise!).
- tedchs 11y agoThis being from 2013 and using Go 1.2, I'd be interested to see an update using Go 1.5. Thanks to the author for including standard deviation numbers and not just the averages like so many other "benchmarks".
- voidlogic 11y agoThanks, came here to say this. Go GC and code generation are improving all the time.
- radmuzom 11y agoIt's interesting to see the lowest throughput provided by the C implementation for the 1k case. Any idea why?
- atilaneves 11y agoNo, I didn't dig deeper.
- merb 11y agodid you use GOMAXPROCS=$YOUR_CORES ? or just the standard in 1.2? Actually golang is really good in multithreading. Next you also have a Java example, did you warm it up or not? Actually thats why I hate benchmarks, since mostly they are screwed since most people who are writing benchmarks actually knowing just a little of each language. (actually they just want to show that "their" language is good)
- atilaneves 11y agoBack then I tried GOMAXPROCS from 1 to 8 and it didn't make much of a difference. Later on with a different version of Go (I don't remember which), increasing GOMAXPROCS to 2 made a difference, but any more than that was pretty much the same. In any case (and again, the last time I tried), the Go implementation with GOMAXPROCS=2 was the slowest even though it was using twice as many threads as the other ones! As for showing "their"/"my" language: there were implementations from 4 different sources, and I didn't write the benchmarks; the guy who wrote the Go implementation did (and that's why the benchmark app is in Go).
- StavrosK 11y agoThis is slightly offtopic, but MQTT says it can provide "exactly once" delivery. I thought that wasn't possible?
- taralx 11y agoIt is possible if you are prepared to wait for an unbounded amount of time.
- Gepsens 11y ago2013 ? Would gladly see the same benches with Go 1.5.