2 ms·
I had a simple rabbitmq -> elasticsearch logger I wrote in scala, which had to handle peaks of about 8000 messages/s on a single t2.large instance (with elastic
by anko 9y ago
I had a simple rabbitmq -> elasticsearch logger I wrote in scala, which had to handle peaks of about 8000 messages/s on a single t2.large instance (with elasticsearch on the same box).
It took about a week and a half to write, and would have taken a bunch more time to fix to apply backpressure properly, so it suffered from getting OOMed. Even it's base memory usage was around 300MB. I rewrote it in elixir + genstage in about 8 hours, now it takes 60MB of ram for basically as many messages as I can throw at it, because 1) beam uses a lot less base memory 2) backpressure concerns are sorted out.
In scala, I would have to use akka streams or similar, and it would take me weeks and still I don't think I could get it to fit in such a small footprint.
I plan to add websockets to view a live stream, but I have a little trouble finding the right genstage patterns (as it's still new).
If I were doing high frequency numerical calculations, maybe scala would be a better fit, but for most high speed messaging elixir is pretty amazing.