4 ms·
> We used our websocket service to publish updates to all the clients. I used /r/place from a few different browsers with a few different accounts, and they al
by eatitraw 10y ago
> We used our websocket service to publish updates to all the clients.
I used /r/place from a few different browsers with a few different accounts, and they all seemed to have slightly different view of the same pixels. Was I the only one who experienced this problem?
When /r/place experiment was still going, I assumed that they grouped updates in some sort of batches, but now it seems like they intended all users to receive all updates more or less immediately.
- programbreeding 10y agoI experienced this as well. I have a different account logged in on mobile than what is logged in on my desktop. I wouldn't say things were drastically different, but when there was a location with a ton of activity (like OSU or the American flag towards the end), I saw different views between them.
- d23 10y agoYeah, we went into it a bit in the "What We Learned" section, but that was most likely during the time we were having issues with RabbitMQ. I believe it was mostly fixed later on, but either way, we found a new pain point in our system we can now work on.
- eatitraw 10y agoAh okay then! BTW, thanks for the great post, it was a very interesting read.
- atombender 10y agoSurprised you're using RabbitMQ. It's one of those things which work great until they don't (clustering is particularly bad), and then you have almost zero insight into the issue, and have to resort to the Pivotal mailing list. Have you looked at NATS at all? We're using it as a message bus for one app and it's been fantastic. It is, however, an in-memory queue, and the current version cannot replace Rabbit for queues that require durability.
- fernandotakai 10y agoi've been using rabbitmq heavily (as in, the whole infrastructure is based on two rabbitmq servers) for a long time and i've never seen it fail. tbh, i never used clustering (because it's one of the shittiest clustering implementations i've ever seen) but we do use two servers (publishers connect to one randomly and consumers connect to both) and it seems to handle millions of messages without any issues. of all servers i've ever used, rabbitmq is by far the most stable (together with ejabberd).
- lobster_johnson 10y agoRabbitMQ is decent if you don't use clustering (which, I agree, is shitty). I have some quibbles with the non-clustered parts, but nothing big. Right now, the main annoyance is that it's impossible, as far as I understand, to limit its memory usage. You can set a "VM high watermark" and some other things, but beyond that, it will — much like, say, Elasticsearch — use a large amount of mysterious memory that you have no control over. You can't just say "use 1GB and nothing more", which is problematic on Kubernetes where you want to pack things a bit tightly. This happens even if all the queues are marked as durable.
- fernandotakai 10y agoyeah we have dedicated machines to rabbitmq because it's basically memory hungry. but i like it that way because it's only going to crash if the machine crashes.
- d23 10y agoHmm, I'll take a look at that. For websockets, non-durability seems like a fine tradeoff, so it sounds interesting. Thanks!
- lobster_johnson 10y agoNote that NATS is currently pub/sub, which is a "if a tree falls in the forest" situation. Messages don't go anywhere if nobody is subscribing. So it's awesome for realtime firehose-type use cases where a websocket client connects, receives messages (every client gets all the messages, although NATS also supports load-balanced fanout) for a while, then eventually disconnects. NATS is ridiculously fast [1], too. There's an add-on currently in beta, NATS Streaming [1], which [2] has durability, acking/redelivery and replay, so covers most of what you get from both RabbitMQ and Kafka. It looks very promising. [1] http://bravenewgeek.com/tag/nats/ http://bravenewgeek.com/tag/nats/ [2] https://nats.io/documentation/streaming/nats-streaming-intro/ https://nats.io/documentation/streaming/nats-streaming-intro...
- not_kurt_godel 10y ago> RabbitMQ You're already on AWS, why not use SQS instead?
- manigandham 10y agoRabbitMQ is absolute crap. Surprised anyone uses it in production. If you already have a good Redis infrastructure then you can just use the pub/sub features built into it for your websockets communication.
- Cthulhu_ 10y agoI had the same thing, my client running out of sync (it seemed to be way behind) the "truth"; had to do a refresh to get it back up.