3 ms·
One thing you don't address: where do messages go while a resource is self-splitting? How much redundancy do you build in to prevent this from being a problem?
by KirinDave 19y ago
One thing you don't address: where do messages go while a resource is self-splitting? How much redundancy do you build in to prevent this from being a problem? How much backlog do you expect to handle?
And have you checked latency serving out of EC2? Our experience is that the connections are fairly fast but the latency establishing the connection is significant, especially from the west coast where twitter is based.
- hank777 19y agoI addressed this in another answer here but happy to do it again. The way shard splitting works is that the initial shard keeps working while the new shard is being built. During this time, all messages to the old shard are written to the old shard and the new shard, so, unless I don't understand your question I dont see there being a backlog. Regarding EC2, latency isnt really an issue because Twitter is far from real-time. That said, haven't really noticed any problems with EC2 connection latency, but we are not live yet so I would defer to you on that.