4 ms·
Indeed it's a nice problem to have :-) we are not yet there, we double every 3 months so far, so we still have 9 months to go :-) however, getting out of cloud
by solso 14y ago
Indeed it's a nice problem to have :-) we are not yet there, we double every 3 months so far, so we still have 9 months to go :-)
however, getting out of cloud does not help. The problem is the high-availability. To maintain 99.9% we cannot rely on a single data-center, no matter what the claims on availability zones say. We have even seen network partition which are the worst of the worst. Once you go on the "Internet" there is no guarantees (unless dedicated lines).
- otterley 14y agoOf course you'll have to have presence in multiple DCs. I didn't mean to imply that one would suffice. I'm not sure whether you've analyzed the bandwidth of the replication stream, but my experience is that if you've got a good circuit provider, you should have adequate bandwidth for 30Mbps+ streams, except under extremely unusual conditions, even over the open Internet.
- solso 14y agoThe replication stream is about 20-25 Mbps for the current throughput of 25K-30K req/s, our peak to valley ratio is low. It seems that we have found this unusual conditions you mention :-) Before compression our replication was often lagging (when below the required bandwidth), which is very dangerous, even if after a minute bandwidth goes well above the threshold. The case of replication is kind of worst case, it's no good to have an average of 30 Mbps if it stays a substantial fraction of the time below the threshold. For every hours at 75% memory grew 2GB, but even one minute below the threshold is bad... if the replication queue builds up and there is a crash, consistency goes out of the window. So better to over-dimension capacity. We even did a quick test with Hetzner instead of Rackspace, it was worse. The average over 4 hours was about 15 Mbps.