3 ms·
Compressing the replication stream will likely buy you time (at the cost of added complexity resulting from having to maintain and monitor the ssh tunnel), but
by otterley 14y ago
Compressing the replication stream will likely buy you time (at the cost of added complexity resulting from having to maintain and monitor the ssh tunnel), but it won't solve your problem in the long term.
If your replication stream bandwidth continues to grow, the returns on compression will eventually diminish. As the cloud providers can't guarantee network throughput, it might be a good time to start planning to evacuate the cloud (or at least this part of the architecture).
If your business is growing that fast, this is a good problem to have. :)
- solso 14y agoIndeed it's a nice problem to have :-) we are not yet there, we double every 3 months so far, so we still have 9 months to go :-) however, getting out of cloud does not help. The problem is the high-availability. To maintain 99.9% we cannot rely on a single data-center, no matter what the claims on availability zones say. We have even seen network partition which are the worst of the worst. Once you go on the "Internet" there is no guarantees (unless dedicated lines).
- otterley 14y agoOf course you'll have to have presence in multiple DCs. I didn't mean to imply that one would suffice. I'm not sure whether you've analyzed the bandwidth of the replication stream, but my experience is that if you've got a good circuit provider, you should have adequate bandwidth for 30Mbps+ streams, except under extremely unusual conditions, even over the open Internet.
- solso 14y agoThe replication stream is about 20-25 Mbps for the current throughput of 25K-30K req/s, our peak to valley ratio is low. It seems that we have found this unusual conditions you mention :-) Before compression our replication was often lagging (when below the required bandwidth), which is very dangerous, even if after a minute bandwidth goes well above the threshold. The case of replication is kind of worst case, it's no good to have an average of 30 Mbps if it stays a substantial fraction of the time below the threshold. For every hours at 75% memory grew 2GB, but even one minute below the threshold is bad... if the replication queue builds up and there is a crash, consistency goes out of the window. So better to over-dimension capacity. We even did a quick test with Hetzner instead of Rackspace, it was worse. The average over 4 hours was about 15 Mbps.