4 ms·
Yes. I once did zero downtime migration first from AWS to Google, then from Google to Hetzner for a client. Mostly for cost reasons: they had a lot of free cred
by vidarh 5y ago
Yes. I once did zero downtime migration first from AWS to Google, then from Google to Hetzner for a client. Mostly for cost reasons: they had a lot of free credits, and moved to Hetzner when they ran out.
Their savings from using the credits were at least 20x what the migrations cost.
We did the migration by having reverse proxies in each environment that could proxy to backends each place, set up a VPN between them, and switched DNS. Trickiest part was the database failover and ensuring updates would be retried transparently after switching master.
Upside was that afterwards they had a setup that was provider agnostic and ready to do transparent failover of every part of the service, all effectively paid for through the free credits they got.
- michealr 5y agoWould you have a write up in more detail of what you did, even high level. Seems cool thing to do
- sruffatti 5y agoBump! This sounds very interesting.
- withinboredom 5y agoHighly recommend WireGuard for this (see kilo for k8s specific that works with whatever network you have setup). Setting up a VPN that just works is super simple.
- johnthescott 5y agoyep, wireguard is the secret for intercloud, for sure.
- czernobog 5y agoSame. Bump!
- vidarh 5y agoUnfortunately not, but it's surprisingly straight-forward, apart from the database bit, but here's a bit more detail from memory. There are many ways of doing this and some will depend strongly on which tools you're comfortable with (e.g. nginx vs. haproxy vs. some other reverse proxy is largely down to which one you know best and/or already have in the mix) [Today I might have considered K8s, but this was before that was even a realistic option, but frankly even with K8s I'm not sure -- the setup in question was very simple to maintain]: * Set up haproxy, nginx or similar as reverse proxy and carefully decide if you can handle retries on failed queries. If you want true zero-downtime migration there's a challenge here in making sure you have a setup that lets you add and remove backends transparently. There are many ways of doing this of various complexity. I've tended to favour using dynamic dns updates for this; in this specific instance we used Hashicorp's Consul to keep dns updated w/services. I've also used ngx_mruby for instances where I needed more complex backend selection (allows writing Ruby code to execute within nginx) * Set up a VPN (or more depending on your networking setup) between the locations so that the reverse proxy can reach backends in both/all locations, and so that the backends can reach databases both places. * Replicate the database to the new location. * Ensure your app has a mechanism for determining which database to use as the master. Just as for the reverse proxy we used Consul to select. All backends would switch on promoting a replica to master. * Ensure you have a fast method to promote a database replica to a master. You don't want to be in a situation of having to fiddle with this. We had fully automated scripts to do the failover. * Ensure your app gracefully handles database failure of whatever it thinks the current master is. This is the trickiest bit in some cases, as you either need to make sure updates are idempotent, or you need to make sure updates during the switchover either reliably fail or reliably succeed. In the case I mentioned we were able to safely retry requests, but in many cases it'll be safer to just punt on true zero downtime migration assuming your setup can handle promotion of the new master fast enough (in our case the promotion of the new Postgres master took literally a couple of seconds, during which any failing updates would just translate to some page loads being slow as they retried, but if we hadn't been able to retry it'd have meant a few seconds downtime). Once you have the new environment running and capable of handling requests (but using the database in the old environment): * Reduce DNS record TTL. * Ensure the new backends are added to the reverse proxy. You should start seeing requests flow through the new backends and can verify error rates aren't increasing. This should be quick to undo if you see errors. * Update DNS to add the new environment reverse proxy. You should start seeing requests hit the new reverse proxy, and some of it should flow through the new backends. Wait to see if any issues. * Promote the replica in the new location to master and verify everything still works. Ensure whatever replication you need from the new master works. You should now see all database requests hitting the new master. * Drain connections from the old backends (remove them from the pool, but leave them running until they're not handling any requests). You should now have all traffic past the reverse proxy going via the new environment. * Update DNS to remove the old environment reverse proxy. Wait for all traffic to stop hitting the old reverse proxy. * When you're confident everything is fine, you can disable the old environment and bring DNS TTL back up. The precise sequencing is very much a question of preference - the point is you're just switching over and testing change by change, and through most of them you can go a step back without too much trouble. I tend to prefer ensuring you do changes that are low effort to reverse first. Need to keep in mind that some changes (like DNS) can take some time to propagate. EDIT: You'll note most of this is basically to treat both sites as one large environment using a VPN to tie them together and ensure you have proper high availability. Once you do, the rest of the migration is basically just failing over.
- AtlasBarfed 5y agoWe haven't done a cloud migration, but I know from zero-downtime DATABASE migrations that if you do this: - don't use any cloud service that isn't a packaged version of an installable/usable OSS project - architect your services to be able to double-write and switchover the read source with A/B deployments If you can migrate your database without downtime this way, then you are much more flexible than if not.
- logifail 5y ago> architect your services to be able to double-write Can you share any details on how to achieve this? For instance, if the first database accepts the write but the second is temporarily inaccessible or throws an error, do you roll back the transaction on the first and throw an error, or <insert_clever_thing_here> ... ?
- vidarh 5y agoYou're right double-writes are flexible, and it's great when it works. With schema migrations I'm fine with it, because you can usually enforce consistency. But with migrations from one database to another at different locations, I'm lukewarm to it because it means having to handle split-brain scenarios, and often that ends up resulting in a level of complexity that costs far more than it's worth. Of course your mileage may vary - there are situations where it's easy enough and/or where the value in doing it is high enough.
- zomglings 5y agoMy estimates - the company had a combined total of $200,000 dollars in credits across both clouds and the OP charged $10,000 for the migrations.
- vidarh 5y agoYou're reasonably in the ballpark. It was complicated a bit because hosting on Hetzner was so much cheaper for them that $1 in credit was not worth $1 to them, as if/when they had to spend cash they spent substantially less than that at Hetzner.
- zomglings 5y agoThanks. Yeah, I totally understand that dynamic. Now that we have run out of credits on AWS and are shifting some of our workloads into a datacenter ourselves. :)
- jrumbut 5y agoSeveral years ago I had a string of clients who also came into enough Google Cloud credits to make a switch worthwhile. For these companies it wasn't a problem to have a few minutes of downtime, so the task was simply recreating their (usually AWS) production environment in Google Cloud.
- vidarh 5y agoIf it had been a one off migration we might have done the same, but since the end goal was Hetzner from the start, and that meant their architecture needed to handle the HA piece anyway, and served as nice validation we really could fail over without anything going down.
- tinco 5y agoAt my previous job we ran a bare metal cluster at Hetzner, and monitoring the hardware was quite an intensive task. Always monitoring hard drives network bottlenecks, CPU usage, etc. This was before K8s, so it might not be comparable to today. Would you say bare metal cost a lot of extra monitoring/maintenance, or is this something you did on the cloud hardware as well anyway? Do you run virtualization on the Hetzner machines?
- hardwaresofton 5y agoJust a note, there are some companies that have been popping up recently to try to bridge the gap for services on clouds like Hetzner. https://elest.io https://elest.io https://nimbusws.com https://nimbusws.com (I'm building this one so I'm biased for it). > Would you say bare metal cost a lot of extra monitoring/maintenance, or is this something you did on the cloud hardware as well anyway? Do you run virtualization on the Hetzner machines? It cost a lot of monitoring/maintenance up front, but once things are purring the costs amortize really well. Hetzner has the occasional hard drive failure[0], but you should be running in a RAIDed setup (that's the default for Hetzner-installed OSes), so you do have some time. They also replace drives very quickly. If you really want to remove this headache, you run something like Ceph and make sure data copies are properly replicated to multiple hosts and you'll be fine if two drives on a single host die at the same time. Obviously nothing is every that easy but I know that I spend pretty much no time thinking about it these days. I run a Kubernetes cluster and most of my downtime/outages have been self-inflicted, I'm wonderfully happy with my cluster now. Also another thing to note is that control plane downtime != workload downtime, which is another nice thing -- and you can hook up grafana etc to monitor it all. [0]: https://vadosware.io/post/handling-your-first-dead-hetzner-hard-drive/ https://vadosware.io/post/handling-your-first-dead-hetzner-h...
- vidarh 5y agoI would say cloud costs a lot of extra maintenance. When I did contracting, those of my clients who insisted on AWS tended to be far more profitable clients for me, because they needed help with so much more stuff. In terms of monitoring, it boils down to picking a solution and building the appropriate monitoring agent into your deployment. I've run basically anything I run in some virtualized env. or other since ~2006 at least, be it OpenVz (ages ago), KVM, or docker. And that goes for Hetzner too. It makes it easy to ensure that the app environment is identical no matter where you move things. I managed on environment where we had stuff running on prem and in several colo's, on dedicated servers at Hetzner, and in VMs, and even on the VMs we still containerised everything - deployment of new containers was identical no matter where you deployed. All of the environment specific details were hidden in the initial host setup.
- teh_klev 5y agoI hate to be the bean counter, but what was the true cost ultimately? As you and your team's cost as resources. It's nice that you ended up with a provider agnostic capability to deploy anywhere, but none of that was free in terms of ownership costs to get there.
- zamadatix 5y agoSee this older sibling comment https://news.ycombinator.com/item?id=30950842 https://news.ycombinator.com/item?id=30950842
- vidarh 5y agoI was the only person doing the migration work and setting up the HA setup and documenting it for them to take over. My fee for setting it up and doing it accounted for that 20x difference. Hetzner in the end was far cheaper for them to run including the devops work they contracted to me. I effectively got paid to reduce my future earnings from them. But that was fine; when I was doing contracting, that was a big part of my pitch - that I'd help drive down their costs and pay for myself many times over - if people were in doubt I'd offer to take payment as a percent of what they saved. So, no, it wasn't free, but it saved them far more money than it cost them, both the initial transition and in ongoing total cost of operation. In fact, my first project for them was to do extensive cost-modelling of their development and operations.
- teh_klev 5y agoAh, now I understand. Sorry it hadn't joined up in my brain you were providing this as a service as a third party. Thanks for the explanation.