21 ms·

Ask HN: Have you ever switched cloud?

274 points by dustinmoris 5y ago

Has anyone ever switched clouds from one service provider to another (e.g. AWS to Azure or vice versa), partially or entirely? If so why? They all offer almost identical services. Do small (but maybe significant?) differences or unique products (e.g. Spanner) make such a big difference that it has swayed someone to switch their cloud infrastructure? I wonder how much these little things matter and how such a transition (in partial or as a whole) went along and how key stakeholders (who were possibly heavily invested in one cloud or felt they were responsible for the initial choice) were convinced to make the switch? I'd love to hear some stories from real world experiences and crucially what it was that pushed the last domino to make the move.

John23832 5y ago
Went from AWS to GCP. Everything we deploy is on Kubernetes, so a combination of Terraform and ArgoCD was pretty easy to move. It was pretty much push button.
- mountainriver 5y ago
  Definitely a space where Kube shines!
deleted 5y ago
[deleted]
intrepidsoldier 5y ago
Some thoughts here: https://www.devprojournal.com/technology-trends/cloud/are-cloud-infrastructure-providers-fulfilling-their-original-promises/ https://www.devprojournal.com/technology-trends/cloud/are-cl...
pettycashstash2 5y ago
I went from Digital Ocean to AmazonWEb Services. Mainly to get familiar and train on the platform.
thatwasunusual 5y ago
Switched from Amazon to Google because I hate Amazon more than Google. Feature-wise, I'm just as happy. However, I trust Google more, but that probably boils to my hatred again. :)
- Derelicts 5y ago
  I've done the same just with a different provider. And I see a trend, people go for the lesser evil out of the ones available.
  - deleted 5y ago
    [deleted]
- xtracto 5y ago
  Interesting that you trust Google more in this regard. Given Google's terrible history of deprecating products, I would not trust any "real world" business to any of their services. Also, at some point I was playing with Serverless in both Google and AWS. Google's Serverless examples were broken (Google cloud was returning 500 errors) while the same stuff in AWS worked smoothly. That left me with a bad taste.
  - milesward 5y ago
    >Dozen of decade+ billon+ customer contracts at this point make shutting GCP down a silly thought. Really crappy that the demos sucked tho, sorry :(
    - xtracto 5y ago
      https://www.aljazeera.com/economy/2020/7/8/google-shut-down-plan-for-cloud-services-in-china https://www.aljazeera.com/economy/2020/7/8/google-shut-down-... https://cloud.google.com/support/docs/shutdown https://cloud.google.com/support/docs/shutdown Also, there have been plenty of Google products with paying customers that Google has shot down. At this point, I have 0 credibility in Google.
fxtentacle 5y ago
Amazon EC2 to OVH AI Docker. Price for GPU instances went down 80%.
- Aachen 5y ago
  > to OVH AI Docker. Can't find an OVH product by that name (would have surprised me), is this a buzzword bingo joke?
  - fxtentacle 5y ago
    I searched for exactly those 3 words on Duckduckgo.com and got https://docs.ovh.com/us/en/publiccloud/ai/training/build-use-custom-image/ https://docs.ovh.com/us/en/publiccloud/ai/training/build-use... which is a tutorial for what I use.
    - sinsterizme 5y ago
      Docker and AI are technologies used on the OVH host, it's not part of the product itself...
      - fxtentacle 5y ago
        Yes and no. OVH has dedicated instances for AI which are different from the regular cloud instances and you don't get a root VM but only root inside docker.
  - nemoniac 5y ago
    Probably OVHcloud AI tools. They're docker based. https://docs.ovh.com/us/en/publiccloud/ai/ https://docs.ovh.com/us/en/publiccloud/ai/
oauea 5y ago
Yes, switched from AWS to Google to Azure. Don't switch to Azure unless your employer forces you, you will regret it. Google is great on a technical level though, especially if you do things with Kubernetes.
- moring 5y ago
  > Don't switch to Azure unless your employer forces you, you will regret it. Can you give any details? Pricing, reliability, weird quirks you have to program around, ...?
  - oauea 5y ago
    Many operations are INCREDIBLY slow. Creating a VM. Mounting a disk to a VM. All take easily 10-30x longer than I'm used to with other clouds. This is especially annoying with Kubernetes as it likes to move disks between VMs which... takes a while on Azure. Many, many, backwards incompatible changes in their Kubernetes platform. I've had to recreate clusters about twice a year so far. Lately it's been better since they finally got node pools working (about 2 years after all of their competitors). They can't get their network stable. Things like `kubectl port-forward` or `kubectl logs` hang after 4-30 minutes of inactivity (i.e. the tunnel is open but no packets actively being sent) which, according to Azure, is "working as expected". This makes the tooling utterly unusable. It has to do with the way Azure's load balancers deal with idle TCP connections. Also, their support engineers are unwilling to help you unless you run Windows. They always insist on remoting into your machine using some Windows utility, even though the issue is with their cloud instead of my machine.
    - mijoharas 5y ago
      > Also, their support engineers are unwilling to help you unless you run Windows. They always insist on remoting into your machine using some Windows utility, even though the issue is with their cloud instead of my machine. That last point sounds like enough of a reason to never touch then with a barge pole.
  - flumpcakes 5y ago
warent 5y ago
Switched from GCP to my own server hardware. After doing the math it came out that it would pay for itself in less than a year. Depending on individual usecase, cloud can be prohibitively expensive and running a server for a small business really isn't nearly as hard as they'd have you believe
- jenscow 5y ago
  Yes, but what about the million new customers you're going to get next week?
  - 10000truths 5y ago
    Funny, but honestly, a single dedicated server should easily be able to handle a million users, for most CRUD apps.
    - Nextgrid 5y ago
      The cloud has normalized terrible underpowered VMs, so many new developers may just not be aware of how much performance a real dedicated machine has - even a relatively "mid range" one (i7 & 16 GB RAM).
      - moooo99 5y ago
        This! I benchmarked a whole bunch of different cloud providers for fun (& my bachelor thesis) and was impressed by how bad some cloud VPS perform. Considering the really steep price to get any kind of significant memory/CPU resources with the major cloud providers as well as the steep bandwidth charges this little experiment was eye opening
        tasubotadas 5y ago
        Could you please share your thesis? Maybe over email?
StreamBright 5y ago
From OVH to AWS. 30% TCO reduction. This was before the fire.
- alberth 5y ago
  I’m really curious how this math adds up given how inexpensive OVH is. Other people in this exact thread are saying they experienced an 80% cost reduction by moving from AWS to OVH. https://news.ycombinator.com/item?id=30943058 https://news.ycombinator.com/item?id=30943058
  - deleted 5y ago
    [deleted]
  - StreamBright 5y ago
    One phrase: right sizing. The original infra on OVH used an instance type that was suboptimal for the workload. Because re-provisioning was not an option due to the size of the cluster, especially without moving hard drives we had only one option. Move the data to S3, find the right instance type and switch over. The ability to right size your infra is grossly underrated. Decoupling data from compute too. We are talking about hundreds of nodes.
    - alberth 5y ago
      Would you mind sharing what size instance types you were on at OVH and are on now at AWS. I ask because I've found that even radically larger sized OVH hardware is still way less expensive than AWS.
      - StreamBright 5y ago
        At the time (many years ago) the largest instance type of OVH was used because that had the required disc capacity that the customer needed. On AWS S3 has a vastly different cost structure and the fact that you can use whatever instance type for your workload enabled us to save up this much. We could use some low spec m5.* instance type that was running with 80% CPU utilization for the workload and also we could use fewer instances. To re-iterate: - the most cost saving comes from the fact that we could de-couple storage from compute - the second part of the cost saving came from the fact that we could use fewer instances with lower spec
someelephant 5y ago
I'd be surprised if more small businesses hadn't, the credits sales people are willing to give you are massive
- lysecret 5y ago
  Yea, that is also my experience working mostly in startups. I wonder if they will keep up this credit strategy.
  - yardie 5y ago
    I've bounced between AWS, Azure, and GCP. Once they get entrenched enough the credits stop but right now it's GCP with the best incentives. Also, don't sleep on Oracle. Their cloud platform is stupidly competitive price-wise but limited feature-wise. If you're just looking for basic compute and storage they can't be beat, sans credits.
    - Reitet00 5y ago
      Good point with Oracle. As much as I don't like their company the pricing for compute is significantly lower than AWS (didn't compare others).
mschuster91 5y ago
As long as you keep on using common services that are available everywhere-ish (=virtual machines, NFS file share, managed mysql/pgsql, managed kubernetes, DNS) it's relatively painless to switch if you're using Terraform. Worst problem in my experience is all the stuff that creeps upon you with time that assumes hardcoded IPs and service names.
Hardik_Shah 5y ago
I have. I was working for a company that had its own data centers, but we were running out of space and time to add more. We were already using Rackspace as a managed hosting provider (we didn't want to manage our own infrastructure), so we decided to move from our data center to Rackspace's data center. The difference in terms of services was negligible, because they all offered almost identical services. But we did it because: * We were looking for someone to manage the hardware and infrastructure for us. * Rackspace's managed hardware offered higher availability than what we were able to achieve on our own. * We had a relationship with Rackspace and they understood our needs, so we felt comfortable switching over entirely.
- bombcar 5y ago
  Interesting - we moved the other direction (to Linode) from Rackspace because their VM/dedicated pricing was just out of whack. Our new box was nominally the same as the old at half the price, but the "4 CPU" was so much newer that it was nearly 2x.
  - vidarh 5y ago
    Whenever I've interacted with Rackspace they've seemed almost proud to be expensive.
  - dx034 5y ago
    At the same scale? I guess if you move a whole data center, they'll quote very different prices from their public ones.
    - bombcar 5y ago
      Yeah, I suspect Rackspace is aimed at "we'll datacenter for you" and not toward small businesses with small needs. And they price themselves accordingly to discourage those customers.
- dx034 5y ago
  Sounds really interesting. I don't understand why so many companies think the alternative to owning your DC must be the cloud. I think the step you describe is where companies can get rid of a lot of overhead and save massive amounts of money. Migrating to the cloud tends to do the opposite.
myzreal2 5y ago
Doing it right now. Entire company is migrating from AWS to Azure for reasons I can't discuss, and I'm currently tasked with this migration in the team I am in. Honestly? It's quite fun. Despite considering myself more of a programmer than devops, I really like the devops stuff - as long as I'm doing it as part of a team and I know the domain and the code - and not being that general devops guy who gets dropped into project, does devops for them and gets pulled into another one to do the same. Working out all those little details of switching from AWS to Azure is fun and enjoyable and I also feel like I'm doing something meaningful and impactful. Luckily there's not much vendor-locking as most of the stuff lives in k8s - otheriwse it would be much trickier.
bartread 5y ago
Yeah, but it was only for a side-project, so I only had a single VM to migrate. I went from AWS (cost ~£25/mo) to Microsoft Azure DE because I didn't want any user data to be subject to search by US law enforcement/national security. I thought the bill would be about the same, but it more than quadrupled almost overnight even though traffic levels, etc., were the same (i.e., practically non-existent). What was happening was Azure was absolutely shafting me on bandwidth, even with Cloudflare in the mix (CF really doesn't help unless you're getting at a decent amount of traffic). In the end I moved to a dedicated server with OVH that was far more powerful and doesn't have the concerns around bandwidth charges (unless I start using a truly silly amount of bandwidth, of course).
- dustinmoris 5y ago
  Oh god quadrupled from £25/month for a small low traffic project as you have described sounds like daytime robbery!
  - dx034 5y ago
    It might be different if you're a developer in SV, but I'd say for most people, that's not insignificant. That's more expensive than most hobbies, I wouldn't spend £100 on a side project if there's an alternative.
  - bartread 5y ago
    Sure, but think about this: that little side project now has about 100x the traffic it did when my bill jumped. That still isn't much traffic (at all) in the grand scheme of things, and Cloudflare's lowest paid tier deals with around 80% of the bandwidth. Still, it's not hard to imagine that bill blowing up to several hundred pounds per month had I chosen not to act. That would translate to several thousand pounds over the course of a year. I don't know very many people for whom such an expenditure, particularly if it's unnecessary and avoidable, would be something they'd regard as insignificant. Putting it into everyday terms: it could have grown into my second largest outgoing after my mortgage. That doesn't really seem proportionate or sensible, so why wouldn't I look for a better deal?
- 5y ago
heywire 5y ago
I know of several retailers and companies serving retailers that switched away from AWS around the time they bought Whole Foods. Before our company switched away, we had multiple retailers say that they would not use our services hosted by one of their biggest competitors.
hpeinar 5y ago
Moved a project with around 600k monthly users from Heroku / Google (split setup) to full AWS setup. Whole process took around 3 months, that was from start of creating AWS account to finish when all production environments were running on AWS and Heroku was "shut down". There was some planning ahead of this as well, so actual time varies. Heroku was heavily limiting platform (for example, they didn't and still don't support http2) and we needed more control over our infrastructure to support further growth without paying enormous costs (for example, redis prices in Heroku are just mind-blowing). Also as we were about to open few new markets, Heroku would have required a lot of manual work to get everything working, something which is really, really simple with Kubernetes. Our monthly costs did go up vs what we had at Heroku at that time, but we're getting a lot more control and bang out of the buck. Regarding convincing stakeholders, you really need to have good reasons to do it. These kind of switches are not cheap nor easy and come with bunch of risks. The easiest thing to sell is always pricing, but in that case you have to show calculations (big guys like AWS and Google have pretty decent calculators you could use) which show the switch is worth it. As I was moving from small player (Heroku) to a big player (AWS) I also had other good reasons (better CI, better logging, better performance overview, more control in general). So it really improved a lot of things for the developers, devops and users.
- safeerm 5y ago
  What are you using for compute EKS and Fargate? We just helped someone switch from Heroku to AWS and they dropped their cloud costs by 67%. This is using ECS + EC2. Fargate is typically 2x more expensive.
  - nijave 5y ago
    Is that taking spot/reserved instances into account (for both). Afaik Fargate supports both of those
    - safeerm 5y ago
      Without Spot/Reserved incentives. Could get even more if that’s done.
eric4smith 5y ago
Moved from Google Cloud -> Digital Ocean -> OVH. Running our own stuff on high powered servers is very easy and less trouble than you think. Sorted out the deploy with a "git push" and build container(s) meant we could just "Set it and forget it". We have a bit under a terabyte of Postgresql data. Any cloud is prohibitively expensive. I think some people think that the cloud is as good a sliced bread. It does not really save any developer time. But it's slower and more expensive than your own server by a huge margin. And I can always do my own stuff on my own iron. Really, I can't see a compelling reason to be in the cloud for the majority of mid-level workloads like ours.
- dx034 5y ago
  I also found that running a PostgreSQL database is really simple. Especially if most of your workload is read only, a few dedicated servers at several providers with a PostgreSQL cluster can deliver 100% uptime and more than enough performance for pretty much any use case. At the same time, this will still be cheaper than one managed database at any cloud provider. I've been running a PostgreSQL cluster with significant usage for a few years now, never had more than a few seconds downtime and I spend next to no time maintaining the database servers (apart from patching). If most requests are read only, clusters are so easy to do in Postgres. And even if one of the providers banned my account, I'd just promote a server at another provider to master and could still continue without downtime. I recently calculated what a switch to a cloud provider would cost, and it was at least 10x of what I pay now, for less performance and with lock-in effects. But I understand that there are business cases and industries where outsourcing makes sense.
  - tomatowurst 5y ago
    can you share more details, because im in teh process of doing the same since having a few terabytes of postgresql / dynamodb is stupid expensive. for a lot big organizations its a matter of accountability. if they say AWS went down vs our dedicated servers went down, it matters a lot for insurance, clients. what i dont get are 4 man startups paying thousands to AWS ... because everybody does it.
nojito 5y ago
Yup moved from AWS --> GCP --> Oracle Most because of cost mitigation.
- dx034 5y ago
  Are you happy with Oracle? I wanted to try it, but the onboarding process (until you're able to actually order some beefy servers) was so problematic (delays, weird error messages, conflicting messages from support) that I didn't dare putting production servers there.
  - nojito 5y ago
    Yup. Pretty much no complaints. We do have a very boring setup though YMMV
kitsune_cw 5y ago
Consolidated all of my small cloud VMs from DigitalOcean, GCP, and AWS into a single dedicated server on Hetzner. It costs way less and I don't have to pay for egress anymore.
dagw 5y ago
If so why? Company policy change. We went from each office (or even department) basically doing their own thing and having their own billing accounts and negotiating (or not) their own deals, to one central cloud deal with central billing and administration. The change from everybody doing their own thing to having a central devops strategy we all had to work within was a much bigger change than the actual changing of cloud providers.
vidarh 5y ago
Yes. I once did zero downtime migration first from AWS to Google, then from Google to Hetzner for a client. Mostly for cost reasons: they had a lot of free credits, and moved to Hetzner when they ran out. Their savings from using the credits were at least 20x what the migrations cost. We did the migration by having reverse proxies in each environment that could proxy to backends each place, set up a VPN between them, and switched DNS. Trickiest part was the database failover and ensuring updates would be retried transparently after switching master. Upside was that afterwards they had a setup that was provider agnostic and ready to do transparent failover of every part of the service, all effectively paid for through the free credits they got.
- michealr 5y ago
  Would you have a write up in more detail of what you did, even high level. Seems cool thing to do
  - sruffatti 5y ago
    Bump! This sounds very interesting.
    - withinboredom 5y ago
      Highly recommend WireGuard for this (see kilo for k8s specific that works with whatever network you have setup). Setting up a VPN that just works is super simple.
      - johnthescott 5y ago
        yep, wireguard is the secret for intercloud, for sure.
    - czernobog 5y ago
      Same. Bump!
  - vidarh 5y ago
brianhorakh 5y ago
I Was a very early aws client (first year of ec2). I moved to azure for better ai/cognitive features, sso, ad blabla, also ms being better open source steward, better docs, better vscode plugins, also felt aws re: kibana/elastic license is/was not aligned with my desire to not live in a dystopian version of the future. It was painful but Az has improved a lot of the sharp edges I encountered.
yonixw 5y ago
In my company, we were aware about the potential honeypots in each cloud and we developed our product from the first commit to be deployed on 3 (!) clouds: AWS, Azure, IBM. And while we made it work by sticking to the least common denominator which was FaaS/IaaS (Lambda, S3, API GW, K8s). It was certainly not easy. We also ignored tools that could've helped us greatly only against a single cloud in order to be multi cloud. The conclusion after 2 years for us is kind of not that exciting. [1] AWS is the most mature one, Azure is best suited for Microsoft products and Old Enterprise features. And IBM is best if you use only K8s. [2] Each cloud has a lot of unique closed code features that are amazing for certain use cases ( Such as Athena for S3 in AWS or Cloud Run in GCP). But relaying on them means you are trapped in that cloud. Looking back, Athena could have simplified our specific solution if we were only on AWS. [3] Moving between clouds, given shared features, is possible, but is definitely not a couple clicks or couple of Jenkins jobs away. Moving between clouds is a full time job. Finding how to do that little VM thing you did in AWS, now in Azure, will take time and learning. And moving between AWS IAM and Azure AD permission? time, time and time. [4] Could we have known which cloud is best for us before? No. Only after developing our product we know exactly which cloud would offer us the most suited set of features. Not to mention different credits and discount we got as a startup. Hope this helps.
- tinco 5y ago
  Why did you feel IBM is best when you only use K8S? Our platform fully K8S and I'm looking to somewhere where we get more performance per buck, probably not IBM but I'm surprised it's even in the list. Do they have an extra nice K8S dashboard or something?
  - OliverGilan 5y ago
    Second this question. Never heard of IBM being exceptional for k8s. Curious to know what makes OP say it’s so good.
    - nijave 5y ago
      Maybe they're using Openshift (OKD). Openshift has a bunch of value add features they add a template marketplace and turn it into a hybrid PaaS with some other multitenancy management bits afaik
jpgvm 5y ago
Yes. Huge AWS -> GCP migration. Why? Incoming CTO signed a massive GCP deal probably because it was marginally cheaper than AWS (while probably ignoring the migration costs).
- amir734jj 5y ago
  Same thing happened to my old company (large insurance company). Moved from Azure to AWS + Terraform/Kubernetes because of cost and "cloud independent" nature of Terraform. The whole IT department spent 2+ years moving hundreds of (relatively modern + legacy) services and applications. Some services couldn't move because they were managed by third-party. I am pretty sure they didn't factor in the cost of migration.
  - jacquesm 5y ago
    Always look for kickbacks if there are cases where the company ends up being out more money all-in. These decisions should not have been made in this way, and it isn't always incompetence that drives this.
uuyi 5y ago
We moved a load of stuff from an acquisition from Azure to AWS only because it's under one cost centre then so we don't have to pay or justify as many invoices. After futzing with this stuff for years though I really would only use the IaaS options in clouds if you want to consider portability. Network, storage, compute and nothing else. The neutral abstraction is Linux for me these days, not a specific vendor!
wazzaps 5y ago
DigitalOcean -> Hetzner Cloud. Simply realized it's much cheaper for my use case (single instance running everything), even doubled my ram from 2GB to 4GB and it was still cheaper. It's also a European company which helps (more trustworthy IMO). Also took the time to simplify my deployment, which was nice.
dijit 5y ago
Yes, I moved from AWS to GCP while a product was in a late release cycle. I talk about it in detail in a google cloud podcast: https://www.gcppodcast.com/post/episode-265-sharkmob-games-with-jan-harasym/ https://www.gcppodcast.com/post/episode-265-sharkmob-games-w... The primary reasons were: Ease of Use, Support and Cost (in that order). I had a bunch of what I call "3am topics" which inhibited our ability to perform stressful operations in the middle of the night meaning we minimise our opportunity of successful outcomes when on-call... I'm not a fan of that. I also argued (quite successfully) the case that AWS was not saving us money or much time when the alternative was renting a handful of servers from a provider. There were attempts by AWS staff to lock us into the platform but those services (cloudfront and ECS being large ones) worked so poorly that caching/reconciliation layers were added into the product to build resiliency: all that was needed was to replace what populated the cache with something else (eventually we moved to Kubernetes which worked much better). Cloudformation was so hard to work with (at least our implementation) that replacing it with terraform was easy: the hard part was understanding what was needed and what was fluff. We also had to care a _lot_ about how the network was setup in AWS, there were issues with MTUs not being aligned by default in some cases for example so we had to write workarounds, and the VPCs being zonal by default meant we had much more complex setups. Another ease of use topic was the sizing of instances, instead of specifically saying what shape of machine most matches your workload in GCP you allocate a number of cores and you're kinda done. Another ease of use topic was that discounts are provided retroactively via sustained use (though, they do have commitments too) where as in AWS you needed to very carefully carve up your requirements to get significant discounts, or write your application to be as stateless as possible (which you can do in GCP too). This is not a lot better than physical machines because the upfront work of trying to capacity plan is still there... at least if you care about cost. Regardless, our dev cycles are much more streamlined now, it's rather easy to deploy an entirely new environment, the operations can be handled by a single individual on a part-time basis, which in my opinion is the point of a cloud provider: to save you time. I can go into much more detail if you have any specific questions.
- philliphaydon 5y ago
  > Cloudformation was so hard to work with (at least our implementation) that replacing it with terraform was easy: the hard part was understanding what was needed and what was fluff. I hated cloudformation when I first started using it. To get started the documentation sucks. Once you get the hang of it the docs are great and is actually really simple. I now quite like it.
busterarm 5y ago
AWS to Google Cloud. Already mature product (public company). Many potential customers are strongly Amazon-averse. Switching to GCP won some deals that were being lost otherwise. Anybody's cloud strategy should try and stick to the most basic services/building-blocks possible (containers/vms, storage, databases, queues, load balancers, dns, etc) to facilitate multi-cloud and/or easy switching. Not that each cloud doesn't have its quirks that you'll have to wrap your head around, but if you go all in with the managed services you're eventually going to have a bad time.
- bgorman 5y ago
  I concur, in my experience the biggest driver of growth for Azure and GCP is that customers of SASS companies and consulting companies make it a requirement to choose anyone but AWS. Legacy companies are terrified of Amazon. Google does have some innovative big data products like BigQuery and Dataflow. In general choosing GCP over AWS shouldn't hinder a companies growth at this point IMO.
  - Cipater 5y ago
    >customers of SASS companies and consulting companies make it a requirement to choose anyone but AWS. Legacy companies are terrified of Amazon. Is there a particular reason for this?
    - judge2020 5y ago
      Apparently Walmart and Target stay away from AWS but most of it seems to be FUD regarding, I guess, Amazon potentially targeting their competitors and their suppliers to steal data stored in AWS/use it for strategic business moves. It would be a swift death for AWS if they were found to be doing this, though, so imo the fear is unfounded. https://www.forbes.com/sites/andriacheng/2019/07/14/amazon-and-rival-retailers-are-much-tighter-than-you-think/?sh=39e3f0e83265 https://www.forbes.com/sites/andriacheng/2019/07/14/amazon-a...
      - dokument 5y ago
kevin_nisbet 5y ago
In startup mode, we switched multiple times, from Azure to Google Cloud to AWS, chasing those startup credits. Things that were easy to move tended to move quickly, but systems that lost owners or priorities didn't tend to get moved all that quickly, so left a fun legacy of a system or two on an old cloud account. Growing out of that mode has the team mostly focused on a single cloud provider, with a few things that'll remain on alternatives because they're better suited, and projects will clean up the rest in a couple of years.
k101 5y ago
We decided on a multi-cloud infrastructure. Now the team is thinking about choosing a tool that would allow you to migrate and do multi-cloud in two clicks. We try to go to not very large clouds, because first we want to support small businesses, and secondly, sometimes it is more convenient for regional legislation. The main clouds are Scaleway, DigitalOcean and your own servers.
cbushko 5y ago
Yes. I built an entire new platform on Google Cloud using Kubernetes (GKE). We moved from AWS. It was a chance to re-architect the platform, make things simpler and cost effective. Huge success on all of those. Costs were reduced by millions a year. Other than GKE, there was not a significant technology reason to move to Google Cloud. AWS didn't even have a managed version of Kubernetes at the time.
cnj 5y ago
Yes. We migrated from Rackspace to GCP. When I joined the company, it was clear Rackspace was loosing the cloud race, but back in 2012/2013 it was a strong contender. Sometimes one adopts a technology too early... We had a longer selection process between AWS, GCP and Azure. AWS was difficult because some of our customers see Amazon as a competitor. However today we also offer the option to run on AWS. GCP won over Azure.
jcal93 5y ago
Moved a SaaS tool from Linode to DigitalOcean. Reason for the move was security driven. DO's K8S volumes are LUKS encrypted at rest by default; one less thing in your security controls to worry about. Prices are higher than Linode's, and have had some reliability issues occasionally with the LoadBalancer, but for the most part it works really well. The SaaS tool was mostly cloud-agnostic, so the changeover was not terrible. Changing the deployments to use DO's CSI storage, setup secrets, deploy services. I stood up the entire infra on DO, then moved the DNS over one subdomain at a time.Took about 4 days to make the move, including validating everything, and finally cutting off Linode totally.
- Aachen 5y ago
  Interesting to hear. A friend also had a lot of trouble with Linode reliability and moved to a bigcorp after that. Running a multiplayer game, even turn-based, had huge hiccups and the problem simply didn't reproduce on the new systems. Iirc support couldn't do anything but I'd have to ask for the details. He spent a lot of time debugging it with a minimal example to rule out other causes, iirc a websocket pinging every few seconds on their kubernetes offering (again, I'd have to ask for the details), and it reproduced on Linode but not on the platform they were considering moving to (with a similar hosted kubernetes offering).
randomtwiddler 5y ago
Moved from AWS to vultr. Primary reason is unsafe billing practices at AWS. I'm a small shop and felt very uncomfortable about the potential of getting a surprise $xx,000 bill from AWS. The hack in December tripled my anxiety. Changed my password and migrated in January. Closed out my AWS account once every last service was migrated. I mostly use only basic services so pretty much any cloud provider can fit my use case. It took some time but I have peace of mind now.
- judge2020 5y ago
  > The hack in December What are you referring to?
dkechag 5y ago
At work we moved from AWS to GCP for pricing reasons. We are still paying loads more than before moving to the cloud, but it's hard to find good sysadmins nowadays that don't want to do everything on the cloud. For personal projects I've moved things to Linode and Digital Ocean, as they provide quite decent value. Actually, for a comparison of AWS/GCP/Azure/Linode/DO/Tencent/Ali value & performance, check out an extensive comparison I ran recently: http://blogs.perl.org/users/dimitrios_kechagias/2022/03/cloud-provider-performance-comparison-gcp-aws-azure-perl.html http://blogs.perl.org/users/dimitrios_kechagias/2022/03/clou...
- greenail 5y ago
  What is the difference in expense between the "loads more" you are spending and hiring and managing a sysadmin? Ballpark is fine, interested to know if it is 1x,10x, 50x?
  - dkechag 5y ago
    So, we went from a cost of a bit over $3k/month for infrastructure, to almost $15k/month (would be more with AWS). Yes, you have to buy the servers, yourself at extra cost, but on Google most people run instances on Haswell/Skylake, which is >5 year old stuff, so a quick calculation for buying top of the line every 5 years (which for the first couple of years at least gives you faster hardware than on GCP) comes to about extra $2-3k/month. Also, we had 1 (very good) sysadmin and 1 other developer who assisted part-time sys-admining, to a cloud that has some extra bells and whistles, but requires 2 full-time sysadmins (we are temporarily left with 1 and he can't keep up). Part of the problem is that the whole architecture was very fine-tuned to run on our customized rack servers (it was doing happily for almost 15 years), so a lot of things did not translate that easily in the cloud. I guess if you have a small system, or designed from scratch for the cloud a single cloud platform person might be enough. Overall, there are some extra disadvantages beyond cost - e.g. Cloud SQL manages some things for you, but there is more lag compared to when our super fast DB server was on the same rack as the application servers, and other such little performance things that we could fine tune when we had control of the hardware.
bajsejohannes 5y ago
I've done so at two companies (without naming names). Not entirely, but enough to have a bargaining chip with AWS. At the largest company, our monthly bill was at least $1M at AWS, so the savings more than made up for the engineering time. Also, in both cases, it was moving from AWS to GCP, and in both cases we were using Kubernetes and not really using much of the provided services of the platforms. I suspect this is the biggest reason for Google to push Kubernetes; abstract away compute so it's easier to switch.
deleted 5y ago
[deleted]
YorickPeterse 5y ago
At GitLab we went from AWS to Azure, then to Google Cloud (this was a few years ago). AWS was what we started with, and I think like most companies very little attention was paid to the costs, setup, etc. The result was that we were basically setting money on fire. I think at some point Azure announced $X in free credits for YC members, and GitLab determined this would save us something like a year's worth in bills (quite a bit of money at the time). Moving over was rather painful, and I think in the months that we used Azure literally nobody was happy with it. In addition, I recall we burned through the free credits _very_ fast. I don't recall the exact reasoning for switching to GCP, but I do recall it being quite a challenging process that took quite some time. Our experiences with GCP were much better, but I wouldn't call it perfect. In particular, GCP had/has a tendency to just randomly terminate VMs for no clear reason whatsoever. Sometimes they would terminate cleanly, other times they would end up in a sort of purgatory/in between state, resulting in other systems still trying to connect to them but eventually timing out, instead of just erroring right away. IIRC over time we got better at handling it, but it felt very Google-like to me to just go "Something broke, but we'll never tell you why". Looking back, if I were to start a company I'd probably stick with something like Hetzner or another affordable bare metal provider. Cloud services are great _if_ you use their services to the fullest extend possible, but I suspect for 90% of the cases it just ends up being a huge cost factor, without the benefits making it worth it.
- zegerjan 5y ago
  Azure to GCP was right after GV became an investor. Again, credits was the reason to change over. Not the poor Azure performance.
- k8sToGo 5y ago
  I wish Hetzner provided RDS
  - hardwaresofton 5y ago
    So it’s a bit in the future but managed postgres is something I want to offer through nimbus[0]. In the Meantime if you’re interested in other managed services, please sign up! [0]: https://nimbusws.com https://nimbusws.com
wink 5y ago
2016. AWS to GCP. Mostly cost reasons and AWS had some annoyances at the time. We also replaced our ~4y old very manual setup with a ground-up rewrite with terraform. Zero regrets, but we didn't have a lot of vendor-lock in with <100 VMs and S3 + a couple of SSL things.
janvdberg 5y ago
Yes from Linode to Hetzner [1] Mainly because of price: more CPU/RAM/Storage for a lower price. I think my previous server was underpowered, because it kept swapping. Now it runs as smooth as it can (it never swaps). Migrating is a bit of a hassle though and things might not work as you expect [2] [1] https://j11g.com/2021/12/28/migrating-a-lamp-vps/ https://j11g.com/2021/12/28/migrating-a-lamp-vps/ [2] https://j11g.com/2022/01/03/bypassing-hetzner-mail-port-block-port-25-465/ https://j11g.com/2022/01/03/bypassing-hetzner-mail-port-bloc...
deleted 5y ago
[deleted]
milesward 5y ago
I've helped around 900 companies make the switch (others to GCP in my case), and I can confirm based on their results over time that the differences are not as small as they might seem. For so many, it's just ease of use and efficiency to get things done; for others its attention and partnership; for a third cohort its absurd cost advantages; and a fourth its performance and reliability. Our customers see gaps in one or many of those four areas? They move.
amir 5y ago
We did, a few times. Only because many of them had generous credits for early stage startups. GCP, Azure, SoftLayer (now IBM Cloud), DigitalOcean. We weren't relying on any particular service except managed Kubernetes, but we were happy to set it up overselves if it wasn't on offer.
thom 5y ago
There have been two circumstances where we’ve seriously thought about it, but as of yet haven’t changed: 1) Under some circumstances we might want to give very stringent uptime guarantees for some systems, and I do not trust providers to have zero global (cross AZ) outages. Having a hot standby or even load balancing across clouds could be tempting. 2) One cloud provider is very keen to get into our sector and has made extremely generous overtures which we’d be silly to completely ignore. As I say, not something we’ve followed through as yet but both are serious considerations.
dataminded 5y ago
Yes. I got a new job at a new company.
pavelevst 5y ago
My team was migrating 1000+ VMs from aws to gcp, mostly for cost efficiency (and adoption of k8s for even better cost) We used kafka mirrormaker 1, with 2 ways sync (new cluster have separate topics for write that are synced to old cluster, and all topics from old cluster synced to new cluster) For postgres failover switch to new master required about 1-2 minutes of downtime We migrated ~80 microservices within 8 months and now our infra cost about 1/4 of what we paid to aws, completely worth the effort!
awinter-py 5y ago
credits and sometimes service reliability
lewisjoe 5y ago
I have a VM running on GCP. I have the world's simplest usecase: how do you export the instance as a snapshot so that I can run it from somewhere else. Say, a virtual box environment or another cloud provider. How do you export a snapshot of your disk and VM? Good luck searching for a solution. I just spent 2 hours trying to figure this out and it seems impossible :(
- j0057 5y ago
  The general answer nowadays is "you don't": you spin up a new VM and provision it using whatever IaC/scripting solution works for you. Edit to add: even if this were possible, it would be made exta challenging due to the cloud provider specific agents that are installed in VMs.
eek2121 5y ago
Yes, I moved a decent sized project from AWS to Digital Ocean. The biggest hurdles I had to deal with were the things Digital Ocean did NOT support. It was worth it in the end, because our bill went from millions of dollars a month to a few hundred thousand a month. Estimated labor cost was about a million.
- intrepidsoldier 5y ago
  Would love to learn more
renewiltord 5y ago
Didn't, but went multi-cloud for performance reasons (we have to be close to the people we're talking to). The most annoying part was the network topology. Used DirectConnect with a provider so that x-cloud latency was as low as it could be. Overall, not too upset. Used terraform to configure things across clouds. Without that, would be too hard to keep track of everything.
0x500x79 5y ago
I had a company that wanted to move from AWS to GCP. This was a top-down decision that was made with an incompetent "tech lead" stating that we could move everything and save money with the 10 year commit hundred million dollar contract in Google. It failed, horrendously. Even though multiple people in the organization were calling out how bad of an idea it was they still moved forward. Google has some nicities with how things connect, setup, etc, but at the end of the day they are cloud providers and not everything they provide is a silver bullet. This project was canceled after 3 years after spending millions on migrating since the migration was not a drop-in replacement (no one thought it was except the "tech lead"). There are a lot of things besides tech that can affect these projects. If you hire AWS experts and they are having to be AWS experts expect to need to hire GCP experts.
martingoodson 5y ago
I once worked with a CTO who decided to move from AWS to GCP (and also move from Spark to Hive and python to scala, at the same time). That guy was an idiot. (At one point somebody accidentally spent £30,000 in data transfer costs with one key press.) The project was never completed and the CTO just moved on to another fancy CTO position.
- klysm 5y ago
  Why on earth would you go from spark to hive?
pnathan 5y ago
yes. I've done Aws, Azure, Google. My basic impression, as a software engineer/site reliability engineer, is that Google >> AWS >> Azure. This relates to sophistication of offering and design of cloud. The dominant questions look like this: - what is the familiarity to the infra people - can we implement appropriate governance concerns - how tightly bound is your important code with the specific cloud? I have generally focused on Kubernetes in the last 5 years, to allow the service layer to be relatively portable. This is very useful in the switching/migrating question. My general thought process is not to use cloud services unless its very obvious (ec2, s3, etc); prefer to have k8s services provide that capability and use the cloud provider as portable COTS.
Sasha_zik85 5y ago
ageyfman 5y ago
We switched 3 times with 0 downtime. Linode > Google > Azure > AWS. Chasing that sweet startup package at each place. We stuck with AWS.
timmit 5y ago
Yes, I changed from AWS to Oracle, from Caprover to k8s, because of the Oracle Cloud Always Free resources, it can save $20/month, and give me 20 times more ram, 6 times more storage I wrote a blog before the implementation. [move to k3s cluster](https://tim.bai.uno/how-to-deploy-a-django-application-to-a-k3s-cluster-re-platform-aceielts-pro-to-k8s/ https://tim.bai.uno/how-to-deploy-a-django-application-to-a-...)
notyourday 5y ago
I have done a few zero downtown migrations. It is not rocket science if you do not use cloud-specific products.
seattle_spring 5y ago
New CTO came into my previous place and forced a switch from AWS to GCP. It took 3 years and an enormous amount of effort just to help the bottom line a small amount. Totally ridiculous.
anonymousDan 5y ago
I'd love to hear any tricks for avoiding punitive egress bandwidth charges needed to get your data out of AWS.
comprev 5y ago
I've done a few zero downtime migrations for clients, mostly due to the common story of the credits when they started out on coming to an end. More often than not they realised a super dynamic cloud infrastructure was completely overkill for their business needs.
wirelesspotat 5y ago
I migrated a previous company's infrastructure from AWS to Fly.io Our AWS bill was the main reason. It was far higher than it should have been for the traffic we were serving. Even after we'd halved our AWS bill (the original bills had been crazy), it was still kinda high Fly was a pretty clear choice when we looked at the lower costs and ease of transitioning from single-region to multi-region infrastructure I'd been nudging the CEO about doing a migration for about a year before we decided to make the move. When I found that I couldn't really get our AWS costs any lower and did a full cost estimate of Fly vs AWS, the wheels moved reasonably quickly The CEO primarily cared about lowering our monthly costs and being able to do the migration reasonably quickly (~1 month)
drusepth 5y ago
I switched from Heroku to AWS, then eventually back to Heroku. Heroku to AWS was for cost reasons (cut monthly costs by roughly 35%) but wasn't enough savings to justify hiring a devops person. As soon as there were too many issues I didn't know how to fix (setting up everything Heroku offers was hard and likely done wrong, which made ongoing maintenance some level of hell), I switched back to Heroku where the lack of devops needs basically paid for itself.
- digianarchist 5y ago
  I would checkout Render and DO App Platform. Heroku has actual competition now.
- safeerm 5y ago
  This is very, very true. One of our customers described it really well “The nightmare of DevOps kept us from managing AWS directly, but with TinyStacks we can scale a billion+ requests a day in audio advertising leveraging the full power of AWS.”
alexnewman 5y ago
I once had to move 30 PB of satellite imagery from Aws to GCP. google gave us all these tools to do it, none of it worked. It was rough
jeffkeen 5y ago
Switched from heroku to a vultr VPS, and AWS S3/Cloudfront -> Backblaze B2 + Bunny CDN. Saved myself a boatload and don’t regret it for a second.
jaimex2 5y ago
AWS to GCP. Google sales people made it worth the companies while and it was a good time for us to refactor the orchestration stack. BigQuery is pretty sweet. Services were done one at a time with a VPN holding the two together till everything was migrated over.
deeteecee 5y ago
For kubernetes, we prefer GCP but have to also support Azure because Microsoft is also our business partner.
kgeist 5y ago
We have to follow GDPR-like laws and in our country providers aren't very reliable ("hey we have a maintenance tomorrow with a 30 minute downtime, sorry") so we migrate between providers a few times per year. There are always at least two providers we use at any given time (exact configuration changes every year), one in standby mode with replication between the two, and various homegrown tools to make the switch instant. Sometimes our application can enter a critical state (hardware errors or a bug in the application) so we switch to a different instance to investigate later without hurry. This complex setup was dictated by the stakeholders after we had a series of painful downtimes in 2019. There also have been cases when we migrated due to pricing changes. For the US region we use AWS and it is the most stable of all, we have zero switches there. Our platform is therefore provider-agnostic mostly.
jesterson 5y ago
Yes, moved away from rackspace to IBM Cloud. Since then use several cloud providers for redundancy. Every provider has severe downtime (when even phone lines are not operational) so we do failover across several providers. Saved a lot of uptime for us. Also we do not use (almost) vendor-specific solutions. Almost everything your cloud provider sells up to you can be achieved without using the provider lock. Will save time later when provider quality goes into sh*t (eventually happens) and you have to migrate your infra somewhere.
christian7007 5y ago
In OpenNebula we're working on trying to ease these use cases by adding an extra layer of abstraction, so you can migrate load as easy as this: https://www.youtube.com/watch?v=IopA_deQK4M&t=1s https://www.youtube.com/watch?v=IopA_deQK4M&t=1s
more_corn 5y ago
I ported a client from OVH to AWS with zero downtime. Built new infrastructure with terraform, flipped dns. Was easier than wading through the existing architecture fixing problems and security vulnerabilities. It was a single web app though. Not too complicated.

deathanatos 5y ago

The biggest problems: things don't work, things don't compose, and support is unknowledgeable & ineffective. Things don't work: outages, bugs. So many I'm not sure where to start: ACR: slow as molasses. I don't know why they didn't build it on blobstore. Many queries are clearly O(images) even when they shouldn't be. Throughput it terrible and made worse by inane pagination; listing images, for example, has a throughput of something like 60kbps. And "b", as in bits. Minutes for 3MB of data. It's absurd; they think that's "working as expected". AKS: they manage the API server in AKS, and we find it is routinely non-responsive. We went through a quarter-long support ticket, which went back and forth between "you're putting too much load on the API server" -> "we don't think we are, what load is there?" "here's the top queries" and they're all queries from like, the cluster controllers — which are also managed by AKS/Azure. -> "well there's too much load!". App Gateway: normally stable, but had an outage when Let's Encrypts old root expired. (We were using a cross signed cert — i.e., our cert was valid, but App Gateway failed it, i.e., a false negative validation.) They never acknowledged the outage, and the support ticket we filed didn't get a response until like days later — missing support's SLA — meanwhile, some engineer somewhere clearly fixed the service, as it started working all on its own. ACR: we used to get 500s. IDK if these still happen or not, as we retry-looped most of the spots that were hitting them. Support response was ridiculous "what's your ISP?" "… this is inside your network, Azure, and my ISP wouldn't cause you to serve a 500." "Maybe the image is too big" "The image is ~100 KiB" … global AAD outages, the status page going down while the twitter account is like "check the status page", the Portal has issues (a few days ago listing subscriptions just returned 0 rows. Like, okay, I guess I'm not looking at those today), the activity log will occasionally just return errors, or today, it returned 0 entries where I knew there to be entries. Composability: a cloud provider's job is to offer bricks from which I can build whatever infra my company's needs demand. But Azure constantly says "well, no, you can't put those bricks together that way." IPv6 anywhere in your vnet? You can't add a managed PostgreSQL server to that vnet, cause not only does it not support IPv6, you can't add it to an IPv4-only subnet in a dual-stack vnet. Like, the entire point of dual-stack…; also, when we attempted this, the API request took 2 hours to fail, and failed with "Internal error, please retry", which we then did, like chumps. 4 hours later, support ticket. AKS will add new features (that ought to have been there from day 1) like nodepools … but only to newly created cluster. You want to take advantage of that? Too bad! Recreate that cluster from scratch! Support: Azure has no meaningful tooling for handling bugs in their services. The only hammer they have is support, and by god it's going to hit that nail. Support might (if you can get them to admit that yes, shit's broken) field a bug report, but then the support ticket is closed. Is the bug fixed? How will you know? IDK. Also, AFAICT, certain products you just can't open a support ticket for; notably, the portal. It isn't an "Azure service", so it isn't in the list of things to select from. Also, they override the mouse wheel on the list of services, so scrolling ~1 "detent" on a trackpad results in the list scrolling at Mach 4. Support tickets lack URLs, so they're unlinkable. Occasionally whatever the agents use to view info on tickets gets desynced from the ticket, and new replies in the portal are black-holed (but email replies still work). You can't put ">" in a support ticket, it's not allowed. You can't upload certain file formats, it's not allowed. (E.g., want to send video of a bug? Not today!) SLAs are regularly missed, and the responses often ask for information included in the opening ticket. And … the agent's grasp on English is frankly terrible. (I'd accuse them of out-sourcing it, but we once had an agent on a ticket go dark on us … because he was in Texas when Texas losts its power.) Honorable mention: My God, AAD is trying to be as complex as can be. Apps, Service Principals (which are needlessly and confusingly just called "Enterprise Apps" in the UI! — oh and the search box for that page doesn't work), Roles, Permissions, Role Assignments, Tenant, Subscriptions … oh my God. Like, AWS IAM is frankly a terrible implementation, but Azure AAD makes AWS IAM look amazing. My entire 2+ year experience with Azure has made me an ardent believer in AWS, and willing to try GCP. I need to write this all in a blog post.

oauea 5y ago
> I need to write this all in a blog post. Please do. It would be very valuable to find these things when searching about Azure, because this absolutely matches my experience.

lapser 5y ago

Cloud isn't about cost though. Cloud is about value. You can scale super easy when you inevitably need it (assuming you've "made it" anyway - whatever that means). You get a burst of new users, it's trivial to add additional nodes (again, assuming you've set up your infra to be easily scalable). With own hardware, scaling is not as easy. You'll have to do a lot more around plumbing too. Networking, security, many other things that you'll have to address. Stuff that has already been solved for you.

izacus 5y ago
That "inevitably" is doing some really heavy lifting in your post. It practically never comes for most companies.
- judge2020 5y ago
  But most companies' dream is to reach that ""inevitable"" point where the cloud saves it; it becomes a bet where they either reach that level of scale, or die trying.
  - jjav 5y ago
    > But most companies' dream is to reach that ""inevitable"" point where the cloud saves it; it becomes a bet where they either reach that level of scale, or die trying. And how many die trying due to bleeding all the funding to AWS, instead of running everything off a couple cheap boxes underneath the CTOs desk? I've been in at least one which ran out of money that way. Don't pay for the future pipedream now, pay for what you need. That "inevitable" dream of near-infinite scale up usually never arrives for most companies. If it does, worry about it then.
    - lapser 5y ago
      AWS gives away literally tens of thousands of pounds to start ups. Granted, it's to lock them in down the line but getting free credits gets you further than not having credits.
      - jjav 5y ago
        AWS credits are a toxic trap. BTDT. They'll get you far enough that you are hopefully (from AWS perspective) locked in pretty tight. And then the high monthly costs start to hit you hard.
        lapser 5y ago
        Sure, that's true. But I've been in companies where they've had major investments and have a pretty significant number of users, and they still had stupid amounts of credits. One of my previous companies was running a third of their infra on AWS credits, and they still long runway in terms of free credits. Meanwhile, they had to shake off a £1 million a year contract for the next 5 years for 2 DCs. With AWS they were using less than half of that per year (this includes the credits they had). But even if it wasn't cheaper, requesting a new server took days, not minutes. Scaling was not possible. I'll take the credit and potentially get trapped than having to deal with an inflexible mess that is in-house managed infrastructure. At least bigger orgs are able to afford it by (hopefully) building a cloud on top of their infrastructure, but outside that, the majority should of companies be looking into the cloud. Whether that be AWS, GCP, or the smaller Clouds like DO, it doesn't matter.
        jjav 5y ago
        > But I've been in companies where they've had major investments and have a pretty significant number of users, and they still had stupid amounts of credits. Your companies have been wiser and more frugal than mine! In every case, I've seen the credits run out before there was a penny of revenue coming in.
  - arka2147483647 5y ago
    Surely if they are that succesfull, they also have the resources to redesign/upgrade their backend?

bombcar 5y ago

There are two kinds of cloud users - those who treat their cloud as a VM to use, and those who actually use all the fancy API features. The first group are almost always better served by dedicated VMs or hardware from a provider specializing in the, if the VM is long-lived.

phillu 5y ago
That's something I find so fascinating. The "cloud" will almost always be more expensive and "not worth it" if you are only using the IaaS services. I mean, look at the numbers, everyone sees that. Cloud only ever is worth it when one uses the higher-tier services, like AWS Lambda and the likes. Even running Kubernetes in the cloud is only semi worth it, because it's not high enough in the stack and still touches too many low-level IaaS services. Of course, higher tier means more vendor lock-in and more knowledge required and all that. But if you are not willing to pay that price, then OVH, Hetzner and the likes will have better offerings for you.
- ChasingEchoes 5y ago
  the problem as many have already pointed out around this thread, is that, in an enterprise env. you cant really do that too much anymore. And as a result that starts being felt by non-enterprise shops too. And you cant really do that because people dont really wanna deal with on-prem shit and server hostings Tehnically speaking, i am rhcsa certified, i know how to do all of this on-prem, hybrid things. I dont even bother looking at job offers from companies that arent cloud based (even if i would get a 10-15% increase, or more if coming from the financial sector) because, i genuinely cant be arsed to deal with all that bullshit again. I'm done with caring about disk space, and hw firewalls and configuring bs in linux. Fuck iptables, let me manage everything from a (network) security group. Fuck Traefik and F5 and all this bs, let me just plop an Application Gatway in Azure or API gateway in AWS. Fuck database clusters. At this point, i havent even configured an apache/nginx server in a couple of years. WebApps in Azure are more than fine; and for the rest K8s. As a result, good "classic" sysadmins are a dying breed even at enterprise level. So they're even more rare and accessible for small/medium sized business. If i go to my IT dept. right now, i can guarantee 80% of them would be completely lost to setup and use an AD, AAD is just too convenient. That basically leaves you with: move to cloud, or learn how to do all of these things by yourself. And those things take time (to learn and to manage) It's like deciding to make apps with Perl. Can you do it? sure. But you'll probably have to do it on your own.
dx034 5y ago
I'm still not a fan of the second way. If you develop all your software to tightly integrate with AWS, you might save time developing the software but create a huge amount of technical debt. Managing your own infrastructure (with dedicated servers, so no hardware management) isn't too hard, even if you're a small shop. And managing a fleet of AWS services isn't necessarily less work. Maybe there's a reason all ads for cloud tend to compare it to running your own data center. Because once you get rid of hardware management, it's not really that much easier being in the cloud, at the risk of lock-in and huge surprise bills.
- philliphaydon 5y ago
  It’s not technical debt if it’s making you money. I would much rather solve the business problems than managing services myself.
  - dx034 5y ago
    It's technical debt if you may need to change your product because a third party makes changes. The beauty of software with little dependencies is that you can run decades old software on a system just fine with no need to regularly refactor. I know how tedious it is to maintain decades old enterprise Java software, but from a cost perspective, it makes much more sense to keep those rather than constantly refactor to chase the newest trend. As an example, if you had a software that was written 20 years ago to store data in a relational DB, updating it to work with current versions of that database system won't be much work (if any). If you rely on managed services, I wouldn't be too sure that you get away that easy.
    - philliphaydon 5y ago
      > As an example, if you had a software that was written 20 years ago to store data in a relational DB, updating it to work with current versions of that database system won't be much work (if any). If you rely on managed services, I wouldn't be too sure that you get away that easy. This very much depends. In the example you gave, nothing changes if you used managed services or not. But you could argue that the 20yo software is technical debt preventing the upgrade of the database due to the source code being lost, the library used to connect to the latest version of the database doesn't exist requiring a rewrite in a moderm language or framework. Etc. Technical debt really is about code that cannot easily be modified to adapt to the requirements of a business. If you wrote some code, and it was trash, made no sense, in an obscure language that few people know, with no comments. Yet it ran for 10 years flawlessly with everyone too scared to look at it, but made the business money. It's not technical debt until it needs to be changed/modified.

lolinder 5y ago

> Really, I can't see a compelling reason to be in the cloud for the majority of mid-level workloads like ours. I work on a very small team. We have a few developers who double as ops. None of us are or want to be sysadmins. For our case, Amazon's ECS is a massive time and money saver. I spent a week or two a few years ago getting all of our services containerized. Since then we have not had a single full production outage (they were distressingly common before), and our sysadmin work has consisted exclusively of changing versions in Dockerfiles from time to time. Yes, most of the problems we had before could have been solved my a competent sysadmin, but that's precisely the point—hiring a good sysadmin is way more expensive for us than paying a bit extra to Amazon and just telling them "please run these containers with this config."

TrickardRixx 5y ago
Very similar experience here. I work on a two person "DevOps" team. Without AWS ECS we would have to have a much higher headcount. I get to spend most of my time solving real problems for the engineers on the product team rather than sysadmin work.
- dijit 5y ago
  What are “real problems” for the engineers or product team?
  - TrickardRixx 5y ago
    Things like automating manual workflows, building small infrastructure debugging tools, or providing infrastructure consultation to an engineer trying to decouple two parts of a legacy code base.
kissgyorgy 5y ago
> None of us are or want to be sysadmins. It's such a huge misconception that by using a cloud provider you can avoid having "sysadmins" or don't need that kind of skills. You still need those, no matter which cloud and which service you use.
- exfascist 5y ago
  In fact, if you're using Linux on your workstation you'll use the same skills locally as you do on the VPS/bare metal (depending on your scale.) Arguably "cloud" services need more sysadmin skills, not less.
  - lolinder 5y ago
    It's far easier to maintain my own Linux workstation than an internet-facing server used daily by customers.
    - exfascist 5y ago
      Absolutely but most of the knowledge translates, it's the procedures that differ.
      - lolinder 5y ago
        It's those different procedures that I'm trying to avoid. It's not that I couldn't do those things or learn to do them, it's that my time is best spent building and improving our applications, not keeping servers running, secured, and up to date. At some point we hope to get to the scale where it makes sense to pay a human to do that, but at this point the additional cost incurred by an ECS instance over an equivalent server is negligible.
  - ChasingEchoes 5y ago
    Thats a very big if. I have yet to work with a $corp that uses Linux for workstations. Overwhelming majority uses Windows. Some use macOS. The ocasional developer that uses linux will usually be in a VM, or if IT policies allow, WSL. So yeah, running cloud services doesnt require sysadmin skills, unless you assume copy pasting from oficial documention "sysadmin skills".
    - xedrac 5y ago
      That's funny... every team I've been on in the last 10 years has used Linux workstations almost exclusively, with a few Macs here and there.
- lolinder 5y ago
  Which skills specifically do you think we might be missing that we would need to run an app on a managed container service and managed database? I know how to configure firewalls, set up a (managed) load balancer, manage DNS, and similar tasks directly related to getting traffic to my app. What I no longer have to know how to do: keep track of drive space, manage database backups, install security updates on a production server without downtime, rotate SSH keys, and a whole bunch of other tasks adjacent to the app but not actually visible to incoming traffic at all.
  - kissgyorgy 5y ago
    You still need to do backups, a database backup is just one part of that, if you are not following the 3-2-1 rules and don't test your restore mechanism, you don't have reliable backups. Those things you listed are sill sysadmin tasks in my eyes, and you are doing them, validating my point. You still have to track storage space, either because you are paying for it and need to expand when necessary, or you have to manage costs at one point, that's not completely out of the picture. It can be easier for sure than building your own storage hardware. You still need to keep systems up-to-date either you are using Docker so you are doing it on your "application level" or you are using Linux VMs and you need to upgrade those systems/images. Even if you are using something like Functions or Lambda, those have their own environment which you need to be aware of and they usually support specific versions of programming languages, so you need to upgrade your own stack when they don't support older versions anymore.
    - lolinder 5y ago
      I tell you that ECS has eliminated a ton of extra work for my team for a bargain price, and your response is "but you still have to do x, y, and z!" It's like saying that I shouldn't buy a dishwasher because I'd still have to wash the pots. Yes, we still need to do some sysadmin-y tasks. But ECS handles so many of them that we actually have the time, energy, and knowledge to take care of the few that remain. (As an aside, keeping language and OS versions up to date becomes a development task rather than an ops task when running Docker + ECS. We increment a version number in the repository and test everything, the same as we do for any library or framework that we depend on.)
- odonnellryan 5y ago
  I think the problem with cloud environments is you do not "need" sysadmins - it is not obvious you need them, so what you end up with is a bunch of systems glued together without much thought, and then crazy things like HTTP logs not being turned on for your various services, insane service costs b/c of not understanding pricing tiers, etc..
- _huayra_ 5y ago
  If you use purely PaaS offerings (or FaaS as well), then you also don't really need sysadmins. That's not to say that you can get away with knowing _no_ sysadmin skills in these scenarios, but you don't need to have someone on staff who knows the ins and outs of Cassandra or Mongo or whatever you're using. In awful workplaces with high turnover, it's worth it for management to opt for these managed services so that when the overworked tech lead decides to rightfully bail on them, she/he doesn't leave them in the lurch. (Note: I'm not defending these workplaces, but just explaining that when they can't keep adequate in-house talent to manage their own services, it makes financial sense to outsource it, and pay the "cloud tax").
- Daishiman 5y ago
  The difference doing ops setting up a couple of Lambdas or Fargate containers vs provisioning your own servers is substantial.
deleted 5y ago
[deleted]
joncrane 5y ago
"Infrastructure is cheaper than developers(sysadmins)" all over again.
deleted 5y ago
[deleted]
ctvo 5y ago
Managed container services (like Amazon ECS) are a sweet spot for me across complexity, cost, and performance. Mid size companies gain much of the modern development workflow (reproducibility, infrastructure as cattle, managed cloud, etc.) using one of these services without the need to go full blown Kubernetes. It's lower level than functions as a service, but much cheaper, more performant, matches local developer setups more closely (allowing for local development vs. trying to debug AWS Lambda or Cloudflare FaaS using an approximation of how the platform would work).
jamesfinlayson 5y ago
Very much agree - due to a coworker leaving recently, I'm looking after two systems. They're both running on ECS and using Aurora Serverless. My company takes security very seriously so if these two systems were running on bare-metal I'd probably be spending one day a week patching servers rather than trying to implement new functionality across two products.
eric4smith 5y ago
I can bet our team is smaller than yours. And yet… Sysadmin tasks take up maybe 2 hours a month. Your theory is right though if no one on your team knows how to setup servers. In your case the cloud makes sense.
holografix 5y ago
To the peeps running ECS? Why not just straight up AKS or GKE? Have you compared ECS to Cloud Run on GCP?
- lolinder 5y ago
  In my case, mostly because it was easier to get buy-in from the rest of the team on ECS than Kubernetes.

Unfortunately not, but it's surprisingly straight-forward, apart from the database bit, but here's a bit more detail from memory. There are many ways of doing this and some will depend strongly on which tools you're comfortable with (e.g. nginx vs. haproxy vs. some other reverse proxy is largely down to which one you know best and/or already have in the mix) [Today I might have considered K8s, but this was before that was even a realistic option, but frankly even with K8s I'm not sure -- the setup in question was very simple to maintain]: * Set up haproxy, nginx or similar as reverse proxy and carefully decide if you can handle retries on failed queries. If you want true zero-downtime migration there's a challenge here in making sure you have a setup that lets you add and remove backends transparently. There are many ways of doing this of various complexity. I've tended to favour using dynamic dns updates for this; in this specific instance we used Hashicorp's Consul to keep dns updated w/services. I've also used ngx_mruby for instances where I needed more complex backend selection (allows writing Ruby code to execute within nginx) * Set up a VPN (or more depending on your networking setup) between the locations so that the reverse proxy can reach backends in both/all locations, and so that the backends can reach databases both places. * Replicate the database to the new location. * Ensure your app has a mechanism for determining which database to use as the master. Just as for the reverse proxy we used Consul to select. All backends would switch on promoting a replica to master. * Ensure you have a fast method to promote a database replica to a master. You don't want to be in a situation of having to fiddle with this. We had fully automated scripts to do the failover. * Ensure your app gracefully handles database failure of whatever it thinks the current master is. This is the trickiest bit in some cases, as you either need to make sure updates are idempotent, or you need to make sure updates during the switchover either reliably fail or reliably succeed. In the case I mentioned we were able to safely retry requests, but in many cases it'll be safer to just punt on true zero downtime migration assuming your setup can handle promotion of the new master fast enough (in our case the promotion of the new Postgres master took literally a couple of seconds, during which any failing updates would just translate to some page loads being slow as they retried, but if we hadn't been able to retry it'd have meant a few seconds downtime). Once you have the new environment running and capable of handling requests (but using the database in the old environment): * Reduce DNS record TTL. * Ensure the new backends are added to the reverse proxy. You should start seeing requests flow through the new backends and can verify error rates aren't increasing. This should be quick to undo if you see errors. * Update DNS to add the new environment reverse proxy. You should start seeing requests hit the new reverse proxy, and some of it should flow through the new backends. Wait to see if any issues. * Promote the replica in the new location to master and verify everything still works. Ensure whatever replication you need from the new master works. You should now see all database requests hitting the new master. * Drain connections from the old backends (remove them from the pool, but leave them running until they're not handling any requests). You should now have all traffic past the reverse proxy going via the new environment. * Update DNS to remove the old environment reverse proxy. Wait for all traffic to stop hitting the old reverse proxy. * When you're confident everything is fine, you can disable the old environment and bring DNS TTL back up. The precise sequencing is very much a question of preference - the point is you're just switching over and testing change by change, and through most of them you can go a step back without too much trouble. I tend to prefer ensuring you do changes that are low effort to reverse first. Need to keep in mind that some changes (like DNS) can take some time to propagate. EDIT: You'll note most of this is basically to treat both sites as one large environment using a VPN to tie them together and ensure you have proper high availability. Once you do, the rest of the migration is basically just failing over.

cosmodisk 5y ago
This was really nice to read. Thanks!
baq 5y ago
People get paid hard cash for lower quality plans than you’ve just provided, thanks a lot! :)
ketzo 5y ago
Damn, this is why I come to HN. This is awesome, thank you so much for taking the time to write it up.
jrumbut 5y ago
Really well done keeping this simple! It's also another one of those situations where good design principles and coding practices pay off. If the app is a tangled mess of interconnected services, scripts, and cron jobs this kind of transition won't be possible.
justsomehnguy 5y ago
> If you want true zero-downtime migration there's a challenge It is astounding how many people require 24/7 ops... while working 8/5. Otherwise this comment is an exemplar on how things should be done. My take on this is what OP is a sysadmin, not a dev. *smug smile*
- throwawayboise 5y ago
  > It is astounding how many people require 24/7 ops Yet when us-east-1 goes offline, it's mostly just shrug wait for it to come back because it's not our fault...
- vidarh 5y ago
  > It is astounding how many people require 24/7 ops... while working 8/5. In this case the client had an actually global audience. They could have afforded downtime for the actual transition, but it was a usual test for the high availability features that mattered for them. I do agree with the overall principle, though - a whole lot of people think they need 24/7 and can't afford downtime, yet almost all of them are a lot less important than e.g. my bank, which do not hesitate to shut down their online banking for maintenance now and again. As it turns out, most people can afford downtime as long as it's planned and announced. Convincing management of that is a whole other issue. > My take on this is what OP is a sysadmin, not a dev. smug smile Hah. I'd say I was devops before devops was a thing. I started out writing code, but my first startup was an ISP where I was thrown head-first into learning networks (we couldn't afford to pay to have our upstream provider help set up our connection, so I learnt to configure cisco routers while having our provider on the phone and feigning troubleshooting with a lot of "so what do you have on your side?") and sysadmin stuff, and I've oscillated back and forth between operations and development ever since. Way too few developers have experienced the sysadmin side, and it's costing a lot of companies a lot of money to have devs that are increasingly oblivious to hardware and networks.

tinco 5y ago

At my previous job we ran a bare metal cluster at Hetzner, and monitoring the hardware was quite an intensive task. Always monitoring hard drives network bottlenecks, CPU usage, etc. This was before K8s, so it might not be comparable to today. Would you say bare metal cost a lot of extra monitoring/maintenance, or is this something you did on the cloud hardware as well anyway? Do you run virtualization on the Hetzner machines?

hardwaresofton 5y ago
Just a note, there are some companies that have been popping up recently to try to bridge the gap for services on clouds like Hetzner. https://elest.io https://elest.io https://nimbusws.com https://nimbusws.com (I'm building this one so I'm biased for it). > Would you say bare metal cost a lot of extra monitoring/maintenance, or is this something you did on the cloud hardware as well anyway? Do you run virtualization on the Hetzner machines? It cost a lot of monitoring/maintenance up front, but once things are purring the costs amortize really well. Hetzner has the occasional hard drive failure[0], but you should be running in a RAIDed setup (that's the default for Hetzner-installed OSes), so you do have some time. They also replace drives very quickly. If you really want to remove this headache, you run something like Ceph and make sure data copies are properly replicated to multiple hosts and you'll be fine if two drives on a single host die at the same time. Obviously nothing is every that easy but I know that I spend pretty much no time thinking about it these days. I run a Kubernetes cluster and most of my downtime/outages have been self-inflicted, I'm wonderfully happy with my cluster now. Also another thing to note is that control plane downtime != workload downtime, which is another nice thing -- and you can hook up grafana etc to monitor it all. [0]: https://vadosware.io/post/handling-your-first-dead-hetzner-hard-drive/ https://vadosware.io/post/handling-your-first-dead-hetzner-h...
vidarh 5y ago
I would say cloud costs a lot of extra maintenance. When I did contracting, those of my clients who insisted on AWS tended to be far more profitable clients for me, because they needed help with so much more stuff. In terms of monitoring, it boils down to picking a solution and building the appropriate monitoring agent into your deployment. I've run basically anything I run in some virtualized env. or other since ~2006 at least, be it OpenVz (ages ago), KVM, or docker. And that goes for Hetzner too. It makes it easy to ensure that the app environment is identical no matter where you move things. I managed on environment where we had stuff running on prem and in several colo's, on dedicated servers at Hetzner, and in VMs, and even on the VMs we still containerised everything - deployment of new containers was identical no matter where you deployed. All of the environment specific details were hidden in the initial host setup.