11 ms·
Story time! A couple of years back, I was working at Mojang (makers of Minecraft). We got purchased by Microsoft, which of course meant we had to at least try
by amarant 2y ago
Story time!
A couple of years back, I was working at Mojang (makers of Minecraft).
We got purchased by Microsoft, which of course meant we had to at least try to migrate away from AWS to Azure. On the surface, it made sense: our AWS bill was pretty steep, iirc into the 6 figures monthly, we could have Azure for free*.
Fast forward about a year, and an uncountable amount of hours spent by both my team, and Azure solutions specialists, kindly lent to us by the Azure org itself, we all agreed the six figure bill to one of corporate daddy's largest competitors would have to stay!
I've written off Azure as a viable cloud provider since then. I've always thought I would have to revaluate that stance sooner or later. Wouldn't be the first time I was wrong!
- stego-tech 2y agoYou've got me curious: what was the single biggest barrier to migration, if you're able to disclose it? I'm guessing it was something proprietary to AWS, like how they handle serverless or something that couldn't translate over directly, but I'm always eager to learn why a migration from X to Y didn't work.
- mi_lk 2y agoCurious about details too. The parent's conclusion is to write off Azure, but I wonder if it's actually AWS or the way they use AWS that makes it hard to migrate. Or put it in another way, if Mojong were to start with Azure but couldn't manage to migrate to AWS, which provider is the parent going to write off?
- noodletheworld 2y agoMy experience has certainly been that AWS is both a) more stable, and b) has more migration resources and guides than the reverse. It was easier to go to AWS than to Azure; and I've done both in the past ~4 years. Migrating to AWS was just technical work. Migrating to azure was 'fight unexpected bugs and the fact it doesn't actually work in surprising situations'. The only reason to go to azure was that Microsoft waved a big fat $$$ discount to use their services. Migrating to AWS was a breeze.
- nosefrog 2y agoYup, the hardest part about migrating to Azure was jumping between all the managed services that work everywhere else but are insanely buggy on Azure. We ended up with the most basic architecture you can imagine (other than AKS, which works great as long as you don't use any plugins) and we're still running into issues. We have a very long list of Azure features and services that we've banned people from using. Just got off a call with someone at Azure today who told us to setup our own NAT gateway instead of using Azure's because of an outage where we made too many requests and then got our NAT Gateway quota taken away for the next 2 hours.
- cj 2y agoTo be fair AWS also has quotas on NAT Gateways. Maximum 55k concurrent connections. After that, they make you deploy NAT gateways in other availability zones. And a max throughput of 10 Gbps. I imagine AWS would also tell you to deploy your own gateway if you were running into the 55k concurrent connection limit of managed NAT. AWS tends to be flexible with their quota enforcement in my experience, though.
- nosefrog 2y agoYea, I don't have a problem with the quota, more that the "out of quota" throttling lasts 2+ hours even after the traffic spike dies down.
- motorest 2y ago> Yup, the hardest part about migrating to Azure was jumping between all the managed services that work everywhere else but are insanely buggy on Azure. Care to point out a concrete example? I've worked with Azure a few years ago and I wouldn't describe it as buggy. At most, I accuse them of not "getting" cloud as well as AWS. For example, the whole concept of a function app is ass-backwards, vs just deploying a Lambda with a specific capacity. That is mainly a reflection of having more years working with AWS, though.
- patmcc 2y agoMy experience with AWS is the documentation can be bad/wrong, it can be difficult to find stuff, but the actual services are very solid. It does what you tell it to do. Stuff works. My experience with Azure is that it simply breaks in ridiculous and frankly unacceptable ways, all the time. It's like someone is unplugging network and power cables every couple hours just for fun.
- smackeyacky 2y agoThe networking is so, so bad in azure. We ran into all kinds of craziness with simple things that kept kicking us in the nuts like port exhaustion between different subscriptions. I quit my last job partly because they were wedded to azure.
- amarant 2y agoThis is a couple of years ago, so I fully expect most of the issues we had back then to be fixed by now, but it was definitely Azure that was the problem. We wanted to use their hosted kubernetes solution(I forget the name) and pods would just randomly lose connection to eachother, everything network related was just ridiculously unstable. Host machines would report their status as healthy, but then be unable to start any pods, making scaling out the cluster unreliable. I also remember a colleague I regarded as a bit of a wizard being very frustrated with cosmosdb, but I cannot for the life of me remember what the specific issue was. Our solution was actually quite well written, if I do say so myself, we had designed it to be cloud agnostic, just on the off chance that something like this would happen (there may have been rumours this acquisition would happen ahead of time). But Azure was just utterly unable to deliver on anything they promised, thus the write-off on my part.
- Too 2y agoPods loosing connection to each other is still very very common in azure. To see the positive side of it, it’s a Chaos Monkey test for free. Everything you deploy must be hardened with reconnections, something you should be doing anyway. What’s frustrating is that it happens rarely enough to give the illusion of being stable, making it easy for PM to postpone hardening work, yet often enough to put a noticeable dent in your uptime if you don’t address it. Perfect degree of gaslighting.
- motorest 2y ago> To see the positive side of it, it’s a Chaos Monkey test for free. Everything you deploy must be hardened with reconnections, something you should be doing anyway. Keep in mind that in Azure it's a must-have, whereas everywhere else it's either a nice-to-have or a sign your system is broken.
- naasking 2y agoAt that point, aren't you better off actually making your own services regularly disconnect clients so you iron out all of the reconnection bugs?
- 2y ago
- kenjackson 2y agoIt has since migrated to Azure. I suspect there was a gap in the technology that was since closed, as AWS certainly had a head start in general.
- sakopov 2y agoIMHO, you're gonna struggle if you move anywhere else from AWS. We're migrating to GCP and there are gaps all over the place.
- borg16 2y agois it because there are features that AWS provides to you that are not available in GCP, or just the fact that setting up exact replicas of processes is hard for migrations like these?
- esprehn 2y agoOut of curiosity, what are the biggest gaps you've hit in GCP?
- notpushkin 2y agoThis is the reason I try to avoid proprietary bullshit services. Use EC2, Postgres, and S3, and you’ll be fine in any cloud or even on bare metal.
- darknavi 2y ago> proprietary bullshit services > S3 Hasn't this recently been an issue where Amazon arbitrarily changes the S3 contract and all software following it as a spec has to play catch-up?
- notpushkin 2y agoI think this happened a few times, but given there are many S3 clients, they can’t keep doing thiat or there will be a backlash. Although perhaps it’s about time we made a proper standard based on S3, yeah.
- immibis 2y agoIt has, but that's not exactly on S3 is it? The S3 system is proprietary to Amazon, and it's your fault if you're not using Amazon but you're relying on Amazon to not change it anyway, because they have no obligation to you. The concept of object storage is not proprietary. You should be able to change your code to use a different object storage provider.
- wayne 2y agoYour story reminds me of when Microsoft acquired Hotmail in the '90s and they tried migrating from FreeBSD & Solaris onto Windows NT/IIS. Having the world's largest email service running on the Windows stack would have been a huge endorsement. It took years until they were successful. https://www.zdnet.com/article/ms-moving-hotmail-to-win2000-servers/ https://www.zdnet.com/article/ms-moving-hotmail-to-win2000-s... https://jimbojones.livejournal.com/23143.html https://jimbojones.livejournal.com/23143.html
- natnatenathan 2y agoHa, I worked on that project. That drove a lot of good requirements into Windows that set us up for web based services (eventually)
- DaiPlusPlus 2y ago> Windows that set us up for web based services (eventually) ...and then .NET and SQL Server started shipping for Linux.
- chupasaurus 2y agoThere are 2 decades between those 2 points, .NET was -4 y.o. at the first one.
- tharkun__ 2y agoSQL Server is really Sybase tho, which was always capable of running on UNIX. Can't say much more, but I worked on a huge (internal) Sybase ASE on Linux based app (you've _all_ bought products administered on this app ;) ) way back (yes, pre-SSD, multi path fiber I/O to get things fast, failover etc.) and T-SQL is really nice, as is/was ASE and the replication server. Been about 20 years tho, so who knows.
- icedchai 2y agoI used to work at a Sybase shop in the late 90's. It was way nicer to work with than Oracle!
- darknavi 2y agoCurrent Mojang employee here, we moved fully onto Azure as of a few years ago AFAIK. Some more game-oriented technologies of course have helped in the years since though. Edit: AWS -> Azure :)
- notpushkin 2y ago> we moved fully onto AWS as of a few years ago Did you mean Azure?
- darknavi 2y agoYes, I do. Whoops!
- inetknght 2y ago> Current Mojang employee here Can I have my Mojang account back? I literally cannot log in to it after it was forcefully migrated to Microsoft. Microsoft doesn't recognize my computer as not-a-bot. Something to do with being Linux, I imagine. Or, can I get a refund?
- mikepurvis 2y agoI bought my kid Minecraft and spent two hours trying to get it set up to run on a Windows 10 machine. In the end I gave up and he plays it on an old Ubuntu laptop instead.
- seasluggy 2y agoI remember that shit show.
- tombert 2y agoWhen I worked at Jet, a shopping website trying to compete with Amazon, we obviously did not want to give money to Amazon, so we used Azure. For the most part it was just fine, until we started using CosmosDB (then called DocumentDB). DocumentDB, in its first incarnation, was utterly terrible. The pricing was extremely hard to predict, so we would end up with ridiculous bills at the end of the week, the provided .NET SDK for it was buggy and horrible, but the very worst part was the WebUI appeared to be directly tied to your particular instance of CosmosDB. Why is this bad? Because if you under-provisioned stuff for your database, it might start going slow, and it would actually lag the web interface you would use to increase the resources! We got into situations where we had to turn off the entire application, just to bump up the resources for Cosmos. It felt like it was a complete amateur hour from Microsoft. My understanding is that Cosmos has gotten a lot better, but man that left a sour taste in my mouth. If I end up getting some free credits or something, maybe I'll give Azure another go but I would definitely not recommend it right now.
- Twirrim 2y ago> Because if you under-provisioned stuff for your database, it might start going slow, and it would actually lag the web interface you would use to increase the resources! What the ever loving heck... seriously?! Why wouldn't this be a control plane API that reconfigures a data plane?!
- tombert 2y agoI think it was querying the database to get usage information, and it was blocking the loading of the page as a result. It's been a few years, but if I recall correctly, the .NET API for changing the database provisioning did work, so some people would hack up a quick script to change resources in an emergency. I think they did fix it eventually, because I know that multiple people on my team complained to Microsoft about it. Very short-sighted decision on their end, but to be fair it was a brand new product.
- vosper 2y agoThis is how hosted Elasticsearch works today, too. Really mess up your ES? Good luck with the web UI!
- abrookewood 2y agoYep, sounds like my experience. Years ago, we migrated of Rackspace to Azure, but the database latency was diabolical. In the end, we got better performance by pointing the Azure web servers to the old database that was still in Rackspace than we did trying to use the database that was supposedly in the same data centre. I kicked up a stink and we migrated everything to AWS in under a week.
- FridgeSeal 2y agoWhen my day job used Azure we had the misfortune of using their managed Postgres instance, the latency of which was _appalling_. We raised support tickets, which were mostly closed to the tune of “seems fine to us”. They seemed to think 20-30ms for a basic Postgres query, which for the same schema, data and hardware was <7ms on RDS.
- motorest 2y ago> Fast forward about a year, and an uncountable amount of hours spent by both my team, and Azure solutions specialists, kindly lent to us by the Azure org itself, we all agreed the six figure bill to one of corporate daddy's largest competitors would have to stay! Being tied to AWS and being unable to shake off a huge bill is not a trait of its competitors. It's a trait of AWS, and stresses the importance of not picking them as your cloud provider. Also, I think it's unbelievable that a monthly 6-figure invoice charged to a company already with cloud engineers in their payroll is not justification enough to shift their workload elsewhere.
- kubb 2y agoLow 6 figures is just a dev. If a team of 5 devs has to work on the problem for 5 years, then it will pay for itself in 25 years. Likely beyond any planning horizon of a company with yearly performance evaluation cycles.
- motorest 2y ago> Low 6 figures is just a dev. ...in the US. In Europe, even the likes of Amazon pays it's SDEs 70k/year. In Sweden, for example, Microsoft pays it's SDEs south of 800k SEK, which is about 70k dollars/year. Low 6 figures is an entire team of Microsoft SDEs working full time for a year.
- pjmlp 2y agoInteresting, because until now I have the same opinion in reverse. Each case is a special case how everything gets configured, but between Azure, GCP, AWS and IBM clouds, the ones with smoother experience on my case, have been on Azure, based on Java and .NET technologies. And we also have our share of support tickets across all of them. Now Azure back in its early days, 2010 - 2016 was kind of ruff, maybe this is the timeframe you're referring to?
- UltraSane 2y agoafter using AWS and Azure extensively AWS seems to be quite well engineered by some very smart people. The isolation between regions is extremely good and the Availability Zone model is quite effective in making very reliable systems if you are willing to pay the cost of inter-AZ data transfer. My company has an Active Directory controller in 3 different AZs Azure is a mess designed by smart people with no time and little budget. Azure flat out lies about the AZs they have by claiming two halved of one data center is two AZs
- 867-5309 2y ago>Microsoft acquired Hotmail in the '90s it was around 2007 (I'm not that old!)
- happymellon 2y agohttps://en.m.wikipedia.org/wiki/Hotmail https://en.m.wikipedia.org/wiki/Hotmail > Founded in 1996 by Sabeer Bhatia and Jack Smith as Hotmail, it was acquired by Microsoft in 1997 for an estimated $400 million Wrong decade, they really did acquire Hotmail in the 90's.
- dartos 2y agoA great story of MS incompetence and Amazon’s vendor lock in.
- pragmatic 2y agoDo you think it would be any different/better if you had to migrate to GCP (for example)? Do you think it was the migration itself or the services on Azure? Having worked with all three, there's certainly things that suck about all of them but I've found aws "most reliable" but also seems to have a large amount of disparate services needed to do things that were simpler on Azure. GCP was pretty meh, but depends on what services you used. Azure is a good choice for .net and sql server (azure sql or whatever it is now) but in but sure a service built for aws is going to "just work" on Azure (or vice versa).