10 ms·

Microsoft Azure data deleted because of DNS outage

217 points by stonewhite 8y ago

lenticular 8y ago
I just don't hear any good things about Azure. That is unfortunate, because I'd love AWS to have some competition.
- briffle 8y ago
  Try google cloud?
  - random3 8y ago
    I'm trying it... starting to have second thoughts though. They make it look like it's all ready to go and then you start uncovering bits and pieces of things that seem to be half-done, half-legacy, half-dead at the same time. TBH it looks like they got spread too thin and can see how a good amount of services are solid while many are just there.
    - lenticular 8y ago
      This was my experience when I briefly considered their relational database solution.
    - outworlder 8y ago
      > I'm trying it... starting to have second thoughts though. They make it look like it's all ready to go and then you start uncovering bits and pieces of things that seem to be half-done, half-legacy, half-dead at the same time. TBH it looks like they got spread too thin and can see how a good amount of services are solid while many are just there. Interestingly, that's my experience on Azure. Seems like a lot of offerings were smashed together in order to try to form a coherent product.
      - ams6110 8y ago
        Seems like a lot of offerings were smashed together in order to try to form a coherent product. Seems like that would apply to many, many web services and products in the last decade.
        outworlder 8y ago
        True. But even the worst AWS offerings ahemEKSahem are still integrated across the product line. You can expect easy integration with other AWS services. And indeed what little it offers, it's integrated (AWS AutoScaling groups, NLBs, IAM, etc) Azure's AKS is way better as a whole, but until recently, wasn't integrated with their own scalesets. How does that even happen? To be fair to Azure, AKS is an example on how to rollout features. All the knobs are there (even more so than GKE, you can even set cluster ip ranges), it just works and no silly surprises. At least, I haven't found surprising behavior yet.
    - dman 8y ago
      Could you be more specific?
    - roneythomas6 8y ago
      Which parts specifically?? I have worked on GCP extensively. Things that annoyed me were appengine standard using Python 2.7 and AWS being more innovative in bringing in new offering much faster than Google can. But Google recently announced Python 3.7 for appengine std.Other than it terms reliability GCP is amazing.
    - random3 8y ago
      1 point by random3 0 minutes ago | edit | delete [-] To be more specific.. Disclaimer: I've built distributed infra on AWS, Azure and GCP as well as on premise. There's a reason I'm using GCP. I'm not stating that reason below. Instead these are a few of the reasons I don't like it. In summary I believe they focus on too many things with too little depth. This is not quite unique to GCP but rather new to GCP IMO. Take endpoints for example. You get a nice feature that documents your API based on the code-level docs. To update it you also get an API, but it only works if you do one manual sync first from the console (https://cloud.google.com/endpoints/docs/grpc/dev-portal-sync... https://cloud.google.com/endpoints/docs/grpc/dev-portal-sync...) Then you discover that your documentation starts disappearing and debug for a while just to figure out that the last refactoring got the total length of your service name, RPC name and parameters over 80 characters and your doc doesn't show up if that's the case. Then you use the tracing capabilities only to discover that the traces don't propagate across services. There's something that ESP (their nginx proxy doesn't do). You take a look at it, try to build it, but discover it uses a Bazel version that is two years old. Then you look at quotas / throttling First you'll notice that the examples don't work. You just get errors (and apparently they also don't get fixed after sending feedback) https://cloud.google.com/endpoints/docs/grpc/quotas-configur.. https://cloud.google.com/endpoints/docs/grpc/quotas-configur.... You look at the example and notice it's copy-pasted-modified from some JSON and the field names are incorrect. Then you see it throttling only works with API keys, but they (and everyone else) don't recommend you use API keys and instead use IAM service accounts. Except that a bunch of features won't work unless you use API keys. So you use API keys and then discover you can't provision them, because there's no API. You have to, again, do it manually through the console. You talk to support and they'll recommend you use IAM service accounts because they are much better, although you won't be able to use the api-key specific features (https://cloud.google.com/endpoints/docs/grpc/when-why-api-ke... https://cloud.google.com/endpoints/docs/grpc/when-why-api-ke...). If you take a look at service accounts and IAM you discover new things.. I won't go into details here, but let’s just say figuring out what if a policy should work in a particular case is more art than science. Then in GKE you want to enable TLS on your gRPC service. It should be easy. https://cloud.google.com/endpoints/docs/grpc/enabling-ssl https://cloud.google.com/endpoints/docs/grpc/enabling-ssl It's just that it doesn't work, or that different documents say both that it works and that it doesn't work.. Take a look at https://github.com/kubernetes/ingress-gce/issues/18 https://github.com/kubernetes/ingress-gce/issues/18 From certificate provisioning to functional service, It took roughly 5 days involving tcpdump, looking at ESP source code, all the logs, raising bugs on github, etc. Then see Service Catalog (https://cloud.google.com/kubernetes-engine/docs/how-to/add-o... https://cloud.google.com/kubernetes-engine/docs/how-to/add-o...). You try to download from the link they provide and it will throw an error. You debug maybe talk to support, and they'll tell you that there was a patch and it fixes it. Just that it wasn't released since April 2018... Then one day your GKE pod won't deploy and you see no errors. It takes a while to realize that if the port name is over a certain number of characters it won't work. There are many of these and I could keep going :)
  - deleted 8y ago
    [deleted]
- outworlder 8y ago
  > I just don't hear any good things about Azure. That is unfortunate, because I'd love AWS to have some competition. Google Cloud is fair competition – provided they have the service you need. AWS and Azure both beat them in number of services. If Google has it, then it should behave as expected, and some are downright impressive (GKE and VM auto migration on GCE). Azure is... infuriating. Inconsistent, unreliable APIs, surprising behavior everywhere (attach an internal load balancer, lose internet connectivity!?), lots of restrictions on which features can be used with which SKUs. I see improvements and it is difficult to beat them in the enterprise, but speaking as an engineer, man Azure is infuriating.
  - dharmab 8y ago
    > attach an internal load balancer, lose internet connectivity!? Ah, yes, the "TCP and UDP egress work unless you define an _ingress_ load balancing rule for either protocol, at which point the other protocol breaks until you create a dummy rule for the other protocol that will trigger your security team to send you tickets every few months."
- itsdrewmiller 8y ago
  We use Azure and love it. We've had more problems over the past few years from downstream services being broken by AWS (S3 outage, etc.) than our primary apps being broken by Azure.
  - gnulinux 8y ago
    > S3 outage Wait what? S3 is 11 nines. That's like 1 file every 8 PB/year-years.
    - sudosteph 8y ago
      11 9s is for durability, not availability. ie, S3 will almost never lose your file, but sometimes you won't be able to retrieve it immediately. There was a pretty significant S3 availability outage not too long ago, which is probably what the parent comment is referring to.
    - viraptor 8y ago
      Availability: in practice 5 hours outage in us-east in 2017: https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ https://www.theregister.co.uk/2017/03/01/aws_s3_outage/
- hbcondo714 8y ago
  There are other large software companies with competing cloud offerings, including Oracle[1], IBM[2], Alibaba[3] and more. [1] https://cloud.oracle.com/home https://cloud.oracle.com/home [2] https://www.ibm.com/cloud/ https://www.ibm.com/cloud/ [3] https://us.alibabacloud.com/ https://us.alibabacloud.com/
  - lenticular 8y ago
    None that can match the maturity or performance of AWS. I always hear really terrible things about Oracle and IBM. edit: Mostly I've heard it just generally is not great, plus you have to deal with typical Oracle badness.
    - tmd83 8y ago
      Do you have specifics to share about Oracle clouds. That's the one you hear the least about publicly I think. You at least can get some ideas of the problems of AWS, GCE and Azure.
      - jstanley 8y ago
        Oracle offer quite a compelling free trial. I got a VM for 30 days with 16 gigabytes (!) of RAM and it didn't cost me a penny. In fact, the machine continued working for a few months after the trial ended, which I thought was very generous. They just blocked port 80 and 443 inbound. They even have a high-touch sales process where a real person sends you emails and replies when you send them questions. Imagine that at AWS! I was actually tempted to keep paying for the VM from Oracle, but the high-touch sales person literally could not tell me how much it was going to cost. She could point me towards the price calculator page, but the VM type I had been given in my free trial did not exist in their price calculator. I asked for just a simple number: how much is this going to cost me per month? She could not give me an answer. So I didn't buy it. I still have no idea whether it would have been a good deal or not.
        outworlder 8y ago
- TheIronYuppie 8y ago
  Disclosure: I work at Azure. We're doing our best, but we're not going to suggest there's not more to do. Every major cloud provider has had issues at one point or another (I formerly worked at Amazon and Google), and I'll just say - we hear you, and we are fiercely committed to earning your trust.
  - lenticular 8y ago
    I'm sure you are, and I wish you the best of luck. I'd rather spend money with Microsoft than Amazon, everything else being equal. One way that Azure could really improve on Amazon is docs/usability, for which AWS is terrible.
    - dpcx 8y ago
      Do I understand your comment to mean that Amazon has terrible documentation around AWS? Because if that's the case, I'm truly curious what documentation you find terrible. It can be confusing at times for me, but from my experience, that's mostly due to the fact that there's so many different options for some things.
      - jstanley 8y ago
        For one, it is practically impossible to work out how much something is going to cost.
      - sl1ck731 8y ago
        I just ran into an issue with the way they blog their documentation. It isn't hierarchical from what I can tell and they don't go back and update blogs to point you at the latest document. I wasted a lot of time going step-by-step through one to find out it was deprecated. It is a problem because a lot of the good documentation about overall usage or architecture for some reason is in this blog format but I can see it just being written off because "its just a blog". They also leave a lot out of the true documentation such as Landing Zone stuff (in fact they hide this one entirely) and Application Discovery Service (they make no mention how the agents work). There are plenty of other minutia that I have had to contact our rep. to get from the product team. The most common or basic stuff has some great documentation. But once you stray into the "weird" it isn't uncommon to see 2 or 3 line documentation pages about something.
- Dayshine 8y ago
  Well, I'm not sure that's true. I only hear bad things about AWS and Google Cloud, and I hear nothing much about Azure.
- shanemhansen 8y ago
  Azure's ILBs are still just bizarre to me. It's the first load balancer I've ever worked with where sometimes a member of the pool can't reach the load balancer. At the packet level given their implementation, this makes sense but tell me how many times you've ever had to write a stats endpoint where the stats nodes had to do workarounds to send their own stats? Source: https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview#limitations https://docs.microsoft.com/en-us/azure/load-balancer/load-ba...
  - outworlder 8y ago
    Do you have any information on their implementation? I only noticed that something was amiss once I connected a load balancer and my instances suddenly lost internet access. And also due to the limitation on how many load balancers you can have. It seems that once you add a load balancer, all traffic gets funneled through it, doesn't matter if it was addressed to it or not. Which is unlike any other load balancer I have ever seen. Coming from other clouds, this was a shock. The only thing comparable is AWS's NLB. Because that load balancer is so transparent, clients appear to be connecting directly, with the original source ip. Which caused issues when I wanted to deploy my own Elasticsearch and use an internal NLB for master discovery (whenever a request got routed to the same machine packets got discarded by the kernel). But you can just switch to another load balancer then.
    - maglite77 8y ago
      Disclosure: Engineer at Microsoft There is an excellent Network Academy video on YouTube [1] that covers some of the internal implementations of the layer-4 load balancers (in the context of HA for NVA's). [1]: https://youtu.be/oYJBrWcOXfY https://youtu.be/oYJBrWcOXfY
- stevenjohns 8y ago
  That's because Azure is a poor service with mediocre support. My anecdotal experience: I spent a couple of weeks (!) setting up our environment (Bitbucket, Django, Ubuntu, Dockerized) on Azure App Service and Azure Pipelines. Their documentation was incomplete, out-of-date and MS support staff struggled to help if you didn't have a Windows machine (their RDP software doesn't support Linux, Skype for Business doesn't support Linux and normal Skype for Linux doesn't support screen sharing). Little things like trying to SSH into any machine so that you can execute commands on your docker container (for, say, database migrations or to check logs) is almost impossible. If it wasn't for the help of a lot of people on #docker in Freenode I would probably still be working on it. I had to use Google Hangouts with a Microsoft support person's personal gmail account, while he was connected over VPN (since he was based in Shanghai), so I could show my issue. The support person was extremely pleasant to deal with and understanding, though, and he went above and beyond to help get my issue resolved even though it turned out to not be from his department. However, after getting set up, I noticed I was getting 12 second (!) responses from an API I had written just to retrieve a logged-in user's first name, last name and email in JSON. This API resolves locally in 20ms - including layers of authentication. This turned out to be a known issue when running a managed "Azure Database for PostgreSQL" service and was common on MS support forums. After reaching out to Microsoft support for Azure Database for PostgreSQL, their response was this, copy-and-pasted: > As you are currently using Basic Tier (2 vCores, 51200MB), the bad performance is expected. > When comparing with the performance in your VM, the on-prem is supposed to be better than cloud even within the same hardware environment. > Please give it a test in higher tier and configure it with a compatible settings compared with your VM. In the meanwhile, you can monitor the slow queries via Query Performance Insight to find out what queries were running at a long time when those API were called. > Pricing tier information can be found at https://docs.microsoft.com/en-us/azure/postgresql/concepts-pricing-tiers#storage https://docs.microsoft.com/en-us/azure/postgresql/concepts-p... . ...they tried to upsell me on the higher tier database 3 times in that email chain, believing that this level of performance was acceptable for my database tier. Of course the next tier up from the $60/month that I was on was $160/month, and since we only have maybe two concurrent users at most it didn't make sense to triple our costs just to avoid 12 second database calls. I moved the entire service to AWS last week. The set up was painless and swift. Using equivalently priced services, the API now resolves in 50ms. I don't think I'll ever go back. Not even for free.
excalibur 8y ago
> The deletions were automated, triggered by a script that drops TDE database tables when their corresponding keys can no longer be accessed in the Key Vault, explained Microsoft in a letter reportedly sent to customers. By what logic is this NOT a terrible idea?
- booi 8y ago
  It sounds like they do this for a data security and compliance reasons but it seems like sloppy engineering to not consider unreachability as possible temporary error.
- mh8h 8y ago
  At least it needs better error handling. A "not found" and "cannot resolve domain name" should behave differently.
  - rcostin2k2 8y ago
    Perhaps that was exactly the response they've got in that moment.
- LoSboccacc 8y ago
  The whole "managed resources have a mandatory public dns and ip" idea is insane Yeah they come with a firewall but still. Imagine competing with everyone else on a single namespace. At least for the s3 bucket is justified because those are meant to be accessible, but the databases?
  - derefr 8y ago
    If the "public dns" are arbitrarily-nested subdomains, and the "ips" are IPv6 addresses, I don't see what the problem is. We've got essentially-unbounded amounts of both of those.
    - LoSboccacc 8y ago
      > If the "public dns" are arbitrarily-nested subdomains, not names. those are the limited conflicting resource, and they aren't arbitrarily nested: "The server can contain only lowercase letters, numbers, and the hyphen (-) character" - "The domain name postgres.database.azure.com is appended to the server name you provide." it also means I cannot just create parallel environments on their own subnet with the same scripts, I've to enter and manage configuration points for all the names, either as dns aliases or straight up in the application connection strings, and it also makes unnecessarily harder to just migrate the stuff around as needed to a failover node i.e. rds advantage was zero downtime upgrades too, then and then sent a mail everyone last week with "we going to have 5 minute downtime to upgrade your db, whether it's multi zone with replicas or not"? you can't trust them with that, the cloud is just somebody else computers', you need your own failover on top, and instead of changing one entry in the namespace I've to redeploy all clients, oh joy.
cddotdotslash 8y ago
Sounds like they built a dead-man's switch and then broke the process through which the man and the switch communicate.
- dtech 8y ago
  Would be a pretty ineffective dead mans switch if a backup from 5 minutes ago is available
  - spydum 8y ago
    The keys didn’t get deleted. Only the encrypted data. If keys are gone - they are GONE. AFAIK you cannot restore the keys
  - eridius 8y ago
    It's presumably intended as way to reclaim space now taken up by effectively-random garbage rather than a security measure.
- ajross 8y ago
  It was garbage collection. They were deleting data that couldn't be accessed (because no key existed in the system to decrypt it), but the DNS failure fooled the detection into thinking that a failed lookup meant "no key exists". Yikes. To be clear: it was ultimately only a 5 minute loss (and the fact that the DNS outage was simultaneous probably meant there wasn't much data being stored anyway) because they had a regular snapshot facility. So defense in depth saved them. Still, yikes. That's a pretty disastrous bug.
  - tybit 8y ago
    I’m really curious as to whether this would be because of REST APIs. Conflating ‘I can’t find that endpoint’ with ‘the endpoint says that resource has been deleted’ via a 404.
snockerton 8y ago
It appears that the SLA guaranteed uptime for Azure SQL Database is 99.9% or 99.99%, depending on tier. That equates to the following allowable downtime per month (which I think is what they base SLA fulfillment on): 99.9: 43m 49.7s 99.99: 4m 23.0s Sounds like they need to cough up some money for their four 9s customers...
- kthejoker2 8y ago
  As the article indicates, MSFT is offering 3 months of free service to affected customers.
m0zg 8y ago
"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." -- Leslie Lamport. I feel this should be rephrased for cloud computing at this point. The more people rely on cloud, the more these global fuck-ups are going affect them. Makes me feel pretty good about that server rack in my garage that addresses most of my own (and my business') compute needs.
- bdibs 8y ago
  Because nothing could happen to your rack (or garage)?
  - mrmondo 8y ago
    I can’t say for sure, but I’m assuming they meant because it is (in comparison) not a distributed (or complex) system.
  - m0zg 8y ago
    Bad things can happen to it, but at least nobody futzes with it all the time. Most of e.g. Google's major outages (Cloud and Borg) are configuration error or human error. I run my own systems very conservatively. I use conservative technology choices, my kernels are years behind bleeding edge (although I do install security updates, of course), and I don't try to fix or "improve" what's not broken because my paycheck does not depend on that. And in the unlikely event something does get fucked up, I can fix it within less than an hour and I'm the only one affected. It's also extremely unlikely to have any kind of "global" outage. Don't get me wrong, I use cloud (GCP, if you must know) too, and if my business grew massively, I'd probably use it more. But frankly I'm more satisfied with my own "on prem" solution. Single rack which basically pays for itself every 3 months or so in cloud costs, what's not to like?
    - joncrane 8y ago
      What if you're on vacation when something happens, such as a power outage? What about a massive event like an earthquake/flood/wind storm/fire/etc?

217 points by stonewhite 8y ago

lenticular 8y ago

I just don't hear any good things about Azure. That is unfortunate, because I'd love AWS to have some competition.

briffle 8y ago
Try google cloud?
- random3 8y ago
  I'm trying it... starting to have second thoughts though. They make it look like it's all ready to go and then you start uncovering bits and pieces of things that seem to be half-done, half-legacy, half-dead at the same time. TBH it looks like they got spread too thin and can see how a good amount of services are solid while many are just there.
  - lenticular 8y ago
    This was my experience when I briefly considered their relational database solution.
  - outworlder 8y ago
    > I'm trying it... starting to have second thoughts though. They make it look like it's all ready to go and then you start uncovering bits and pieces of things that seem to be half-done, half-legacy, half-dead at the same time. TBH it looks like they got spread too thin and can see how a good amount of services are solid while many are just there. Interestingly, that's my experience on Azure. Seems like a lot of offerings were smashed together in order to try to form a coherent product.
    - ams6110 8y ago
      Seems like a lot of offerings were smashed together in order to try to form a coherent product. Seems like that would apply to many, many web services and products in the last decade.
      - outworlder 8y ago
        True. But even the worst AWS offerings ahemEKSahem are still integrated across the product line. You can expect easy integration with other AWS services. And indeed what little it offers, it's integrated (AWS AutoScaling groups, NLBs, IAM, etc) Azure's AKS is way better as a whole, but until recently, wasn't integrated with their own scalesets. How does that even happen? To be fair to Azure, AKS is an example on how to rollout features. All the knobs are there (even more so than GKE, you can even set cluster ip ranges), it just works and no silly surprises. At least, I haven't found surprising behavior yet.
  - dman 8y ago
    Could you be more specific?
  - roneythomas6 8y ago
    Which parts specifically?? I have worked on GCP extensively. Things that annoyed me were appengine standard using Python 2.7 and AWS being more innovative in bringing in new offering much faster than Google can. But Google recently announced Python 3.7 for appengine std.Other than it terms reliability GCP is amazing.
  - random3 8y ago
    1 point by random3 0 minutes ago | edit | delete [-] To be more specific.. Disclaimer: I've built distributed infra on AWS, Azure and GCP as well as on premise. There's a reason I'm using GCP. I'm not stating that reason below. Instead these are a few of the reasons I don't like it. In summary I believe they focus on too many things with too little depth. This is not quite unique to GCP but rather new to GCP IMO. Take endpoints for example. You get a nice feature that documents your API based on the code-level docs. To update it you also get an API, but it only works if you do one manual sync first from the console (https://cloud.google.com/endpoints/docs/grpc/dev-portal-sync... https://cloud.google.com/endpoints/docs/grpc/dev-portal-sync...) Then you discover that your documentation starts disappearing and debug for a while just to figure out that the last refactoring got the total length of your service name, RPC name and parameters over 80 characters and your doc doesn't show up if that's the case. Then you use the tracing capabilities only to discover that the traces don't propagate across services. There's something that ESP (their nginx proxy doesn't do). You take a look at it, try to build it, but discover it uses a Bazel version that is two years old. Then you look at quotas / throttling First you'll notice that the examples don't work. You just get errors (and apparently they also don't get fixed after sending feedback) https://cloud.google.com/endpoints/docs/grpc/quotas-configur.. https://cloud.google.com/endpoints/docs/grpc/quotas-configur.... You look at the example and notice it's copy-pasted-modified from some JSON and the field names are incorrect. Then you see it throttling only works with API keys, but they (and everyone else) don't recommend you use API keys and instead use IAM service accounts. Except that a bunch of features won't work unless you use API keys. So you use API keys and then discover you can't provision them, because there's no API. You have to, again, do it manually through the console. You talk to support and they'll recommend you use IAM service accounts because they are much better, although you won't be able to use the api-key specific features (https://cloud.google.com/endpoints/docs/grpc/when-why-api-ke... https://cloud.google.com/endpoints/docs/grpc/when-why-api-ke...). If you take a look at service accounts and IAM you discover new things.. I won't go into details here, but let’s just say figuring out what if a policy should work in a particular case is more art than science. Then in GKE you want to enable TLS on your gRPC service. It should be easy. https://cloud.google.com/endpoints/docs/grpc/enabling-ssl https://cloud.google.com/endpoints/docs/grpc/enabling-ssl It's just that it doesn't work, or that different documents say both that it works and that it doesn't work.. Take a look at https://github.com/kubernetes/ingress-gce/issues/18 https://github.com/kubernetes/ingress-gce/issues/18 From certificate provisioning to functional service, It took roughly 5 days involving tcpdump, looking at ESP source code, all the logs, raising bugs on github, etc. Then see Service Catalog (https://cloud.google.com/kubernetes-engine/docs/how-to/add-o... https://cloud.google.com/kubernetes-engine/docs/how-to/add-o...). You try to download from the link they provide and it will throw an error. You debug maybe talk to support, and they'll tell you that there was a patch and it fixes it. Just that it wasn't released since April 2018... Then one day your GKE pod won't deploy and you see no errors. It takes a while to realize that if the port name is over a certain number of characters it won't work. There are many of these and I could keep going :)
- deleted 8y ago
  [deleted]
outworlder 8y ago
> I just don't hear any good things about Azure. That is unfortunate, because I'd love AWS to have some competition. Google Cloud is fair competition – provided they have the service you need. AWS and Azure both beat them in number of services. If Google has it, then it should behave as expected, and some are downright impressive (GKE and VM auto migration on GCE). Azure is... infuriating. Inconsistent, unreliable APIs, surprising behavior everywhere (attach an internal load balancer, lose internet connectivity!?), lots of restrictions on which features can be used with which SKUs. I see improvements and it is difficult to beat them in the enterprise, but speaking as an engineer, man Azure is infuriating.
- dharmab 8y ago
  > attach an internal load balancer, lose internet connectivity!? Ah, yes, the "TCP and UDP egress work unless you define an _ingress_ load balancing rule for either protocol, at which point the other protocol breaks until you create a dummy rule for the other protocol that will trigger your security team to send you tickets every few months."
itsdrewmiller 8y ago
We use Azure and love it. We've had more problems over the past few years from downstream services being broken by AWS (S3 outage, etc.) than our primary apps being broken by Azure.
- gnulinux 8y ago
  > S3 outage Wait what? S3 is 11 nines. That's like 1 file every 8 PB/year-years.
  - sudosteph 8y ago
    11 9s is for durability, not availability. ie, S3 will almost never lose your file, but sometimes you won't be able to retrieve it immediately. There was a pretty significant S3 availability outage not too long ago, which is probably what the parent comment is referring to.
  - viraptor 8y ago
    Availability: in practice 5 hours outage in us-east in 2017: https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ https://www.theregister.co.uk/2017/03/01/aws_s3_outage/
hbcondo714 8y ago
There are other large software companies with competing cloud offerings, including Oracle[1], IBM[2], Alibaba[3] and more. [1] https://cloud.oracle.com/home https://cloud.oracle.com/home [2] https://www.ibm.com/cloud/ https://www.ibm.com/cloud/ [3] https://us.alibabacloud.com/ https://us.alibabacloud.com/
- lenticular 8y ago
  None that can match the maturity or performance of AWS. I always hear really terrible things about Oracle and IBM. edit: Mostly I've heard it just generally is not great, plus you have to deal with typical Oracle badness.
  - tmd83 8y ago
    Do you have specifics to share about Oracle clouds. That's the one you hear the least about publicly I think. You at least can get some ideas of the problems of AWS, GCE and Azure.
    - jstanley 8y ago
      Oracle offer quite a compelling free trial. I got a VM for 30 days with 16 gigabytes (!) of RAM and it didn't cost me a penny. In fact, the machine continued working for a few months after the trial ended, which I thought was very generous. They just blocked port 80 and 443 inbound. They even have a high-touch sales process where a real person sends you emails and replies when you send them questions. Imagine that at AWS! I was actually tempted to keep paying for the VM from Oracle, but the high-touch sales person literally could not tell me how much it was going to cost. She could point me towards the price calculator page, but the VM type I had been given in my free trial did not exist in their price calculator. I asked for just a simple number: how much is this going to cost me per month? She could not give me an answer. So I didn't buy it. I still have no idea whether it would have been a good deal or not.
      - outworlder 8y ago
TheIronYuppie 8y ago
Disclosure: I work at Azure. We're doing our best, but we're not going to suggest there's not more to do. Every major cloud provider has had issues at one point or another (I formerly worked at Amazon and Google), and I'll just say - we hear you, and we are fiercely committed to earning your trust.
- lenticular 8y ago
  I'm sure you are, and I wish you the best of luck. I'd rather spend money with Microsoft than Amazon, everything else being equal. One way that Azure could really improve on Amazon is docs/usability, for which AWS is terrible.
  - dpcx 8y ago
    Do I understand your comment to mean that Amazon has terrible documentation around AWS? Because if that's the case, I'm truly curious what documentation you find terrible. It can be confusing at times for me, but from my experience, that's mostly due to the fact that there's so many different options for some things.
    - jstanley 8y ago
      For one, it is practically impossible to work out how much something is going to cost.
    - sl1ck731 8y ago
      I just ran into an issue with the way they blog their documentation. It isn't hierarchical from what I can tell and they don't go back and update blogs to point you at the latest document. I wasted a lot of time going step-by-step through one to find out it was deprecated. It is a problem because a lot of the good documentation about overall usage or architecture for some reason is in this blog format but I can see it just being written off because "its just a blog". They also leave a lot out of the true documentation such as Landing Zone stuff (in fact they hide this one entirely) and Application Discovery Service (they make no mention how the agents work). There are plenty of other minutia that I have had to contact our rep. to get from the product team. The most common or basic stuff has some great documentation. But once you stray into the "weird" it isn't uncommon to see 2 or 3 line documentation pages about something.
Dayshine 8y ago
Well, I'm not sure that's true. I only hear bad things about AWS and Google Cloud, and I hear nothing much about Azure.
shanemhansen 8y ago
Azure's ILBs are still just bizarre to me. It's the first load balancer I've ever worked with where sometimes a member of the pool can't reach the load balancer. At the packet level given their implementation, this makes sense but tell me how many times you've ever had to write a stats endpoint where the stats nodes had to do workarounds to send their own stats? Source: https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview#limitations https://docs.microsoft.com/en-us/azure/load-balancer/load-ba...
- outworlder 8y ago
  Do you have any information on their implementation? I only noticed that something was amiss once I connected a load balancer and my instances suddenly lost internet access. And also due to the limitation on how many load balancers you can have. It seems that once you add a load balancer, all traffic gets funneled through it, doesn't matter if it was addressed to it or not. Which is unlike any other load balancer I have ever seen. Coming from other clouds, this was a shock. The only thing comparable is AWS's NLB. Because that load balancer is so transparent, clients appear to be connecting directly, with the original source ip. Which caused issues when I wanted to deploy my own Elasticsearch and use an internal NLB for master discovery (whenever a request got routed to the same machine packets got discarded by the kernel). But you can just switch to another load balancer then.
  - maglite77 8y ago
    Disclosure: Engineer at Microsoft There is an excellent Network Academy video on YouTube [1] that covers some of the internal implementations of the layer-4 load balancers (in the context of HA for NVA's). [1]: https://youtu.be/oYJBrWcOXfY https://youtu.be/oYJBrWcOXfY
stevenjohns 8y ago
That's because Azure is a poor service with mediocre support. My anecdotal experience: I spent a couple of weeks (!) setting up our environment (Bitbucket, Django, Ubuntu, Dockerized) on Azure App Service and Azure Pipelines. Their documentation was incomplete, out-of-date and MS support staff struggled to help if you didn't have a Windows machine (their RDP software doesn't support Linux, Skype for Business doesn't support Linux and normal Skype for Linux doesn't support screen sharing). Little things like trying to SSH into any machine so that you can execute commands on your docker container (for, say, database migrations or to check logs) is almost impossible. If it wasn't for the help of a lot of people on #docker in Freenode I would probably still be working on it. I had to use Google Hangouts with a Microsoft support person's personal gmail account, while he was connected over VPN (since he was based in Shanghai), so I could show my issue. The support person was extremely pleasant to deal with and understanding, though, and he went above and beyond to help get my issue resolved even though it turned out to not be from his department. However, after getting set up, I noticed I was getting 12 second (!) responses from an API I had written just to retrieve a logged-in user's first name, last name and email in JSON. This API resolves locally in 20ms - including layers of authentication. This turned out to be a known issue when running a managed "Azure Database for PostgreSQL" service and was common on MS support forums. After reaching out to Microsoft support for Azure Database for PostgreSQL, their response was this, copy-and-pasted: > As you are currently using Basic Tier (2 vCores, 51200MB), the bad performance is expected. > When comparing with the performance in your VM, the on-prem is supposed to be better than cloud even within the same hardware environment. > Please give it a test in higher tier and configure it with a compatible settings compared with your VM. In the meanwhile, you can monitor the slow queries via Query Performance Insight to find out what queries were running at a long time when those API were called. > Pricing tier information can be found at https://docs.microsoft.com/en-us/azure/postgresql/concepts-pricing-tiers#storage https://docs.microsoft.com/en-us/azure/postgresql/concepts-p... . ...they tried to upsell me on the higher tier database 3 times in that email chain, believing that this level of performance was acceptable for my database tier. Of course the next tier up from the $60/month that I was on was $160/month, and since we only have maybe two concurrent users at most it didn't make sense to triple our costs just to avoid 12 second database calls. I moved the entire service to AWS last week. The set up was painless and swift. Using equivalently priced services, the API now resolves in 50ms. I don't think I'll ever go back. Not even for free.

excalibur 8y ago

> The deletions were automated, triggered by a script that drops TDE database tables when their corresponding keys can no longer be accessed in the Key Vault, explained Microsoft in a letter reportedly sent to customers. By what logic is this NOT a terrible idea?

booi 8y ago
It sounds like they do this for a data security and compliance reasons but it seems like sloppy engineering to not consider unreachability as possible temporary error.
mh8h 8y ago
At least it needs better error handling. A "not found" and "cannot resolve domain name" should behave differently.
- rcostin2k2 8y ago
  Perhaps that was exactly the response they've got in that moment.
LoSboccacc 8y ago
The whole "managed resources have a mandatory public dns and ip" idea is insane Yeah they come with a firewall but still. Imagine competing with everyone else on a single namespace. At least for the s3 bucket is justified because those are meant to be accessible, but the databases?
- derefr 8y ago
  If the "public dns" are arbitrarily-nested subdomains, and the "ips" are IPv6 addresses, I don't see what the problem is. We've got essentially-unbounded amounts of both of those.
  - LoSboccacc 8y ago
    > If the "public dns" are arbitrarily-nested subdomains, not names. those are the limited conflicting resource, and they aren't arbitrarily nested: "The server can contain only lowercase letters, numbers, and the hyphen (-) character" - "The domain name postgres.database.azure.com is appended to the server name you provide." it also means I cannot just create parallel environments on their own subnet with the same scripts, I've to enter and manage configuration points for all the names, either as dns aliases or straight up in the application connection strings, and it also makes unnecessarily harder to just migrate the stuff around as needed to a failover node i.e. rds advantage was zero downtime upgrades too, then and then sent a mail everyone last week with "we going to have 5 minute downtime to upgrade your db, whether it's multi zone with replicas or not"? you can't trust them with that, the cloud is just somebody else computers', you need your own failover on top, and instead of changing one entry in the namespace I've to redeploy all clients, oh joy.

cddotdotslash 8y ago

Sounds like they built a dead-man's switch and then broke the process through which the man and the switch communicate.

dtech 8y ago
Would be a pretty ineffective dead mans switch if a backup from 5 minutes ago is available
- spydum 8y ago
  The keys didn’t get deleted. Only the encrypted data. If keys are gone - they are GONE. AFAIK you cannot restore the keys
- eridius 8y ago
  It's presumably intended as way to reclaim space now taken up by effectively-random garbage rather than a security measure.
ajross 8y ago
It was garbage collection. They were deleting data that couldn't be accessed (because no key existed in the system to decrypt it), but the DNS failure fooled the detection into thinking that a failed lookup meant "no key exists". Yikes. To be clear: it was ultimately only a 5 minute loss (and the fact that the DNS outage was simultaneous probably meant there wasn't much data being stored anyway) because they had a regular snapshot facility. So defense in depth saved them. Still, yikes. That's a pretty disastrous bug.
- tybit 8y ago
  I’m really curious as to whether this would be because of REST APIs. Conflating ‘I can’t find that endpoint’ with ‘the endpoint says that resource has been deleted’ via a 404.

snockerton 8y ago

It appears that the SLA guaranteed uptime for Azure SQL Database is 99.9% or 99.99%, depending on tier. That equates to the following allowable downtime per month (which I think is what they base SLA fulfillment on): 99.9: 43m 49.7s 99.99: 4m 23.0s Sounds like they need to cough up some money for their four 9s customers...

kthejoker2 8y ago
As the article indicates, MSFT is offering 3 months of free service to affected customers.

m0zg 8y ago

"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." -- Leslie Lamport. I feel this should be rephrased for cloud computing at this point. The more people rely on cloud, the more these global fuck-ups are going affect them. Makes me feel pretty good about that server rack in my garage that addresses most of my own (and my business') compute needs.

bdibs 8y ago
Because nothing could happen to your rack (or garage)?
- mrmondo 8y ago
  I can’t say for sure, but I’m assuming they meant because it is (in comparison) not a distributed (or complex) system.
- m0zg 8y ago
  Bad things can happen to it, but at least nobody futzes with it all the time. Most of e.g. Google's major outages (Cloud and Borg) are configuration error or human error. I run my own systems very conservatively. I use conservative technology choices, my kernels are years behind bleeding edge (although I do install security updates, of course), and I don't try to fix or "improve" what's not broken because my paycheck does not depend on that. And in the unlikely event something does get fucked up, I can fix it within less than an hour and I'm the only one affected. It's also extremely unlikely to have any kind of "global" outage. Don't get me wrong, I use cloud (GCP, if you must know) too, and if my business grew massively, I'd probably use it more. But frankly I'm more satisfied with my own "on prem" solution. Single rack which basically pays for itself every 3 months or so in cloud costs, what's not to like?
  - joncrane 8y ago
    What if you're on vacation when something happens, such as a power outage? What about a massive event like an earthquake/flood/wind storm/fire/etc?