5 ms·

> Since Microsoft’s other businesses are either hardware or office software, with the exception of maybe XBox’s gaming, it generally lacks operational experienc

> Since Microsoft’s other businesses are either hardware or office software, with the exception of maybe XBox’s gaming, it generally lacks operational experience in running “always on” services and preparing for unanticipated traffic surges. Uh, as a former Microsoft equivalent of a SRE this is a utterly ridiculous statement. I was in the Windows division on a team handling all the incoming telemetry - we definitely knew how to prepare for traffic surges and keep our services "always on". Has the author never seen Office 365? Bing? The entirety of Azure in general that has dozens if not hundreds of publicly facing always on services?

jscholes 7y ago
Yeah, really weird statement for the author to make, particularly given that at the top of the very same section they list a bunch of cloud-based MS products.
sokoloff 7y ago
Office 365 was derisively referred to as “Office 360” with some good reason years ago. (I use it and the uptime has gotten noticeably better. That it could get noticeably better is the problem.)
- quaquaqua1 7y ago
  HAH this has been my experience as well. And the system often times out even when the main screen is "accessible".
- fulafel 7y ago
  I think this was less than a year ago. eg https://www.reddit.com/r/sysadmin/comments/dyu69c/office365_outage/ https://www.reddit.com/r/sysadmin/comments/dyu69c/office365_...
- mike503 7y ago
  And now it’s going to be Microsoft 365!
deleted 7y ago
[deleted]
TallGuyShort 7y ago
>> we definitely knew how to prepare for traffic surges and keep our services "always on" Okay, but as a heavy customer of AWS for 10 years and Azure & GCP for 6 years... there's some very valid criticism of the capacity planning and elasticity. I hit capacity, provisioning and scaling issues all the time in Azure regions. And many times the cause is not obvious because most services seem to be unaware that the resources are exhausted, and fail with seemingly unrelated errors and partial failure. edit: and to be fair, I'm not saying it's easy to build something like Azure or that I would do better. And I have a huge list of cases where Amazon's product engineering is terrible compared to Azure's. But... the problems I've seen all week in the UK aren't imaginary. I've had no such problems on Amazon.
- outworlder 7y ago
  > there's some very valid criticism of the capacity planning and elasticity. Indeed. Getting quota increases is also a much slower and involved process, compared to AWS or GCP. > and fail with seemingly unrelated errors and partial failure. Oh, don't get me started! If managers were forced to work with the Azure API, they would not choose it as the provider. The unclear error responses are the most aggravating part.
- Thaxll 7y ago
  AWS have the same sort of problems sometimes, try to use 100/1000s of some C5 instances in some regions.
  - gregw2 7y ago
    even getting 20 r4.4xlarges in us-east1 in some AZs was a problem for our workloads a few months ago
- deleted 7y ago
  [deleted]
- godtoldmetodoit 7y ago
  I don't disagree with anything you are saying here, but I was heavily taken back by the statement the author made as it felt right out of the early 2000's, ignoring the last 20 years of Microsoft's shift to services and the cloud. Azure clearly prioritized massively increasing their geographic footprint without the availability zones that AWS almost always deploys when they enter a new region, and I have no doubt they run leaner with less compute available then AWS. When I was on the telemetry team about 3 years ago, we were in the early days of transitioning from the internal only "physical cloud" that the Bing team had created years before into Azure, and it was full of pain for a service our size at that time.
thoraway1010 7y ago
I think they have the experience, but the success has not always been great. I used to have a hotmail account -> not impressed with how microsoft managed that vs things like gmail (downtime etc). Same thing during the early Office 365 days. These systems were rough. Zoom and AWS seem to be handling scaling pretty well - I'm a bit confused though as to how much extra capacity it would make sense for AWS to be carrying or maybe they aren't being asked to scale the way microsoft is which is my guess.
- outworlder 7y ago
  > I'm a bit confused though as to how much extra capacity it would make sense for AWS to be carrying or maybe they aren't being asked to scale the way microsoft is which is my guess. AWS has a crapton of capacity. So much so that they were the first ones to come up with the 'spot' instance concept. We could easily get hundreds of spot instances most days, even in a popular region like US-West 2. Those were relatively big ones too, with a minimum 16 VCPUs and 64GB – but given the spot pricing, we would request even bigger ones if they were available. Even when they were terminated, usually we would get a replacement instance in a few minutes. Even faster if the fleet request was flexible and didn't need a particular instance type. AWS will also honor pretty large quota increases without batting an eye, while Azure (and sometimes even GCP) would send similar numbers "for review" and take their sweet time. I can only remember a couple of times, years ago, where requests for normal, on-demand instances failed to get satisfied, in a particular availability zone. But the other zones would be fine. Most Azure regions do not even _have_ availability zones, so that's out.
  - cc81 7y ago
    AWS is bigger but I doubt they have seen the same type of change in traffic as I don't think they have the same traffic patterns that would be affected by something like this. Teams was used before but suddenly in my company you had 100k office workers working from home and all meetings converted to Teams meetings, a lot with video streaming. People also seem to react by booking even more meetings now when the casual talk was gone (instead of chatting casually).
kerng 7y ago
Agreed, Microsoft has been running cloud services for long time, even before the cloud got its name. Things like Windows Update, Xbox Live, MSN, Hotmail... some of them have been around for 20+ years.
btilly 7y ago
You are judging by what actions you were taking to make things better. Everyone else judges by what results they see. Based on my experience, Azure is not as good at AWS and Google at actually keeping things up and running when the shit hits the fan. And AWS is not as good as Google. (That said, AWS exposes a lot more to customers than Google does. Google may stay up, but I prefer running software in the AWS cloud.) That isn't to say that all three aren't doing a lot right. Clearly they all are. And clearly they are all doing a lot right that I don't know how to do. But the fact that they do a lot right doesn't mean that they are equal.
- lostmsu 7y ago
  Google? Did not they have a multi-hour outage just a few days ago?
  - briffle 7y ago
    they had some issues in the IAM services, causing retries and latency, but none of my gce and gke infrastructure showed anything on our monitoring.
  - btilly 7y ago
    They did. But which has had more outages in the last 5 years?
    - posnet 7y ago
      The biggest problem with GCP outages is that they've had multiple network issues that affected multiple north American regions at the same time. So maybe there are less of them on a per service basis compared the AWS, but when they do happen they are far more serious. The only AWS has had that came close was the us-east-1 s3 outage. But even that didn't break networking to running instances.
  - ajaimk 7y ago
    Google employee here (my own statements, not the company's, blah blah blah). The incident last week was due to hardware failure and completely unrelated to Capacity or COVID. https://status.cloud.google.com/incident/zall/20002 https://status.cloud.google.com/incident/zall/20002
nrki 7y ago
> I was in the Windows division on a team handling all the incoming telemetry How did your team live with yourselves? Dedicated to scaling a backend to ensure Windows could adequately spy on users.
- dang 7y ago
  Please don't attack other users. It leads to worse discussion, not better. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

jscholes 7y ago

Yeah, really weird statement for the author to make, particularly given that at the top of the very same section they list a bunch of cloud-based MS products.

sokoloff 7y ago

Office 365 was derisively referred to as “Office 360” with some good reason years ago. (I use it and the uptime has gotten noticeably better. That it could get noticeably better is the problem.)

quaquaqua1 7y ago
HAH this has been my experience as well. And the system often times out even when the main screen is "accessible".
fulafel 7y ago
I think this was less than a year ago. eg https://www.reddit.com/r/sysadmin/comments/dyu69c/office365_outage/ https://www.reddit.com/r/sysadmin/comments/dyu69c/office365_...
mike503 7y ago
And now it’s going to be Microsoft 365!

deleted 7y ago

[deleted]

TallGuyShort 7y ago

>> we definitely knew how to prepare for traffic surges and keep our services "always on" Okay, but as a heavy customer of AWS for 10 years and Azure & GCP for 6 years... there's some very valid criticism of the capacity planning and elasticity. I hit capacity, provisioning and scaling issues all the time in Azure regions. And many times the cause is not obvious because most services seem to be unaware that the resources are exhausted, and fail with seemingly unrelated errors and partial failure. edit: and to be fair, I'm not saying it's easy to build something like Azure or that I would do better. And I have a huge list of cases where Amazon's product engineering is terrible compared to Azure's. But... the problems I've seen all week in the UK aren't imaginary. I've had no such problems on Amazon.

outworlder 7y ago
> there's some very valid criticism of the capacity planning and elasticity. Indeed. Getting quota increases is also a much slower and involved process, compared to AWS or GCP. > and fail with seemingly unrelated errors and partial failure. Oh, don't get me started! If managers were forced to work with the Azure API, they would not choose it as the provider. The unclear error responses are the most aggravating part.
Thaxll 7y ago
AWS have the same sort of problems sometimes, try to use 100/1000s of some C5 instances in some regions.
- gregw2 7y ago
  even getting 20 r4.4xlarges in us-east1 in some AZs was a problem for our workloads a few months ago
deleted 7y ago
[deleted]
godtoldmetodoit 7y ago
I don't disagree with anything you are saying here, but I was heavily taken back by the statement the author made as it felt right out of the early 2000's, ignoring the last 20 years of Microsoft's shift to services and the cloud. Azure clearly prioritized massively increasing their geographic footprint without the availability zones that AWS almost always deploys when they enter a new region, and I have no doubt they run leaner with less compute available then AWS. When I was on the telemetry team about 3 years ago, we were in the early days of transitioning from the internal only "physical cloud" that the Bing team had created years before into Azure, and it was full of pain for a service our size at that time.

thoraway1010 7y ago

I think they have the experience, but the success has not always been great. I used to have a hotmail account -> not impressed with how microsoft managed that vs things like gmail (downtime etc). Same thing during the early Office 365 days. These systems were rough. Zoom and AWS seem to be handling scaling pretty well - I'm a bit confused though as to how much extra capacity it would make sense for AWS to be carrying or maybe they aren't being asked to scale the way microsoft is which is my guess.

outworlder 7y ago
> I'm a bit confused though as to how much extra capacity it would make sense for AWS to be carrying or maybe they aren't being asked to scale the way microsoft is which is my guess. AWS has a crapton of capacity. So much so that they were the first ones to come up with the 'spot' instance concept. We could easily get hundreds of spot instances most days, even in a popular region like US-West 2. Those were relatively big ones too, with a minimum 16 VCPUs and 64GB – but given the spot pricing, we would request even bigger ones if they were available. Even when they were terminated, usually we would get a replacement instance in a few minutes. Even faster if the fleet request was flexible and didn't need a particular instance type. AWS will also honor pretty large quota increases without batting an eye, while Azure (and sometimes even GCP) would send similar numbers "for review" and take their sweet time. I can only remember a couple of times, years ago, where requests for normal, on-demand instances failed to get satisfied, in a particular availability zone. But the other zones would be fine. Most Azure regions do not even _have_ availability zones, so that's out.
- cc81 7y ago
  AWS is bigger but I doubt they have seen the same type of change in traffic as I don't think they have the same traffic patterns that would be affected by something like this. Teams was used before but suddenly in my company you had 100k office workers working from home and all meetings converted to Teams meetings, a lot with video streaming. People also seem to react by booking even more meetings now when the casual talk was gone (instead of chatting casually).

kerng 7y ago

Agreed, Microsoft has been running cloud services for long time, even before the cloud got its name. Things like Windows Update, Xbox Live, MSN, Hotmail... some of them have been around for 20+ years.

btilly 7y ago

You are judging by what actions you were taking to make things better. Everyone else judges by what results they see. Based on my experience, Azure is not as good at AWS and Google at actually keeping things up and running when the shit hits the fan. And AWS is not as good as Google. (That said, AWS exposes a lot more to customers than Google does. Google may stay up, but I prefer running software in the AWS cloud.) That isn't to say that all three aren't doing a lot right. Clearly they all are. And clearly they are all doing a lot right that I don't know how to do. But the fact that they do a lot right doesn't mean that they are equal.

lostmsu 7y ago
Google? Did not they have a multi-hour outage just a few days ago?
- briffle 7y ago
  they had some issues in the IAM services, causing retries and latency, but none of my gce and gke infrastructure showed anything on our monitoring.
- btilly 7y ago
  They did. But which has had more outages in the last 5 years?
  - posnet 7y ago
    The biggest problem with GCP outages is that they've had multiple network issues that affected multiple north American regions at the same time. So maybe there are less of them on a per service basis compared the AWS, but when they do happen they are far more serious. The only AWS has had that came close was the us-east-1 s3 outage. But even that didn't break networking to running instances.
- ajaimk 7y ago
  Google employee here (my own statements, not the company's, blah blah blah). The incident last week was due to hardware failure and completely unrelated to Capacity or COVID. https://status.cloud.google.com/incident/zall/20002 https://status.cloud.google.com/incident/zall/20002

nrki 7y ago

> I was in the Windows division on a team handling all the incoming telemetry How did your team live with yourselves? Dedicated to scaling a backend to ensure Windows could adequately spy on users.

dang 7y ago
Please don't attack other users. It leads to worse discussion, not better. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html