11 ms·

Azure cracks, AWS strong: Covid-19 stress-tests the cloud

155 points by ceohockey60 7y ago

luhn 7y ago
> One misleading headline that’s been pushed out by Microsoft and unfortunately picked up by various tech media outlets, is the “775% increase” in Azure cloud service usage in geographical areas that are most committed to some form of social distancing or shelter-in-place policy. It's way worse than that. The original statement: > We have seen a 775 percent increase of our cloud services in regions that have enforced social distancing or shelter in place orders The updated statement: > We have seen a 775 percent increase in Teams' calling and meeting monthly users in a one month period in Italy, where social distancing or shelter in place orders have been enforced. Somehow an increase in usage of a single feature of a single product in a single country was miscommunicated as an increase in demand for all of Azure's cloud services. Microsoft's PR team royally screwed up on this one. https://azure.microsoft.com/en-us/blog/update-2-on-microsoft-cloud-services-continuity/ https://azure.microsoft.com/en-us/blog/update-2-on-microsoft...
- NicoJuicy 7y ago
  This "single feature" is video-conference and is used by almost every Office 365 business customer. In chat propositions, it's Slacks biggest competitor. There is no feature bigger in bandwidth than this one. I believe the numbers are similar in Europe. By reference, I think it's similar as having a Netflix business appear from nowhere.
  - tw04 7y ago
    >By reference, I think it's similar as having a Netflix business appear from nowhere. Only without all the edge caching devices Netflix has had a decade to deploy.
  - JeremyNT 7y ago
    But is this a feature many actually used before? If it was used e.e. 10 hours prior and now 7.75 hours, it's insignificant. A percent increase is a meaningless metric without the baseline. If it's "all azure" then we immediately know it's a huge increase! But if it's just a possibly seldom used feature it's minimal.
samspenc 7y ago
> limited to places where shelter-in-place is in full force, which at least in the United States, is only a small part of the country with densely populated urban areas: New York, California, New Jersey, Michigan, etc. Ummm, it is NOT a "small part" of the country that is under "shelter-in-place". By geographic area, maybe, but California, New York and New Jersey alone include some of the largest cities by population and population density. And there are now several other states with large populations (Illinois, Florida, Washington, for example) not listed here that have shelter-in-place as well. UPDATE: I found an interactive map that shows which states have statewide shelter-in-place orders in the United States; it looks like all the states with the largest populations (and more than 50% of the country's total population) are covered, which is actually significantly larger than I had actually thought before I looked this up: https://www.nytimes.com/interactive/2020/us/coronavirus-stay-at-home-order.html https://www.nytimes.com/interactive/2020/us/coronavirus-stay...
- madhadron 7y ago
  By population, New York + California + New Jersey is 22% of the USA.
  - altoidaltoid 7y ago
    and have 6% representation in the Senate...
    - rjbwork 7y ago
      Yes, that's how the Senate works.
      - MrZongle2 7y ago
        More specifically: every state has 2 US Senators, whereas Representatives in the US House of Representatives are allocated to states based upon population.
        7y ago
uji 7y ago
A better comparison would be between Azure and GCP, both of them provide office suite (SAAS) to customers. Google recently released its side of the numbers (https://cloud.google.com/blog/topics/inside-google-cloud/how-google-cloud-is-helping-during-covid-19 https://cloud.google.com/blog/topics/inside-google-cloud/how...)
- gallego2007 7y ago
  From the link, Google has seen the following for Meet usage: “Over the last few weeks, Meet’s day-over-day growth surpassed 60%, and as a result, its daily usage is more than 25 times what it was in January. Despite this growth, the demand has been well within the bounds of our network’s ability.”
- all_blue_chucks 7y ago
  AWS also provides office suite to customers: WorkMail, WorkDocs, Chime messaging, etc.
  - reallydontask 7y ago
    Are they even in the same league as O365 and G-suite, usage wise? Only asking because I'd never heard of these services before
    - cc81 7y ago
      Are G-suite even in the same league as O365? I have no numbers and might be incredibly biased by location but I feel that all the "boring" office companies in Europe has moved towards Office 365 as a natural step. That is of course not true and I'm sure there are plenty of G-suite companies but for the large ones it seems very common that they have just followed Microsoft into the cloud.
godtoldmetodoit 7y ago
> Since Microsoft’s other businesses are either hardware or office software, with the exception of maybe XBox’s gaming, it generally lacks operational experience in running “always on” services and preparing for unanticipated traffic surges. Uh, as a former Microsoft equivalent of a SRE this is a utterly ridiculous statement. I was in the Windows division on a team handling all the incoming telemetry - we definitely knew how to prepare for traffic surges and keep our services "always on". Has the author never seen Office 365? Bing? The entirety of Azure in general that has dozens if not hundreds of publicly facing always on services?
- jscholes 7y ago
  Yeah, really weird statement for the author to make, particularly given that at the top of the very same section they list a bunch of cloud-based MS products.
- sokoloff 7y ago
  Office 365 was derisively referred to as “Office 360” with some good reason years ago. (I use it and the uptime has gotten noticeably better. That it could get noticeably better is the problem.)
  - quaquaqua1 7y ago
    HAH this has been my experience as well. And the system often times out even when the main screen is "accessible".
  - fulafel 7y ago
    I think this was less than a year ago. eg https://www.reddit.com/r/sysadmin/comments/dyu69c/office365_outage/ https://www.reddit.com/r/sysadmin/comments/dyu69c/office365_...
  - mike503 7y ago
    And now it’s going to be Microsoft 365!
- deleted 7y ago
  [deleted]
Havoc 7y ago
Doesn't seem like a particularly balanced article. Plus some downright bizarre statements. GCP is "not big enough to be stress tested on the same level."??? Even smaller operations are subject to stress proportionately and I'm using a hell of a lot of google hangouts conferences...
- smt88 7y ago
  Do major Google services even run on GCP? My understanding was that they don't...
  - theginger 7y ago
    It is more that GCP runs on google's infrastructure, i think you can think of GCP as customer friendly layer on what they are using internally.
    - rstupek 7y ago
      Is that really the case though? I've never seen any discussion on how GCP is architected and distributed on the google network. Does a GCP server run next to one serving up google.com results?
  - saberience 7y ago
    No, this is why GCP is fairly crap when comparing to AWS. Google doesn't actually use GCP for building anything so they have less incentive to actually make it better, that or it means they don't understand the issues with it. At least with AWS you know that Amazon dog-foods the hell out of it and have a major incentive to improve it.
    - DangitBobby 7y ago
      In what way is GCP crap?
  - Havoc 7y ago
    I don't think at that scale it's realistic to think of one "running on" the other - for either google or MS.
purple-again 7y ago
Eh. A legion of non remote, non technology, companies were forced to quickly thrust themselves into the web. Want to bet how many of those companies were already running as a Microsoft shop? Nearly 100%. I would love to see details on the total requests placed on AWS and Azure during the time frame of COVID. My instinct tells me that Azure got hammered driven by those microsoft shops scrambling to get into the cloud while AWS had only a small increase, driven by technology companies (a small portion of companies) where the devs have final say and prefer AWS.
- outworlder 7y ago
  > where the devs have final say Highlighting the important part.
  - cc81 7y ago
    If devs are the ones generating revenue for the company. Not all companies are software companies.
- tw04 7y ago
  100% echo this statement. I have dozens of customers who went to VDI in Azure as their immediate work from home solution. I can't think of a single one that went to AWS. The article makes a bunch of baseless claims to get to an unproven conclusion. >It’s safe to say that AWS runs a much bigger, if not more critical, chunk of the digital world than Azure. Based on?? If o365 goes down for any extended period of time, most of the fortune 100 can't do business. > it generally lacks operational experience in running “always on” services and preparing for unanticipated traffic surges. Ya... Microsoft lacks experience running always-on services... they'd only been managing hotmail for a decade before AWS was even in beta. They have no experience running always-on services AT ALL.
  - mastry 7y ago
    I have seen the same. One of our clients stood up 25,000 Azure VMs (Win 10) in a single day. No problems so far.
blntechie 7y ago
Very subjective article trying to pass off as an objective report.
tinyhouse 7y ago
One thing that probably helps AWS is that they have experience preparing for peak usage given Amazon's peak traffic (and others) on some events during the year, such as black Friday, Prime Day, major holidays, etc.
- scarface74 7y ago
  Please don’t start down the false road that AWS came out of excess capacity that Amazon had. You didn’t state that explicitly, just making sure that no one rose thinks that.
sgt101 7y ago
"It’s safe to say that AWS runs a much bigger, if not more critical, chunk of the digital world than Azure" noooo... it isn't. It's just that the people we know use AWS. Most people use office, sharepoint, outlook and exchange - especially exchange.
- scarface74 7y ago
  AWS even admits that only 5% of enterprise IT spend is in the cloud.
  - moooo99 7y ago
    I think its safe to say that more consumer services are using AWS than Azure, things like Spotify, Netflix, Snapchat, etc. But when it comes to large enterprises, I'd bet that Azure is first place by a significant margin. Given that these enterprises power a significant chunk of the worlds supply chains, manufacturing processes and so on, I'd say that the piece of the digital world that Azure offers is way more critical.
    - kazen44 7y ago
      Also, don't understimate the amount of old, unsupported or ancient hardware that is still churning along in factories and supply companies keeping the supply chain running. It's terrifying how much of the world is held together by (literal sometimes) duct tape.
    - scarface74 7y ago
      Amazon always brags that they have twice as many Windows VMs running as Azure. I’m by no means a Windows hater. I’ve never developed on anything professionally besides a Windows computer - these days .Net Core, Python and Node. But once I started working with AWS, I avoided deploying to Windows servers like the plague. Anytime you add Windows to the mix you basically more than double your cost between Windows licenses and the increased hardware required over running Linux. All that to say, no one chooses to use Windows on AWS. So if there are that many Windows instances on AWS, they are more than likely old enterprise workloads.
curiousgal 7y ago
The entirety of French universities are using Microsoft Teams for their online classes, that's a very serious workload. Previously almost no one used that tool.
gigel82 7y ago
Where exactly are the "cracks" of which the author speaks? There are some links in the article pointing to generic MSDN help articles but I'm not seeing this claim substantiated by data (reports of downtime for example). No one is saying online services and cloud providers aren't affected by the surge in traffic but I'm not seeing where the cracks are, so I'm calling the title BS (or clickbait at best). Hate on M$ all you want, the worlds' enterprises run on their software (and on their cloud).
naringas 7y ago
Seems to me like microsoft probably: underestimated their customer's potential demand and/or oversold their cloud capacity.
dkhenry 7y ago
I have had a Service Down Ticket open with Azure now for four days with out as much as a call back from them. Just today I finally got the automated email saying they acknowledge my issue. My system is totally dead in the water on Azure. On the other hand amazon just called me to make sure everything was ok. Their system has been a rock.
afpx 7y ago
Except for EMR
- contol-m 7y ago
  Could you explain further?
  - afpx 7y ago
    When we started using EMR, we'd never exhaust the number of available instances (in both spot and on-demand). Now, we frequently do. And, for the last 3 weeks, it's been close to impossible to launch a huge cluster.
xkgt 7y ago
MS has more user facing SaaS offerings than Amazon, so both are not impacted the same way due to the lock down. What is the AWS equivalent of Teams & Office 365? Chime and work docs? The comparison is apples to oranges.

155 points by ceohockey60 7y ago

luhn 7y ago

> One misleading headline that’s been pushed out by Microsoft and unfortunately picked up by various tech media outlets, is the “775% increase” in Azure cloud service usage in geographical areas that are most committed to some form of social distancing or shelter-in-place policy. It's way worse than that. The original statement: > We have seen a 775 percent increase of our cloud services in regions that have enforced social distancing or shelter in place orders The updated statement: > We have seen a 775 percent increase in Teams' calling and meeting monthly users in a one month period in Italy, where social distancing or shelter in place orders have been enforced. Somehow an increase in usage of a single feature of a single product in a single country was miscommunicated as an increase in demand for all of Azure's cloud services. Microsoft's PR team royally screwed up on this one. https://azure.microsoft.com/en-us/blog/update-2-on-microsoft-cloud-services-continuity/ https://azure.microsoft.com/en-us/blog/update-2-on-microsoft...

NicoJuicy 7y ago
This "single feature" is video-conference and is used by almost every Office 365 business customer. In chat propositions, it's Slacks biggest competitor. There is no feature bigger in bandwidth than this one. I believe the numbers are similar in Europe. By reference, I think it's similar as having a Netflix business appear from nowhere.
- tw04 7y ago
  >By reference, I think it's similar as having a Netflix business appear from nowhere. Only without all the edge caching devices Netflix has had a decade to deploy.
- JeremyNT 7y ago
  But is this a feature many actually used before? If it was used e.e. 10 hours prior and now 7.75 hours, it's insignificant. A percent increase is a meaningless metric without the baseline. If it's "all azure" then we immediately know it's a huge increase! But if it's just a possibly seldom used feature it's minimal.

samspenc 7y ago

> limited to places where shelter-in-place is in full force, which at least in the United States, is only a small part of the country with densely populated urban areas: New York, California, New Jersey, Michigan, etc. Ummm, it is NOT a "small part" of the country that is under "shelter-in-place". By geographic area, maybe, but California, New York and New Jersey alone include some of the largest cities by population and population density. And there are now several other states with large populations (Illinois, Florida, Washington, for example) not listed here that have shelter-in-place as well. UPDATE: I found an interactive map that shows which states have statewide shelter-in-place orders in the United States; it looks like all the states with the largest populations (and more than 50% of the country's total population) are covered, which is actually significantly larger than I had actually thought before I looked this up: https://www.nytimes.com/interactive/2020/us/coronavirus-stay-at-home-order.html https://www.nytimes.com/interactive/2020/us/coronavirus-stay...

madhadron 7y ago
By population, New York + California + New Jersey is 22% of the USA.
- altoidaltoid 7y ago
  and have 6% representation in the Senate...
  - rjbwork 7y ago
    Yes, that's how the Senate works.
    - MrZongle2 7y ago
      More specifically: every state has 2 US Senators, whereas Representatives in the US House of Representatives are allocated to states based upon population.
      - 7y ago

uji 7y ago

A better comparison would be between Azure and GCP, both of them provide office suite (SAAS) to customers. Google recently released its side of the numbers (https://cloud.google.com/blog/topics/inside-google-cloud/how-google-cloud-is-helping-during-covid-19 https://cloud.google.com/blog/topics/inside-google-cloud/how...)

gallego2007 7y ago
From the link, Google has seen the following for Meet usage: “Over the last few weeks, Meet’s day-over-day growth surpassed 60%, and as a result, its daily usage is more than 25 times what it was in January. Despite this growth, the demand has been well within the bounds of our network’s ability.”
all_blue_chucks 7y ago
AWS also provides office suite to customers: WorkMail, WorkDocs, Chime messaging, etc.
- reallydontask 7y ago
  Are they even in the same league as O365 and G-suite, usage wise? Only asking because I'd never heard of these services before
  - cc81 7y ago
    Are G-suite even in the same league as O365? I have no numbers and might be incredibly biased by location but I feel that all the "boring" office companies in Europe has moved towards Office 365 as a natural step. That is of course not true and I'm sure there are plenty of G-suite companies but for the large ones it seems very common that they have just followed Microsoft into the cloud.

godtoldmetodoit 7y ago

> Since Microsoft’s other businesses are either hardware or office software, with the exception of maybe XBox’s gaming, it generally lacks operational experience in running “always on” services and preparing for unanticipated traffic surges. Uh, as a former Microsoft equivalent of a SRE this is a utterly ridiculous statement. I was in the Windows division on a team handling all the incoming telemetry - we definitely knew how to prepare for traffic surges and keep our services "always on". Has the author never seen Office 365? Bing? The entirety of Azure in general that has dozens if not hundreds of publicly facing always on services?

jscholes 7y ago
Yeah, really weird statement for the author to make, particularly given that at the top of the very same section they list a bunch of cloud-based MS products.
sokoloff 7y ago
Office 365 was derisively referred to as “Office 360” with some good reason years ago. (I use it and the uptime has gotten noticeably better. That it could get noticeably better is the problem.)
- quaquaqua1 7y ago
  HAH this has been my experience as well. And the system often times out even when the main screen is "accessible".
- fulafel 7y ago
  I think this was less than a year ago. eg https://www.reddit.com/r/sysadmin/comments/dyu69c/office365_outage/ https://www.reddit.com/r/sysadmin/comments/dyu69c/office365_...
- mike503 7y ago
  And now it’s going to be Microsoft 365!
deleted 7y ago
[deleted]

Havoc 7y ago

Doesn't seem like a particularly balanced article. Plus some downright bizarre statements. GCP is "not big enough to be stress tested on the same level."??? Even smaller operations are subject to stress proportionately and I'm using a hell of a lot of google hangouts conferences...

smt88 7y ago
Do major Google services even run on GCP? My understanding was that they don't...
- theginger 7y ago
  It is more that GCP runs on google's infrastructure, i think you can think of GCP as customer friendly layer on what they are using internally.
  - rstupek 7y ago
    Is that really the case though? I've never seen any discussion on how GCP is architected and distributed on the google network. Does a GCP server run next to one serving up google.com results?
- saberience 7y ago
  No, this is why GCP is fairly crap when comparing to AWS. Google doesn't actually use GCP for building anything so they have less incentive to actually make it better, that or it means they don't understand the issues with it. At least with AWS you know that Amazon dog-foods the hell out of it and have a major incentive to improve it.
  - DangitBobby 7y ago
    In what way is GCP crap?
- Havoc 7y ago
  I don't think at that scale it's realistic to think of one "running on" the other - for either google or MS.

purple-again 7y ago

Eh. A legion of non remote, non technology, companies were forced to quickly thrust themselves into the web. Want to bet how many of those companies were already running as a Microsoft shop? Nearly 100%. I would love to see details on the total requests placed on AWS and Azure during the time frame of COVID. My instinct tells me that Azure got hammered driven by those microsoft shops scrambling to get into the cloud while AWS had only a small increase, driven by technology companies (a small portion of companies) where the devs have final say and prefer AWS.

outworlder 7y ago
> where the devs have final say Highlighting the important part.
- cc81 7y ago
  If devs are the ones generating revenue for the company. Not all companies are software companies.
tw04 7y ago
100% echo this statement. I have dozens of customers who went to VDI in Azure as their immediate work from home solution. I can't think of a single one that went to AWS. The article makes a bunch of baseless claims to get to an unproven conclusion. >It’s safe to say that AWS runs a much bigger, if not more critical, chunk of the digital world than Azure. Based on?? If o365 goes down for any extended period of time, most of the fortune 100 can't do business. > it generally lacks operational experience in running “always on” services and preparing for unanticipated traffic surges. Ya... Microsoft lacks experience running always-on services... they'd only been managing hotmail for a decade before AWS was even in beta. They have no experience running always-on services AT ALL.
- mastry 7y ago
  I have seen the same. One of our clients stood up 25,000 Azure VMs (Win 10) in a single day. No problems so far.

blntechie 7y ago

Very subjective article trying to pass off as an objective report.

tinyhouse 7y ago

One thing that probably helps AWS is that they have experience preparing for peak usage given Amazon's peak traffic (and others) on some events during the year, such as black Friday, Prime Day, major holidays, etc.

scarface74 7y ago
Please don’t start down the false road that AWS came out of excess capacity that Amazon had. You didn’t state that explicitly, just making sure that no one rose thinks that.

sgt101 7y ago

"It’s safe to say that AWS runs a much bigger, if not more critical, chunk of the digital world than Azure" noooo... it isn't. It's just that the people we know use AWS. Most people use office, sharepoint, outlook and exchange - especially exchange.

scarface74 7y ago
AWS even admits that only 5% of enterprise IT spend is in the cloud.
- moooo99 7y ago
  I think its safe to say that more consumer services are using AWS than Azure, things like Spotify, Netflix, Snapchat, etc. But when it comes to large enterprises, I'd bet that Azure is first place by a significant margin. Given that these enterprises power a significant chunk of the worlds supply chains, manufacturing processes and so on, I'd say that the piece of the digital world that Azure offers is way more critical.
  - kazen44 7y ago
    Also, don't understimate the amount of old, unsupported or ancient hardware that is still churning along in factories and supply companies keeping the supply chain running. It's terrifying how much of the world is held together by (literal sometimes) duct tape.
  - scarface74 7y ago
    Amazon always brags that they have twice as many Windows VMs running as Azure. I’m by no means a Windows hater. I’ve never developed on anything professionally besides a Windows computer - these days .Net Core, Python and Node. But once I started working with AWS, I avoided deploying to Windows servers like the plague. Anytime you add Windows to the mix you basically more than double your cost between Windows licenses and the increased hardware required over running Linux. All that to say, no one chooses to use Windows on AWS. So if there are that many Windows instances on AWS, they are more than likely old enterprise workloads.

curiousgal 7y ago

The entirety of French universities are using Microsoft Teams for their online classes, that's a very serious workload. Previously almost no one used that tool.

gigel82 7y ago

Where exactly are the "cracks" of which the author speaks? There are some links in the article pointing to generic MSDN help articles but I'm not seeing this claim substantiated by data (reports of downtime for example). No one is saying online services and cloud providers aren't affected by the surge in traffic but I'm not seeing where the cracks are, so I'm calling the title BS (or clickbait at best). Hate on M$ all you want, the worlds' enterprises run on their software (and on their cloud).

naringas 7y ago

Seems to me like microsoft probably: underestimated their customer's potential demand and/or oversold their cloud capacity.

dkhenry 7y ago

I have had a Service Down Ticket open with Azure now for four days with out as much as a call back from them. Just today I finally got the automated email saying they acknowledge my issue. My system is totally dead in the water on Azure. On the other hand amazon just called me to make sure everything was ok. Their system has been a rock.

afpx 7y ago

Except for EMR

contol-m 7y ago
Could you explain further?
- afpx 7y ago
  When we started using EMR, we'd never exhaust the number of available instances (in both spot and on-demand). Now, we frequently do. And, for the last 3 weeks, it's been close to impossible to launch a huge cluster.

xkgt 7y ago

MS has more user facing SaaS offerings than Amazon, so both are not impacted the same way due to the lock down. What is the AWS equivalent of Teams & Office 365? Chime and work docs? The comparison is apples to oranges.