18 ms·
Microsoft: Cloud services demand up, prioritization rules in place
- gundmc 7y agoWhy would Microsoft be disproportionately affected by this? Are we expecting similar decrees from AWS and GCP? Or was Microsoft operating with less runway before this began?
- rightbyte 7y ago"Rampant" work from home with Office 365 I guess. I guess most of the load is all these Team video calls and VPN services?
- daxfohl 7y agoYeah as a member of Azure networking, it's odd: HR is communicating we should do video calls for meetings instead of just audio, to keep the personal touch, and I'm wondering if that's really the best use of our bandwidth. I keep my video off anyway.
- rightbyte 7y agoImagine thousand and thousand video calls with multiple receivers for each transmitter ... over VPN. I don't use video since I feel weird doing it. Also I don't want to do my hair and change from pyjamas.
- simplezeal 7y agoCompromise is to turn on video only when talking. Also works as #raisehand ..
- jedieaston 7y agoOffice 365 runs on Azure in the same regions other workloads do (There's probably some replication going on behind the scenes too, since you don't pick your instance), so I'd bet that tons of new Office 365 subscriptions in the past month are what is causing this, not people deciding to lift-and-shift their app workloads because of the virus. Teams adoption has probably shot way up too.
- maxerickson 7y agoUsage went up 40% last week: https://www.theverge.com/2020/3/19/21186452/microsoft-teams-new-features-noise-supression-user-increase-coronavirus https://www.theverge.com/2020/3/19/21186452/microsoft-teams-...
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- breakingcups 7y agoI imagine companies are flocking to Microsoft's virtual desktop solutions more than AWS or Google. It is the most well-known provider among managers who need to make a snap decision right now on how to have people work from home, if I had to guess.
- salex89 7y agoWorking with all three of them, so here's my two cents: 1. Mostly traditional, "legacy" companies have been hit hard by this. Ones that don't have culture or technology of work-from-home. Those companies use some Microsoft products. Also, Microsoft has been poaching them heavily, handing out trials, bundling licenses and so on. A lot of them don't actually buy stuff from Microsoft, but through 3rd party vendors which have incentives of their own. Some of the end users don't actually want to use AWS, also. 2. I actually think Microsoft has much less runway. From what I understand, AWS has more modern infrastructure and backend, and they shuffle resources easily around, between services, and I think they have much more in reserve. Microsoft has concentrated much more on the sheer number of regions. 3. Azure has a strange way of handling quotas, if you ask me. Up until now, once you provision a VM, it is deducted from a quota and stays like that as long as it exists. It has never been an issue to actually power it on (unlike AWS), once you have it. It's not billed, but we always thought it stays like that. Since last week, you can see failures not only when provisioning VM's (even within your quota) but also when starting them. Nevertheless, I also think a lot of users had larger quotas allocated then they actually use. So they just started creating more VMs or other resources (because they could), and the thing came crashing. I think that's just poor planning on Microsoft's side. But the thing I'm mostly pissed of is the status page. VM's are failing left, right and center and everything is nice and green on the status page. Once you open a ticket, they send you an incident-in-progress report.
- pwarner 7y agoI think Azure also has more regions than AWS, but yet a smaller overall capacity probably, so it's gotta be harder to keep space capacity. It's notable the shortages are isolated to specific regions. They were possibly small to start and so it probably didn't take as much to hit capacity there? Just a guess.
- salex89 7y agoThat's my hunch also... Speaking for someone interested in IaaS/VDI, not all instances are available in all regions. For example, instances with GPUs are available only in North and West Europe. If you need them, and they are out of stock, you have no value of any UK or France region.
- hn_throwaway_99 7y agoA huge portion of this has got to be MS's own services, e.g. tons of people who normally use Office and are at work are now use Office 365 online and doing a ton more videoconferencing with Teams.
- onetimemanytime 7y ago>>Why would Microsoft be disproportionately affected by this? Nobody gets fired for buying Microsoft..er IBM. Inertia too.
- dehrmann 7y agoIf you're shopping for cloud providers, nobody gets fired for picking AWS. Microsoft is if you're looking for email, spreadsheets, and desktop publishing.
- dodobirdlord 7y agoUnless you're a vendor for Walmart.
- empath75 7y agoA significant percent of Microsoft’s ‘cloud’ division is office and sharepoint and the like. I’m sure thatusage is through the roof right now.
- RustyBucket 7y agoSharepoint Online runs on separate infrastructure from Azure in their own data centers.
- deleted 7y ago[deleted]
- ma2rten 7y agoMy theory is that Amazon and Google are more prepared to handle surges because they run some of the largest web properties. This like every day is Black Friday for Amazon. It's rough, but not something they have never dealt with before.
- leesalminen 7y agoGCP and GSuite had outages this week listed on their status pages. AWS has been rock solid for us in the 3 regions we operate in.
- dodobirdlord 7y agoThe GCP outages don't seem to have been capacity related though, they were both networking issues. In one of them they broke their traffic routing to a datacenter in Atlanta, and in the other one an unnamed cloud provider (that pretty much has to be AWS) broke inbound inter-cloud traffic in us-east1 (so presumably AWS borked the link from Google's us-east1 to AWS us-east-1). These seem like normal sorts of cloud issues that happen to coincide with covid-19.
- pjscott 7y agoIf anybody at AWS is reading this: congratulations! Thank you! Keeping infrastructure running is hard in the best of times, and these times definitely are not the best.
- Craighead 7y agoFind the team outages that have been happening for weeks on any public tracker
- bmurphy1976 7y agoWe've anecdotally seen a huge increase in spot market flakiness, but basic functionality and reserved instances have been solid.
- traskjd 7y ago"This like every day is Black Friday for Amazon." So, a pretty big deal then: https://www.theverge.com/2018/7/16/17577654/amazon-prime-day-website-down-deals-service-disruption https://www.theverge.com/2018/7/16/17577654/amazon-prime-day...
- Maakuth 7y agoAll Office 365 company customers were also dormant Teams customers. There's a lot of these companies and some non-trivial portion of those have now become active Teams users. That probably drives a big portion of the increased conference traffic at least.
- Kneecaps07 7y agoBecause a lot of people have their Office 365 subscription and maybe just use email. All of a sudden they realized they are paying for all this other stuff they aren't using so they're leveraging it now that they need it.
- Vrondi 7y agoA lot of companies and educational institutions already had Teams licenses through their Office 365 contracts, but weren't really using Teams. Now they're using it, a lot, because "it's already paid for". Simple as that.
- crazygringo 7y agoWow. First of all, I'm curious to what proportions this is driven primarily by remote office work (O365), videoconferencing streaming, recreational video streaming (does Disney+ run on Azure?), or what. But second... I'm fascinated by the concept of prioritization rules in place rather than simply raising prices. I wonder if it "looks bad" to raise prices, or if the vast majority of customers already have locked-in prices contractually so that raising prices has little effect. But I'd always thought that with AWS's spot pricing and so forth, that auction-style dynamic pricing was a core feature of clouds.
- deleted 7y ago[deleted]
- stefan_ 7y agoI don't think 1) locking in customers via proprietary APIs 2) limiting supply of resources 3) auctioning off to the highest bidder among your locked in customers was entirely the core feature of clouds. Rather, the idea was "we'll charge so much margin you never need to worry about anything physical, another machine is just an API call away". You don't need to overprovision for peak times and there is less physical stuff on your balance sheet, ez cash flow, win win! Of course now everyone realizes it's a big sham and neither Amazon nor Microsoft are stupid enough to spend a trillion capex on building the datacenters, fiber lines and so on you need to handle a temporary 700% pandemic situation. In a year from now, they don't want that stuff on their quarterly report just as companies going "cloud only" don't want it.
- bobthepanda 7y agoTo be fair, a situation where every single company in the world needed to ramp up on the cloud at the same time is completely unprecedented, and spending the money to do it would've been considered a fools' errand. If people weren't on the cloud, you'd see them scrambling for physical hardware, which also isn't exactly easy to come by on extremely short notice when the world's logistic chains are under stress as well.
- deleted 7y ago
- hestefisk 7y agoTime to buy MS stock.
- gruez 7y agopriced in
- jbverschoor 7y agoNo it's not.. The market is not sane atm. It's 100% emotion.
- __blockcipher__ 7y agoWhat makes you think that? Have you seen a discrepancy between your projections of future cash flows and therefore your valuation, as compared to the value the market is at right now?
- jbverschoor 7y agoThey’re mainly a software / services company. There’s no reason their revenue will drop, the contrary.. Yet the stock price has the exact same changes as the rest of the entire market. Their cloud demand is up x7.. it’s not priced in. Amazon same thing, although less so imho Their current p/e is 26. Remove cash and it’s nearing 20.
- deleted 7y ago[deleted]
- dumbfoundded 7y agoI think it's safe to say the market isn't rational when Zoom Technologies is up nearly 900% when it's completely unrelated to Zoom Video and not actually operating. There's a different HN post about it: https://www.nasdaq.com/articles/the-sec-really-wants-investors-to-stop-buying-the-wrong-zoom-stock-2020-03-27 https://www.nasdaq.com/articles/the-sec-really-wants-investo...
- Mountain_Skies 7y agoThe difficult question is how quickly can capacity be increased and given uncertainty about how long this demand increase will last, does it make financial sense to do so?
- _sword 7y agoSo that's why my OneDrive sync hasn't been working at all.
- greatgib 7y agoGood time to remind that 'the cloud is just some else computer' ...
- lkbm 7y agoThe cloud is someone else's collection of computers operating at a very large scale. If my load changes in a way uncorrelated with other users', that's a very good thing. If my load is correlated with other users', that's may be bad. But it could still be good, because at that scale, they can justify having robust contingency plans -- back-up supply contracts, extra engineers around to ramp stuff up, or relatively unimportant stuff they can de-prioritize. (If you're relatively unimportant, that's a bummer; if you're a hospital getting priority, that's a plus.)
- johannes1234321 7y agoWhile true: only few individual companies would have enough computing power to handle their suddenly increasing load ready ... or capability to setup the services now in demand
- raghavtoshniwal 7y agoIt would be interesting to know how many standard deviations is 775% extra demand from the norm. More than six standard deviations is rarer than a 1 in 506797346 event.
- Simon_says 7y agoSamples getting 6 standard deviations away from the mean can happen 1/6² ≈ 2.7% of the time. You might be assuming properties of the distribution that aren't true. https://en.wikipedia.org/wiki/Chebyshev%27s_inequality https://en.wikipedia.org/wiki/Chebyshev%27s_inequality
- yread 7y agoIsn't that an upper bound? It says that it happens at most 2.7% of the time
- Simon_says 7y agoYea, it's an upper bound. The actual percentage depends on the distribution and can be anywhere from 0-2.7%. The Wikipedia link has more details.
- techntoke 7y agoLeave it up to Microsoft to find out ways to profit more during a pandemic, rather than open sourcing their software and platform to be used anywhere and everywhere. People forgot what hosting is like without these massively overpriced cloud services that nickel and dime for every feature. Want encryption which basically costs nothing? Oh, you'll pay an extra $0.10 per hour on top of your $.08GB of outbound traffic. God forbid you load balance your service.
- jedieaston 7y agoOffice (/Microsoft) 365 is a pretty good value for what it is. It isn’t free, but $20 per user per month for Windows + Office + Exchange + OneDrive + Azure AD and support is pretty good. Microsoft saying “Ahh! Crisis! Here’s the Windows Server codebase!” wouldn’t help, since you’d have to still find a host and time to make sure your code works on this upstream variant without all of the licensed DLLs that makes Windows work.
- techntoke 7y agoGoogle offers the same for less than $12 a month with unlimited storage for large companies and $12 a month for everyone else. Plus you get an operating system that is based off Linux and open source. Windows is a giant PITA when it comes to deploying and setting up mobile device management. Matrix offers free decentralized video/voice communication and there are lots of self-hosted alternatives that often provide better features than things like Teams. With NextCloud you basically get a free office solution that just needs to be deployed. With Kubernetes you can quickly deploy a Nextcloud cluster. These services should be free and open source because the goal should be helping build a better world, not building better profits through backroom deals with tax dollars.
- freeone3000 7y agoYou've just named and listed the services. Odd why people would choose Office365 instead of hosting their own nextcloud distribution cluster in their own colocation space.
- 7y ago
- dabeeeenster 7y agoHonest question; how do you define "demand"?
- KennethSRoberts 7y agoMaybe start by clicking the link? > In a March 28 blog post, officials said that demand for its new Windows Virtual Desktop usage has grown by more than three times. They also said government use of public Power BI for sharing COVID-19 dashboads is up 42 percent in a week. (As is the case with Microsoft's overall cloud services figure, we don't have a base number for WVD and Power BI from which to calculate these percentages.)
- dabeeeenster 7y agoIt still doesnt define demand. Is that new sales or what?
- 9nGQluzmnq3M 7y agoUtilization.
- dabeeeenster 7y agoTheir utilization has gone up 700%? Bullshit.
- swrj 7y ago>"We have seen a 775 percent increase of our cloud services in regions that have enforced social distancing or shelter in place orders." Seems to be a cherry picked statistic, not an overall increase in utilization.
- 9nGQluzmnq3M 7y agoNot for Azure as a whole, no, but that sounds entirely plausible for specific services like Teams.
- oneplane 7y agoI wonder if this is because a lot of companies suddenly have to make a change they didn't feel necessary in the past and now rush to compensate for a lack of planning. While it technically doesn't matter where you run, there are a lot of choices that have a different answer depending on if you ask your vendor or if you do your own checks and research. Scaling purely because you needed to scale up would be a different story, and then you're expect overall service demands to go up that steep. This seems more like an unpreparedness peak to me.
- williesleg 7y agoGotta get me some MSFT stock.
- gameswithgo 7y agovideo conferencing is way up online food ordering is way up buying stuff online in general is up i imagine all video streaming and gaming is up
- spencerwgreene 7y agoThe Microsoft blog post that the article is using as a source: https://azure.microsoft.com/en-us/blog/update-2-on-microsoft-cloud-services-continuity/ https://azure.microsoft.com/en-us/blog/update-2-on-microsoft....
- enlyth 7y agoAfter reading this, it seems the headline is a bit misleading. The Microsoft blog post states: "We have seen a 775 percent increase of our cloud services in regions that have enforced social distancing or shelter in place orders." So it's not an overall increase, just in certain regions?
- loeg 7y agoA lot of the value of hyperscaler clouds is the lowish latency local datacenters. Also, everywhere's going to have enforced S.D. or S.I.P. sooner or later.
- ineedasername 7y agoYes, and they also say that if you're hitting quotas, you can try another less congested region. If you have established workloads in a region though, I'm not sure how quickly you can migrate a complex interplay of services to another region-- are there simple tools to do so in Azure?
- bowmessage 7y agohopefully you are writing infrastructure as code - would be easy then.
- wutwutwutwut 7y agoFor toy projects, sure.
- mr_toad 7y agoI doubt they’re talking about their compute services. The increases are more likely in Microsoft teams and Office 365. And as far as I know you can’t choose a region for those services.
- 867-5309 7y agoI read and executed but did not write that percentage
- smcleod 7y agoOffice365 and AzureAD has been even more unreliable than usual over the past week or so, plenty of "an error has occurred" pages and delayed mail - as usually Microsoft didn't update the status dashboards (the ones you can see without being able to log in anyway), The other client I work with uses GCP and Google Mail/Docs for business which I haven't noticed any issues with.
- leesalminen 7y agoGSuite and GCP had impactful outages for us this past week and were listed on the status dashboards.
- smcleod 7y agoWe might have been lucky with the region and timezone we're in - australia-southeast1
- what_ever 7y agoLooks like you are repeating the same comment everywhere without linking at what kind of outages GCP had. Disc: Googler
- kekskeks 7y agohttps://www.ciodive.com/news/google-cloud-platform-outage/575047/ https://www.ciodive.com/news/google-cloud-platform-outage/57...
- what_ever 7y agoYup, that was my point but I failed to clarify in the first comment.
- yjftsjthsd-h 7y agoIt made the front page, so most readers here are already at least passingly aware of it.
- ineedasername 7y agoHow much excess capacity would a cloud provider typically have (as in %) to accommodate normal peak loads and still have a comfortable margin? Or maybe that's not something providers reveal, so let me as the sys-admins of this thread: For on-premise resources, what is your % margin of excess capacity reserved?
- thanksforfish 7y agoFor specific ec2 instance types the excess capacity is sometimes zero, which customers sometimes see. I've needed to swap instant types to get a deploy unstuck or to scale up in the past when capacity didn't exist. Because of that I always recommend that a project reserve instances to ensure capacity exists when you need it. It's also cheaper, but that risk is important to mitigate.
- rsync 7y ago"How much excess capacity would a cloud provider typically have (as in %) to accommodate normal peak loads and still have a comfortable margin?" We (rsync.net) run a sort-of just-in-time deployment model wherein as our storage arrays approach 80% capacity, we completely remove-and-replace the oldest storage array at a location[1]. The 80% number is related to performance penalties that become extant when you fill up zpools too full. In the old days, that is the threshold beyond which we would see significant performance degradation. ZFS behaves better now and you can avoid a fair amount of that degradation by having read and write cache devices (L2ARC and SLOG) but we still hang onto that 80% number ... When we do such a replacement, we are often replacing a 2-3 year old storage array with a brand new one, which means that the drive capacity of the refreshed system is about double what it was before - so we have a relative glut of space for a time. This allows us to onboard larger (50-100 TB) clients with no lead-time. FWIW, we have seen a relative increase in signups since lockdowns began. My current theory is that a lot of people are sitting at home with a lot of free time and thinking about risks ... and some of them think about backups. [1] Denver, San Diego, Fremont, Zurich, and Hong Kong.
- justlexi93 7y agoPeople may be quick to complain, but for an infrastructure the size of Microsoft's to support a sudden 775% increase in demand, you've got to give it to them as that is incredible.
- metreo 7y agoAnd 77.5% of the new demand turns out is from the Microsoft employees sent home :)
- holri 7y agoThere is no cloud, only other people computers. (c) FSFE
- MUHAMMADRIZWAN 7y agohttps://bit.ly/3dH1Dxu https://bit.ly/3dH1Dxu