6 ms·
Microsoft’s Azure portal down following new claims of DDoS attacks
- tatersolid 3y agoCan confirm the impact appears total for the Azure portal; Microsoft sells DDoS protection as part of Front Door and other Azure services. Do they not use it themselves?
- donutshop 3y agoAm wondering the same thing. I have yet to meet anybody that does aside from Microsoft themselves.
- tatersolid 3y agoWe used Front Door for a while for our SaaS application, but actually had to turn it off because of a large customer whose janky internal IDP broke because of the mere presence of IPv6 on the Azure Front Door CNAME records. It worked well for about a week for everyone else.
- nullindividual 3y agoI use it for a number of web properties.
- nwah1 3y agoI worked at a commodities brokerage that used it. No problems.
- nullindividual 3y agoYou can tell the error on the Portal is coming from AFD thanks to the style/format of it.
- re-thc 3y ago> Microsoft sells DDoS protection as part of Front Door They got in via a Back Door. A lot of places sell DDoS protection. Most of it only ever cover the basics. With the size of botnets, hacked computers, etc that are easily and cheaply available for rent - a lot of so called DDoS protection services can't compete. You really need to dig deep into what they mean by DDoS protection. What layers are covered? What type of services are covered? Also e.g. when they mention they cover volumetric attacks, it's often marketing more than the real deal. E.g. a provider sells $x of total protection (that number means across all PoPs, so usually $x / 30 or less). It can still go down if you focus all your attacks on a few PoPs that matter.
- MrOwnPut 3y agoIt's also important whether it's active or passive. Really important for game servers to have active DDoS protection, like what OVH offers. Not one that kicks in after the attack is detected, otherwise your players get booted, which is the goal of the attacker.
- re-thc 3y ago> Not one that kicks in after the attack is detected You can't even detect the attack. I've seen 1s that are well disguised or the patterns are so short there's not enough data to tell if it's an attack or real traffic. And yet it's enough to crash your connection or the service itself.
- stevefan1999 3y agoAlso remember that Spamhaus attack shows upstream providers can also be the crack in the armor, too. They could've just knock out a whole regional network access given the huge volume of traffic.
- ojintoad 3y agoIt's very odd because it's only new page loads. Existing metrics pages I've been refreshing keep showing new data.
- tatersolid 3y agoWe see the same thing. One of my team is sailing around in the portal, even opening new tabs and it all is just working. But I can't create a new session, even when remoted into an Azure-hosted VM. All our servers and services seem to be running fine; it just seems to be the portal.azure.com website that is impacted.
- mynameisvlad 3y agoThat would be hitting separate API endpoints which are not hosted in the same place as the portal itself.
- Izikiel43 3y agoProbably because the actual portal service call happens at new page load. Metric pages and everything else are data plane/control plane info most likely, which is coming from other services. An easy way to validate this would be using fidler or similar to analyze the traffic that happens in the loaded page.
- mixdup 3y agoThis is exactly the case. API calls/cli tools never went down because they tickle the control plane APIs directly, and the Portal also tickles those APIs directly. The Portal seems to just be a client-side web app, and the CDN serving that and what ever minimal server-side infra that makes login work, etc was what was targeted by the ddos
- error9348 3y agoInterestingly, https://preview.portal.azure.com/ https://preview.portal.azure.com/ is up.
- AtNightWeCode 3y agoI don't really understand how a DDoS can happen when the portal is using MS routing. Should be a simple fix. More likely some misconfiguration.
- MuffinFlavored 3y agoi wonder how much traffic it would take to DDoS Azure's Web UI. You're talking about like 1,000,000+ orchestrated/coordinated silently hijacked machines all... what? curl'ing in a loop some API of theirs?
- re-thc 3y agoMost people over-estimate what DDoS protection provides. There are all sorts of tricky attacks. The APIs are now so complex there are lots of layers - load balancers, gateways, kubernetes, etc etc. The attackers can exploit any amount of them until they find it. It could be a bug e.g. integer overflow on a specific header or the lower TCP layers, etc. It can also be very tricky to detect and stop. You may imagine the attacks to be constant but they are not. They do enough to deny service but not enough to be detected. It makes it very hard to differentiate from real traffic.
- AtNightWeCode 3y agoOne problem could be JWT. Anybody can use the bearer tokens from the portal and then use K6 to flood any endpoint. If you are willing to pay for the attack which states often do.
- MuffinFlavored 3y agoWhat's K6 in this context?
- tejado 3y agoLikely Grafana K6: “ A modern load testing tool, using Go and JavaScript - https://k6.io https://k6.io “
- soco 3y agoI'm working on it right now, 45 minutes after the submission. Maybe not everywhere? Maybe fixed already?
- jakedata 3y agoFlawless instant access for me. Clearly not a total impact. Didn't even notice until I saw the headline.
- NicoJuicy 3y agoIt was fixed an hour ago :)
- AtNightWeCode 3y agoYou are too late to the party. The problem was between 15:10 UTC and 17:30 UTC on 9 Jun 2023.