4 ms·
Multiple Digital Ocean services down
- jsheard 1y agoThey don't name names but it's probably due to the ongoing Cloudflare explosion. I know the DigitalOcean Spaces CDN is just Cloudflare under the hood.
- matt-p 1y agoJust spaces CDN, not spaces - you'd think they'd just turn the CDN off for a bit.
- potato3732842 1y agoYou can't just "turn off CDN" on the modern internet. You'd instantly DDOS your customers' origins. They're not provisioned to handle it, and even if they were the size of the pipe going to them isn't. The modern internet is built around the expectation that everything is distributed via CDN. Some more "traditional" websites would probably be fine.
- matt-p 1y agoIf you are DO you could, you just decided not to bother. They control the origins it's spaces (s3), so they could absolutely spin up further gateways or a cache layer and then turn the CDN off.
- graemep 1y agoEither you are wrong and they do not have the capacity to do that, or they have decided it is acceptable to be down because a major provider is down I imagine a cache layer cannot be that easy to spin up - otherwise why would they outsource it?
- matt-p 1y agoYou outsource it because clouflare have more locations than you so offer lower latency and can offer it at a cost that's cheaper or the same price as doing it yourself.
- graemep 1y agoWhich suggests its expensive enough for it to be unlikely they just have the capacity lying around to spin up.
- oasisbob 1y agoTo the contrary, CDN pricing will usually beat cloud provider egress fees. Common example: you can absolutely serve static content from an S3 bucket worldwide without using a CDN. It will usually scale OK under load. However, you're going to pay more for egress and give your customers a worse experience. Capacity isn't the problem you're engineering around. For a site serving content at scale, a CDN is purpose-built to get content around the world efficiently. This is usually cheaper and faster than trying to do it yourself.
- graemep 1y agoThat is not what I said. I said DO will not have the spare capacity because its too expensive. Can you please tell me who DO pay egress fees to?
- matt-p 1y agoThey will be doing a mix of peering both across free PNIs and very low cost IXP ports, with the reminder going down transit like Colt or cogent. Probably average cost of the order of about $1 per 20TB of egress in Europe and NA markets. The thing is with edge capacity is that you massively overbuild on the basis that; It's generally a long ISH lead time to scale capacity (days not minutes or hours). Transit contracts are usually 12-60 months Traffic is variable, not all ports cover all destinations Redundancy So if you are doing say 100Gbps 95%ile out of say London then you will probably have at least 6+ 100Gb ports, so you do have quite a bit of latent capacity if you just need it for a few hours.
- oasisbob 1y agoMight be just me, but I can think of many origins under my control which could live without a (non-functional) CDN for a while. CDN is great for peak-load, latency reductions, and cost - but not all sites depend on it for scale 24/7
- tgma 1y agonit: that's more DoS (from a handful of DO LBs) than DDoS.
- TechRemarker 1y agoYes all sites showing the CloudFlare error due to the massive outage. Seems their outages are getting more frequent and taking down the internet in new ways each time.
- archerx 1y agoI have two projects on DO using droplets and they are still running fine.
- hshdhdhj4444 1y agoIt seems mostly a CludFlare related issue. My DOs are working fine as well.
- sgc 1y agoAre you using their "reserved IPs"? I was thinking of starting to use them, but now I wonder if it is part of their load balancing stack under the hood.
- soheilpro 1y agoDroplets are fine. > This incident affects: API, App Platform (Global), Load Balancers (Global), and Spaces (Global).
- giancarlostoro 1y agoSo yesterday Azure got hit hard, today CF and DO are down, bad week or something else?
- zx8080 1y agoYear-end promotion cycle is the worst time for end-users and the best one for engineers greedy for promotions.
- Lammy 1y agoDon't blame individual engineers who want to do what will be rewarded instead of company performance policies that reward this type of behavior.
- fridder 1y agoshoot, there are also end of year layoffs and reorgs to pump up those year end numbers
- red-iron-pine 1y agowhat engineers, mate? they AI now and they're doing just spectacular
- BubbleRings 1y agoI would also like to know people’s opinion on this.
- matt-p 1y agoDDOS, but I don't really understand why in particular.
- giancarlostoro 1y agoHaving known people like this, its either flexing about who has the more powerful botnet or advertising who can do what.
- 1y ago
- showerst 1y agoI use DO's load balancers in a couple of projects, and they don't list Cloudflare as an upstream dependency anywhere that I've seen. It's so frustrating to think you're clear of a service then find out that you're actually in their blast radius too through no fault of your own.
- wesammikhail 1y agoslight off topic: I used DO LBs for a little while but found myself moving away from that toward a small droplet with haproxy or nginx setup. Worked much better for me personally!
- showerst 1y agoThe point of an LB for these projects is to get away from a single point of failure, and I find configuring HA and setting up the networking and everything to be a pain point. These are all low-traffic projects so it's more cost effective to just throw on the smallest LB than spend the time setting it up myself.
- grayhatter 1y agoIf they are small projects, why are they behind a load balancer to begin with?
- nickmonad 1y agoUsually because of SSL termination. It's generally "easier" to just let DO manage getting the cert installed. Of course, there are tradeoffs.
- showerst 1y agoI use the LB's for high availability rather than needing load balancing. The LB + 2 web back-ends + Managed DB means a project is resilient to a single server failing, for relatively low devops effort and around $75/mo.
- drob518 1y agoDominos falling into dominos falling into dominos…
- aforty 1y agoCloudflare outage.
- igtztorrero 1y agoI knew it, DigitalOcean CDN is using Cloudflare behind the scenes. Why DO ?
- mrkramer 1y agoWho is next?
- red-iron-pine 1y agomy guesses would be look at who has a FedRAMP capable service first. maybe also GCP, hetzner, akamai
- foxyv 1y agoMan, it really seems like the cloud providers are having some tough times lately. Azure, AWS, and Cloudflare! Is everything just secretly AWS?