4 ms·
Can we at some point acknowledge that constant cloud disruptions are too costly, and can we then finally move all of our hosting back on-prem?
by lpcvoid 11mo ago
Can we at some point acknowledge that constant cloud disruptions are too costly, and can we then finally move all of our hosting back on-prem?
- donglong 11mo agocan you define "constant"
- lpcvoid 11mo agoWell, between AWS US EAST 1 killing half the internet, and this incident, not even a month passed. Meanwhile, my physical servers don't care and happily serve many people at a cheaper cost than any cloud offer.
- alt227 11mo agoWe had an Azure outage in between those 2 as well.
- chistev 11mo agoHow do you back up?
- lpcvoid 11mo agoWe have a few colocated servers offsite, each in a different region, each with a zpool of mirrored spinning rust. We use rsync across those at different times.
- donglong 11mo agonever build on us-east-1, everyone knows that ;)
- vlovich123 11mo agoYou realize these are two different companies right? If you’re saying “I’m an AWS customer with cloudflare in front” I think you’ve failed to realize that two 99.9% available services in series have a combined availability of ~99.8% - that’s just math. Your physical servers should have similar issues if you put a CDN in front unless the physical server is able to achieve a 100% uptime (100% * 3 9s = 3 9s). Or you don’t have a CDN but can be trivially knocked offline by the tiniest botnet (or even hitting hacker news front page)
- lpcvoid 11mo agoI do. But I put both into the "cloud offering off-prem for very much money" shoebox. I setup a CDN once using VPS from different hosting providers for under 100 USD a month, which I would vastly prefer over trusting anything cloud. And yes, I know that there's sites that need the scale of an operation like Cloudflare or AWS. But 99.9(...)% of pages don't, and people should start realizing that.
- mallets 11mo agoPeople who don't need that, also don't care much for an hour or two of service disruption. Most users will have far worse disruptions with the alternatives.
- darkwater 11mo agoFunnily and ironically enough, I was trying to check out a few things on Ansible Galaxy and... I ended up here trying to submit the link for the CF ongoing incident
- DC-3 11mo agoIt's the old IBM thing. If your website goes down along with everyone else's because of Cloudflare, you shrug and say "nothing we could do, we were following the industry standard". If your website goes down because of on-prem then it's very much your problem and maybe you get to look forward to an exciting debrief with your manager's manager.
- lpcvoid 11mo agoThat's lazy engineering and I don't think we as technical, rational people should make that our way of working. I know the saying, but I disagree with it. My fuckups, my problem, but at least I can avoid fuckups actively if I am in charge.
- reassess_blind 11mo agoHow do you mitigate large scale DDoS?
- lpcvoid 11mo agoI don't, since my stuff is reachable only within the company network/VPN. If I needed to though, I would consult the BSI list of official DDOS mitigation services [0] and evaluate each one before deciding. I would not auto-pick Cloudflare. [0] (German) https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Cyber-Sicherheit/Themen/Dienstleister-DDos-Mitigation-Liste.pdf?__blob=publicationFile&v=21 https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Cyber-Si...
- reassess_blind 11mo agoWhen the solution you pick inevitably has downtime too you’re in the same boat. DDoS mitigation is one of those areas that an on-prem solution just isn’t well suited to solve.
- lpcvoid 11mo agoYeah, but people aren't using Cloudflare just for DDOS Mitigation. Some are running pretty much everything over it, from DNS to edge caching to load balancing and even hosting. That's what I oppose mainly.
- rkangel 11mo agoI would only consider doing stuff on-prem because of services like Cloudflare. You can have some of the global features like edge-caching while also getting the (cost) benefits of on-prem.