4 ms·
Availability is overblown these days IMO. Ten or twenty years ago I would have agreed with you: the internet was new, and when the site went down, people blame
by floating-io 3y ago
Availability is overblown these days IMO.
Ten or twenty years ago I would have agreed with you: the internet was new, and when the site went down, people blamed the site, and things got nasty fast. Nowadays, though?
People are much more likely to blame their internet provider first, or just plain try again later. They're used to the unreliable nature of the internet now.
Unless you're Google, but how many businesses are Google?
As someone else on this thread said, simpler systems are less likely to fail anyway. Keep backups, replicate your DB to a cold DR site if you're truly worried. That will handle the majority of non-FAANG businesses in the majority of non-FAANG cases.
And if that failure occurs? So long as you don't lose data, it will be a short-lived blip for most businesses. Even being down for a few days can be weathered if you have good PR, provided that it's a once in a blue moon sort of event. For that matter, data losses can be weathered in many cases.
What was that saying? Each nine of reliability doubles the cost or some such?
That needs to be accounted for when determining the ROI. How reliable does your business actually need to be?
That depends on your target market. The default answer from IT, however, is an automatic "100%!". This is wrong, IMO.
- datascienced 3y agoWat? If the site is down I’ll probably wont be back, and if I am paying for a service I am angry and might unsubscribe. Let alone lost revenue for the company from new sign ups or readers.
- jakewins 3y agoYeah agree - if a website doesn’t respond I immediately open google in another tab; if google doesn’t respond I blame my internet and swap to my phone. We’re all on redundant internet connections now.
- kidintech 3y agoYour sentence ANDs together an overlap of a bunch of improbable scenarios: - if the site is down for more than a few seconds - if you attempt to access the site during its downtime - if you pay an ongoing subscription (this could very well be a non-profit site, or one that doesn't primarily drive its revenue from subscriptions) - if you feel like unsubscribing How much does this *really* affect a non-moonshot business?
- datascienced 3y agoThe article talks about a high traffic site, maybe a top 10 one or something like that. It is already a moon business, not a moonshot. For a small blog or local business, sure a bit of downtime is OK but why are they even bare metal when squarespace etc. exists.
- kidintech 3y ago> The article talks about a high traffic site, maybe a top 10 one The article explicitly restricts the problem space to top 1000 websites. I've taken 30 seconds to browse through top 1000 websites and outside of the usual suspects up top, most of them look like https://www.pro-football-reference.com https://www.pro-football-reference.com or https://www.tempo.co https://www.tempo.co which to me are not far off blogs and local businesses.
- ReflectedImage 3y agoWell for a top 10 site, it's $1000 a month dedicated or $1,000,000 a month cloud.
- iforgotpassword 3y agoGithub was down for the nth time a few days ago. Yet I'm back. Who comes up with these stupid claims? I'm not going to ditch <messenger everyone in my friends circle is using> if they're down for a few minutes once a month. The supermarket at the corner had problems with their payment system for three days(!) last year and only took cash. I'm still a regular there. My favorite sushi place was closed for two months after a fire; I'm still going there. If people stop using your site after it's been down once it can't be that good.
- datascienced 3y agoThere are counterexamples yes. The almost monopoly on open source hosting. If Github goes down every day for an hour, companies will move to Gitlab. Obviously in business there are power dynamics. I queued for an hour to get an ice cream once. That seller was in a kids water park. Yet McDonalds did a hockey stick because they solved fast food. There are always exceptions where a business has you. Nice to own such a business!
- iforgotpassword 3y ago> If Github goes down every day for an hour Now that's what I call moving the goalposts :) suddenly we're at an hour a day. Github wasn't the first OSS hosting platform, yet they managed to steal the show, and let me tell you a secret, it wasn't because they had less downtime than others. But if you want to keep telling yourself five nines is of utmost importance to your startup, more power to you.
- datascienced 3y agoYes the goalposts move if you run a near monopoly with a huge mindshare. Let’s say you want an electric toothbrush. Amazon is down, you order from kmart or something. If Github is down you don’t simply migrate your entire Ci/Cd/code/issues/actions to something else. You are locked in! So they can afford crappier goalposts, but only so crappy.
- 3y ago
- ffsm8 3y agoWhy are you on HN? It had quiet a lot of downtimes/service degradations over the last year, and according to your claim you don't use services that aren't 100% available.
- datascienced 3y agoYou snookered me there! Nice. But I will say that I have only seen it degraded, i.e. slow, not completely down. Also it may matter less to HN which is a cult following. For a FAANGy type company of course it matters.
- ffsm8 3y agoSo you sold all your apple products last week? They had a full downtime after all.
- datascienced 3y agoWell maybe on your splunk dashboard but I have not experienced downtime of any Apple services.
- ffsm8 3y agoIt's almost like short downtimes don't matter, isn't it? And depending on the service, sometimes even multi-hour downtimes that make almost all of your services (store, music, arcade, fit,...) unusable like in apples case. Most of the users won't even notice, and if they do... They still come back.
- floating-io 3y agoThen I have to ask if you're a customer I want to have. Customers with unreasonable demands (especially demands for absolute perfection) are a cost, not a revenue stream IMO.
- sisve 3y agoYes if the site is down you loose reader/buyers (in the end money) But keeping the site up also cost money. So context is really important here. How much money do you spend on the last % of availability vs how much money so you loose on downtime. In my experience people have a more complicated setup then needed because of availability, but at the same time have not tested a real DR scenario.
- zokier 3y ago> So long as you don't lose data The article advocates storing data in local sqlite on single server. That means that when the server goes poof you will lose data. How much data, depends on how frequently you run backups. But it will be non-zero amount. And backups bring the other point. It will never be just one server like the author proclaims. You need another server for the backups. Now you have two servers, and a dependency between them. The dreaded complexity is creeping in again, almost if it was inevitable
- geysersam 3y agoThe article suggest using litestream to continuously back up the data to remote storage. Sure you'll still loose some data but that's always the case whatever solution you choose.
- erik_seaberg 3y agoA quorum across isolated facilities doesn't lose data unless a disaster hits all of them at once. Unfortunately, "disaster" includes a bad commit that is deployed too quickly.
- Sammi 3y agoSqlite has exactly the same backup solutions as other sql dbs and the author specifically recommends using litestream to backup.
- TacticalCoder 3y ago> What was that saying? Each nine of reliability doubles the cost or some such? I don't know but if there's a domain where the five nines are a pipe dream it's the cloud: we've seen a shitload of major outages where this or that cloud provider had services down for hours. This blows the five nines for years, if not decades (6 minutes max downtime per year: good luck with your five nines when you've got a 300 minutes cloud downtime). The cloud is so unreliable people are now simply used to the Web not working: "Oh shit, it's down again, I'll try again later today or tomorrow". That's how bad the "x nines" expectation became because of the cloud.
- nh2 3y agoMy experience is: Downtime does matter. Customers ignore downtime when you're a service with big lock-in or no real alternative (GitHub, Facebook, etc). Because then they can't do anything about it. When you're a small/medium service, and you're down for 3 hours during a time where 4 employees of a company need you for productive work, they ask what's up, and if they must expect that this will happen again. Generally, shorter, one-off downtimes are forgiven, but longer or recurring ones are not. We run on Hetzner dedicated, and I think the best way to run most sensible software is to run on 3 or 5 beefy machines. If you only have 1 machine, the following will take you down, all of which have happened in my production systems (except the last, which I only read about in the news for 3 hosters): * You need to reboot for kernel security updates. A reboot causes 3 minutes downtime because that's how long a reboot takes on server hardware, including Hetzner's (multiple POST style screens, netboot timeouts and so on, most of which make sense). Kernel updates come in approximately every 3 weeks. You may skip some of them that are not security-relevant for you, but this requires careful analysis, which is more effort than just installing. * The mainboard dies. It needs to be replaced. Immediately causes 3 hours of downtime. * Some disk or RAM dies in a way that prevents the mainboard from booting. This should not happen, but it does. Also 3 hours of downtime. * One of these happens at 1 am on Saturday. You are in Europe or Asia. If you are not set up to be paged out of bed, your US customer will see 8 hours of downtime. * The top-of-rack or datacenter building core router has some trouble, and it takes the hoster 3 hours to fix it. 3 hours of downtime. * There's a fire or sprinkler flooding in your data center. It takes the hoster 2 weeks to clean up the physical mess. 2 weeks of downtime, unless you manage to re-build the infra somewhere else and restore from backups, then maybe 48 hours of downtime. Note most of this also applies if you run a single VM in AWS. If you're lucky, the restore times will be faster by being a VM. But that is not guaranteed; I've had instances not do their work for some hours and afterwards got the email "We noticed a problem with the physical host of your VM, we've automatically migrated it somewhere else now". Most of these problems go away if you have 3 servers (of the type the original post describes) instead of one, which is only slightly more effort.
- floating-io 3y agoI largely agree with this. I didn't take "one server" literally; more as hyperbole. Concern about availability is overblown, not irrelevant! There are shades of gray, though. Most sites don't need to engineer their system to hide catastrophic database failures from the user. Not being able to tolerate a single web server failure, though... that's just careless, and will end up giving you a bad reputation. You probably don't actually need five nines -- but three probably won't go amiss...