29 ms·
Ask HN: How does HN manage to be always online?
How does HN manage to be always online?
- notlukesky 4y agoGuess they don’t use Cloudflare: https://news.ycombinator.com/item?id=31820635 https://news.ycombinator.com/item?id=31820635
- ignoramous 4y agonews.yc might not be, but ycombinator.com and startupschool.org are on Cloudflare.
- nik736 4y agoA single bare metal server is more reliable than most people think it is. Complexity adds a lot of overhead and layer after layer that could possibly fail.
- jazzyjackson 4y agoObligatory Lamport quote: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."
- srer 4y agoA single server is much faster than most people think, too! In the microservice or serverless arrangements I've seen, data is scattered across the cloud. It's common for the dominant factor in performance to be data locality, most times this talk about data locality is about avoiding trips to RAM, or worse disk. But in our "modern" distributed cloud things, finding a bit of data frequently involves a trip over the network. In the monolith world what was once invoking a method on an account object, has become making a HTTP POST to the accounts microservice. What might have been a microseconds operation in the single server world, might become hundreds of milliseconds in the distributed cloud world. While you can't horizontally scale a single server, your 1000x head start in performance might delay scaling issues for a very long time. A most excellent paper related to this topic that I think should be mandatory reading before allowing anyone an AWS account is http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf :)
- bayindirh 4y agoWhen you give half the effort to set things up properly, a single server can handle a lot of load and traffic, and get a lot of things done. If you know some details of the services you're going to host on that hardware, the things you can do while saving a lot of resources is considered as black magic by many people who only deploys microservices to K8S systems. ...and you don't need VMs, containers, K8S and anything.
- GGfpc 4y agoWhat are those details?
- bayindirh 4y agoIt generally boils down to three things: 1) How much resource a service need to run well, 2) How much the service wants to consume if left unchecked, 3) How performant you want your service to be. After understanding these parameters, you can limit the resources of your application by running it under a cgroup. Doing this won't allow a service to surpass the limits you've put onto it, and cgroup will pressure your service when it nears its resource limits. Also, sharing resources is good. Instead of having 10 web server containers, you can host all of them under a single one with virtual hosts, most of the time. This allows good resource sharing and doing more with less processes. On the extreme case, I'm running a home server (DNS server, torrent client, synching client, a small HTTP server and an FTP server with some other services) under a 512MB OrangePi Zero. The guy works well, and never locks up. It has plenty of free RAM, and none of the services are choking.
- iso1631 4y agoI think the main risk of the HN architecture is that (I believe) it's a single datacentre in a single location. Hopefully they have offsite backups. The other risk I guess would be that all the NS are AWS/route53 severs, so if that went down they'd be up for about a minutes (looks like the DNS TTL is 2 minutes). You could host your own NS servers in two different locations on two different providers, you could have a third hot spare server ready to go too, that would allow the service to survive an earthquake flattening San Diego (off site server ready to go), and cope with the loss of AWS DNS. Whether the cost/benefit ratio is there is another matter. I think the serving side of route53 is fairly reliable though (even when the interface to update records fails on a frequent basis), and the cost of being down isn't particularly terrible.
- barelysapient 4y ago> I think the main risk of the HN architecture is that (I believe) it's a single datacentre in a single location. Exactly, Amazon US EAST makes the same point several times a year! :)
- systemvoltage 4y agoWe should ask this question to ourselves: As a user, if I have to manage the risk of US-East-1 going down, may be we should question this entire Cloud thing. It was marketed as "100% bulletproof" in 2005 when we discarded on-prem.
- iso1631 4y agoTo be fair to amazon I don't think they've had an outage that affected different data centres at the same time, but yes I would far rather have my main and backup on two providers in two appropriately diverse locations.
- dopp0 4y agobut never forget that AWS uses us-east-1 to host core services that, when they go down, they take almost the whole thing down, no matter which region.
- tomsmeding 4y ago> A single bare metal server is more reliable than most people think it is. Well, two, presumably. Otherwise you can't reboot without taking the website offline.
- stavros 4y agoSometimes that's fine, we can go without HN for five minutes once every few months.
- theandrewbailey 4y agoFew months? I notice HN down for a few minutes about once a week.
- NorwegianDude 4y agoIf you want it to be online, why turn it off or reboot in the first place? You can run updates without rebooting, at least on Linux.
- tomsmeding 4y agoCan't replace the kernel though, and updating the web server application itself is going to be hard too. Unless it's some Erlang like hotpluggable server I guess.
- fragmede 4y ago> Can't replace the kernel though Yes you can, since 2008. https://en.wikipedia.org/wiki/Ksplice https://en.wikipedia.org/wiki/Ksplice
- rvz 4y agoMaybe they are intentionally using boring tech, not using Cloudflare, AWS, etc and are self hosting somewhere, hopefully.
- capableweb 4y ago> Maybe they are intentionally using boring tech This might be true on the infrastructure layer, but HN definitly uses "fancy" technology as Paul developed his own lisp that powers HN, Arc :) http://arclanguage.org/ http://arclanguage.org/ > Arc is designed for exploratory programming: the kind where you decide what to write by writing it. A good medium for exploratory programming is one that makes programs brief and malleable, so that's what we've aimed for. This is a medium for sketching software.
- joshxyz 4y agoPowered by Illuminati technology.
- ImpulseGuided 4y ago>Powered by Illuminati technology. PG operating the switchboard.
- lazybreather 4y agoI would love to know tech stack and architecture of HN. And how it evolved (if it did). And what resources they spend (money, manhours etc) to maintain it.
- beaconstudios 4y agoIt's written in arc, a custom dialect of lisp that Paul Graham made.
- sokoloff 4y agoThere are micro-outages somewhat frequently; I don’t mean that as a criticism but merely as an observation.
- sshine 4y agoI'm reading through hckrnews.com, as I'm sure others have their favorite readers. Those function as proxies and lower the perception of downtime as well. I've never experienced the website being inaccessible even for short periods.
- globular-toast 4y agoAn interesting note would be that multiple 9s of uptime isn't actually necessary to keep users happy on a site like HN. I used to use sites in the early 2000s that were down a lot more regularly than HN but it didn't put me off using them.
- 6510 4y agoI sometimes wonder if shutting down a website [say] with exception of Mondays or between 22:00 and 8:00 would add to the experience.
- viraptor 4y agoWhose 22-08? Whose Monday? For anything other than very local groups, I don't think that would be a good idea.
- globular-toast 4y agoA minor spoiler I guess, but that's what they decided to do at the end of Ready Player One (the film, not sure about the book).
- aaaaaaaaata 4y agoThere are government and personal websites like this! https://freakonomics.com/2012/08/this-website-only-open-during-business-hours/ https://freakonomics.com/2012/08/this-website-only-open-duri... I hope another elder will drop others here.
- revskill 4y agoThey read file, save data to file. That's it.
- koolba 4y agoHopefully there’s an fsync as well.
- sgt 4y agoLife is too short to wait for a fsync.
- somat 4y agoThere are some environments where making sure your transaction is secure on non volatile storage is critically important. And then there is storing comments on a news aggregation site. Hell, this is one environment where I bet you could get away with a full async mount.
- koolba 4y agoSure, but the default should be erring on the side of persistence and consistency. The throughput for a site like HN of actual writes is next to nil. It’s short text only comments averaging a couple hundred bytes and you only have to pay for the fsync once anyway. If you have to turn off fsync to make it perform then you’re doing something horribly wrong.
- 5id 4y agoAccording to @dang (https://news.ycombinator.com/item?id=28479595 https://news.ycombinator.com/item?id=28479595) via @sctb (https://news.ycombinator.com/item?id=16076041 https://news.ycombinator.com/item?id=16076041) We’re recently running two machines (master and standby) at M5 Hosting. All of HN runs on a single box, nothing exotic: CPU: Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz (3500.07-MHz K8-class CPU) FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads Mirrored SSDs for data, mirrored magnetic for logs (UFS)
- likortera 4y agoNo microservices on top of kubernetes? no SPA with SSR? You are doing it wrong. I'm gonna write an alternative which will be WebScale. j/k of course.
- yodsanklai 4y ago(beginner question) How do they store the data? is an SQL db on overkill for such a use case? what would be the alternative? an ad-hoc filesystem based solution? then how do the two servers share the db? and is there redundancy at the db level? is it replicated somehow?
- krapp 4y agoData is stored in flat text files containing Arc Lisp tables, or in RAM. There is no 'database' per se, unless they've added one and not mentioned it. You can get the software and language HN is based on here: http://arclanguage.org http://arclanguage.org
- aristofun 4y agoWe managed to run a successful bootcamp school LMS on a single cheapest 1gb ram hetzner vps with rails app, redis cache, postgres, backups, staging env and near zero production issues. Not hi loaded of course, but still hundreds active users every single day interacting with the app. Recently had to upgrade to the next tier because of growth. Modern servers are super fast and reliable as long as you know what you’re doing and don’t waste it on unnecessary overheads like k8s etc.
- kuschku 4y agoEven k8s only costs about 10% overhead. It's kind of incredible that "a few hetzner dedis running k8s" still has better reliability than the Cloud™ does.
- capableweb 4y agoRemove Kubernetes and "A few Hetzner dedicated servers running just your application" has even better reliability as you remove more points of failures than "A few Hetzner dedicated servers running Kubernetes".
- carlmr 4y agoSo much this. System failure is a statistics game. Which is why I like the Erlang approach of having isolated state that you assume will corrupt after a while and you can restart that part of your application with a fresh state. K8s does this, kind of, on a higher level, where the downtime is often more noticeable because there's more to restart. But if your application is already highly fault-tolerant, this is just another point of failure.
- sammyo 4y agoWhy hasn't most of the web moved to Erlang (or erlang like) architecture, actually distributed?
- 4y ago
- jve 4y agoIt IS down occasionally: https://twitter.com/hnstatus https://twitter.com/hnstatus Not all "downs" are reflected there. Last time I remember having really bad perf or non-workable, don't remember - but opening in incognito, you would get cached results fast.
- thatoneguytoo 4y agomost of the outages usually happen when there are changes. hn has no changes.
- Gatsky 4y agoRelated question: How is HN funded? I presume through YC.
- killingtime74 4y agoProbably smaller than the snack budget for YC
- jasode 4y ago> always online? It depends on what you mean by "online" and the service level : - "online" meaning a HN server responds with something. In this more literal sense, HN always seems to be up. - "online" meaning normal page load response times. In this sense, HN sometimes times out with "sorry we can't serve your request right now". That seems to happen once a week or once a month. Another example is a super popular thread (e.g. "Trump wins election") that hammers the server and threads take a minute or more to load. This prompts dang to write a comment in the thread asking people to "log out" if they're not posting. This reduces load on the server as rendering pages of signed-out users don't need to query individual user stats, voting counts, hidden comments, etc. This would be a form of adhoc community behavior cooperating to reduce workload on the server rather than spin up extra instances on AWS. The occasional disruptions to the 2nd meaning of "online" is ok since this is a discussion forum and nobody's revenue is dependent on it. Therefore, it doesn't "need" more uptime than it already has.
- MarcScott 4y agoLast time I remember it being down for anything other than a few minutes was back in 2014. https://twitter.com/Coding2Learn/status/420298797462593536 https://twitter.com/Coding2Learn/status/420298797462593536
- _benj 4y agoI guess because of sanity and simplicity in its architecture? I once wrote a software in Rust, a simple API, one binary in one DigitalOcean instance started by systemd, and nothing else. The things has been working nonstop for years making it the most stable piece of long running software I’ve ever written, and I think it all comes from it being simple without any extra/unnecessary complexity added. I’m not bragging btw, I actually had to contact the user years after I wrote that because I couldn’t believe that the thing was still working but I hadn’t heard from them in years!
- thdxr 4y agoHN is a mature product! Most common software failures are from a deployment of changes, I don't think HN is being deployed to every day We're all used to effectively beta software that's constantly being updated every day and never final
- ericpauley 4y agoHN is down and has loading issues pretty often. AFAIK they now run behind a CDN so if you're not logged in you won't notice this.
- mbrodersen 4y agoI have never experienced HN being down.
- jjice 4y agoThere was a post somewhere else in this thread that said they use Nginx caching for non-logged in users, so not even a CDN it seems.
- jmartens 4y agoI've literally never experienced this.
- EduardoBautista 4y agoThere have been times when we have been asked if we can log out if we won’t be commenting because this will then allow the servers to serve straight from cache. I have only seen that happen a few times though.
- eimrine 4y agoI have seen an error message once, instead of content. I did not remember it but the wording was amazingly perfect. And there is one minor issue when "reply" button to a specific comment had not been loaded by the time the comment has rendered.
- pictur 4y ago0 features = 0 problems
- jmartens 4y agoI'd guess that HN has nearly zero dependencies, other than the bare metal box we've been told about. That makes things a lot more simple!