6 ms·
Does all of HN run on one single server?
by blantonl 14y ago
Does all of HN run on one single server?
- deleted 14y ago[deleted]
- abstractbill 14y agoNot just a single server, a single process on a single core (last time I asked pg at least).
- iamwil 14y agoAnd I recall the last time I looked at the source (when it was released with Arc), it didn't use a database either. All the data is stored as files.
- spoiledtechie 14y agoFiles???? That's a joke right???
- abstractbill 14y agoWhy should it be? You can get a long long way by treating the filesystem as a database. The first engineers at Amazon used the same technique a lot, as do I.
- kansface 14y agoBecause you always end up building your own database out of flat files and that is always worse than using an existing one.
- saraid216 14y agoAnd yet, here you are, on a site run off a flat file database.
- deleted 14y ago[deleted]
- badgar 14y agoA commenting site that: - Has average latency over 500ms when not under load - Performs quite poorly under load (I hate to bring it up, but the most recent example was Aaron Swartz's passing. Anyone who used HN then to get news knows how poorly HN performs under load) - Is restarted every week or two because it leaks memory - Keeps XSRF tokens in memory and loses them across restarts - Doesn't have a full markup language HN is quite poorly-featured compared to typical commenting sites. People use HN because pg is here. He could remove half the features on the site (bold & italics... what features are there even to remove beside nested commenting?) and retain 90% of the audience.
- Steko 14y agoYou're probably seeing the artificial delay introduced for commenters that can't maintain at least a 4.0 comment point average or people who aren't signed in. Site runs like butter for me.
- incongruity 14y ago>People use HN because pg is here. Nothing personal against pg - but I'm here more because of everyone else - the caliber of the discussion, for a news/tech site is quite high, IMHO - and that's due to everyone, not just pg.
- nonamegiven 14y ago
- robinh 14y agoWell, no. There are some exceptions, but most databases add a whole lot of bloat you don't necessarily need. Simple files can be just as fast or even faster than using a big database - which is the most important metric to me.
- rogerbinns 14y agoIf it was always worse then every developer doing this must be stupid. Here are some ways in which a filesystem is "better": - Zero administration - Only configuration setting is the directory - Trivial to test - Trivial to examine with existing tools, backup, modify etc - Works with any operating system, language, platform, libraries etc - Good performance characteristics and well tuned by the operating system - Easy for any developer to understand - No dependencies - Security model is trivial to understand and is a base part of operating system - Data is not externally accessible Many existing databases have attributes that aren't desirable. For example they tend to care about data integrity and durability, at the expense of other things (eg increased administration, performance). For a use case like HN, losing 1 out of every 1,000 comments wouldn't be that big a deal - it isn't a bank. Consider the development, deployment and administrative differences between doing "hello world" with a filesystem versus an existing database. Of course this doesn't always mean filesystems should be used. Developers should be practical and prudent. TLDR: YAGNI, KISS, DTSTTCPW
- nonamegiven 14y agoTIL: DTSTTCPW
- keeperofdakeys 14y agoYou also get an automated buffer system for the data, due to the OS.
- JohnHaugeland 14y agoACID transactionality. Thank you, drive through.
- abstractbill 14y agoEven if that was true, I'd still tell people to start with flat files for a new project. It's like the advice to do a job yourself before hiring for it: You'll be better equipped to judge how well a database is managing your data if you've already done it yourself.
- jlgreco 14y agoJust as one example, I think you could make a fairly convincing case that the official version of git uses a "filesystem as a database" system with great success. I doubt it would be improved by using something more "proper".
- deleted 14y ago[deleted]
- InclinedPlane 14y agoI find it interesting how easy it is to criticize a functioning system based on some aspect of non-conformance with some hypothetical ideal. The idea that HN wouldn't run on a database is no less astounding than the idea that much of facebook runs on PHP. Design of real systems is often messy and imperfect and deviates from the ideal due to necessity of optimizing one or another factor that may not be obvious.
- spoiledtechie 14y agoFacebook doesn't run on PHP and hasn't for a while... Its compiled C.
- btown 14y agoA link to (possibly an old version of) the source for those interested: https://github.com/nex3/arc/blob/master/lib/news.arc https://github.com/nex3/arc/blob/master/lib/news.arc
- taylorbuley 14y agoInteresting read. I'd love to see my median karma on my profile. Much more robust to outliers than average. https://github.com/nex3/arc/blob/master/lib/news.arc#L2613 https://github.com/nex3/arc/blob/master/lib/news.arc#L2613
- dylangs1030 14y agoJust like Viaweb.
- googoobaby 14y agoIt actually runs secretly on a large number of other people's servers. Rtm was a pioneer in this field.
- shiftpgdn 14y agoI think a lot of people underestimate how powerful a single well configured server can be. You don't need Heroku/AWS for everything.
- nostrademons 14y agoAlso, people underestimate the power of serving out of RAM. It's not unreasonable to serve 20-30K QPS off a single server if the work it needs to do is limited to minimal request parsing and fetching some data from main memory. That's about 2.5 billion requests/day, fully loaded. Granted, I'm thinking something more like memcached than a fully-formed webserver, but an in-memory webserver that stores its data in hashtables (like news.yc) and has a really fast templating languages, or just writes output directly to the socket, could probably come close.
- self 14y agoI use redis for this exact reason -- I prerender over 2,000 page templates twice a day, and store them in RAM. The app server has to do a little processing before sending the pages to users -- it picks a different template depending on whether the user's logged in or not, and then substitutes the user's info into the template (for logout/profile links). The session info is also stored in redis. This lets me reboot the server and be ready to serve pages again almost as soon as it's back up. With all the data, redis uses about 300-400MB RAM on a 64bit Debian VM. I use a VPS for my site, and on a VPS, the only thing you're allocated that you can depend on always being available is RAM. The processor cores might be shared with a busy user, and you can't always depend on high disk I/O speeds.
- mbell 14y agoKeep in mind that a single modern physical server that is decently configured (12-16 cores, 128GB of ram) is the processing equivalent of about 16 EC2 m1.large instances.