36 ms·
Tell HN: Server Status
HN went down for nearly all of Monday the 6th. I suspected failing hardware.
I configured a new machine that is nearly identical to the old one, but using ZFS instead of UFS. This machine can tolerate the loss of up to two disks. I switched over to it early morning on the 16th, around 1AM PST.
Performance wasn't great. Timeouts were pretty frequent. I looked into it quickly, couldn't see anything obvious, and decided to sleep on it. I switched back to the old server, expecting to call it a night.
Then the old server went down. Again. The filesystem was corrupted. Again. So I switched back to the new server. During this switch some data was lost, but hopefully no more than an hour.
And here we are. I'm sorry that performance is poor, but we're up. I'll work to speed things up as soon as I can, and I'll provide a better write-up once things are over. I'm also really sorry for the data loss, both on the 6th and today.
- nmc 13y agoThanks for the info! Out of curiosity, do you have an idea about the source of the corruption problems?
- jffry 13y agoThanks for the writeup.
- hartator 13y agoNot really related but any update on releasing the HN code again? [the current release is pretty old: https://github.com/wting/hackernews https://github.com/wting/hackernews]
- cincinnatus 13y agoI'm sure it has been asked many times before, but I'd love to hear the latest thinking... Why in 2013 is HN still running on bespoke hardware and software? If a startup came to you with this sort of legacy thinking you'd laugh them out of the room.
- raimue 13y agoIt's 2014 already!
- oneeyedpigeon 13y agoHN isn't a mission-critical business. It's a fun, experimental distraction (albeit one that we all rate very highly, of course).
- lazyjones 13y ago> It's a fun, experimental distraction It seems to incur maintenance costs and reputation hits that are avoidable though, isn't that reason enough to invest some time in building a reliable solution?
- user24 13y agoNo-one's going to say "Yeah I was going to apply to YC but HN keeps going down so I don't think I will after all".
- lmm 13y agoSpeak for yourself
- randallsquared 13y agoCould you elaborate on why that's a factor at all? Given that YC's core business has nothing to do with HN, I wonder if the idiosyncrasies of HN aren't actually contributing something like intermittent reinforcement training to community members.
- ishener 13y agomay i ask where are the machines hosted? is that on AWS? if not, why don't you move to a more reliable hosting, like AWS?
- ishener 13y agofrom the downvotes i gather i may not ask...
- oneeyedpigeon 13y agoProbably due to the fact that you levelled an unqualified, unsubtantiated claim that AWS is better than any alternative
- viraptor 13y agoThat's not what he said. There's nothing about AWS being better than alternatives to AWS. > "a more reliable hosting, like AWS"
- staz 13y agoAlso because if you had RTFA you would have already known it's not on AWS
- defcon84 13y agoon a nginx server on cloudflare
- thejosh 13y ago>if not, why don't you move to a more reliable hosting, like AWS? Ah yes, because AWS has worked out so well for the reddit folk. :-)
- meddlepal 13y agoReddit is also a couple orders more active than HN. Also I thought Reddit's AWS issues were resolved years ago.
- smalu 13y agoThe world would be better place if software could exist without hardware.
- rincebrain 13y agoZFS instead of UFS on what, an Illumos derivative, FBSD, or actual Oracle Solaris?
- pmarin 13y agoFreeBSD.
- lewq 13y agoI'd love to show you how HybridCluster, which automates ZFS replication and failover (FreeBSD + ZFS + jails), might be able to help. Relatedly, we've just announced free non-commercial licences which would be perfect for HN: http://www.hybridcluster.com/blog/containers-distributed-storage-future-now-free-hybridcluster-non-commercial-licenses/ http://www.hybridcluster.com/blog/containers-distributed-sto...
- erkkie 13y agoThis reminds me I'm still looking for a (pki?-)encrypted zfs snapshots as a backup service, /wink-wink @anyone Hoping the box has ECC ram, otherwise zfs, too, can be unreliable (http://research.cs.wisc.edu/adsl/Publications/zfs-corruption-fast10.pdf http://research.cs.wisc.edu/adsl/Publications/zfs-corruption...)
- barrkel 13y agoBy tolerating the loss of two disks, do you mean raidz2 or do you mean 3-way mirror? Raidz2 is not fast. In fact, it is slow. Also, it is less reliable than a two way mirror in most configurations, because recovering from a disk loss requires reading the entirety of every other disk, whereas recovering from loss in a mirror requires reading the entirety of one disk. The multiplication of the probabilities don't work out particularly well as you scale up in disk count (even taking into account that raidz2 tolerates a disk failure mid-recovery). And mirroring is much faster, since it can distribute seeks across multiple disks, something raidz2 cannot do. Raidz2 essentially synchronizes the spindles on all disks. Raidz2 is more or less suitable for archival-style storage where you can't afford the space loss from mirroring. For example, I have an 11 disk raidz2 array in my home NAS, spread across two separate PCIe x8 8-port 6Gbps SAS/SATA cards, and don't usually see read or write speeds for files[1] exceeding 200MB/sec. The drives individually are capable of over 100MB/sec - in a non-raidz2 setup, I'd be potentially seeing over 1GB/sec on reads of large contiguous files. Personally I'm going to move to multiple 4-disk raid10 vdevs. I can afford the space loss, and the performance characteristics are much better. [1] Scrub speeds are higher, but not really relevant to FS performance.
- JoeAltmaier 13y agoReading 2 disks is not very much slower than 1. Its always the I/o bandwidth that is at issue, so it very much depends on the connectivity. And it all pretty much impacts your customer. We've all suffered under the 4-hour mirror rebuild, the whole machine made inoperable by the constant disk load. The only way to alleviate that, is to design extra bandwidth (e.g. another cable and controller) that's used exclusively for recovery. I've designed/build storage that worked that way, but it was for enterprise. The home user doesn't want to pay for extra bandwidth and then not use it most of the time. And if the DO use it, then they notice when a rebuild is triggered, and complain. Its catch-22.
- chrissnell 13y agoCan you elaborate on the second controller? Are you referring to two disks on two controllers, using mdadm (or similar) to create a software RAID 1 across the two?
- rdl 13y agoI still like hardware RAID because it's conceptually simple and nicely isolated. Sometimes horrible things happen to it, though, too. I didn't realize HN had enough disk storage needs to need more than one drive. I guess you could have 1+2 redundancy or something.
- avifreedman 13y agoAssuming the disk footprint is small... Would recommend a new SSD-based ZFS box (Samsung 840 Pros have been great even for pretty write-intesive load), with raidz3 for protection and zfs send (and/or rsync from hourly/N-minute snapshot for data protection which should eliminate copying FS metadata corruption, as not sure if zfs send will). Happy to provide and/or host such a box or two if helpful.
- carsonreinke 13y agoMaybe you could provide details on the current configuration and architecture and some suggestions could be made on how to improve. Just a thought.
- lukasm 13y agoHow about error page show the last static HN page? Most people just need likns
- codfrantic 13y agoI used http://www.hckrnews.com/ http://www.hckrnews.com/ for that :-) But I actually usually prefer reading the comments more than the links ^_^
- waxzce 13y agoHi, I'm the CEO of http://www.clever-cloud.com/ http://www.clever-cloud.com/ and I'll be happy to help you on this, ping me on twitter : @waxzce
- richardw 13y agoThanks for the update. No worries, it's just a news message board and no businesses are hurt when it's down. I quite enjoy seeing how these things are solved and I'm sure all will be forgiven if you post a meaty post-mortem.
- makmanalp 13y agoThe trend I'm noticing is people mentioning that if only HN was moved to <insert-cloud-provider>, problems would go away. Instead of doing that, they probably dropped a bit more than a thousand dollars on a box, and are probably saving thousands in costs per year. This is money coming out of someone's pocket. This site is here, and it's a charity, being provided free of cost, to you. Who cares if HN is down for a few hours? Seriously? Has anyone been hurt because of this, yet?
- loceng 13y agoNot to discount the value of HackerNews, however there is great benefit to YCombinator being associated with it; They could let it go down and never host it again but they would lose value.
- annnnd 13y agoYes, but I don't think such shortages do them much harm.
- loceng 13y agoI think it's pretty bad that something run by YCombinator, a company trying to build billion dollar companies, has outages - and where they don't have immediate redundancy.
- colinbartlett 13y agoSadly, I agree. But only because there's not a viable alternative. It's like if YouTube goes down for an hour, everyone whines but really nothing changes.
- annnnd 13y agoI wouldn't say that. I think they are competitive because they focus on what is important and disregard the rest as much as they can. And high availability is clearly not what is important for success in this case. With Youtube I wouldn't be so sure...
- shawn-butler 13y agoUsing DTrace to profile zfs: http://dtrace.org/blogs/brendan/files/2011/02/DTrace_Chapter_5_File_Systems.pdf http://dtrace.org/blogs/brendan/files/2011/02/DTrace_Chapter... I'm sure other more experienced DTrace users can offer tips but I remember reading this book and learning a lot. And I believe all the referenced scripts were open source and available.
- JayNeely 13y agoBeing the sysadmin on a site frequented by sysadmins has to be frustrating at times. Thanks for all you do!
- conorh 13y agoHave you thought about perhaps open sourcing the server setup scripts for HN? I'd love (and I'm sure many others here) to help with the configuration. Perhaps a github repo for some chef recipies that people could work on given the current servers?
- Goladus 13y agoI've been reading this site regularly for almost 7 years. 6-Jan-2014 is the only downtime I remember, and it was really a very minor inconvenience. Sucks about the data loss though, always hard to own that when doing system administration. Thanks for the explanation.
- elwell 13y agoIt's not the only one I remember. It seems like it's down for several hours every couple months.
- rrpadhy 13y agoI am curious to know the server configuration, architecture and the number of hits it is getting. If someone does offer a new software architecture, and hosting, would people be open to move hackernews there?
- superice 13y agoGood you posted this, but it came a little late. After the first series of timeouts you could've posted an update so everybody knew what was going on. But hey, thanks for the update, this clears up a lot.
- whalesalad 13y agoThis passive-aggressive attitude is going to get you nowhere in life. Back handed complements are only cool in pre-teen TV shows.
- scurvy 13y agoWhy on earth are you not using SSD's? The HN footprint can't be that large. The extra speed and reliability from a pair of SSD's has to far outweigh the costs.
- cperciva 13y agoFailing SSDs get data corruption too.
- ivoras 13y agoI'd guesstimate that the READ load is served practically entirely from RAM (file cache) and the WRITE load is non-critical enough that it's done "eventually consistent" (e.g. synchronous_commit=off in PostgreSQL, or fsync=off elsewhere) - or at least that's how I'd run it. YMMV.
- 0xdeadbeefbabe 13y agoDon't worry about it. I visited facebook for the first time in years when hn went down. Is hn on linux using zfs or bsd?
- whalesalad 13y agoThere's a lot of tuning that can be done on a ZFS setup to improve performance. I'm not a pro, so others will have more feedback and knowledge, but some things off the top of my head to get you started: Add a flash memory based (SSD) ZIL or L2ARC or both to the box. That'll help improve read/write performance. I believe the ZIL (ZFS intent log) is used to cache during writes, and the L2ARC is used during reads. You might want to look into disabling atime, so that the pool isn't wasting energy keeping access times on files up to date. Not sure if this is relevant with the architecture of HN or not. This can be done with zfs set atime=off srv/ycombinator Finally, ZFS needs a LOT of memory to be a happy camper. Like 3-5GB of RAM per TB of storage. I actually think you'll probably have a lot of fun with ZFS tuning, if that's the problem with news.yc. FreeBSD's page is pretty detailed: https://wiki.freebsd.org/ZFSTuningGuide https://wiki.freebsd.org/ZFSTuningGuide
- stock_toaster 13y ago> I believe the ZIL (ZFS intent log) is used to cache during writes, and the L2ARC is used during reads. I think the ZIL (zfs intent log) is an intermediary for synchronous writes only. My understanding is that it effectively turns the sync write into an async write (from the standpoint of the zpool) -- this is why it requires a faster device than the pool it is used with. If it is absent, the pool itself houses the zil.
- takeda 13y agoARC basically is a read cache. ZIL improves performance of writes, but by itself is almost never read (only written to) except on failures. It will be read for example on power failure to finish writing the data to the disk. It is used to speed up synchronous writes.
- lsc 13y agoAre you bottlenecking on high iowait? or something else? just one random bit to try... Obviously, I have no insight into your system and I'm not saying I know more than you or anything, but I've been seeing more situations lately where I had massive latency but reasonable throughput and the disks mostly looked okay wrt. smart, and I mostly just wanted to write about it: [lsc@mcgrigor ~]$ sudo iostat -x /dev/sda /dev/sdb /dev/sdc /dev/sdd Linux 2.6.18-371.3.1.el5xen (mcgrigor.prgmr.com) 01/16/2014 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.05 0.02 0.00 99.93 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.70 75.11 35.66 1.38 4568.62 611.67 139.85 0.36 9.61 0.53 1.95 sdb 0.46 75.10 35.62 1.39 4566.77 611.67 139.89 0.22 5.89 0.45 1.66 sdc 0.80 75.14 35.63 1.35 4569.63 611.63 140.10 0.64 17.18 0.57 2.10 sdd 0.46 75.09 35.62 1.40 4566.60 611.63 139.87 0.13 3.47 0.40 1.49 (this is a new server built out of older disks that appears to have the problem. It's not so bad that I get significant iowait when idle, but if you try to do anything, you are in a world of hurt.) Check out the await value. re-do the same command with a '1' after /dev/sdd and it will repeat every second. If sdd consistently has a much worse await, it is what is killing your RAID. Drop the drive from the raid. If performance is better, replace the drive. If performance is worse (and with raid z2, it should be worse if you killed the drive) the drive was fine. (Of course you want to do the usual check with smart and the like before this) The interesting part of this failure mode that I have seen is that /throughput/ isn't that much worse than healthy. You get reasonable speeds on your dd tests. but latency makes the whole thing unusable.