13 ms·
Cause of YC/HN outage discovered
- bshep 17y ago"We just figured out what caused the problem. Apparently Pair Networks' procedure for requesting that users upgrade to a dedicated server is to shut down their site without warning...." I think you should upgrade to a more "dedicated provider" rather than a "dedicated server"... But seriously, not even a warning?
- eli 17y agoI think this is pretty common. When Joyent shut down my shared account way back when, I discovered it in a comment in my httpd.conf
- swombat 17y agoShared hosting is generally sucky for anything remotely successful. I'm amazed you got away with storing your static assets there so long! When I got 100k hits in a day on my first blog, Dreamhost promptly shut it down without warning (in the middle of a slashdotting!)
- PostOnce 17y agoDreamhost? Their main advertising point is UNLIMITED TRAFFIC!!! I guess I'll scratch them off the list of potential hosts.
- jackowayed 17y agoNo, they they're careful to say unlimited bandwith (and storage and such), but not unlimited traffic. When I was looking around dreamhost docs/FAQs/something either right before or right after signing up, it did say somewhere that they might ask you to move up to their VPS if you're using too much CPU, which could happen just from a whole bunch of static hits. It's nice for your hosting all of your little sites and getting them up in basically 0 time, even if you need wordpress or something. The panel's pretty nice. I'd at least sign up when they have one of their crazy deals (I got a year of hosting with a free domain for $9.something, which basically means you pay for the domain and get the hosting for free. It's a good way to give them a try with the main cost being that they'll suck you in and get you paying $9/month after your cheap price expires.)
- yellowbkpk 17y agoI was told in a friendly conversation with one of their service reps that shared accounts get roughly 200-250MB of total memory usage (Ruby on Rails bumped me over, for example) and a "certain amount" of 95%ile bandwidth averaged over some time. I have a feeling they sort their customer list based on these two values every once and a while and send the top X accounts a message.
- axod 17y agoNothing is unlimited. It's just sales tactics.
- byoung2 17y agoThe terms and conditions pages for shared hosting sites usually outline that bandwidth, disk space, number of sites, and number of databases are unlimited, but CPU and memory are not. There's your bottleneck. You can put thousands of sites on shared hosting, but get one or two with decent traffic or heavy scripts and you get the upsell or leave email. That's why I host at Rackspace Cloud now.
- froo 17y agoI actually don't mind Dreamhost for very targeted uses. I jumped on a deal at the beginning of this year for ~$9 for 1 year of shared hosting. I now use that account purely for running things like scripts through SSH for doing basic data grabs with wget for further analysis elsewhere. I've more than got my money's worth and I've yet to see any kind of email message asking me to upgrade.
- deleted 17y ago[deleted]
- DrewHintz 17y agoDreamhost also repeatedly shut down my sites, such as mapwow.com, due to too much traffic. One time they even denied they shut it down -- even though my web directory had been chowned to root, which requires root privilege. I moved to slicehost and appengine[disclaimer,etc] and have been relatively happy with them.
- voxio 17y agoNo company worth their weight should be using off the shelf shared hosting.
- stuntgoat 17y agoSo your network traffic is guilty and sentenced until proven innocent.
- a2tech 17y agoThats some good customer service there Lou. (Said in Chief Wiggims voice) Seriously-they couldn't just email the account holder?
- deleted 17y ago[deleted]
- pg 17y agoThings should be back to normal now. Trevor just moved www.ycombinator.com to Slicehost, and I just told HN to refer to static stuff there instead of serving it locally. Interesting how easy it is to move your whole web site. Good customer service is important when users can switch so easily.
- othello 17y agoThank you ! Procrastination becomes much harder when HN does not provide a continuous flow of interesting distraction.
- nfnaaron 17y agoHacker News: We get shit done, so you don't have to.
- axod 17y ago>> "Things should be back to normal now." Hopefully that's the normal of months ago when loading comments/threads etc was quick? Still taking 10-20 seconds to load most pages. One thing I haven't tried yet is just creating a fresh account - maybe mine just has too much associated with it (Maybe I comment too much etc). edit: ah I see it only affected static content. Shame :/
- Locke1689 17y agoI've had some speed issues too. My account isn't tiny, but it is a 12th of your upvote count.
- carbocation 17y agoAny particular reason for Slicehost as opposed to any of the other similar options (Linode, prgmr)? I'm not affiliated with any, though I run my site on VPSes from Linode. I was convinced by a Dec 2009 post by an HNer, uggedal: http://journal.uggedal.com/vps-performance-comparison http://journal.uggedal.com/vps-performance-comparison
- 17y ago
- fletchowns 17y agoThis site was on a shared hosting box? And of course they are going to disable it without warning if it's causing problems for all the other customers on that box...
- pg 17y agoNot this site, www. There was no sudden spike in traffic. If they'd bothered to check the logs they'd have found that the load, whatever it was, was no higher than it had been.
- archon810 17y agoThis is almost as bad as the reason for my server's downtime recently: http://beerpla.net/wp-content/uploads/img_2958.JPG http://beerpla.net/wp-content/uploads/img_2958.JPG My server literally had its plug pulled.
- nkassis 17y agoAre we at the point were we have to deal with robotic janitors too? Geez, and I thought that watching the janitors while the work was over the top, now I gotta watch the rumba too :(
- eli 17y agoAnd this is why I don't recommend shared hosting for anyone
- bwb 17y agoBecause everyone you know has one of the busiest sites on the net?
- eli 17y agoAnyone can get slashdotted. And my time is valuable -- I'd much rather pay more for better service.
- bwb 17y agoYa but if your dad wants to put up his pictures you go tell him to buy a nice scalable hosting product instead of a cheap shared hosting account for $5 to $10 a month?
- ahi 17y agoYou can get a nice scalable hosting product for $5 to $10 a month.
- Xichekolas 17y agoNo. I'd tell him to get a Facebook/Picasa/Flickr/Twitpic account. Dad used to get a shared hosting account for things like that, but now we have services that host content for free. They are much easier for Dad to use as well. In this age of $20/mo VPSs and free content hosting, I'm honestly not sure how shared hosting survives. It's not as flexible as a VPS for hosting sites (and with more than two sites, isn't even as cheap), and it's not as easy to use (or as free) as Facebook.
- patio11 17y agoI'm honestly not sure how shared hosting survives. John Smith Plumbing's five page brochureware site doesn't need the headaches of having a VPS but also won't accomplish business needs on Flickr.
- deleted 17y ago[deleted]
- samd 17y agoI'm not sure what is more ridiculous: that they disabled the site without warning or that they were too lazy to look at their logs or do a simple Google search to find out that Hacker News is real site with lots of regular traffic.
- VBprogrammer 17y agoWhat I really loved is that they just demonstrated a fair degree of incompetence to a site used pretty much exclusively by people who are very good potential customers! This site is a wet-dream for their marketing department. What they should have done is upgrade the website to a dedicated server for free and let that news hit the front page.
- nfnaaron 17y agoWell, maybe upgrade for free, with a time limit, and an email saying "if we don't hear from you in x days, we'll shut 'er down."
- rrhyne 17y agoReally, we are all guilty. If we'd just get back to work, HN wouldn't have this problem. ;)
- sh1mmer 17y agoDear Pair Networks, I'd like to highlight this lesson in how to loose (or not gain) customers by randomly shutting down technology sites that serve the decision makers you wish to influence. Tom P.S. Well done Slicehost.
- gojomo 17y ago150K hits in 30 minutes is 83 hits/second. That's a lot to ask for from a shared-hosting account. When the traffic starts to impair neighboring sites, something has to be done. Just about any ISP will do the same thing: block the site with the surge, that could possibly make other arrangements, rather than inconvenience other customers whose traffic is as expected/usual. The detail missing so far is why Pair noticed today, if it was the same level of traffic as before, or a slow build. Was a new threshold crossed? (Did someone's HN-focused tool go haywire?) The Pair message suggests end-of-day logs will be the way to tell for sure.
- sh1mmer 17y agoI was being tongue-in-cheek, but there is a real point here. As you say I doubt today was unusual for HN, it's the way Pair went from zero to shut-down. One of the things I like about the way that Joyent operate as a cloud host is that they allow you burst on shared boxes because of those times you need it. At the same time they'll let you know you need to think about buying more resources without just slamming on the brakes. Pair should do a much better job of noticing a soft limit earlier, so if a heavy traffic day had hit HN PG et al should have been already aware they were overusing their shared hosting and were planning a route out.
- bwb 17y agoThat technology is not out there... A static site might be fun at that level and not causing problems, where as a WordPress site with super cache is fine, but then a Drupal site can only handle 1/4th of that. It currently is not really possible to track that per user in a shared environment and see who is using what resources from an individual perspective of MySQL / Apache / CPU / Mem.
- moe 17y agoAnd the Darwin award, category marketing, goes to...
- howcool 17y agoI had the same treatment from Site5 a while back before moving to a dedicated server. They just pointed to their hidden TOS page and laughed
- dirtyhand 17y agoThe thing I can't believe is that nobody at Pair Networks that was involved with this actually knew about Ycombinator or Hacker News... otherwise they wouldn't have shut down such a popular site. Somebody needs to hire real hackers
- bwb 17y agoGuys, shared hosting is designed for the 99% of non busy sites, say under 1,500 unique visitors a day, with the level of traffic HN is doing I'm surprised Pair didn't warn you earlier. A site this busy needs a dedicated server or virtual server. That is why shared hosting is cheap, you start with it and once you are successful or starting to get slashdotted you buy something bigger that can scale.
- bwb 17y agoAlso, just going to point out here that Pair.com has had a great reputation in the hosting industry, and has for the last 10 years. I've been in the industry for 10 years and worked for quite a number of hosting companies, not Pair though, and when you have 150 shared clients on a machine and 1 client is causing the problems you do your best to deliver the warning before it gets out of hand but it is very hard to do.
- rg 17y agoWhen I emailed Pair.com to tell them that my site was about to get a writeup in the Wall Street Journal, they moved my site to a dedicated server within an hour, left it there until the spike of traffic was over, then moved my site back to a shared server--all at no charge. If you use Pair.com properly, they are more than excellent.
- euccastro 17y agoThey could have a small pool of servers where they could move problematic sites until the customer has had a warning and a fair chance to react to it.
- jseliger 17y agoI don't think anyone will disagree. What they will assert, however, is that shutting down a site with zero notice is a jerky thing to do. As the other commenters have said, the nicer/smarter thing to do would've been to send an e-mail that says "Hey, you're using too much bandwidth/whatever." The smartest thing of all probably would've been, "You're using too much. Want to by more?"
- jqueryin 17y agoI actually find it quite embarrassing that the guy had no clue what HN or YC were. He had absolutely no idea... and he works for a hosting company. It's quite a shame. I really got a kick out of the fact he questioned the legitimacy of traffic as well.
- MikeCapone 17y agoThere are probably a lot more hosting company employees out there than there are HN regulars (or even occasional) users.
- ramchip 17y agoI don't really see why every hosting company employee out there is supposed to know about a certain small american investment firm started in 2005 and its associated news aggregator... we're probably an order of magnitude or so less popular than Reddit, which is itself rather niche. It's a small dot in the web world.
- feverishaaron 17y ago"Biting the hand that feeds you" seems like an appropriate analogy for arbitrarily shutting down this site without warning.
- ck2 17y agoTo be fair, if I was another customer on that shared host and you made my side crawl through no fault of my own, I'd want your site disabled asap too. VPS or cloud is dirt cheap these days, no excuses for not planning growth?
- CGamesPlay 17y agoThe real WTF here is why HN considered it appropriate to run a production site on a shared host. Shared hosting is a bad idea all around if you are even slightly concerned about reliability. You don't control the server, so any 60 year old woman running a photoblog with a vulnerable WordPress version can bring your site down just by getting hacked.
- marcamillion 17y agoHrmm....HN was hosted on a shared server. That's pretty impressive..if I must say so myself.
- RyanMcGreal 17y agoThe email sounds pretty generic and policyish, which leads me to suspect it was auto-generated rather than typed out by some hapless sysadmin who's never heard of HN.
- InclinedPlane 17y agoThere are too many grammar errors for it to be auto-generated (I hope).
- mikestanley 17y agoKinda funny and sad that so many people here only see this as some sort of stupid or unfair action against HN, seemingly without even acknowledging that every single other customer on that shared server had as much right individually, and more right collectively, to not have their performance negatively impacted by HN. Yeah, it sucks that one of our favorite tech news sites was impacted by this, but how impacted were all those other customers? It is easy to make a smartaleck comment about how Pair was trying to upsell by doing this, which is preposterous. Pair is a well-respected provider with many more years providing good service at a fair price than HN has existed, and I'd be willing to bet will be around after HN has peaked and begins to move back to the traffic load that might make sense on a shared system. But the fault here ultimately lies with the folks running HN who thought it was wise or appropriate to host any of its content on a shared server that likely cost them less money per month than most of us spend on soft drinks in a week.
- euccastro 17y agoStrawman. Nobody's saying HN/YC should be allowed to overuse resources (then again, I don't know the terms they had agreed to). But not giving a warning is crappy customer service, no matter how many other folks do likewise, how many years Pair has been doing other stuff well, or however else you want to spin it. In this case, it also happens to be a big sales screwup.
- mikestanley 17y agoWell, you didn't identify what I said that you think is a strawman, but I'll point out yours. Fine, nobody's saying HN/YC should be allowed to overuse resources. I didn't say anybody was saying that. Not giving a warning probably doesn't count as great customer service, but then again, once the problem had been identified by Pair, and once they knew of the negative impact HN was having on every other paying customer on that server, what kind of customer service to those other customers would it have been for Pair to fire off an email to HN then wait an hour, or thirty minutes, or ten minutes, before shutting it down? How long should Pair have allowed HN to impact other customers to satisfy folks here? And what makes HN more important than any other paying customer on that server? Oh right, it's because you read and like HN, which, ironically, so do I. As for it being a sales screwup, maybe. I kinda doubt there is a great deal of overlap between HN readership and the average potential Pair customer. We could also suggest that Pair taking action to protect all those other customers on the server is an example of how they would provide good service to the many when they're being hammered by one overpowering fellow customer.
- anotherpaulg 17y agoThis makes me think of a little side project I've been chipping away at called InstaCDN. It makes it easy to minify, combine, gzip and push your css, js and image assets into the Amazon Cloudfront CDN with far-future expiration headers. It also automagically detects background images referenced in your css, and puts them in the CDN. It rewrites the css to use the new CDN image urls. It's all done through a trivial REST API. Would love some feedback, and to find out if/how it's breaking any of your complex css/js. http://www.instacdn.com/ http://www.instacdn.com/
- _pius 17y agoComplete failure on the part of Pair Networks.
- ivankirigin 17y agoIf I ran biz dev at _any other hosting company_ I would offer to host the site for free in exchange for a little logo on the bottom of the page.
- mstevens 17y agoI'd suggest this as a business opportunity, but I suspect non-sucky shared hosting is a hard sell.
- patio11 17y agoI typically don't tell other people how to run their businesses, but if a similar issue brought my website down and I were to post about the causes, I might focus more on my failures in capacity planning, vendor selection, and monitoring rather than on my vendor's lackluster customer service. User-visible failures are, ultimately, process failures on my part, regardless of the surface cause. A nice side effect of this philosophy is that improvements to my processes help with all sorts of surface causes whereas if I were to address surface causes individually it would be like playing whack-a-mole. Bad vendor whack, hard drive failed whack, traffic spike whack, poor customer service whack, out of memory exception whack whack whack -- why is everyone conspiring to keep me from getting any work done.
- davidw 17y agoOne obvious point is that HN is a peripheral part of YC's business, mostly aimed at recruiting, rather than something that's critical.
- deleted 17y ago[deleted]
- pg 17y agoIt didn't bring HN down. HN deliberately didn't rely on that server for anything except hosting static content that was also duplicated on this server. I planned in advance for the possibility that the other server wouldn't be usable, by writing the code so that I could switch to serving the same content off news by changing one variable, which I did. As a result service was barely affected. In short, Pair flaked, but we had in fact planned the system in a way that protected us against it.
- dnsworks 17y agoHN has been pretty sluggish for a little while now. If you rewrote it in Rails you could it host it on Heroku and get back to not worrying about scaling!
- alnayyir 17y agoPlease tell me you're not serious.