22 ms·

HN is up again

1010 points by tpmx 4y ago

edgartaor 4y ago
HN is so reliable that I distrusted my wife first. "Did you unplug the router?"
known 4y ago
pjbeam 4y ago
Just when I was starting to get some work done. Thank you!
thebitstick 4y ago
I nearly died... of boredom...
collegeburner 4y ago
Ong i couldnt stand being that productive so relieved
Pr0ject217 4y ago
Yay. Thank you HN.
deleted 4y ago
[deleted]
elcapitan 4y ago
Oh noes, now my productivity is in the toilet.
- bell-cot 4y ago
  127.0.0.1 news.ycombinator.com # REQUIRED for meaningful productivity
deleted 4y ago
[deleted]
jdoliner 4y ago
That was a rough 6 hours
- tpmx 4y ago
  It really was.
  - sabjut 4y ago
    Yeah, I had to _actually work_ for once. Glad that I can finally distract myself again :)
- hansword 4y ago
  I went for a long walk. When I came back, HN was still down. :(
- dboreham 4y ago
  Someone suggested lobste.rs, which I hadn't come across before. It needs an invite though so read-only today for me.
deleted 4y ago
[deleted]
crorella 4y ago
it is back! I wonder what happened.
- jmt_ 4y ago
  https://twitter.com/HNStatus/status/1545409429113229312 https://twitter.com/HNStatus/status/1545409429113229312 https://twitter.com/HNStatus/status/1545461511870566400 https://twitter.com/HNStatus/status/1545461511870566400 Disk and fallback server failure. Was definitely a long day for their ops team, on a Friday no less.
ricefield 4y ago
thank god
gruturo 4y ago
Thanks for all the daily work in running and keeping up the site, which we too easily take for granted, even those of us with similar jobs.
function_seven 4y ago
Everyone, you're welcome. My last F5 must've jogged it loose finally.
- proactivesvcs 4y ago
  You can't fool us, we know you're an F7.
- raydiatian 4y ago
  Thank you
- _8j50 4y ago
  Thank you for your service.
- deleted 4y ago
  [deleted]
- leeoniya 4y ago
  are you wearing a cape, or are you not?
- DamnInteresting 4y ago
  A refreshing take!
- gedy 4y ago
  Name almost checks out
- hguant 4y ago
  07
  - smarx007 4y ago
    007
dom96 4y ago
Hurray! Is this the longest downtime ever for HN? I don't remember it being down for this long in the past, anyone know?
deleted 4y ago
[deleted]
Jcampuzano2 4y ago
My routine is usually to check HN first thing in the morning when sitting down at my computer before work. I definitely spent a non-reasonable amount of time thinking my internet had a problem trying to open HN since it's always just been so constant.
ImpulseGuided 4y ago
Life felt so meaningless. Don't ever do this to me again.
ghostoftiber 4y ago
RCA? AAR? "Ooops someone tripped over the cord?"
joshstrange 4y ago
I don't think I had ever fully internalized how often I open this site throughout the day. Finish a task? HN. Got frustrated/stuck on a problem? HN break. Waiting for something to install/upload/compile/etc? HN. Needless to say I opened a new tab, typed "n", and hit enter countless times today before my brain caught up with my muscle memory.
- agnos 4y ago
  Wow, I've never realized just how often I do this. There's some sort of a reward pathway in my brain connecting my right index finger to seeing that orange bar and lines of text.
- pmoleri 4y ago
  That used to happen to me with Slashdot too. I'm so happy that today I didn't even realize HN was down, I think I tried once, it didn't load and went do something else, I assumed it was some local DNS or internet issue.
- bragr 4y ago
  noprocrast exists for a reason :)
  - slategruen 4y ago
    Does it work even when you're logged out?
    - timeon 4y ago
      Even if it did. Second browser will help with the urge. But it can help anyway. Some people just need the reminder.
- m4tthumphrey 4y ago
  I realised this today too. I imagine pretty much all of us are the same!
- beckingz 4y ago
haswell 4y ago
This downtime made me realize (again) how much I appreciate the kind of interesting topics that show up here, the depth of discussion, and a general attitude of good faith that (most) engage with here. I realized how little of this I find elsewhere in my life - whether through Reddit or even my IRL friend circles. This realization saddens me - I feel like I shouldn’t have to rely on HN so much to scratch this particular itch. Perhaps I need to get out more.
- telesilla 4y ago
  Enjoy it while it's here. When it's gone or you have to move on, we'll miss you too and that's the meaningful part of life.
- rco8786 4y ago
  > This realization saddens me - I feel like I shouldn’t have to rely on HN so much to scratch this particular itch. > Perhaps I need to get out more. Another way to look at it is that you have a particular set of interests and HN is the online outlet that serves those interests. There's nothing wrong with that, at all and you don't need to have multiple sources for it. No different than someone who likes to ride bikes owning one bike, or someone who likes to read going to the same local library every week for 10 years.
  - avalys 4y ago
    It is very different from your examples. Even if you only own one bike, there are innumerable others in existence and companies making new ones every day, if yours is destroyed or lost. Similarly, there are plenty of local libraries to choose from, even if your favorite one closes. Whereas, if HN closes, there is no equivalent replacement available.
    - rco8786 4y ago
      What I am saying is that you don't need to worry about any of that. Sure, if HN permanently shutters - you'll need to go find a replacement. But HN isn't going anywhere, as far as I am aware. You don't need redundancy for your online community/content consumption
alberth 4y ago
I’d love to learn more about HN hardware & software stack. I know old posts indicate it’s running on a low core count but high frequency Intel CPUs on FreeBSD and no database (just flat files). I wonder if it’s still the same.
- oblio 4y ago
  Ah, that's easy. HN is running on an old laptop from Viaweb. Arc is running under the pg user and it's used as the process supervisor. The actual web server is a VB app running on Linux through Wine. The flat files have been migrated to an MS Access DB, also running through Wine.
  - fartcannon 4y ago
    My employer is taking notes.
    - tbyehl 4y ago
      Even IBM is leaving Notes.
  - deleted 4y ago
    [deleted]
TIPSIO 4y ago
Went to Reddit instead today more than I should (not proud). Anyone been on Slashdot lately? Checked it out too was really nice.
- doublerabbit 4y ago
  Slashdot is a slurry of Reddit and HN. My goto when bored with reddit and hn...
- autoexec 4y ago
  I used to spend a lot of time there in the 2000s, but when they changed the site's layout and it suddenly needed a bunch of JS I stopped going. Maybe I'll give it a shot again though... the outage made me think I need more options.
anttiharju 4y ago
I've been so used to using hn as my "is internet working" page that for a while I thought my internet was just down and tried to troubleshoot it haha.
bob1029 4y ago
Even accounting for this outage, most other SAAS platforms still can't compete with HN's non-existent SLA. Thank you to everyone who keeps this thing running.
- vntok 4y ago
  > Even accounting for this outage, most other SAAS platforms still can't compete with HN's non-existent SLA. 8 hours of downtime in a given year is 99.9%, so only three nines. The major SaaS platforms all are basically at least as resilient as this, and most have more stringent SLAs.
  - metadat 4y ago
    Most SaaS platforms don't really measure it. They just "do their best". Plus, it's hard to quantify many cases because there is hard-down and soft-down (partial interruptions).
thamer 4y ago
The last post[1] before this one was posted at 12:45:10 UTC. This current post was made at 20:30:55 UTC, so that's a gap of 7 hours 45 minutes and 45 seconds. [1] https://news.ycombinator.com/item?id=32026548 https://news.ycombinator.com/item?id=32026548
- TimWolla 4y ago
  There's this comment which is newer: https://news.ycombinator.com/item?id=32026565 https://news.ycombinator.com/item?id=32026565. There's also comments on some test submission: https://news.ycombinator.com/item?id=32026568 https://news.ycombinator.com/item?id=32026568 that itself got deleted: https://news.ycombinator.com/item?id=32026567 https://news.ycombinator.com/item?id=32026567
- deleted 4y ago
  [deleted]
damsta 4y ago
It was really funny to read advices to move HN to cloud from all those "experts".
booleandilemma 4y ago
This was a rough one. I actually opened slashdot at one point.
- ddingus 4y ago
  Same Early on /. was amazing! Remember Cmdrtaco it all out, often taking us for the ride? Good times, frequently good discussion. HN has been better for years now, was better at inception, for the most part. /. has improved a bit. Good to see, or I caught it on a good day.
sillysaurusx 4y ago
HN was down because the failover server also failed: https://twitter.com/HNStatus/status/1545409429113229312 https://twitter.com/HNStatus/status/1545409429113229312 Double disk failure is improbable but not impossible. The most impressive thing is that there seems to be no dataloss, almost whatsoever. Whatever the backup system is, it seems rock solid.
- hunterb123 4y ago
  What was the test to determine the dataloss?
  - Cerium 4y ago
    I came to the same conclusion by observing that there are posts and comments from only eight hours ago.
    - jbverschoor 4y ago
      So that means dataloss.. Probably restored from backup. Good news for people who were banned, or for posts that didn't get enough momentum :) edit: Was restored from backup.. so def. dataloss
      - joshuamorton 4y ago
        > So that means dataloss.. Probably restored from backup. If the server went down at XX:XX, and the backup they restored from is also from XX:XX, there isn't dataloss. If the server was down for 8 hours, the last data being 8 hours old isn't dataloss, it's correct.
      - dang 4y ago
        8 hours of downtime, but not data loss, since there was no data to lose during the downtime. Last post before we went down (2022-07-08 12:46:04 UTC): https://news.ycombinator.com/item?id=32026565 https://news.ycombinator.com/item?id=32026565 First post once we were back up (2022-07-08 20:30:55 UTC): https://news.ycombinator.com/item?id=32026571 https://news.ycombinator.com/item?id=32026571 (hey, that's this thread! how'd you do that, tpmx?) So, 7h 45m of downtime. What we don't know is how many posts (or votes, etc.) happened after our last backup, and were therefore lost. The latest vote we have was at 2022-07-08 12:46:05 UTC, which is about the same as the last post. There can't be many lost posts or votes, though, because I checked HN Search (https://hn.algolia.com/ https://hn.algolia.com/) just before we brought HN back up, and their most recent comment and story were behind ours. That means our last backup on the ill-fated server was taken after the last API update (HN Search relies on our API), and the API gets updated every 30 seconds. I'm not saying that's a rock-solid argument, but it suggests that 30 seconds is an upper bound on how much data we lost.
jchw 4y ago
I assume this post will be flagged as off topic, but I actually just went to visit moments before it came back up. I didn't figure out that it wasn't an issue with my internet connection until I saw this post.
codegeek 4y ago
This was traumatic :). Felt like I had lost a part of me lol. Glad to be back.
cableshaft 4y ago
How many times did people refresh to check this today? I did at least four times. I might have a bit of a problem.
- beckingz 4y ago
  like 40 times. Probably more.
- marcosdumay 4y ago
  What? I refreshed way more than 4 times before I believed it was offline. At the early 00's, when Google went offline I wouldn't believe it, and go check my connection (even if I was fetching other sites at the same time). Looks like nowadays HN is in that place.
- timbit42 4y ago
  My RSS app refreshes every hour.
nickmyersdt 4y ago
The muscle memory that kept opening HN was strong today. Thank you for bringing it back.
Bayart 4y ago
Did you turn it off and on again ?
MrBlueIncognito 4y ago
Top post in just 3 minutes Edit: 100 points in 4 minutes
erwincoumans 4y ago
Thanks for fixing the site! Keep up the good work.
drpgq 4y ago
I was wondering if HN was running on Rogers
- UncleOxidant 4y ago
  Same here. HN is from Canada? Who knew?
gmiller123456 4y ago
Thought my RSS reader was broken because I came to the site and saw it was up, then saw this headline.
cryptoz 4y ago
Obligatory "can't wait for postmortem" comment
lnalx 4y ago
I finally started to have a social life... it didn't last.
CSMastermind 4y ago
A huge thank you to everyone who keeps this amazing site running :)
humanwhosits 4y ago
Spent so long debugging my home network.. Couldn't possibly have been HN that was the problem haha
- D-Coder 4y ago
  All sites broke on my machine this morning. I reset the router... and HN was still down. <sniff>
- pbhjpbhj 4y ago
  Same, I ended up updating my pihole which was long overdue. Just finished, loaded up HN and it worked - "huh, wonder what the problem was with my pihole" I thought ... well it needed doing anyway.
- tekacs 4y ago
  I always use https://isup.me https://isup.me (https://downforeveryoneorjustme.com/ https://downforeveryoneorjustme.com/) in these situations, to find out if I'm losing my mind. :)
onenukecourse 4y ago
I'm traveling, so I just assumed my internet had died instead and didn't refresh all day. I literally just found out HN was down by this post.
nyadesu 4y ago
Just installed a new router and was trying to browse HN to check if my internet connection was working properly, lol
anewpersonality 4y ago
This downtime is a wakeup call for many
- m0llusk 4y ago
  a call to go back to sleep for others
srj 4y ago
My first thought was that my own Internet was down. My second was that HN was somehow dependent on Rogers. Hard to go through a Friday afternoon without HN!
beckingz 4y ago
Previous post on the HN status twitter account is from June 2021. Over a year with no issues. Impressive. https://twitter.com/HNStatus https://twitter.com/HNStatus
- jve 4y ago
  Twitter doesn't bother with minor issues, but there were some for sure. This logs lesser ones: https://hn.hund.io/ https://hn.hund.io/
johnsutor 4y ago
Worst couple of hours of my life.
ithinkso 4y ago
I've realized that I check if my internet works by opening HN...
jeffbee 4y ago
I wonder if things like double disk failures and restores from backup make HNers of the "bare metal 4ever" tribe revisit their cloud hatred.
- wumpus 4y ago
  No.
onion2k 4y ago
I did some work. Please don't let this happen again.
O__________O 4y ago
Thank you Dang!! ____________ Related: https://www.newyorker.com/news/letter-from-silicon-valley/the-lonely-work-of-moderating-hacker-news/ https://www.newyorker.com/news/letter-from-silicon-valley/th... https://news.ycombinator.com/user?id=dang https://news.ycombinator.com/user?id=dang
jacquesm 4y ago
And all those poor project managers at the end of 2022 wondering what on earth they did right on the 8th of July that caused productivity to reach previously unthinkable heights.
Johnny555 4y ago
Will you be posting a postmortem? Not that I deserve or expect one from a free service, but because I enjoy reading postmortems from failures where both the primary and backup systems failed, I like to see what holes I might have in my own failover setup.
- dang 4y ago
  The postmortem is sort of dissolved into the bloodstream of these threads: Tell HN: HN Moved from M5 to AWS - https://news.ycombinator.com/item?id=32030400 https://news.ycombinator.com/item?id=32030400 - July 2022 (116 comments) Ask HN: What'd you do while HN was down? - https://news.ycombinator.com/item?id=32026639 https://news.ycombinator.com/item?id=32026639 - July 2022 (218 comments) HN is up again - https://news.ycombinator.com/item?id=32026571 https://news.ycombinator.com/item?id=32026571 - July 2022 (314 comments) I'd particularly look here: https://news.ycombinator.com/item?id=32026606 https://news.ycombinator.com/item?id=32026606 and here: https://news.ycombinator.com/item?id=32031025 https://news.ycombinator.com/item?id=32031025. If you scroll through my comments from today via https://news.ycombinator.com/comments?id=dang&next=32039936 https://news.ycombinator.com/comments?id=dang&next=32039936, there are additional details. (Sorry for recommending my own comments.) If you (or anyone) skim through that stuff and have a question that isn't answered there, I'd be happy to take a crack at it.
olingern 4y ago
While the naysayers will say, "Why isn't this in the cloud?," I think the response times and uptime of hackernews is really impressive. If anyone has a write-up of the infrastructure that runs HN, I would be interested. Maybe startups really can be run off of a rasberry pi
- pyb 4y ago
  AWS has had more outages than HN in recent times
  - mpyne 4y ago
    AWS also operates at a significantly larger scale. When was the last AWS outage due to two critical disks failing at the same time?
    - hosteur 4y ago
      Failures due to increasing complexity are still failures.
      - mpyne 4y ago
        Sure, but you're talking about all of AWS as if every customer is impacted when any part of AWS suffers a failure. But that's not the case, which makes it quite an apples/oranges comparison. But even comparing the apples to the oranges, this HN status page someone else pointed out https://hn.hund.io/ https://hn.hund.io/ seems to show that HN has had more than one outage in just the past month. All but today's and last night's being quite short, but still. Sometimes you need some extra complexity if you want to make it to zero downtime overall. That's not something the HN website needs but I think AWS is doing fine even if that's your point of comparison.
  - deleted 4y ago
    [deleted]
  - fartcannon 4y ago
    Agree. Same with places like github.
lkxijlewlf 4y ago
Feels snappier. :)
SilasX 4y ago
I think I found the last one before the outage: https://news.ycombinator.com/item?id=32026565 https://news.ycombinator.com/item?id=32026565 The ones after it are hours later and usually deleted, until this post (...71).
rob_c 4y ago
So, why and what lessons were learnt? We get to post about others having that problems
rr808 4y ago
Whenever I get frustrated by cloud complexity I wonder if its all worth it, as HN, stackoverflow, camelx3 etc are still on real servers. Maybe it is worth it after all.
- geoffeg 4y ago
  I've wondered about this a lot recently as I've spent a lot of time recently fighting with Terraform, Atlantis, CircleCI, github and AWS. The first half of my career was deploying code to various UNIX machines. When there was a problem I could login to that machine and use various (common) tools to diagnose the issue. I may not have had root but I was at least able to gain insight into what was likely the culprit. The interface was immediate and allowed quick iteration to test out a solution. It feels like we've lost a lot of that observability and immediacy with the cloud. It's not as easy to quickly understand the larger picture. You can understand the state of various services with the web console or command line tools but tracing a path through those services is much less obvious and efficient. I'm kind of nervous to even discuss this as I wonder if it's just my age showing, especially since I see very people mention this as one of the downsides of various cloud solutions. Maybe I'm just jaded?
  - rr808 4y ago
    Lol, same boat. Not sure if I'm the old wise guy who really knows his sh*t about what's important, or the old useless guy rambling and moaning in the corner.
  - bbkane 4y ago
    Oh my goodness yes. I had the "great" idea to use Azure Functions to do a task at work. It's **ing insane how difficult it is to specify an Azure Function all in code with reasonable CI/CD, AD permissions, logging, and dev/prod instances. I wrote about what it takes at https://www.bbkane.com/blog/azure-functions-with-terraform/ https://www.bbkane.com/blog/azure-functions-with-terraform/ but the experience really soured me on cloud services.
    - jiggawatts 4y ago
jonnycomputer 4y ago
Did global productivity go up today, or were we all hopelessly clicking refresh to see if HN was back up?
- bawolff 4y ago
  Rogers engineers too busy refreshing ;)
Sjonny 4y ago
HN was down?
nmajor25 4y ago
Phew, we're back!
nimbius 4y ago
but why was it down though
rvz 4y ago
No news reports, press coverage or the majority of world caring about this 'site' going down for longer than usual. Does that mean nothing of value was lost?
qwertox 4y ago
What a ride. HN should shut down once a month on purpose on a random working day just to allow us to recalibrate our inner compass a bit.
smn1234 4y ago
Not S3 static site hosting + DynamoDB?
Amfy 4y ago
Interesting though, HN is now hosted on AWS and no longer on bare metal (m5 hosting). % host news.ycombinator.com news.ycombinator.com has address 50.112.136.166 and also interesting: DNS TTL is set to 1 (one).
- bestinterest 4y ago
  Oh that is interesting, I guess they just spun up a beefy EC2 instance. I'm noticing slower performance, I used to get about <200ms for front page. Now it's 500ms-1s? Or is this placebo with my bias to thinking AWS is slow? NetRange: 50.112.0.0 - 50.112.255.255 CIDR: 50.112.0.0/16 NetName: AMAZON-EC2-USWESTOR NetHandle: NET-50-112-0-0-1 Parent: NET50 (NET-50-0-0-0-0) NetType: Direct Allocation OriginAS: AS14618 Organization: Amazon.com, Inc. (AMAZO-47)
  - doublerabbit 4y ago
    I hope it's temporary. Would hate HN to move to the "cloud" from bare metal.
    - dang 4y ago
      Us too.
    - GekkePrutser 4y ago
      This, totally. It's great that they were able to spin it up in the cloud for recovery purposes. But it's more legendary on a real server <3 Yes I'm old :P
  - deleted 4y ago
    [deleted]
cpdean 4y ago
Its weird how I saw this headline, thought to myself "Oh good! It's back!" before realizing that I was using HN to see this very headline...
jandrusk 4y ago
Had to take a bunch of Xanex to get through the data without HN.
divbzero 4y ago
Thank you HN admins for bringing the site back online with everything restored from backup. Thank you also for the 99.99% of the time that HN just runs and runs without issue.
jhatax 4y ago
It might just be my devices (iPhone, iPad), but the fonts for stories and comments have changed since the restore. Anyone else noticing this?
clintonwoo 4y ago
When HN went down I was reading the stuff that was up before it went down on https://remix.hnclone.win https://remix.hnclone.win GitHub page for that project: https://github.com/clintonwoo/hackernews-remix-react https://github.com/clintonwoo/hackernews-remix-react
throw7 4y ago
thanks!
aasasd 4y ago
You don't say.
droptablemain 4y ago
Hypothesis: a link from HN hit the front-page of HN and inception ensued. In short, HN got HN'd.
I_complete_me 4y ago
I really enjoyed the outage. Disclaimer: I did not cause it.
sam0x17 4y ago
> HN Is Up Again I'm not sure I believe you what is your source for this ;)
ru552 4y ago
grats on the reload
peter_retief 4y ago
What happened? wasn't resolving at all.
abarrak 4y ago
For me, HN is one of the default pages when browser's open.
ddingus 4y ago
Whew! Thanks Dang and company. I appreciate you all. @my HN peers: Have a great weekend and thank you all for being you. I learn a ton here and enjoy the perspectives often found on these pages. It is all high value.
afrcnc 4y ago
I'm sorry... I tripped.
ilikeitdark 4y ago
I've been noticing internet problems here and there all week, and was getting a little sketched out (is the heat waves, Russia, or alien attack:) so this really got me worried.
robotsquidward 4y ago
HN is my default 'is the internet working' site and that genuinely threw me for a loop across multiple devices today dealing with hotspots while our power was out.
caycep 4y ago
granted, this is probably the most exciting post here all day!
swat535 4y ago
Great! Just in time for the Friday evening to kick in as well. I was worried that I may actually have to go out and do things instead of lurking here this weekend..
zacharycohn 4y ago
I have a flakey internet connection. Because HN loads so fast, I use it to test if my internet is working/is back up after it disconnects. WELL TODAY WAS VERY INCONVENIENT LET ME TELL YOU! :)
Mockapapella 4y ago
Well on the plus side I got a lot done today
bevdecloud 4y ago
Didn't even notice
ahepp 4y ago
I read an article recently on avoiding fallback in distributed systems.[0] Is it more appropriate to call the strategy in this case fallback, or failover? Since the secondary server wasn't running in production until the first one failed, it sounds like fallback? Perhaps higher reliability strategies would have been instead of having a secondary server, just have more mirrored disks on the main server, to reduce the likelihood of the array being compromised? Alternatively, to run both the primary and secondary servers in production all the time. But that would presumably merely move the single point of failure to the proxy? [0] https://aws.amazon.com/builders-library/avoiding-fallback-in-distributed-systems/ https://aws.amazon.com/builders-library/avoiding-fallback-in...
hbn 4y ago
Prove it
akomtu 4y ago
Mr. dang, can I use the opportunity to suggest to turn the tiny upvote and downvote arrows into links, separated by sufficient distance? My fingertip is 15x larger than these arrows and it takes quite a bit of precision to hit the right one. I bet, half of upvotes and downvotes are erroneous for this reason.
- mkl 4y ago
  The "undown" or "unvote" link that appears tells you which one you hit. I appreciate how little space they take up. If you're doing it with touch just zoom in.
coding123 4y ago
Damn, and I got a LOT done today.
willhinsa 4y ago
I just figured my internet wasn't working! It's a testament to how well this site works!
bane 4y ago
Echoing lots of comments here, HN being down is a real bummer to my day. I feel less informed, less engaged, and frankly bored without it in my life. True appreciation to the team who works to keep it up, high-quality, and impactful.
unixhero 4y ago
Phewwwwwww, I seriously haven't been this worried since What.CD disappeared.
wawjgreen 4y ago
let's not forget that aside from content signal-to-noise ratio of the links, HN comments provide nearly half of the value because one can see the "other" side of the "story", sort of like visceral critical thinking...
soupbowl 4y ago
I was hoping they just screwed up dns for a short time while launching hacker news ipv6 support!
lambda_dn 4y ago
Hackernews, still using ancient tech, what a joke. It's trivial to host on a cloud provider and get 100% uptime for pennies.
- lta 4y ago
  My dear Sir/Madam, Would you please stop spamming your opinion ? You wrote it once, it's enough. Thank you very much <3
jonbaer 4y ago
Things learned today: HN is not listed on DD ... https://downdetector.com/search/?q=hacker+news https://downdetector.com/search/?q=hacker+news
- s1mon 4y ago
  https://twitter.com/HNStatus https://twitter.com/HNStatus
lambda_dn 4y ago
HN just lost 10% of it's users. Being down for hours is unforgivable in this day and age with cloud providers. Please get with the times, your a tech site for god sake!!!
- gitgud 4y ago
  > Being down for hours is unforgivable in this day and age with cloud providers. Wasn't cloud flare down for a few hours recently... Cloud providers don't magically fix outages...
- lta 4y ago
  Cloud providers mostly suck. Not that I deserve to feel that, I feel proud that HN was/is run from bare metal
boosteri 4y ago
Must admit it was really hard to find out what was going on. I stumbled upon a twitter link somewhere deep down a list of search results. Doe HN have a status page of any kind?
- bolasanibk 4y ago
  I believe this is the official twitter account: https://twitter.com/HNStatus https://twitter.com/HNStatus
mise_en_place 4y ago
Most productive day I’ve had in a while (I kid).
O__________O 4y ago
Appears HN is currently on AWS: https://check-host.net/ip-info?host=https://news.ycombinator.com/ https://check-host.net/ip-info?host=https://news.ycombinator... http://50.112.136.166/ http://50.112.136.166/ https://search.arin.net/rdap/?query=50.112.136.166 https://search.arin.net/rdap/?query=50.112.136.166 Note: HN has been on M5 hosting for years and they were still there as of 16-hours ago per Dang: https://news.ycombinator.com/item?id=32024105 https://news.ycombinator.com/item?id=32024105 During the outage, listed places to check HN related systems, posted them here: https://news.ycombinator.com/item?id=32029014 https://news.ycombinator.com/item?id=32029014
naikrovek 4y ago
oh thank god you put this here, I was unable to determine this for myself. I'm sorry, this kind of thing reeks of point "whoring" to me, and I consider that to be an indefensible thing to do; it's pollution. We can see the site. We know it's up. We don't need to be told. Stop doing things purely to increase your score. This isn't a game. Etc.
- deleted 4y ago
  [deleted]
stereoradonc 4y ago
I slept through the downtime on the other side of the planet.
okonomiyaki3000 4y ago
Prove it.
danellis 4y ago
Maybe I'm missing something, but what's the point of posting this here? Anyone will already know it's back up before they see this post.
magundu 4y ago
I thought HN is immortal. Visiting a HN is everyday morning ritual to me.
stevenjgarner 4y ago
Clearly a correlation between 2 hard drives failing at HN [1] and the Canadian Rogers network outage [2]. [1] https://twitter.com/HNStatus https://twitter.com/HNStatus [2] https://www.reuters.com/business/media-telecom/rogers-communications-services-down-thousands-users-downdetector-2022-07-08/ https://www.reuters.com/business/media-telecom/rogers-commun...
Chirael 4y ago
My first thought was “oh sh*t, they finally added this to the list of time-wasting social media blocked sites” :( It was only when I saw it also didn’t work on my phone, that I realized HN itself was actually down
forgotpwd16 4y ago
Problem with HN being down is can't check HN for whether HN is down.
Thetawaves 4y ago
Imagine running a website that earns you millions and millions of dollars and it all gets taken out by a couple disk failures? Amateurs.
- CRConrad 4y ago
  Imagine running a website that doesn't earn you millions and millions of dollars and it gets taken out for a mere eight hours by a couple disk failures. Perfectly acceptable.

HN is up again

1010 points by tpmx 4y ago

edgartaor 4y ago

HN is so reliable that I distrusted my wife first. "Did you unplug the router?"

known 4y ago

pjbeam 4y ago

Just when I was starting to get some work done. Thank you!

thebitstick 4y ago

I nearly died... of boredom...

collegeburner 4y ago

Ong i couldnt stand being that productive so relieved

Pr0ject217 4y ago

Yay. Thank you HN.

deleted 4y ago

[deleted]

elcapitan 4y ago

Oh noes, now my productivity is in the toilet.

bell-cot 4y ago
127.0.0.1 news.ycombinator.com # REQUIRED for meaningful productivity

deleted 4y ago

[deleted]

jdoliner 4y ago

That was a rough 6 hours

tpmx 4y ago
It really was.
- sabjut 4y ago
  Yeah, I had to _actually work_ for once. Glad that I can finally distract myself again :)
hansword 4y ago
I went for a long walk. When I came back, HN was still down. :(
dboreham 4y ago
Someone suggested lobste.rs, which I hadn't come across before. It needs an invite though so read-only today for me.

deleted 4y ago

[deleted]

crorella 4y ago

it is back! I wonder what happened.

jmt_ 4y ago
https://twitter.com/HNStatus/status/1545409429113229312 https://twitter.com/HNStatus/status/1545409429113229312 https://twitter.com/HNStatus/status/1545461511870566400 https://twitter.com/HNStatus/status/1545461511870566400 Disk and fallback server failure. Was definitely a long day for their ops team, on a Friday no less.

ricefield 4y ago

thank god

gruturo 4y ago

Thanks for all the daily work in running and keeping up the site, which we too easily take for granted, even those of us with similar jobs.

function_seven 4y ago

Everyone, you're welcome. My last F5 must've jogged it loose finally.

proactivesvcs 4y ago
You can't fool us, we know you're an F7.
raydiatian 4y ago
Thank you
_8j50 4y ago
Thank you for your service.
deleted 4y ago
[deleted]
leeoniya 4y ago
are you wearing a cape, or are you not?
DamnInteresting 4y ago
A refreshing take!
gedy 4y ago
Name almost checks out
hguant 4y ago
07
- smarx007 4y ago
  007

dom96 4y ago

Hurray! Is this the longest downtime ever for HN? I don't remember it being down for this long in the past, anyone know?

deleted 4y ago

[deleted]

Jcampuzano2 4y ago

My routine is usually to check HN first thing in the morning when sitting down at my computer before work. I definitely spent a non-reasonable amount of time thinking my internet had a problem trying to open HN since it's always just been so constant.

ImpulseGuided 4y ago

Life felt so meaningless. Don't ever do this to me again.

ghostoftiber 4y ago

RCA? AAR? "Ooops someone tripped over the cord?"

joshstrange 4y ago

I don't think I had ever fully internalized how often I open this site throughout the day. Finish a task? HN. Got frustrated/stuck on a problem? HN break. Waiting for something to install/upload/compile/etc? HN. Needless to say I opened a new tab, typed "n", and hit enter countless times today before my brain caught up with my muscle memory.

agnos 4y ago
Wow, I've never realized just how often I do this. There's some sort of a reward pathway in my brain connecting my right index finger to seeing that orange bar and lines of text.
pmoleri 4y ago
That used to happen to me with Slashdot too. I'm so happy that today I didn't even realize HN was down, I think I tried once, it didn't load and went do something else, I assumed it was some local DNS or internet issue.
bragr 4y ago
noprocrast exists for a reason :)
- slategruen 4y ago
  Does it work even when you're logged out?
  - timeon 4y ago
    Even if it did. Second browser will help with the urge. But it can help anyway. Some people just need the reminder.
m4tthumphrey 4y ago
I realised this today too. I imagine pretty much all of us are the same!
beckingz 4y ago

haswell 4y ago

This downtime made me realize (again) how much I appreciate the kind of interesting topics that show up here, the depth of discussion, and a general attitude of good faith that (most) engage with here. I realized how little of this I find elsewhere in my life - whether through Reddit or even my IRL friend circles. This realization saddens me - I feel like I shouldn’t have to rely on HN so much to scratch this particular itch. Perhaps I need to get out more.

telesilla 4y ago
Enjoy it while it's here. When it's gone or you have to move on, we'll miss you too and that's the meaningful part of life.
rco8786 4y ago
> This realization saddens me - I feel like I shouldn’t have to rely on HN so much to scratch this particular itch. > Perhaps I need to get out more. Another way to look at it is that you have a particular set of interests and HN is the online outlet that serves those interests. There's nothing wrong with that, at all and you don't need to have multiple sources for it. No different than someone who likes to ride bikes owning one bike, or someone who likes to read going to the same local library every week for 10 years.
- avalys 4y ago
  It is very different from your examples. Even if you only own one bike, there are innumerable others in existence and companies making new ones every day, if yours is destroyed or lost. Similarly, there are plenty of local libraries to choose from, even if your favorite one closes. Whereas, if HN closes, there is no equivalent replacement available.
  - rco8786 4y ago
    What I am saying is that you don't need to worry about any of that. Sure, if HN permanently shutters - you'll need to go find a replacement. But HN isn't going anywhere, as far as I am aware. You don't need redundancy for your online community/content consumption

alberth 4y ago

I’d love to learn more about HN hardware & software stack. I know old posts indicate it’s running on a low core count but high frequency Intel CPUs on FreeBSD and no database (just flat files). I wonder if it’s still the same.

oblio 4y ago
Ah, that's easy. HN is running on an old laptop from Viaweb. Arc is running under the pg user and it's used as the process supervisor. The actual web server is a VB app running on Linux through Wine. The flat files have been migrated to an MS Access DB, also running through Wine.
- fartcannon 4y ago
  My employer is taking notes.
  - tbyehl 4y ago
    Even IBM is leaving Notes.
- deleted 4y ago
  [deleted]

TIPSIO 4y ago

Went to Reddit instead today more than I should (not proud). Anyone been on Slashdot lately? Checked it out too was really nice.

doublerabbit 4y ago
Slashdot is a slurry of Reddit and HN. My goto when bored with reddit and hn...
autoexec 4y ago
I used to spend a lot of time there in the 2000s, but when they changed the site's layout and it suddenly needed a bunch of JS I stopped going. Maybe I'll give it a shot again though... the outage made me think I need more options.

anttiharju 4y ago

I've been so used to using hn as my "is internet working" page that for a while I thought my internet was just down and tried to troubleshoot it haha.

bob1029 4y ago

Even accounting for this outage, most other SAAS platforms still can't compete with HN's non-existent SLA. Thank you to everyone who keeps this thing running.

vntok 4y ago
> Even accounting for this outage, most other SAAS platforms still can't compete with HN's non-existent SLA. 8 hours of downtime in a given year is 99.9%, so only three nines. The major SaaS platforms all are basically at least as resilient as this, and most have more stringent SLAs.
- metadat 4y ago
  Most SaaS platforms don't really measure it. They just "do their best". Plus, it's hard to quantify many cases because there is hard-down and soft-down (partial interruptions).

thamer 4y ago

The last post[1] before this one was posted at 12:45:10 UTC. This current post was made at 20:30:55 UTC, so that's a gap of 7 hours 45 minutes and 45 seconds. [1] https://news.ycombinator.com/item?id=32026548 https://news.ycombinator.com/item?id=32026548

TimWolla 4y ago
There's this comment which is newer: https://news.ycombinator.com/item?id=32026565 https://news.ycombinator.com/item?id=32026565. There's also comments on some test submission: https://news.ycombinator.com/item?id=32026568 https://news.ycombinator.com/item?id=32026568 that itself got deleted: https://news.ycombinator.com/item?id=32026567 https://news.ycombinator.com/item?id=32026567
deleted 4y ago
[deleted]

damsta 4y ago

It was really funny to read advices to move HN to cloud from all those "experts".

booleandilemma 4y ago

This was a rough one. I actually opened slashdot at one point.

ddingus 4y ago
Same Early on /. was amazing! Remember Cmdrtaco it all out, often taking us for the ride? Good times, frequently good discussion. HN has been better for years now, was better at inception, for the most part. /. has improved a bit. Good to see, or I caught it on a good day.

sillysaurusx 4y ago

HN was down because the failover server also failed: https://twitter.com/HNStatus/status/1545409429113229312 https://twitter.com/HNStatus/status/1545409429113229312 Double disk failure is improbable but not impossible. The most impressive thing is that there seems to be no dataloss, almost whatsoever. Whatever the backup system is, it seems rock solid.

hunterb123 4y ago
What was the test to determine the dataloss?
- Cerium 4y ago
  I came to the same conclusion by observing that there are posts and comments from only eight hours ago.
  - jbverschoor 4y ago
    So that means dataloss.. Probably restored from backup. Good news for people who were banned, or for posts that didn't get enough momentum :) edit: Was restored from backup.. so def. dataloss
    - joshuamorton 4y ago
      > So that means dataloss.. Probably restored from backup. If the server went down at XX:XX, and the backup they restored from is also from XX:XX, there isn't dataloss. If the server was down for 8 hours, the last data being 8 hours old isn't dataloss, it's correct.
    - dang 4y ago
      8 hours of downtime, but not data loss, since there was no data to lose during the downtime. Last post before we went down (2022-07-08 12:46:04 UTC): https://news.ycombinator.com/item?id=32026565 https://news.ycombinator.com/item?id=32026565 First post once we were back up (2022-07-08 20:30:55 UTC): https://news.ycombinator.com/item?id=32026571 https://news.ycombinator.com/item?id=32026571 (hey, that's this thread! how'd you do that, tpmx?) So, 7h 45m of downtime. What we don't know is how many posts (or votes, etc.) happened after our last backup, and were therefore lost. The latest vote we have was at 2022-07-08 12:46:05 UTC, which is about the same as the last post. There can't be many lost posts or votes, though, because I checked HN Search (https://hn.algolia.com/ https://hn.algolia.com/) just before we brought HN back up, and their most recent comment and story were behind ours. That means our last backup on the ill-fated server was taken after the last API update (HN Search relies on our API), and the API gets updated every 30 seconds. I'm not saying that's a rock-solid argument, but it suggests that 30 seconds is an upper bound on how much data we lost.

jchw 4y ago

I assume this post will be flagged as off topic, but I actually just went to visit moments before it came back up. I didn't figure out that it wasn't an issue with my internet connection until I saw this post.

codegeek 4y ago

This was traumatic :). Felt like I had lost a part of me lol. Glad to be back.

cableshaft 4y ago

How many times did people refresh to check this today? I did at least four times. I might have a bit of a problem.

beckingz 4y ago
like 40 times. Probably more.
marcosdumay 4y ago
What? I refreshed way more than 4 times before I believed it was offline. At the early 00's, when Google went offline I wouldn't believe it, and go check my connection (even if I was fetching other sites at the same time). Looks like nowadays HN is in that place.
timbit42 4y ago
My RSS app refreshes every hour.

nickmyersdt 4y ago

The muscle memory that kept opening HN was strong today. Thank you for bringing it back.

Bayart 4y ago

Did you turn it off and on again ?

MrBlueIncognito 4y ago

Top post in just 3 minutes Edit: 100 points in 4 minutes

erwincoumans 4y ago

Thanks for fixing the site! Keep up the good work.

drpgq 4y ago

I was wondering if HN was running on Rogers

UncleOxidant 4y ago
Same here. HN is from Canada? Who knew?

gmiller123456 4y ago

Thought my RSS reader was broken because I came to the site and saw it was up, then saw this headline.

cryptoz 4y ago

Obligatory "can't wait for postmortem" comment

lnalx 4y ago

I finally started to have a social life... it didn't last.

CSMastermind 4y ago

A huge thank you to everyone who keeps this amazing site running :)

humanwhosits 4y ago

Spent so long debugging my home network.. Couldn't possibly have been HN that was the problem haha

D-Coder 4y ago
All sites broke on my machine this morning. I reset the router... and HN was still down. <sniff>
pbhjpbhj 4y ago
Same, I ended up updating my pihole which was long overdue. Just finished, loaded up HN and it worked - "huh, wonder what the problem was with my pihole" I thought ... well it needed doing anyway.
tekacs 4y ago
I always use https://isup.me https://isup.me (https://downforeveryoneorjustme.com/ https://downforeveryoneorjustme.com/) in these situations, to find out if I'm losing my mind. :)

onenukecourse 4y ago

I'm traveling, so I just assumed my internet had died instead and didn't refresh all day. I literally just found out HN was down by this post.

nyadesu 4y ago

Just installed a new router and was trying to browse HN to check if my internet connection was working properly, lol

anewpersonality 4y ago

This downtime is a wakeup call for many

m0llusk 4y ago
a call to go back to sleep for others

srj 4y ago

My first thought was that my own Internet was down. My second was that HN was somehow dependent on Rogers. Hard to go through a Friday afternoon without HN!

beckingz 4y ago

Previous post on the HN status twitter account is from June 2021. Over a year with no issues. Impressive. https://twitter.com/HNStatus https://twitter.com/HNStatus

jve 4y ago
Twitter doesn't bother with minor issues, but there were some for sure. This logs lesser ones: https://hn.hund.io/ https://hn.hund.io/

johnsutor 4y ago

Worst couple of hours of my life.

ithinkso 4y ago

I've realized that I check if my internet works by opening HN...

jeffbee 4y ago

I wonder if things like double disk failures and restores from backup make HNers of the "bare metal 4ever" tribe revisit their cloud hatred.

wumpus 4y ago
No.

onion2k 4y ago

I did some work. Please don't let this happen again.

O__________O 4y ago

Thank you Dang!! ____________ Related: https://www.newyorker.com/news/letter-from-silicon-valley/the-lonely-work-of-moderating-hacker-news/ https://www.newyorker.com/news/letter-from-silicon-valley/th... https://news.ycombinator.com/user?id=dang https://news.ycombinator.com/user?id=dang

jacquesm 4y ago

And all those poor project managers at the end of 2022 wondering what on earth they did right on the 8th of July that caused productivity to reach previously unthinkable heights.

Johnny555 4y ago

Will you be posting a postmortem? Not that I deserve or expect one from a free service, but because I enjoy reading postmortems from failures where both the primary and backup systems failed, I like to see what holes I might have in my own failover setup.

dang 4y ago
The postmortem is sort of dissolved into the bloodstream of these threads: Tell HN: HN Moved from M5 to AWS - https://news.ycombinator.com/item?id=32030400 https://news.ycombinator.com/item?id=32030400 - July 2022 (116 comments) Ask HN: What'd you do while HN was down? - https://news.ycombinator.com/item?id=32026639 https://news.ycombinator.com/item?id=32026639 - July 2022 (218 comments) HN is up again - https://news.ycombinator.com/item?id=32026571 https://news.ycombinator.com/item?id=32026571 - July 2022 (314 comments) I'd particularly look here: https://news.ycombinator.com/item?id=32026606 https://news.ycombinator.com/item?id=32026606 and here: https://news.ycombinator.com/item?id=32031025 https://news.ycombinator.com/item?id=32031025. If you scroll through my comments from today via https://news.ycombinator.com/comments?id=dang&next=32039936 https://news.ycombinator.com/comments?id=dang&next=32039936, there are additional details. (Sorry for recommending my own comments.) If you (or anyone) skim through that stuff and have a question that isn't answered there, I'd be happy to take a crack at it.

olingern 4y ago

While the naysayers will say, "Why isn't this in the cloud?," I think the response times and uptime of hackernews is really impressive. If anyone has a write-up of the infrastructure that runs HN, I would be interested. Maybe startups really can be run off of a rasberry pi

pyb 4y ago
AWS has had more outages than HN in recent times
- mpyne 4y ago
  AWS also operates at a significantly larger scale. When was the last AWS outage due to two critical disks failing at the same time?
  - hosteur 4y ago
    Failures due to increasing complexity are still failures.
    - mpyne 4y ago
      Sure, but you're talking about all of AWS as if every customer is impacted when any part of AWS suffers a failure. But that's not the case, which makes it quite an apples/oranges comparison. But even comparing the apples to the oranges, this HN status page someone else pointed out https://hn.hund.io/ https://hn.hund.io/ seems to show that HN has had more than one outage in just the past month. All but today's and last night's being quite short, but still. Sometimes you need some extra complexity if you want to make it to zero downtime overall. That's not something the HN website needs but I think AWS is doing fine even if that's your point of comparison.
- deleted 4y ago
  [deleted]
- fartcannon 4y ago
  Agree. Same with places like github.

lkxijlewlf 4y ago

Feels snappier. :)

SilasX 4y ago

I think I found the last one before the outage: https://news.ycombinator.com/item?id=32026565 https://news.ycombinator.com/item?id=32026565 The ones after it are hours later and usually deleted, until this post (...71).

rob_c 4y ago

So, why and what lessons were learnt? We get to post about others having that problems

rr808 4y ago

Whenever I get frustrated by cloud complexity I wonder if its all worth it, as HN, stackoverflow, camelx3 etc are still on real servers. Maybe it is worth it after all.

geoffeg 4y ago
I've wondered about this a lot recently as I've spent a lot of time recently fighting with Terraform, Atlantis, CircleCI, github and AWS. The first half of my career was deploying code to various UNIX machines. When there was a problem I could login to that machine and use various (common) tools to diagnose the issue. I may not have had root but I was at least able to gain insight into what was likely the culprit. The interface was immediate and allowed quick iteration to test out a solution. It feels like we've lost a lot of that observability and immediacy with the cloud. It's not as easy to quickly understand the larger picture. You can understand the state of various services with the web console or command line tools but tracing a path through those services is much less obvious and efficient. I'm kind of nervous to even discuss this as I wonder if it's just my age showing, especially since I see very people mention this as one of the downsides of various cloud solutions. Maybe I'm just jaded?
- rr808 4y ago
  Lol, same boat. Not sure if I'm the old wise guy who really knows his sh*t about what's important, or the old useless guy rambling and moaning in the corner.
- bbkane 4y ago
  Oh my goodness yes. I had the "great" idea to use Azure Functions to do a task at work. It's **ing insane how difficult it is to specify an Azure Function all in code with reasonable CI/CD, AD permissions, logging, and dev/prod instances. I wrote about what it takes at https://www.bbkane.com/blog/azure-functions-with-terraform/ https://www.bbkane.com/blog/azure-functions-with-terraform/ but the experience really soured me on cloud services.
  - jiggawatts 4y ago

jonnycomputer 4y ago

Did global productivity go up today, or were we all hopelessly clicking refresh to see if HN was back up?

bawolff 4y ago
Rogers engineers too busy refreshing ;)

Sjonny 4y ago

HN was down?

nmajor25 4y ago

Phew, we're back!

nimbius 4y ago

but why was it down though

rvz 4y ago

No news reports, press coverage or the majority of world caring about this 'site' going down for longer than usual. Does that mean nothing of value was lost?

qwertox 4y ago

What a ride. HN should shut down once a month on purpose on a random working day just to allow us to recalibrate our inner compass a bit.

smn1234 4y ago

Not S3 static site hosting + DynamoDB?

Amfy 4y ago

Interesting though, HN is now hosted on AWS and no longer on bare metal (m5 hosting). % host news.ycombinator.com news.ycombinator.com has address 50.112.136.166 and also interesting: DNS TTL is set to 1 (one).

bestinterest 4y ago
Oh that is interesting, I guess they just spun up a beefy EC2 instance. I'm noticing slower performance, I used to get about <200ms for front page. Now it's 500ms-1s? Or is this placebo with my bias to thinking AWS is slow? NetRange: 50.112.0.0 - 50.112.255.255 CIDR: 50.112.0.0/16 NetName: AMAZON-EC2-USWESTOR NetHandle: NET-50-112-0-0-1 Parent: NET50 (NET-50-0-0-0-0) NetType: Direct Allocation OriginAS: AS14618 Organization: Amazon.com, Inc. (AMAZO-47)
- doublerabbit 4y ago
  I hope it's temporary. Would hate HN to move to the "cloud" from bare metal.
  - dang 4y ago
    Us too.
  - GekkePrutser 4y ago
    This, totally. It's great that they were able to spin it up in the cloud for recovery purposes. But it's more legendary on a real server <3 Yes I'm old :P
- deleted 4y ago
  [deleted]

cpdean 4y ago

Its weird how I saw this headline, thought to myself "Oh good! It's back!" before realizing that I was using HN to see this very headline...

jandrusk 4y ago

Had to take a bunch of Xanex to get through the data without HN.

divbzero 4y ago

Thank you HN admins for bringing the site back online with everything restored from backup. Thank you also for the 99.99% of the time that HN just runs and runs without issue.

jhatax 4y ago

It might just be my devices (iPhone, iPad), but the fonts for stories and comments have changed since the restore. Anyone else noticing this?

clintonwoo 4y ago

When HN went down I was reading the stuff that was up before it went down on https://remix.hnclone.win https://remix.hnclone.win GitHub page for that project: https://github.com/clintonwoo/hackernews-remix-react https://github.com/clintonwoo/hackernews-remix-react

throw7 4y ago

thanks!

aasasd 4y ago

You don't say.

droptablemain 4y ago

Hypothesis: a link from HN hit the front-page of HN and inception ensued. In short, HN got HN'd.

I_complete_me 4y ago

I really enjoyed the outage. Disclaimer: I did not cause it.

sam0x17 4y ago

> HN Is Up Again I'm not sure I believe you what is your source for this ;)

ru552 4y ago

grats on the reload

peter_retief 4y ago

What happened? wasn't resolving at all.

abarrak 4y ago

For me, HN is one of the default pages when browser's open.

ddingus 4y ago

Whew! Thanks Dang and company. I appreciate you all. @my HN peers: Have a great weekend and thank you all for being you. I learn a ton here and enjoy the perspectives often found on these pages. It is all high value.

afrcnc 4y ago

I'm sorry... I tripped.

ilikeitdark 4y ago

I've been noticing internet problems here and there all week, and was getting a little sketched out (is the heat waves, Russia, or alien attack:) so this really got me worried.

robotsquidward 4y ago

HN is my default 'is the internet working' site and that genuinely threw me for a loop across multiple devices today dealing with hotspots while our power was out.

caycep 4y ago

granted, this is probably the most exciting post here all day!

swat535 4y ago

Great! Just in time for the Friday evening to kick in as well. I was worried that I may actually have to go out and do things instead of lurking here this weekend..

zacharycohn 4y ago

I have a flakey internet connection. Because HN loads so fast, I use it to test if my internet is working/is back up after it disconnects. WELL TODAY WAS VERY INCONVENIENT LET ME TELL YOU! :)

Mockapapella 4y ago

Well on the plus side I got a lot done today

bevdecloud 4y ago

Didn't even notice

ahepp 4y ago

I read an article recently on avoiding fallback in distributed systems.[0] Is it more appropriate to call the strategy in this case fallback, or failover? Since the secondary server wasn't running in production until the first one failed, it sounds like fallback? Perhaps higher reliability strategies would have been instead of having a secondary server, just have more mirrored disks on the main server, to reduce the likelihood of the array being compromised? Alternatively, to run both the primary and secondary servers in production all the time. But that would presumably merely move the single point of failure to the proxy? [0] https://aws.amazon.com/builders-library/avoiding-fallback-in-distributed-systems/ https://aws.amazon.com/builders-library/avoiding-fallback-in...

hbn 4y ago

Prove it

akomtu 4y ago

Mr. dang, can I use the opportunity to suggest to turn the tiny upvote and downvote arrows into links, separated by sufficient distance? My fingertip is 15x larger than these arrows and it takes quite a bit of precision to hit the right one. I bet, half of upvotes and downvotes are erroneous for this reason.

mkl 4y ago
The "undown" or "unvote" link that appears tells you which one you hit. I appreciate how little space they take up. If you're doing it with touch just zoom in.

coding123 4y ago

Damn, and I got a LOT done today.

willhinsa 4y ago

I just figured my internet wasn't working! It's a testament to how well this site works!

bane 4y ago

Echoing lots of comments here, HN being down is a real bummer to my day. I feel less informed, less engaged, and frankly bored without it in my life. True appreciation to the team who works to keep it up, high-quality, and impactful.

unixhero 4y ago

Phewwwwwww, I seriously haven't been this worried since What.CD disappeared.

wawjgreen 4y ago

let's not forget that aside from content signal-to-noise ratio of the links, HN comments provide nearly half of the value because one can see the "other" side of the "story", sort of like visceral critical thinking...

soupbowl 4y ago

I was hoping they just screwed up dns for a short time while launching hacker news ipv6 support!

lambda_dn 4y ago

Hackernews, still using ancient tech, what a joke. It's trivial to host on a cloud provider and get 100% uptime for pennies.

lta 4y ago
My dear Sir/Madam, Would you please stop spamming your opinion ? You wrote it once, it's enough. Thank you very much <3

jonbaer 4y ago

Things learned today: HN is not listed on DD ... https://downdetector.com/search/?q=hacker+news https://downdetector.com/search/?q=hacker+news

s1mon 4y ago
https://twitter.com/HNStatus https://twitter.com/HNStatus

lambda_dn 4y ago

HN just lost 10% of it's users. Being down for hours is unforgivable in this day and age with cloud providers. Please get with the times, your a tech site for god sake!!!

gitgud 4y ago
> Being down for hours is unforgivable in this day and age with cloud providers. Wasn't cloud flare down for a few hours recently... Cloud providers don't magically fix outages...
lta 4y ago
Cloud providers mostly suck. Not that I deserve to feel that, I feel proud that HN was/is run from bare metal

boosteri 4y ago

Must admit it was really hard to find out what was going on. I stumbled upon a twitter link somewhere deep down a list of search results. Doe HN have a status page of any kind?

bolasanibk 4y ago
I believe this is the official twitter account: https://twitter.com/HNStatus https://twitter.com/HNStatus

mise_en_place 4y ago

Most productive day I’ve had in a while (I kid).

O__________O 4y ago

Appears HN is currently on AWS: https://check-host.net/ip-info?host=https://news.ycombinator.com/ https://check-host.net/ip-info?host=https://news.ycombinator... http://50.112.136.166/ http://50.112.136.166/ https://search.arin.net/rdap/?query=50.112.136.166 https://search.arin.net/rdap/?query=50.112.136.166 Note: HN has been on M5 hosting for years and they were still there as of 16-hours ago per Dang: https://news.ycombinator.com/item?id=32024105 https://news.ycombinator.com/item?id=32024105 During the outage, listed places to check HN related systems, posted them here: https://news.ycombinator.com/item?id=32029014 https://news.ycombinator.com/item?id=32029014

naikrovek 4y ago

oh thank god you put this here, I was unable to determine this for myself. I'm sorry, this kind of thing reeks of point "whoring" to me, and I consider that to be an indefensible thing to do; it's pollution. We can see the site. We know it's up. We don't need to be told. Stop doing things purely to increase your score. This isn't a game. Etc.

deleted 4y ago
[deleted]

stereoradonc 4y ago

I slept through the downtime on the other side of the planet.

okonomiyaki3000 4y ago

Prove it.

danellis 4y ago

Maybe I'm missing something, but what's the point of posting this here? Anyone will already know it's back up before they see this post.

magundu 4y ago

I thought HN is immortal. Visiting a HN is everyday morning ritual to me.

stevenjgarner 4y ago

Clearly a correlation between 2 hard drives failing at HN [1] and the Canadian Rogers network outage [2]. [1] https://twitter.com/HNStatus https://twitter.com/HNStatus [2] https://www.reuters.com/business/media-telecom/rogers-communications-services-down-thousands-users-downdetector-2022-07-08/ https://www.reuters.com/business/media-telecom/rogers-commun...

Chirael 4y ago

My first thought was “oh sh*t, they finally added this to the list of time-wasting social media blocked sites” :( It was only when I saw it also didn’t work on my phone, that I realized HN itself was actually down

forgotpwd16 4y ago

Problem with HN being down is can't check HN for whether HN is down.

Thetawaves 4y ago

Imagine running a website that earns you millions and millions of dollars and it all gets taken out by a couple disk failures? Amateurs.

CRConrad 4y ago
Imagine running a website that doesn't earn you millions and millions of dollars and it gets taken out for a mere eight hours by a couple disk failures. Perfectly acceptable.