13 ms·
Server is smoking, trying to figure out why
- Hublium 5y ago> It's a difficult habit to quit LOL
- bilalq 5y agoI interpreted this much more literally. When the updated tweet about a code revert fixing it came out, it took me a second to figure out what was meant.
- AceJohnny2 5y agohttps://en.wikipedia.org/wiki/Halt_and_Catch_Fire_(computing) https://en.wikipedia.org/wiki/Halt_and_Catch_Fire_(computing...
- seryoiupfurds 5y ago> Something went wrong. Try reloading. Seems like Twitter isn't doing much better.
- nomoreplease 5y agoUpdate provided > Welp, figured out why - because I am idiot. Routine code change exploded performance. Only took me 2 hours to think of reverting it. Sorry! https://mobile.twitter.com/HNStatus/status/1402011274104360960 https://mobile.twitter.com/HNStatus/status/14020112741043609...
- bryan_w 5y agoIf they are reading this and need an SRE to help out let me know. We are really good at reverting the shit out of an release when it breaks prod.
- jrockway 5y agoMy team has also become good at that. We do "game days" where we push a simulated breakage and the team tries to fix it, but after a few "faulty" code pushes, everyone checks the rollout history for recent code or configuration changes and suggests a rollback as the first step. Having this log makes it very hard to break things! This just means the breakers have to be more clever. My favorites are taking a global lock in Postgres (all reads and writes instantly stop, and someone has to figure out how to kill the session that holds the lock), and installing a service worker that blocked traffic to our domain. (You can roll back, but the service worker is still there!)
- Guest42 5y agoDoes fixing constitute bringing back everything to the state things were at when the global lock was taken or does it also need to handle requests that occurred after? (I’m more of a db person with a small understanding of web apps)
- jrockway 5y agoFixing restores the ability of the application to handle future requests. Past requests were already replied to with an error.
- MontyCarloHall 5y agoBesides applying security patches, what exactly has needed to be changed in the HN codebase over the last 14 years? Featurewise, it's been essentially static for its whole existence (which is a large part of its appeal to many, including myself).
- xeromal 5y agoIf the number of users has grown over the last 14 years, I'm sure some of the code areas have needed to be tuned over 14 years as well.
- deleted 5y ago[deleted]
- jolmg 5y agoI think I've noticed changes a few times. For example, - the relative timestamps would go up to days, so you'd see e.g. "3121 days ago", and now they're replaced with dates at some point. - we're now able to escape * in comments. It used to be impossible (?) to use asterisks with consecutive characters, so you'd either have to put spaces around it or substitute with something like <asterisk>. This was unfortunate for inline code. - code blocks no longer scroll horizontally and wrap instead. This is my favorite change because many code blocks were actually indented bullet paragraphs. - the "past" link in the navbar was added - the subthread collapse link [-] used to use a - (ASCII 0x2d) and now it uses – (Unicode U+2013). This is my least favorite change because I used to use that for keyboard navigation. It was pretty convenient to Ctrl+F [- <Enter>... to navigate between comments then <Esc> <Enter> to collapse a thread and Ctrl+F <Enter>... to keep navigating. These are relatively recent changes.
- jensenbox 5y agoNicotine is a hell of a drug.
- raverbashing 5y agoIf you think about it, it's impressive how such a high traffic site works out using a niche language which what I assume is not such a big ecosystem. (On the other hand there's moderate complexity but low page sizes and nothing like modern websites with hundreds of cookies, heavy js, etc)
- smegcicle 5y agosomething about deep comment thread folding?
- 1vuio0pswjnm7 5y agoYesterday I saw a strange artifact in a comment. Instead of the usual "[-]", I saw "[2 more]". Did not seem right.
- extra88 5y agoThat's what it says after you use the [-] to collapse a comment and it has a single reply. It also counts replies to replies (and so on).
- deleted 5y ago[deleted]
- 1vuio0pswjnm7 5y agoBut I do not use Javascript and I only ever see "[-]". I use a text-only browser and all comments are left-justified. A flat view, no hidden comments. Looks very different than in a browser that runs Javascript. Seeing "[2 more]" seemed unusual.
- cesarb 5y agoI also saw that yesterday. What's worse, the only way to expand the comments hidden behind that seems to be to temporarily unblock Javascript, reload the page, and click on that.
- jscipione 5y agoBeOS came up with an function for this: https://www.haiku-os.org/legacy-docs/bebook/TheKernelKit_SystemInfo.html#is_computer_on_fire https://www.haiku-os.org/legacy-docs/bebook/TheKernelKit_Sys... but “Smoldering doesn't count.”
- ballenf 5y agoI really thought it was going to be someone complaining about their restaurant server having a smoke break. Thought it must have been covid related to make the front page. But don't ask me how.
- bombcar 5y agoI had an older Dell give up the magic smoke recently and literally - https://i.imgur.com/WHYODQS.jpeg https://i.imgur.com/WHYODQS.jpeg Amusingly enough I smelled it weeks ago and it finally shut off a few days ago.
- lamontcg 5y agoYeah I've been paged before due to a crashed server failing ping and found out the datacenter tech was already on the job because it was suffering from actual smoking. I was remote so I don't know what they did, presumably pull the power and hit it with a CO2 extinguisher?
- userbinator 5y agoThose are buck converters, the MOSFETs in them tend to die either due to overheating or being run at the edge of design limits for long enough. It's rare that they die excitingly, however.
- oxfordmale 5y agoDid you try nicotine patches?
- loves_mangoes 5y ago...nicotine? =)
- el_benhameen 5y agoIf it’s any consolation, I thought my wireless provider had booted me before I realized HN could be having issues. It’s generally so fast and reliable that I use it as my “is my internet broken?” sanity check page.
- lapp0 5y agoAhahah, similar situation here. I thought maybe our IT department had noticed my activity here and decided to throttle my connection (which would still be an odd solution instead of just outright blocking it), but still it took me a few minutes to suspect that maybe the problem was with HN itself.
- whoisburbansky 5y agoAlso assumed this, was only convinced otherwise when I ran into the same issue on my personal mobile phone, on a data network separate from my work network.
- masswerk 5y agoSame here. :-)
- jtbayly 5y agoIt was worse for me, as I had just replaced my router!
- phendrenad2 5y ago"Smoking" meaning running high CPU, presumably, not literally smoldering
- vlmutolo 5y agoWas the last outage really in 2015
- Izkata 5y agoNo, but three months ago it wasn't their fault: https://news.ycombinator.com/item?id=26469116 https://news.ycombinator.com/item?id=26469116
- jpeter 5y agoI thought Replit started ddosing the site
- shkkmo 5y agoMy google foo failed and I wasn't able to find the correct @hnstatus account when I was having trouble. Perhaps a ”status" link in the footer would make this easier to find?
- sfeng 5y agoI don’t mean to be rude, but is HN a refutation of the Lispy vision it was meant to demonstrate? There has been a bug where comment pages can only show a limited number of entries for years, and the wonderful mods like dang continually report it is being worked on. It’s hard for me to believe it wouldn’t be fixed much faster if it occurred in a more practical language.
- Waterluvian 5y agoWhat's the bug? Isn't there a "more..." At the bottom of those long comment pages?
- tzs 5y agoI wouldn't necessarily call it a bug, but the way the pagination works does not work well for active threads which is probably why they want to replace it at some point with code than can handle long threads on a single page. A problem with pagination as implemented on HN is that the algorithm to display page N seems to be functionally (no pun intended!) equivalent to this: Render all the comments as one long virtual page Split it into smaller pages Render the Nth page from that set of smaller pages Note! I'm not saying that is how it works. Just that it produces the same result as that. So you read page 1 of an active discussion. You get to the end of the page and hit "more". But while you were reading page 1, other people were commenting, and other people were voting. Now, when it goes to render page 2 for you other comments have moved to the top, pushing the stuff you already read on page 1 down to page 2. Or maybe what was just after page 1 when you were reading page 1 has moved up due to voting, and when you go to page 2 you end up past it. It's like if you had a book that went through several editions, with significant changes between editions (including rearranging the order of chapters), and someone put together a printing by taking the first 10 pages from the first edition, the second 10 pages from the second edition, and so on. This could be fixed while retaining pagination by saving a timestamp when you go to page 1, and then when you go to subsequent pages use a snapshot of the comment database from that timestamp instead of the current comments. If you want to update to the current comments you'd have to hit refresh or go back to page 1. Another approach while retaining pagination would be keep track of what top level comments you've seen. If when rendering page N it sees that due to comment order changes one of the comments that is going to end on the page is one you've already seen it could omit it. Or maybe omit it unless the tree starting at that comment has changed since you last saw it. (If going this route, I'd also suggesting adding something to the UI to highlight what has changed in the tree and/or dehighlight what you've already seen). That second approach could still be annoying. You could still get stuck on a single comment that is getting a lot of relies. That could be fixed, but then you are getting into probably needing to add new controls or settings. I'm sure they have thought of both of those approaches and more, but probably want to keep the interface clean and simple and retain minimal state, hence the plan to someday make it handle very long comment threads on a single page.
- kristianp 5y agoI wonder what database or file schema they use. Many would find that interesting, given the scale that hn has reached.
- NullPrefix 5y agoOh how ironic. Twitter errored out with > Something went wrong. Try reloading.