7 ms·

Update provided > Welp, figured out why - because I am idiot. Routine code change exploded performance. Only took me 2 hours to think of reverting it. Sorry!

Update provided > Welp, figured out why - because I am idiot. Routine code change exploded performance. Only took me 2 hours to think of reverting it. Sorry! https://mobile.twitter.com/HNStatus/status/1402011274104360960 https://mobile.twitter.com/HNStatus/status/14020112741043609...

bryan_w 5y ago
If they are reading this and need an SRE to help out let me know. We are really good at reverting the shit out of an release when it breaks prod.
- jrockway 5y ago
  My team has also become good at that. We do "game days" where we push a simulated breakage and the team tries to fix it, but after a few "faulty" code pushes, everyone checks the rollout history for recent code or configuration changes and suggests a rollback as the first step. Having this log makes it very hard to break things! This just means the breakers have to be more clever. My favorites are taking a global lock in Postgres (all reads and writes instantly stop, and someone has to figure out how to kill the session that holds the lock), and installing a service worker that blocked traffic to our domain. (You can roll back, but the service worker is still there!)
  - Guest42 5y ago
    Does fixing constitute bringing back everything to the state things were at when the global lock was taken or does it also need to handle requests that occurred after? (I’m more of a db person with a small understanding of web apps)
    - jrockway 5y ago
      Fixing restores the ability of the application to handle future requests. Past requests were already replied to with an error.
MontyCarloHall 5y ago
Besides applying security patches, what exactly has needed to be changed in the HN codebase over the last 14 years? Featurewise, it's been essentially static for its whole existence (which is a large part of its appeal to many, including myself).
- xeromal 5y ago
  If the number of users has grown over the last 14 years, I'm sure some of the code areas have needed to be tuned over 14 years as well.
  - deleted 5y ago
    [deleted]
- jolmg 5y ago
  I think I've noticed changes a few times. For example, - the relative timestamps would go up to days, so you'd see e.g. "3121 days ago", and now they're replaced with dates at some point. - we're now able to escape * in comments. It used to be impossible (?) to use asterisks with consecutive characters, so you'd either have to put spaces around it or substitute with something like <asterisk>. This was unfortunate for inline code. - code blocks no longer scroll horizontally and wrap instead. This is my favorite change because many code blocks were actually indented bullet paragraphs. - the "past" link in the navbar was added - the subthread collapse link [-] used to use a - (ASCII 0x2d) and now it uses – (Unicode U+2013). This is my least favorite change because I used to use that for keyboard navigation. It was pretty convenient to Ctrl+F [- <Enter>... to navigate between comments then <Esc> <Enter> to collapse a thread and Ctrl+F <Enter>... to keep navigating. These are relatively recent changes.
  - viraptor 5y ago
    There's also the "second chance" list (few months ago?)
    - jolmg 5y ago

bryan_w 5y ago

If they are reading this and need an SRE to help out let me know. We are really good at reverting the shit out of an release when it breaks prod.

jrockway 5y ago
My team has also become good at that. We do "game days" where we push a simulated breakage and the team tries to fix it, but after a few "faulty" code pushes, everyone checks the rollout history for recent code or configuration changes and suggests a rollback as the first step. Having this log makes it very hard to break things! This just means the breakers have to be more clever. My favorites are taking a global lock in Postgres (all reads and writes instantly stop, and someone has to figure out how to kill the session that holds the lock), and installing a service worker that blocked traffic to our domain. (You can roll back, but the service worker is still there!)
- Guest42 5y ago
  Does fixing constitute bringing back everything to the state things were at when the global lock was taken or does it also need to handle requests that occurred after? (I’m more of a db person with a small understanding of web apps)
  - jrockway 5y ago
    Fixing restores the ability of the application to handle future requests. Past requests were already replied to with an error.

MontyCarloHall 5y ago

Besides applying security patches, what exactly has needed to be changed in the HN codebase over the last 14 years? Featurewise, it's been essentially static for its whole existence (which is a large part of its appeal to many, including myself).

xeromal 5y ago
If the number of users has grown over the last 14 years, I'm sure some of the code areas have needed to be tuned over 14 years as well.
- deleted 5y ago
  [deleted]
jolmg 5y ago
I think I've noticed changes a few times. For example, - the relative timestamps would go up to days, so you'd see e.g. "3121 days ago", and now they're replaced with dates at some point. - we're now able to escape * in comments. It used to be impossible (?) to use asterisks with consecutive characters, so you'd either have to put spaces around it or substitute with something like <asterisk>. This was unfortunate for inline code. - code blocks no longer scroll horizontally and wrap instead. This is my favorite change because many code blocks were actually indented bullet paragraphs. - the "past" link in the navbar was added - the subthread collapse link [-] used to use a - (ASCII 0x2d) and now it uses – (Unicode U+2013). This is my least favorite change because I used to use that for keyboard navigation. It was pretty convenient to Ctrl+F [- <Enter>... to navigate between comments then <Esc> <Enter> to collapse a thread and Ctrl+F <Enter>... to keep navigating. These are relatively recent changes.
- viraptor 5y ago
  There's also the "second chance" list (few months ago?)
  - jolmg 5y ago