6 ms·

That's definitely very fair! You could throw it into the constructor of your app right after you init log4j, at least. I've been surprised (and horrified) to l

That's definitely very fair! You could throw it into the constructor of your app right after you init log4j, at least. I've been surprised (and horrified) to learn that there are companies out there that _can't reboot their servers_ because they simply don't know how to bring the app back up. For them... live patching is the only options. Blows my mind, but that's reality (apparently)!

NaturalPhallacy 5y ago
I would run screaming from such an organization.
- stickfigure 5y ago
  I run screaming towards such organizations. Higher bill rates!
  - hvgk 5y ago
    This person gets it. As always if it’s a shit show, charge by the hour and get double time! If it’s easy charge by the job and spend two months getting paid to play solitaire!
- shakna 5y ago
  I've helped out orgs where they had a listing of servers, but couldn't tell me what purpose, if any, they served. You have multiple entire architecture rebuilds, deprecating various parts, whilst people come and go and take their knowledge with them leaving documentation behind that is outdated enough to be more harmful than helpful. Reality is a messy thing.
  - kubanczyk 5y ago
    Usually there is no universal way to find the latest version of internal documentation across teams, hence updating such documentation is barely worth the time. People notice the fact.
    - nefitty 5y ago
      In your mind, do hosted wikis and KBs like Stack Exchange, Wikimedia or Confluence address the discovery problem? If not, is there anything close?
      - nitrogen 5y ago
        Not the OP, but I am a fan of leaving breadcrumbs everywhere. That is, every bit of documentation should point to other bits of documentation. Code comments point to the corporate wiki and vice versa, commit messages point to issues, issues refer to commit hashes, log messages might say who wrote a service, bespoke server and container images could have a README in /, internal DNS can have text records, etc. We have to accept that we will sometimes play digital archaeologist. That's a good thing! It means that the technology we're studying was successful. So given that, it's better to have out of date documentation than zero documentation, so that we can pick up the trail somewhere.
  - nisegami 5y ago
    In cases like these, I'm a strong proponent of turning it off and seeing who yells.
    - shakna 5y ago
      In an org that named their servers after planets, we turned off "Mars" (which was actually completely and utterly undocumented) and discovered it was the bridge between the intranet and the internet. However, upon turning it back on... Nothing came up again. There were no logs to try and work out which processes needed to launch for it to do its job, causing several days of website downtime (not a simple website - dozens of interconnected services). We ended up taking the phone off the hook because we kept getting calls of "is it fixed yet?"
      - jatone 5y ago
        future reference: next time just firewall the network and see what blows up rofl.
        hamburglar 5y ago
        A bit of network analysis to identify services that are being called before turning things off would probably be wise, too.
        shakna 5y ago
        The story with Mars is quite a bit more complicated than I've written, but we did scan it, and then try and isolate it, for a few weeks before we tried the restart. Let's just say there was a lot of redundancy that only broke once it couldn't ping at a very specific time.
iovrthoughtthis 5y ago
the pyramids of our time
plafl 5y ago
> _can't reboot their servers_ because they simply don't know how to bring the app back up A hundred years later IT priests will demand human sacrifices to appease the server gods
- HowTheStoryEnds 5y ago
  So firms will need Ceremoniously Executed Offerings, eh?
staticassertion 5y ago
Server uptime used to be a point of pride. Now people try to shut them down as often as they can. Funny how things change.
- viraptor 5y ago
  We moved a lot of things from "that one server that's setup just-so by hand" to automated and repeatable setups. If we had more/ cheaper servers before, we'd likely get there earlier.
- Sitio 5y ago
  My server had 80kbyte bandwidth and doing very little in a much slower time. Kernel patches were not on top of my head 15 years ago.
- Bluecobra 5y ago
  Indeed, I remember going though great lengths to keep the Solaris hosts up that I used to administer. The graybeards I worked with treated rebooting as last resort, like admitting defeat. It was a dirty word, only reserved for lesser operating systems like Windows 95, that needed rebooting often. There was an art to systems administration that seems to be lost now. Or maybe I am just looking back though rose colored glasses. :)
  - count 5y ago
    To be fair, it also used to take sometimes an hour+ to reinit a system. All those SCSI busses had to be enumerated, all the RAM checked for error, ah, the old days...
- joshstrange 5y ago
  That seems to go hand-in-hand with the "Cattle not pets" concept, which I'm a huge fan of.

NaturalPhallacy 5y ago

I would run screaming from such an organization.

stickfigure 5y ago
I run screaming towards such organizations. Higher bill rates!
- hvgk 5y ago
  This person gets it. As always if it’s a shit show, charge by the hour and get double time! If it’s easy charge by the job and spend two months getting paid to play solitaire!
shakna 5y ago
I've helped out orgs where they had a listing of servers, but couldn't tell me what purpose, if any, they served. You have multiple entire architecture rebuilds, deprecating various parts, whilst people come and go and take their knowledge with them leaving documentation behind that is outdated enough to be more harmful than helpful. Reality is a messy thing.
- kubanczyk 5y ago
  Usually there is no universal way to find the latest version of internal documentation across teams, hence updating such documentation is barely worth the time. People notice the fact.
  - nefitty 5y ago
    In your mind, do hosted wikis and KBs like Stack Exchange, Wikimedia or Confluence address the discovery problem? If not, is there anything close?
    - nitrogen 5y ago
      Not the OP, but I am a fan of leaving breadcrumbs everywhere. That is, every bit of documentation should point to other bits of documentation. Code comments point to the corporate wiki and vice versa, commit messages point to issues, issues refer to commit hashes, log messages might say who wrote a service, bespoke server and container images could have a README in /, internal DNS can have text records, etc. We have to accept that we will sometimes play digital archaeologist. That's a good thing! It means that the technology we're studying was successful. So given that, it's better to have out of date documentation than zero documentation, so that we can pick up the trail somewhere.
- nisegami 5y ago
  In cases like these, I'm a strong proponent of turning it off and seeing who yells.
  - shakna 5y ago
    In an org that named their servers after planets, we turned off "Mars" (which was actually completely and utterly undocumented) and discovered it was the bridge between the intranet and the internet. However, upon turning it back on... Nothing came up again. There were no logs to try and work out which processes needed to launch for it to do its job, causing several days of website downtime (not a simple website - dozens of interconnected services). We ended up taking the phone off the hook because we kept getting calls of "is it fixed yet?"
    - jatone 5y ago
      future reference: next time just firewall the network and see what blows up rofl.
      - hamburglar 5y ago
        A bit of network analysis to identify services that are being called before turning things off would probably be wise, too.
      - shakna 5y ago
        The story with Mars is quite a bit more complicated than I've written, but we did scan it, and then try and isolate it, for a few weeks before we tried the restart. Let's just say there was a lot of redundancy that only broke once it couldn't ping at a very specific time.

iovrthoughtthis 5y ago

the pyramids of our time

plafl 5y ago

> _can't reboot their servers_ because they simply don't know how to bring the app back up A hundred years later IT priests will demand human sacrifices to appease the server gods

HowTheStoryEnds 5y ago
So firms will need Ceremoniously Executed Offerings, eh?

staticassertion 5y ago

Server uptime used to be a point of pride. Now people try to shut them down as often as they can. Funny how things change.

viraptor 5y ago
We moved a lot of things from "that one server that's setup just-so by hand" to automated and repeatable setups. If we had more/ cheaper servers before, we'd likely get there earlier.
Sitio 5y ago
My server had 80kbyte bandwidth and doing very little in a much slower time. Kernel patches were not on top of my head 15 years ago.
Bluecobra 5y ago
Indeed, I remember going though great lengths to keep the Solaris hosts up that I used to administer. The graybeards I worked with treated rebooting as last resort, like admitting defeat. It was a dirty word, only reserved for lesser operating systems like Windows 95, that needed rebooting often. There was an art to systems administration that seems to be lost now. Or maybe I am just looking back though rose colored glasses. :)
- count 5y ago
  To be fair, it also used to take sometimes an hour+ to reinit a system. All those SCSI busses had to be enumerated, all the RAM checked for error, ah, the old days...
joshstrange 5y ago
That seems to go hand-in-hand with the "Cattle not pets" concept, which I'm a huge fan of.