6 ms·

Just another reminder of how systems that you'd think are rock solid often aren't. In my previous life working with telcos, I once tried to teach a particularl

Just another reminder of how systems that you'd think are rock solid often aren't. In my previous life working with telcos, I once tried to teach a particularly huge customer how to use CVS how to manage configurations across a 10+ machine cluster of machines. They didn't see any value in it, so they stuck to their good old process of SSHing into each machine individually, "cp config.xml config.xml.20131022", and then editing the configs by hand. Didn't take too long until a typo in a chmod command took down the whole thing (= the node couldn't take down a network interface anymore, so failover stopped working), and they spent several weeks flying in people from all over the planet to debug it... and they still didn't learn their lesson!

fosk 13y ago
Ha, probably no one ever got fired for manually SSHing into each machine. It seems like a joke, but it really isn't.
- rev_null 13y ago
  They can't fire that person. He's the only one who knows the state of the servers.
- falcolas 13y ago
  Considering that I've watched a new automated deployment process nuke an entire network due to a mistake in the config... Sometimes slow and plodding and potentially losing a node (as opposed to 100 nodes) is seen as preferable to the PTBs. Of course intelligent, incremental deployment works too, but let's not confuse those poor suits...
- mathattack 13y ago
  Is that job security?
- gngeal 13y ago
  What about SSHing into each machine not manually, but automatically with OKing all the changes before reloading the configuration? SSH is great at least in the sense that it's configuration-format-agnostic, mostly-OS-agnostic, client-side-automation-agnostic, and many other things like that.
lhaussknecht 13y ago
Reminds me of a colleague who RDPed into each of our 140 subsidiaries to change a config file. He had a list of servers on his desk and ticked off every server. Took him the whole day to apply the changes.
- cstross 13y ago
  Two words: job security. (Also: lots of folks really don't enjoy learning new stuff. Or new ways of working. No, they really don't. Even if the new techniques are vastly better and more efficient. Put this down to a human cognitive bias favouring the tried-and-trusted over new-and-untested. There's a lot to be said for that when you're a neolithic hunter-gatherer or an iron-age peasant -- if you try something new and it fails, you maybe get to watch your family starve next spring -- but it's a bit less useful as a rule of thumb in the data centre.)
  - pnathan 13y ago
    Totally true. Even when the path is seamless, initial costs are nearly zilch... learning new things and applying them is anathema to many devs.
    - mathattack 13y ago
      You can get blamed for changing something! The power if inertia....
- dijit 13y ago
  I did this with nagios configs in my first job :(
- alephnil 13y ago
  When the service guy in a former job quit, the databases at the customers' production servers started to be corrupted. It appeared that he had manually logged into the database at every single customer regularly and fixed the problem on site. I don't know if had told the responsible developers (I worked on a different product), but after it was apparent that the system was not stable.
sasas 13y ago
We often get frustrated with strict change management processes, excruciating verification etc .. but there is a reason why the most mature (not always the biggest..) operators have these processes in place.
hrjet 13y ago
I heard similar stories from a friend working for a big telco. The other day he was describing a strange bug which was triggering in the field of a large telco. It was strange because it used to get triggered after exactly 85 days of deployment. It turned out to be a debug script that was pinging a development server and timing out (because the development server was not accessible from the field). The series of retrials and timeouts totaled to exactly 85 days, after which the rest of the script would activate and uninstall crucial dependencies!
- mathattack 13y ago
  Legacy telcos are the epitome of large big institutions where many of the best talents leave. After just a few years of working with them, I couldn't believe they could get a dial tone. How did they produce so much great R&D?
  - AsymetricCom 13y ago
    wining and dining college grads and telling them all sorts of great stories then laying the workload on them until they burn out or quit.
  - Zigurd 13y ago
    > How did they produce so much great R&D? By having negotiated profit margins. Bell Labs was meant to soak up excess money. What we are now nostalgic for would never have happened without an interval during which a telco was considered a "natural monopoly."
    - mathattack 13y ago
      But didn't Bell Labs have the telco mentality? Or how did they protect themselves from it?
rhapsodyv 13y ago
the guy responsible for deployment doesn't even know the repository version that is running in production... I'm really tired of fight against this jerks...

fosk 13y ago

Ha, probably no one ever got fired for manually SSHing into each machine. It seems like a joke, but it really isn't.

rev_null 13y ago
They can't fire that person. He's the only one who knows the state of the servers.
falcolas 13y ago
Considering that I've watched a new automated deployment process nuke an entire network due to a mistake in the config... Sometimes slow and plodding and potentially losing a node (as opposed to 100 nodes) is seen as preferable to the PTBs. Of course intelligent, incremental deployment works too, but let's not confuse those poor suits...
mathattack 13y ago
Is that job security?
gngeal 13y ago
What about SSHing into each machine not manually, but automatically with OKing all the changes before reloading the configuration? SSH is great at least in the sense that it's configuration-format-agnostic, mostly-OS-agnostic, client-side-automation-agnostic, and many other things like that.

lhaussknecht 13y ago

Reminds me of a colleague who RDPed into each of our 140 subsidiaries to change a config file. He had a list of servers on his desk and ticked off every server. Took him the whole day to apply the changes.

cstross 13y ago
Two words: job security. (Also: lots of folks really don't enjoy learning new stuff. Or new ways of working. No, they really don't. Even if the new techniques are vastly better and more efficient. Put this down to a human cognitive bias favouring the tried-and-trusted over new-and-untested. There's a lot to be said for that when you're a neolithic hunter-gatherer or an iron-age peasant -- if you try something new and it fails, you maybe get to watch your family starve next spring -- but it's a bit less useful as a rule of thumb in the data centre.)
- pnathan 13y ago
  Totally true. Even when the path is seamless, initial costs are nearly zilch... learning new things and applying them is anathema to many devs.
  - mathattack 13y ago
    You can get blamed for changing something! The power if inertia....
dijit 13y ago
I did this with nagios configs in my first job :(
alephnil 13y ago
When the service guy in a former job quit, the databases at the customers' production servers started to be corrupted. It appeared that he had manually logged into the database at every single customer regularly and fixed the problem on site. I don't know if had told the responsible developers (I worked on a different product), but after it was apparent that the system was not stable.

sasas 13y ago

We often get frustrated with strict change management processes, excruciating verification etc .. but there is a reason why the most mature (not always the biggest..) operators have these processes in place.

hrjet 13y ago

I heard similar stories from a friend working for a big telco. The other day he was describing a strange bug which was triggering in the field of a large telco. It was strange because it used to get triggered after exactly 85 days of deployment. It turned out to be a debug script that was pinging a development server and timing out (because the development server was not accessible from the field). The series of retrials and timeouts totaled to exactly 85 days, after which the rest of the script would activate and uninstall crucial dependencies!

mathattack 13y ago
Legacy telcos are the epitome of large big institutions where many of the best talents leave. After just a few years of working with them, I couldn't believe they could get a dial tone. How did they produce so much great R&D?
- AsymetricCom 13y ago
  wining and dining college grads and telling them all sorts of great stories then laying the workload on them until they burn out or quit.
- Zigurd 13y ago
  > How did they produce so much great R&D? By having negotiated profit margins. Bell Labs was meant to soak up excess money. What we are now nostalgic for would never have happened without an interval during which a telco was considered a "natural monopoly."
  - mathattack 13y ago
    But didn't Bell Labs have the telco mentality? Or how did they protect themselves from it?

rhapsodyv 13y ago

the guy responsible for deployment doesn't even know the repository version that is running in production... I'm really tired of fight against this jerks...