5 ms·

So... a tech would just show up at your office, ready to fix the machine? On the one hand, that is some fantastic service when you're running mission critical s

by rashkov 10y ago

So... a tech would just show up at your office, ready to fix the machine? On the one hand, that is some fantastic service when you're running mission critical software. On the other hand, that must cost a whole lot of money and is quite devious of IBM.

jonknee 10y ago
> that must cost a whole lot of money and is quite devious of IBM. You have essentially described IBM's business model. You pay more, but if something goes wrong someone shows up.
- deleted 10y ago
  [deleted]
seanp2k2 10y ago
Yeah, that type of service and phone-home is also common with $$$ enterprise storage (SANs). Funny story: in one instance I know of first-hand, the people running a NetApp filer setup with a redundant pair of systems had chosen to disable AutoSupport (NetApp's version of the phone-home support system where a tech would just show up with replacement disks when some failed) and take on support themselves. They did not test the alerting well enough after some network changes, which left the poor SAN unable to get distress signals through. One day, the shared mount on a ton of systems around the company just stopped working. The master node (these have internal redundancy at the network, controller, shelf, and disk level) had failed over to the redundant node, which after time had so many disks fail without being replace that it went down hard. They had to work with NetApp support (at great expense) to get it back up, which took a few days. The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed. The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. The other lesson here is that such systems don't exist in isolation and are only as good as their weakest link, which in this case was a now-unreachable SMTP server. Lastly, although the LCD displays were trying their best to also signal this fault condition, people who would walk by these physical machines were used to this, so they didn't bother to actually read the errors. Do that at the peril of anything dependent on that system.
- hackuser 10y ago
  > The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. Agreed. Another moral: Don't believe anything works until you see it with your own eyes.
  - protomyth 10y ago
    Had a friend unplug the UPS to test it while I was still configuring the new machine. "Can't really be sure unless you test it"
  - throwawayish 10y ago
    Especially true for backups. Make sure those aren't write-only!
- shubb 10y ago
  Note to self, if need to socially engineer my way into the equipment room at a large company, pretend to be an IBM tech answering a phone home...
  - protomyth 10y ago
    Good luck with that, as we do check these things before we allow folks to get to the servers. IBM has been doing this for a fair bit and we aren't exactly blind idiots. It would be interesting where I am since you skip local crime and go straight to federal offense. Kept me away from booze in high school since its not minor in possessions, its federal bootlegging. I can only imagine what trying to enter a server room nets a person.
- syntheticnature 10y ago
  "The Fail-Safe Theorem: When a Fail-Safe system fails, it fails by failing to fail safe." --https://en.wikipedia.org/wiki/Systemantics#System_failure https://en.wikipedia.org/wiki/Systemantics#System_failure
protomyth 10y ago
It actually isn't that bad on the budget. We don't have the same day service (I think its 24 - 48 hour response time). It sure beats the heck out of waiting for the disaster as the thing is pretty proactive. Much, much cheaper than service and support from Oracle, and Oracle doesn't visit or monitor the machine.
teh_klev 10y ago
We just decommissioned a pair of EMC CX's. These things had phone home to EMC (and then eventually a third party maintainer) if something was about to go bad or had failed. We'd usually have an engineer onsite within four hours at the DC with all the right parts to carry out a fix. By the time it was being looked after by the third party maintainer they'd just leave us a few disks and a couple of fibre channel controllers onsite and we'd get the DC remote hands to swap them out. The faulty parts were then boxed up and couriered back to the maintenance company all on their dime. There was not an awful lot of involvement on our end except for authorising access to the DC when an engineer needed to attend. None of this was massively expensive (and this was 24x7x365 cover) in the great scheme of things given their mission critical status.
mavrc 10y ago
> So... a tech would just show up at your office, ready to fix the machine? Yes. They are expensive and deviously good at it. Or, at least they used to be many years ago. Many years back I worked in a tiny town (1000 people?) four hours from the closest depot and we even got same day service. This cost a king's ransom. But the AS/400 ran most of the business...

by rashkov 10y ago

jonknee 10y ago

> that must cost a whole lot of money and is quite devious of IBM. You have essentially described IBM's business model. You pay more, but if something goes wrong someone shows up.

deleted 10y ago
[deleted]

seanp2k2 10y ago

Yeah, that type of service and phone-home is also common with $$$ enterprise storage (SANs). Funny story: in one instance I know of first-hand, the people running a NetApp filer setup with a redundant pair of systems had chosen to disable AutoSupport (NetApp's version of the phone-home support system where a tech would just show up with replacement disks when some failed) and take on support themselves. They did not test the alerting well enough after some network changes, which left the poor SAN unable to get distress signals through. One day, the shared mount on a ton of systems around the company just stopped working. The master node (these have internal redundancy at the network, controller, shelf, and disk level) had failed over to the redundant node, which after time had so many disks fail without being replace that it went down hard. They had to work with NetApp support (at great expense) to get it back up, which took a few days. The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed. The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. The other lesson here is that such systems don't exist in isolation and are only as good as their weakest link, which in this case was a now-unreachable SMTP server. Lastly, although the LCD displays were trying their best to also signal this fault condition, people who would walk by these physical machines were used to this, so they didn't bother to actually read the errors. Do that at the peril of anything dependent on that system.

hackuser 10y ago
> The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. Agreed. Another moral: Don't believe anything works until you see it with your own eyes.
- protomyth 10y ago
  Had a friend unplug the UPS to test it while I was still configuring the new machine. "Can't really be sure unless you test it"
- throwawayish 10y ago
  Especially true for backups. Make sure those aren't write-only!
shubb 10y ago
Note to self, if need to socially engineer my way into the equipment room at a large company, pretend to be an IBM tech answering a phone home...
- protomyth 10y ago
  Good luck with that, as we do check these things before we allow folks to get to the servers. IBM has been doing this for a fair bit and we aren't exactly blind idiots. It would be interesting where I am since you skip local crime and go straight to federal offense. Kept me away from booze in high school since its not minor in possessions, its federal bootlegging. I can only imagine what trying to enter a server room nets a person.
syntheticnature 10y ago
"The Fail-Safe Theorem: When a Fail-Safe system fails, it fails by failing to fail safe." --https://en.wikipedia.org/wiki/Systemantics#System_failure https://en.wikipedia.org/wiki/Systemantics#System_failure

protomyth 10y ago

It actually isn't that bad on the budget. We don't have the same day service (I think its 24 - 48 hour response time). It sure beats the heck out of waiting for the disaster as the thing is pretty proactive. Much, much cheaper than service and support from Oracle, and Oracle doesn't visit or monitor the machine.

teh_klev 10y ago

We just decommissioned a pair of EMC CX's. These things had phone home to EMC (and then eventually a third party maintainer) if something was about to go bad or had failed. We'd usually have an engineer onsite within four hours at the DC with all the right parts to carry out a fix. By the time it was being looked after by the third party maintainer they'd just leave us a few disks and a couple of fibre channel controllers onsite and we'd get the DC remote hands to swap them out. The faulty parts were then boxed up and couriered back to the maintenance company all on their dime. There was not an awful lot of involvement on our end except for authorising access to the DC when an engineer needed to attend. None of this was massively expensive (and this was 24x7x365 cover) in the great scheme of things given their mission critical status.

mavrc 10y ago

> So... a tech would just show up at your office, ready to fix the machine? Yes. They are expensive and deviously good at it. Or, at least they used to be many years ago. Many years back I worked in a tiny town (1000 people?) four hours from the closest depot and we even got same day service. This cost a king's ransom. But the AS/400 ran most of the business...