11 ms·

OpenVMS or AS/400. Very common for admins to report systems running for 5+ years without crashes while doing real work. People occasionally forget where they're

OpenVMS or AS/400. Very common for admins to report systems running for 5+ years without crashes while doing real work. People occasionally forget where they're at and have to go looking for them. My company has an AS/400 employees never saw go down or get maintenance over 10 years. It's in use 24/7. They might have some fail-over setup or upgrade when nobody is using it due to traffic patterns. We've just never seen it fail. Their other benefit is they're very hands off with it nearly self-administering.

protomyth 10y ago
Or that AS/400 calls the tech to fix it and s/he just does it while no one is around. Had a call once from IBM asking for directions since the AS/400 RAID cache battery needed replacing. No danger of losing data, the AS/400 was just running slower because it was bypassing the cache.
- rashkov 10y ago
  So... a tech would just show up at your office, ready to fix the machine? On the one hand, that is some fantastic service when you're running mission critical software. On the other hand, that must cost a whole lot of money and is quite devious of IBM.
  - jonknee 10y ago
    > that must cost a whole lot of money and is quite devious of IBM. You have essentially described IBM's business model. You pay more, but if something goes wrong someone shows up.
    - deleted 10y ago
      [deleted]
  - seanp2k2 10y ago
    Yeah, that type of service and phone-home is also common with $$$ enterprise storage (SANs). Funny story: in one instance I know of first-hand, the people running a NetApp filer setup with a redundant pair of systems had chosen to disable AutoSupport (NetApp's version of the phone-home support system where a tech would just show up with replacement disks when some failed) and take on support themselves. They did not test the alerting well enough after some network changes, which left the poor SAN unable to get distress signals through. One day, the shared mount on a ton of systems around the company just stopped working. The master node (these have internal redundancy at the network, controller, shelf, and disk level) had failed over to the redundant node, which after time had so many disks fail without being replace that it went down hard. They had to work with NetApp support (at great expense) to get it back up, which took a few days. The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed. The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. The other lesson here is that such systems don't exist in isolation and are only as good as their weakest link, which in this case was a now-unreachable SMTP server. Lastly, although the LCD displays were trying their best to also signal this fault condition, people who would walk by these physical machines were used to this, so they didn't bother to actually read the errors. Do that at the peril of anything dependent on that system.
    - hackuser 10y ago
      > The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. Agreed. Another moral: Don't believe anything works until you see it with your own eyes.
      - protomyth 10y ago
        Had a friend unplug the UPS to test it while I was still configuring the new machine. "Can't really be sure unless you test it"
      - throwawayish 10y ago
        Especially true for backups. Make sure those aren't write-only!
    - shubb 10y ago
      Note to self, if need to socially engineer my way into the equipment room at a large company, pretend to be an IBM tech answering a phone home...
      - protomyth 10y ago
        Good luck with that, as we do check these things before we allow folks to get to the servers. IBM has been doing this for a fair bit and we aren't exactly blind idiots. It would be interesting where I am since you skip local crime and go straight to federal offense. Kept me away from booze in high school since its not minor in possessions, its federal bootlegging. I can only imagine what trying to enter a server room nets a person.
    - syntheticnature 10y ago
      "The Fail-Safe Theorem: When a Fail-Safe system fails, it fails by failing to fail safe." --https://en.wikipedia.org/wiki/Systemantics#System_failure https://en.wikipedia.org/wiki/Systemantics#System_failure
  - protomyth 10y ago
    It actually isn't that bad on the budget. We don't have the same day service (I think its 24 - 48 hour response time). It sure beats the heck out of waiting for the disaster as the thing is pretty proactive. Much, much cheaper than service and support from Oracle, and Oracle doesn't visit or monitor the machine.
  - teh_klev 10y ago
    We just decommissioned a pair of EMC CX's. These things had phone home to EMC (and then eventually a third party maintainer) if something was about to go bad or had failed. We'd usually have an engineer onsite within four hours at the DC with all the right parts to carry out a fix. By the time it was being looked after by the third party maintainer they'd just leave us a few disks and a couple of fibre channel controllers onsite and we'd get the DC remote hands to swap them out. The faulty parts were then boxed up and couriered back to the maintenance company all on their dime. There was not an awful lot of involvement on our end except for authorising access to the DC when an engineer needed to attend. None of this was massively expensive (and this was 24x7x365 cover) in the great scheme of things given their mission critical status.
  - mavrc 10y ago
    > So... a tech would just show up at your office, ready to fix the machine? Yes. They are expensive and deviously good at it. Or, at least they used to be many years ago. Many years back I worked in a tiny town (1000 people?) four hours from the closest depot and we even got same day service. This cost a king's ransom. But the AS/400 ran most of the business...
- nickpsecurity 10y ago
  It's possible. I do want to try to get the uptime info from the people that run it if I ever see them. We regularly get techs come in to improve or fix IT systems. Just not that one.
  - protomyth 10y ago
    I have had multiple years of uptime until I needed to do an OS upgrade for some new part of the financial system.
    - nickpsecurity 10y ago
      That's what vast majority of my feedback has said. Stays up until OS upgrade. Whereas, my research on VMS side indicated they do clusters that support rolling upgrades of the individual machines to not affect application. Might be around that remaining limitation of AS/400 uptime if they have software for that. Do they? Or are customers forced to upgrade to mainframe with Sysplex?
      - protomyth 10y ago
        OpenVMS has an amazing cluster system, and yes they did upgrade one machine at a time in the cluster and lose no overall uptime. I think the bigger IBM z machines do allow zero-time upgrades, but the iSeries (new name for AS/400) is basically a reboot the machine affair. I've never heard of Sysplex in regards to the iSeries. They sent me two DVDs with 12 pages of instructions. The unnumbered first page is release notes and a notice that physical media deliveries would be an average of 5 to 9 business days on Jan 31, 2007 and beyond. (odd notice for a 2016 document) Page 1 has my customer information with all sorts of tracking numbers. It also include the number of selected files (1,360), Kilo-bytes (sp) of data (3,963,529), fixes (3,394 fixes with 815 Kilo-bytes of data were superceded), etc. Pre-install instructions finally show up on page 3 and the big warning about f'n it up shows on page 4. Page 6 brings us to the actual install which is written to be typed exactly with exact detailed results of each step. to give an example 2. Load the image catalog into the virtual optical device using the following command: LODIMGCLG IMGCLG(ptfcatalog) DEV(OPTVRTxx) OPTION(*LOAD) 3. Type GO PTF and press the Enter key 4. Take menu option 8 and press Enter key Took about an two hours counting the the 45 minute reboot. We have a very old and slow machine (need to buy a new one in the next two years, we are at a decade of use now) so reboots are a bit slow.
skissane 10y ago
Do AS/400 and OpenVMS systems support live kernel patching? Can you patch the entire OS (even the lowest layers) without any reboot? If they don't, then you can't have zero downtime for years unless patches aren't being applied, which is rather bad for security.
- gaius 10y ago
  With VMS you can hot add a machine to the cluster, migrate all running processes to it, then shut down the first node without any interruption. People kept systems running while moving buildings like this. And this was 20+ years ago.
  - youdontknowtho 10y ago
    Also supports cluster nodes with different CPU architectures. It was/is really slick.
    - nickpsecurity 10y ago
      That's something people keep forgetting that should be in any "Required Features for Good, Clustering Software" document. That they could move the software from VAX to Alpha to Itanium within same cluster was pretty amazing. Also a requirement for high-availability if a processor goes EOL. The other strategy is to bet on the market leader. Something they missed. ;)
      - youdontknowtho 10y ago
        Yeah, they (the new owner) is finally porting OpenVMS to x64. Their roadmap shows that this is going to be a long road, though. I can't remember their name...the company that bought VMS from HP. HP is licensing it from them for their own products.
        nickpsecurity 10y ago
        It's this company: https://www.vmssoftware.com/about_faq.html#faq1 https://www.vmssoftware.com/about_faq.html#faq1 They didn't buy it. HP is too greedy about the lucrative revenue & maybe patents they get off VMS that made me wonder why they killed it in the first place. Instead, it appears (not certain) that VMS Software Inc gets some OEM-like license that puts them in control of it while HP still owns it, can license it themselves (probably existing customers), and likely gets a portion of VMS Software Inc's licensing revenue. So, they've pushed off almost all responsibility for it onto another company without actually selling it or losing all the revenue. Still a good development for VMS customers. Both legacy and people who will suffer through archaic stuff to reap benefits of bulletproof clusters. I'd be happy to. :) Only problem is it's basically got no security attention over the years with plenty of zero days or configuration issues lurking in there. If I deployed in a company, I plan to put network guards in front of it to absorb attacks while converting the traffic into something easy to process. Ideally, a PCI-based guard that forces it to only talk to the application's memory instead of rest of system. Plus safe-coded apps (Rust, Ada) and API wrappers to try to spot BS. That VMS supports cross-language development will probably help. EDIT: The weird thing is they finished a port to Alpha ISA before they got to x86 ISA for migrations. I thought Alpha's already migrated to Itanium. Didn't see that coming lol.
bluedino 10y ago
Most people recommend an IPL of an iSeries (AS/400) at least every 1-3 months. Did they just do that after hours?

protomyth 10y ago

Or that AS/400 calls the tech to fix it and s/he just does it while no one is around. Had a call once from IBM asking for directions since the AS/400 RAID cache battery needed replacing. No danger of losing data, the AS/400 was just running slower because it was bypassing the cache.

rashkov 10y ago
So... a tech would just show up at your office, ready to fix the machine? On the one hand, that is some fantastic service when you're running mission critical software. On the other hand, that must cost a whole lot of money and is quite devious of IBM.
- jonknee 10y ago
  > that must cost a whole lot of money and is quite devious of IBM. You have essentially described IBM's business model. You pay more, but if something goes wrong someone shows up.
  - deleted 10y ago
    [deleted]
- seanp2k2 10y ago
  Yeah, that type of service and phone-home is also common with $$$ enterprise storage (SANs). Funny story: in one instance I know of first-hand, the people running a NetApp filer setup with a redundant pair of systems had chosen to disable AutoSupport (NetApp's version of the phone-home support system where a tech would just show up with replacement disks when some failed) and take on support themselves. They did not test the alerting well enough after some network changes, which left the poor SAN unable to get distress signals through. One day, the shared mount on a ton of systems around the company just stopped working. The master node (these have internal redundancy at the network, controller, shelf, and disk level) had failed over to the redundant node, which after time had so many disks fail without being replace that it went down hard. They had to work with NetApp support (at great expense) to get it back up, which took a few days. The system had been attempting to warn the admins as best it could of the impending doom for the better part of a year, and no one noticed. The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. The other lesson here is that such systems don't exist in isolation and are only as good as their weakest link, which in this case was a now-unreachable SMTP server. Lastly, although the LCD displays were trying their best to also signal this fault condition, people who would walk by these physical machines were used to this, so they didn't bother to actually read the errors. Do that at the peril of anything dependent on that system.
  - hackuser 10y ago
    > The moral of this story is that you can have big-dollar redundant systems, but if you don't also have some dollars going to the care and feeding of them, it's wasted. Agreed. Another moral: Don't believe anything works until you see it with your own eyes.
    - protomyth 10y ago
      Had a friend unplug the UPS to test it while I was still configuring the new machine. "Can't really be sure unless you test it"
    - throwawayish 10y ago
      Especially true for backups. Make sure those aren't write-only!
  - shubb 10y ago
    Note to self, if need to socially engineer my way into the equipment room at a large company, pretend to be an IBM tech answering a phone home...
    - protomyth 10y ago
      Good luck with that, as we do check these things before we allow folks to get to the servers. IBM has been doing this for a fair bit and we aren't exactly blind idiots. It would be interesting where I am since you skip local crime and go straight to federal offense. Kept me away from booze in high school since its not minor in possessions, its federal bootlegging. I can only imagine what trying to enter a server room nets a person.
  - syntheticnature 10y ago
    "The Fail-Safe Theorem: When a Fail-Safe system fails, it fails by failing to fail safe." --https://en.wikipedia.org/wiki/Systemantics#System_failure https://en.wikipedia.org/wiki/Systemantics#System_failure
- protomyth 10y ago
  It actually isn't that bad on the budget. We don't have the same day service (I think its 24 - 48 hour response time). It sure beats the heck out of waiting for the disaster as the thing is pretty proactive. Much, much cheaper than service and support from Oracle, and Oracle doesn't visit or monitor the machine.
- teh_klev 10y ago
  We just decommissioned a pair of EMC CX's. These things had phone home to EMC (and then eventually a third party maintainer) if something was about to go bad or had failed. We'd usually have an engineer onsite within four hours at the DC with all the right parts to carry out a fix. By the time it was being looked after by the third party maintainer they'd just leave us a few disks and a couple of fibre channel controllers onsite and we'd get the DC remote hands to swap them out. The faulty parts were then boxed up and couriered back to the maintenance company all on their dime. There was not an awful lot of involvement on our end except for authorising access to the DC when an engineer needed to attend. None of this was massively expensive (and this was 24x7x365 cover) in the great scheme of things given their mission critical status.
- mavrc 10y ago
  > So... a tech would just show up at your office, ready to fix the machine? Yes. They are expensive and deviously good at it. Or, at least they used to be many years ago. Many years back I worked in a tiny town (1000 people?) four hours from the closest depot and we even got same day service. This cost a king's ransom. But the AS/400 ran most of the business...
nickpsecurity 10y ago
It's possible. I do want to try to get the uptime info from the people that run it if I ever see them. We regularly get techs come in to improve or fix IT systems. Just not that one.
- protomyth 10y ago
  I have had multiple years of uptime until I needed to do an OS upgrade for some new part of the financial system.
  - nickpsecurity 10y ago
    That's what vast majority of my feedback has said. Stays up until OS upgrade. Whereas, my research on VMS side indicated they do clusters that support rolling upgrades of the individual machines to not affect application. Might be around that remaining limitation of AS/400 uptime if they have software for that. Do they? Or are customers forced to upgrade to mainframe with Sysplex?
    - protomyth 10y ago
      OpenVMS has an amazing cluster system, and yes they did upgrade one machine at a time in the cluster and lose no overall uptime. I think the bigger IBM z machines do allow zero-time upgrades, but the iSeries (new name for AS/400) is basically a reboot the machine affair. I've never heard of Sysplex in regards to the iSeries. They sent me two DVDs with 12 pages of instructions. The unnumbered first page is release notes and a notice that physical media deliveries would be an average of 5 to 9 business days on Jan 31, 2007 and beyond. (odd notice for a 2016 document) Page 1 has my customer information with all sorts of tracking numbers. It also include the number of selected files (1,360), Kilo-bytes (sp) of data (3,963,529), fixes (3,394 fixes with 815 Kilo-bytes of data were superceded), etc. Pre-install instructions finally show up on page 3 and the big warning about f'n it up shows on page 4. Page 6 brings us to the actual install which is written to be typed exactly with exact detailed results of each step. to give an example 2. Load the image catalog into the virtual optical device using the following command: LODIMGCLG IMGCLG(ptfcatalog) DEV(OPTVRTxx) OPTION(*LOAD) 3. Type GO PTF and press the Enter key 4. Take menu option 8 and press Enter key Took about an two hours counting the the 45 minute reboot. We have a very old and slow machine (need to buy a new one in the next two years, we are at a decade of use now) so reboots are a bit slow.

skissane 10y ago

Do AS/400 and OpenVMS systems support live kernel patching? Can you patch the entire OS (even the lowest layers) without any reboot? If they don't, then you can't have zero downtime for years unless patches aren't being applied, which is rather bad for security.

gaius 10y ago
With VMS you can hot add a machine to the cluster, migrate all running processes to it, then shut down the first node without any interruption. People kept systems running while moving buildings like this. And this was 20+ years ago.
- youdontknowtho 10y ago
  Also supports cluster nodes with different CPU architectures. It was/is really slick.
  - nickpsecurity 10y ago
    That's something people keep forgetting that should be in any "Required Features for Good, Clustering Software" document. That they could move the software from VAX to Alpha to Itanium within same cluster was pretty amazing. Also a requirement for high-availability if a processor goes EOL. The other strategy is to bet on the market leader. Something they missed. ;)
    - youdontknowtho 10y ago
      Yeah, they (the new owner) is finally porting OpenVMS to x64. Their roadmap shows that this is going to be a long road, though. I can't remember their name...the company that bought VMS from HP. HP is licensing it from them for their own products.
      - nickpsecurity 10y ago
        It's this company: https://www.vmssoftware.com/about_faq.html#faq1 https://www.vmssoftware.com/about_faq.html#faq1 They didn't buy it. HP is too greedy about the lucrative revenue & maybe patents they get off VMS that made me wonder why they killed it in the first place. Instead, it appears (not certain) that VMS Software Inc gets some OEM-like license that puts them in control of it while HP still owns it, can license it themselves (probably existing customers), and likely gets a portion of VMS Software Inc's licensing revenue. So, they've pushed off almost all responsibility for it onto another company without actually selling it or losing all the revenue. Still a good development for VMS customers. Both legacy and people who will suffer through archaic stuff to reap benefits of bulletproof clusters. I'd be happy to. :) Only problem is it's basically got no security attention over the years with plenty of zero days or configuration issues lurking in there. If I deployed in a company, I plan to put network guards in front of it to absorb attacks while converting the traffic into something easy to process. Ideally, a PCI-based guard that forces it to only talk to the application's memory instead of rest of system. Plus safe-coded apps (Rust, Ada) and API wrappers to try to spot BS. That VMS supports cross-language development will probably help. EDIT: The weird thing is they finished a port to Alpha ISA before they got to x86 ISA for migrations. I thought Alpha's already migrated to Itanium. Didn't see that coming lol.

bluedino 10y ago

Most people recommend an IPL of an iSeries (AS/400) at least every 1-3 months. Did they just do that after hours?