4 ms·

My advice: 1. Suggest that the work to replace it is prioritized commensurate with the business impact caused whilst re-establishing service as if it went down

My advice: 1. Suggest that the work to replace it is prioritized commensurate with the business impact caused whilst re-establishing service as if it went down right now 2. Remind them that it will go down at the worst possible time. 3. Ensure your name is attached to these two warnings. 4. Promise yourself you wouldn't run your business this way. 5. Get on with your life.

jlokier 7y ago
I would add to that: - Make a plan of exploratory steps that could trigger a failure, in order of increasing risks. - Then, if necessary, get a formal sign-off from someone with authority (perhaps a director of the company) to proceed with each step. Steps like that, which are usually very low risk but do risk taking down an unknown service, might include things like (just ideas from other comments): - Logging in to the server at all. - Splicing something into the network switch. - Cloning filesystems or disk images. - Cloning the process memory or system memory. - Cloning the image or other things if it's in a VM. - Running strace gdb, or packet tracing. - Swapping the power source live. These days, it is often possible to transfer a full working system into a VM on a more modern, powerful machine, and after doing that, it's a great relief to everyone because it's no longer hardware dependent. I've done that with some legacy systems that were originally on bare metal, and are now still running nearly 20 years later in reliable VMs, with no change to the running software. Usually running faster and with more memory doesn't break a working system. But doing it on a service that can't be taken down even for a moment is quite an interesting adventure! :-) It is possible, but technically challenging, as long as you can obtain disk and memory images from the live system and then capture changes fast enough to perform a hot transfer.
- deleted 7y ago
  [deleted]
- Apofis 7y ago
  I read earlier that some people were putting legacy apps on VMs, but I didn't realize that you can transfer a full working system into a VM including memory... which is pretty much a full state snapshot. Awesome. This is this guy's solution right there, until a replacement is in place.
CareyB 7y ago
What verytrivial said!!!! I’ve been in similar situations, but never without an immediately obvious solution. That system WILL FAIL. Even as we speak the time-to-failure is shrinking. Even if one of the solutions described below actually works, you won’t get 100% recovery. I recall a story years ago - from MIT, if memory serves - where they rebooted a system because they had many generations of Sybase backups. When they tried, it didn’t work. Nobody had actually tested recovering from a backup. Grit your teeth; cover your ass; and get on with it. The clock IS ticking.
- pawelos 7y ago
  Aah, my favorite write-only backups.
- mrweasel 7y ago
  > That system WILL FAIL. I think that the key point. We have a system that can't easily be restarted and won't automatically start after a server is rebooted. The developers of the software don't care, because "what are the chances of a virtual machine spontaneously restarting". Turns out, those chances are rather good. Servers, virtual machines, containers, doesn't matter, unless that thing is running on a mainframe, it will crash.
  - andai 7y ago
    What's special about mainframes in this context?
    - DuskStar 7y ago
      Mainframes tend to be designed with extreme uptime in mind - to the point of often having hot-swappable processors.
      - marcosdumay 7y ago
        The software doesn't care. It will crash anyway.
deleted 7y ago
[deleted]
3fe9a03ccd14ca5 7y ago
Follow up advice: After sufficient CYA steps are taken, don’t offer any help on this problem unless explicitly asked to do so. And even then wait until there’s actually desperation. Feel free to think about it so you have some ideas when the time comes, but if this service isn’t something you are responsible for then stay away. Or else it will be a “you touched it last you broke it” regardless of what actually happened. And if you make too much ruckus about it, they’ll say “why didn’t you do anything to fix it??”
Nextgrid 7y ago
I would say keep your own copy of the warnings - email them to yourself or similar. When shit hits the fan if the company wanted to throw you under the bus they could delete them remotely from your machine or work mailbox and then pretend the warnings never existed.

jlokier 7y ago

I would add to that: - Make a plan of exploratory steps that could trigger a failure, in order of increasing risks. - Then, if necessary, get a formal sign-off from someone with authority (perhaps a director of the company) to proceed with each step. Steps like that, which are usually very low risk but do risk taking down an unknown service, might include things like (just ideas from other comments): - Logging in to the server at all. - Splicing something into the network switch. - Cloning filesystems or disk images. - Cloning the process memory or system memory. - Cloning the image or other things if it's in a VM. - Running strace gdb, or packet tracing. - Swapping the power source live. These days, it is often possible to transfer a full working system into a VM on a more modern, powerful machine, and after doing that, it's a great relief to everyone because it's no longer hardware dependent. I've done that with some legacy systems that were originally on bare metal, and are now still running nearly 20 years later in reliable VMs, with no change to the running software. Usually running faster and with more memory doesn't break a working system. But doing it on a service that can't be taken down even for a moment is quite an interesting adventure! :-) It is possible, but technically challenging, as long as you can obtain disk and memory images from the live system and then capture changes fast enough to perform a hot transfer.

deleted 7y ago
[deleted]
Apofis 7y ago
I read earlier that some people were putting legacy apps on VMs, but I didn't realize that you can transfer a full working system into a VM including memory... which is pretty much a full state snapshot. Awesome. This is this guy's solution right there, until a replacement is in place.

CareyB 7y ago

What verytrivial said!!!! I’ve been in similar situations, but never without an immediately obvious solution. That system WILL FAIL. Even as we speak the time-to-failure is shrinking. Even if one of the solutions described below actually works, you won’t get 100% recovery. I recall a story years ago - from MIT, if memory serves - where they rebooted a system because they had many generations of Sybase backups. When they tried, it didn’t work. Nobody had actually tested recovering from a backup. Grit your teeth; cover your ass; and get on with it. The clock IS ticking.

pawelos 7y ago
Aah, my favorite write-only backups.
mrweasel 7y ago
> That system WILL FAIL. I think that the key point. We have a system that can't easily be restarted and won't automatically start after a server is rebooted. The developers of the software don't care, because "what are the chances of a virtual machine spontaneously restarting". Turns out, those chances are rather good. Servers, virtual machines, containers, doesn't matter, unless that thing is running on a mainframe, it will crash.
- andai 7y ago
  What's special about mainframes in this context?
  - DuskStar 7y ago
    Mainframes tend to be designed with extreme uptime in mind - to the point of often having hot-swappable processors.
    - marcosdumay 7y ago
      The software doesn't care. It will crash anyway.

deleted 7y ago

[deleted]

3fe9a03ccd14ca5 7y ago

Follow up advice: After sufficient CYA steps are taken, don’t offer any help on this problem unless explicitly asked to do so. And even then wait until there’s actually desperation. Feel free to think about it so you have some ideas when the time comes, but if this service isn’t something you are responsible for then stay away. Or else it will be a “you touched it last you broke it” regardless of what actually happened. And if you make too much ruckus about it, they’ll say “why didn’t you do anything to fix it??”

Nextgrid 7y ago

I would say keep your own copy of the warnings - email them to yourself or similar. When shit hits the fan if the company wanted to throw you under the bus they could delete them remotely from your machine or work mailbox and then pretend the warnings never existed.