6 ms·
Actually, I have experience here. The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is
by dijit 7y ago
Actually, I have experience here.
The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely.
First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM because there might be vMotion or some kind of live-migration in place meaning hardware failure might not actually bring the service down.
Next thing is you absolutely have to plan for failure because the one thing that I learn as I learn more about computers is that they're basically working "by accident", so much error correction goes into systems that it absolutely boggles my mind and failures still get through. So, it's certainly not a question of "if" but "when" and plan for it being soon.
Now, the obvious technical things are:
* Dump the memory
* Grab any files it has open from its file-descriptors (/proc/<pid>/fd/) Your config file might be there... but somehow I doubt it.
Attach a debugger and dump as much state as possible
Be sure to cleanly detatch : https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_22.html https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_22.htm...
Don't use breakpoints! They will obviously halt execution.
If it was my service I would also capture packet's and see what kind of traffic it receives most commonly; and I would make some kind of stub component to replace it before working on the machine. Just in case I break it and it causes everything to go down that depends on it.
But, this is a horrible situation, reverse engineering this thing is going to be a pain. Good Luck.
- scandox 7y ago> because the one thing that I learn as I learn more about computers is that they're basically working "by accident" So true. Thanks this is good advice. As it happens it is a physical server. I'm not actually mandated to do anything except replace it with a comparable service but this makes me feel like I'm racing against time!
- deleted 7y ago[deleted]
- arethuza 7y agoHave you tried decompiling the executable to see if any sensible information can be gained that way?
- scandox 7y agoEveryone's afraid to do anything on that machine at all. So even getting agreement to login or install software or run tcpdump or whatever is fraught.
- arethuza 7y agoWhat I was meaning was that if you can get a hold of the binary that is running you could take it to some other environment and have a look (starting with a hex dump and working up from there...). Mind you if things are that bad then maybe I'd follow my own advice (in another comment) and stay well away from the currently running thing.
- celticmusic 7y agothis would certainly help recreate the configuration.
- nerpderp82 7y agoYou need to setup an identical-ish host, could even be in VM so that you can build software that will run on the Ghost Ship. This in and of itself could be difficult, Debian 4? Using this Shadow Ghost Ship you can build a copy of tcpdump that you can just scp over and run in place. Don't touch the package manager, /etc, /var, anything!
- mst 7y agoIt might not help much but maybe you can get permission to run tcpdump on a machine that talks to the service?
- arethuza 7y agoI have a personal theory that Really Bad Things (TM) usually tend to happen during the response to other, relatively minor, problems. So whatever is done cannot risk the running process. So personally I'd be loathed to try anything that touches the running process - even if they seem low risk. Edit: One thing I would do is make it clear that building a replacement and keeping this thing running are different responsibilities and if you are doing the former then you really shouldn't be held responsible for the latter (politics I know).
- lonelappde 7y agoYep. Responding to an outage increases the probability of having an outage.
- pjc50 7y agoAttaching a debugger potentially disrupts the service. I would suggest trying to live-image it with some other technique first.
- fxtentacle 7y agoAgree. Attaching a debugger can cause multi-threaded processes to behave differently and/or deadlock if you are unlucky. Also, some closed-source software components deliberately crash if they detect a debugger and since you don't have source code for the binary, there is a risk that one such component was used during development.
- dijit 7y agoYeah, that's pretty much the last resort once you have fallbacks in place, might be the last/only way to glean info about the config that it's running.
- lmilcin 7y agoI would add: overcommunicate everything you plan before you do and give others plenty of time to react to even simplest operations. The company may have failed but when the proverbial truckload of shit hits the proverbial supersonic fan there will be tremendous pressure to find a culprit. A small peon might find himself/herself outnumbered and outgunned.
- lallysingh 7y agoYou can use /proc to get what you need. It has a memory map and the memory itself. Read the map, pull the heap segments out, and read them into files named after the addresses. On an identical machine, load the process up, attach via gdb, and (this is the tricky part) then mmap in the files saved from the other side at the right addresses. If you can get the roots into your heap (tough), you can use gdb'd python API to write a traverser to get the data you need back out.
- dbancajas 7y agoSeems like a checkpoint restore process. Do you have experience of this or actual guide? I'm really curious as to the underlying mechanism.
- lallysingh 7y agoNothing so formal. The tricky parts are the non-memory program state: registers and instruction pointers for all threads.
- adrianmonk 7y agoI would add this: whatever you decide to do to the running binary, create a mock binary on another computer and try the exact sequence of commands/steps first. And verify that it doesn't disturb the mock. This should reduce risk a bit because you are not experimenting and creating your procedure on the fly when you do the actual thing. Instead, you are applying a procedure that you have rehearsed, and you have gained partial confidence that it won't disturb a similar running binary. The mock doesn't have to be a perfect reproduction of the real binary, but it should do similar things where relevant. For example, if the service listens on a TCP port, so should the mock. You might not have to code the mock from scratch; maybe some existing software is similar enough. Or maybe you can even run a non-production copy of your real binary, if you are really confident you can configure it to not interact with anything it shouldn't.
- Nextgrid 7y agoI would also recommend trying the process on a wide range of existing software you can get your hands on (let's say PostgreSQL or something similarly big). This will reduce the risk of any edge-case that your mock didn't cover.