4 ms·
We were doing a cleanup of VMs. The network had been rebuilt four times by three different people, and only half documented each time. One time, each VM had b
by shakna 10y ago
We were doing a cleanup of VMs.
The network had been rebuilt four times by three different people, and only half documented each time.
One time, each VM had been named after planetary bodies. Sol was the AD, Jupiter the print server, etc.
We found one called Mars. Completely undocumented. Doesn't exist so far as the docs knew. The previous admin didn't remember it.
I ran Wireshark, and got nothing.
So... I didn't just shut it down, but I deleted it.
Took 10 minutes for mass panic to hit the office.
Mars was the gateway for our publicly exposed servers.
No website, no email, no VPN.
Our daily backup only copied data, not actual images.
So, just hoping, I threw a reverse pass through proxy up on the same IP, with routes for our servers.
Quiet returned, as I went about recovering the Mars image I had deleted.
Lesson learned: if you are working in unknown territory, let it break before deleting. Also, add VM images to the backup routine.
- dawnerd 10y agoAlso name things so they're a little more obvious.
- shakna 10y agoNew name for Mars: extern-gateway01-<domain> And a page in the documentation.
- shermanyo 10y agoWe had a pair of VMs that switched hostnames in DNS at one point. (ie. the hostname 'test-1' resolved to the VM with the image named 'test-2') someone would inevitably restart the wrong one from the VM host, thinking it was the one they'd been SSHing to :/
- shakna 10y agoTwitching, that would make me rather upset, and maybe some host-file changes. Though we did, in the same mess of network, have dc01-<domain> and dc02-<domain>. dc01 was the Domain Controller. dc02 was... The backup of ad01-<domain>...
- danieltillett 10y agoThe name should have been a hint as well - I am not sure I would want to kill anything named after the god of war.
- shermanyo 10y agoThere should be a name specifically for this, like "VM Host Archaeology"