7 ms·
This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no
by Nextgrid 6y ago
This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains which should at least bring their internal services back online and allow them to proceed further.
- arethuza 6y agoI guess if you are in the business of selling services that include custom domains that isn't quite as crazy as it initially sounds... You are, of course, assuming they have a backup and the backup can actually be restored and that the restore contains the required information.
- WJW 6y agoIt does say in the original post that: > Our backups are fucked. So I assume that they have already tried and they didn't work.
- arethuza 6y agoEarly in my career (>30 years) ago I nuked a companies salary database (a missing $ in a shell script moved everything to the file i) No problem they said - we have backups. They had three tapes. First tape failed. Second tape failed. Third tape worked. In retrospect that was a useful, if rather stressful, lesson.
- lupire 6y ago> nuked a companies salary database > a missing $ Bravo!
- artursapek 6y ago>a missing $ in a shell script moved everything to the file i I've always had anxiety when running a shell script, this is a perfect example of why
- noisy_boy 6y agoI always wrap the actual cp/mv/rm etc in echo first before actually running it e.g. running echo "mv $src i" first would have immediately highlighted the problem visually.
- ansible 6y agoIt is also helpful to have at least two tape drives, and check the backup tapes against both. Early in my career, we had a storm that caused an electrical surge. In spite of the power protection, it damaged the tape drive on the main server (unknown to us at the time). The entire rest of the system (CPU, RAM, hard drives (RLL!)) were fine. Listening to the backups being run, I noticed it sounded a little different. Eventually I determined that the tape drive (old QIC, maybe 40MB capacity?) worked, but it would not switch tracks when reading / writing. So the backup ran, but it just overwrite the same track rather than write to subsequent tracks. A short backup / restore wouldn't have caught the problem, it was necessary to try to read the whole backup to see. This wasn't caught for a while, so all the backups were mostly useless.
- pc86 6y ago> > All domain information was wiped out and records became null. > > Every person who knew how the custom DNS system worked has left the company years ago. Our backups are fucked. Our records are wiped from all domain servers out there. Sure sounds like the data is lost to me.
- buro9 6y agoTo me it sounds like those people who left can command a very high day rate to help with the immediate aftermath. A high enough rate that even if they are doing hours elsewhere some will be tempted to sign in after their work hours to help out. A company does not just stop existing or being able to function, deleting DNS records didn't empty the bank account or the ability of the accountants to be directed to pay someone... there are lots of options.
- goatinaboat 6y agoTo me it sounds like those people who left can command a very high day rate to help with the immediate aftermath Those people are probably employed somewhere else and would need very significant incentives. More than likely their current employers would take a dim view of them freelancing to help a competitor, so: enough to take a year off or even to retire on. Basically the idea of hiring back former employees as consultants is a non-starter.
- IgorPartola 6y agoWe don’t know that the records are still in the database. This could be a matter of ON DELETE CASCADE followed by a delete of what looked like innocent data and then boom it’s all gone. Then the integration updates bind which helpfully erases all records and then once there are no domain names to talk to everything is down. I would be looking at the Internet Archive and Whois too figure out some basic domains and what the registrars are and go from there but of course depends on what business they are in and what their domain names were doing.
- ragnese 6y agoYeah, the poster didn't make totally clear whether all of their business records are totally gone, but it does sound like that's the case. The ERP was very likely the source of truth for everything- including employee information that the poster says is gone (can't send SMS to remote employees). So, it probably is that bad...
- Nextgrid 6y agoNothing in his post suggests the ERP's DB is gone, merely that the mechanism which populates DNS records from the ERP has broken down, which probably has ripple effects on everything else as their internal DNS is broken and services can't talk to each other. But the data should still be there, and it's a matter of reverse-engineering how to go from "ERP DB" to "DNS zone file".
- ragnese 6y agoOkay, fair enough. I probably skimmed the post too quickly.
- Semaphor 6y ago> Skimmed through the MySQL and Postgres databases, no signs of anything domain-related. I'm betting it's all stored in the custom integrated [and encrypted] KV database.. -- https://old.reddit.com/r/sysadmin/comments/ma4mwl/the_absolute_worst_case_scenario_happened_what/grqu1i3/ https://old.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
- rebuilder 6y agoFrom a reply the poster made in that thread: "So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don't have an authentication key or the scripts used to pull the DNS domain / record information from the ERP system anymore. The API's are undocumented, and there's only one key that can access those records." Elsewhere, they say that two years ago, they found they were unable to restore from backups...
- ansible 6y ago> Elsewhere, they say that two years ago, they found they were unable to restore from backups... So their business actually failed a couple years ago, and they have only noticed this month.
- lupire 6y agoThey noticed 2 years ago, and the ship sank below the water line this month.
- TeMPOraL 6y agoThey were already dead 2 years ago, the body just didn't realize it and kept moving until last Friday...
- ptero 6y agoThat isn't the failure of the business; backup failure is a screwup of the IT, which I think is not that uncommon. But the fact that after this event the management did not get it fixed (and requested that IT demo it, three times, with no failures) is an existential failure.
- koonsolo 6y agoThe technical people noticed, left, and they had to hire new ones who didn't know yet. Now the new ones also know.
- batch12 6y agoI agree. My first reaction was, "I think I could fix this". Worst case this may be be proceeded by a forensic recovery of the data. This doesn't seem impossible at all unless there is some other factor at play like ransomware/physical damage.
- Mauricebranagh 6y agoI did something similar for a customer of ours who was a major UK cinema chain - they had had a falling out with some small company that had built an online booking system. Highlights: 1 Bookings where made at individual cinemas 2 The network was a packet radio system that blocked :-) only Transaction could be in flight (Leicester square had two channels) 3 Core of system was Perl running on three windows NT machines which manipulated a shonky screen based system (Pacer Cats) 4 Trying to sell tickets online for Phantom menace system crashed at midnight (when tickets went on sale) we made 1 transaction at 6am % And no credentials so I had to research how to crack NT - I did consider setting up or fleet of a dozen or so of our suns as a cluster to crack NT, but not sure if the internal security would have liked that (aka secret squirrels and the bit that Bruce Shenier worked for
- Nextgrid 6y agoI'd love to read a detailed write-up about this. Why packet radio - what was the real-time requirement and why couldn't it be fulfilled by an internet connection (I understand internet was primitive back then, but ISDN should've been able to give you enough bandwidth?).
- Mauricebranagh 6y agoWe never understood why the where using packet radio as you say in 1999 ip via ISDN would have been fine. Its complicated by at the time the total lock on ticket booking systems cinema and theatre by a particular company - who wanted 1/4 mill plus to update the system. It was an interesting project I had to get cover from a v senior manger to investigate how to break into NT.