3 ms·

When I worked at IBM Research we got access to the latest hardware (IBM RS/6000) and the latest software (AIX), written by guys down the hall. This machine was

by daly 9y ago

When I worked at IBM Research we got access to the latest hardware (IBM RS/6000) and the latest software (AIX), written by guys down the hall. This machine was "blazingly fast" for its time and had a useful feature for us. You could add a new disk drive and dynamically extend your filesystem over all the available drives. We grew to a 3-drive system which contained all our master source code. One day a hard disk failed. The system couldn't reboot because the operating system needed "quorum" (all the drives) and, in particular, it needed part of the operating system on the failed drive. So we lost ALL of our source code. You didn't keep sources on your own drive because you used SCCS to check them in/out and you didn't have enough space for very much. All eyes turn to me. Sigh. The game was to hook up a disk drive to another machine, open the drive for raw read, and read the drive sector by sector. The theory was that the failed drive could still be read, just not certain sectors. Anyone who tries this will discover that there are a LOT of details about drives they never knew, such as "overflow sectors" and file-system layouts and log-file formats. Plus there are multiple copies of code pieces because the new version is allocated but the old version is in "free space" but not reclaimed. All in all, it was a long time recovering the source tree. Raw read works. You knew about that, right? You can handle head/track/sector maps on your SSD, right? You know your B-trees cold and your log-structured filesystems, right? Forget the Order of an algorithm. Can you recover vital company assets from a failed hard drive? Fun times. Don't you miss them?

DrScump 9y ago
I had to do that with a mag tape once, but it was just a case of monkeying around with JCL parameters and DDing a few hundred bytes into files one by one until I figured out where the surviving data was, then just use the right JCL to make it transparent to the program. I couldn't figure out why I couldn't get BLP to work, though (NL did the trick).
cat199 9y ago
Miss those times? Nothing specific to those times about having a terrible backup strategy.. Magtape existed from the very early days and definitely was around during this fiasco.. similarly people still don't run backups when they should.. which reminds me, I need to kick off an offsite backup :)
- daly 9y ago
  So, backup to tape... Story there. I did a backup system on the mainframe to tapes. The first issue was that the machine room operators would 'recycle' the tapes they dismounted. So while we thought we had backups, the operators just assumed they were 'scratch tapes'. Second, after having solved that problem we found that a tape was giving us hundreds of thousands of duplicate records. When I visited the actual tape drive I could see that the drive found a bad spot on the tape, did 'auto-recovery' to reread the tape, rinse and repeat. The end result was a physically transparent section of tape. I used to carry it around in my wallet to explain that even backups fail.
  - DrScump 9y ago
    Was that before tape catalogs and volume labels? That was well before my time. In the OS/MVS environment I was in in the 80s, the job's JCL would identify a data set and a relative version number (+1 when creating, 0 for current, -1 for its predecessor, etc.), then the tape catalog would look up the volume number, which was written on the tape case itself (and usually correct). If the operator mounted a wrong tape (either wrong dataset or version), the tape would be kicked off and you'd be prompted again ("you dummy humans can't even handle unpacked decimal?!") It was typical that backup copies be made via an additional DD pass.

by daly 9y ago

DrScump 9y ago

I had to do that with a mag tape once, but it was just a case of monkeying around with JCL parameters and DDing a few hundred bytes into files one by one until I figured out where the surviving data was, then just use the right JCL to make it transparent to the program. I couldn't figure out why I couldn't get BLP to work, though (NL did the trick).

cat199 9y ago

Miss those times? Nothing specific to those times about having a terrible backup strategy.. Magtape existed from the very early days and definitely was around during this fiasco.. similarly people still don't run backups when they should.. which reminds me, I need to kick off an offsite backup :)

daly 9y ago
So, backup to tape... Story there. I did a backup system on the mainframe to tapes. The first issue was that the machine room operators would 'recycle' the tapes they dismounted. So while we thought we had backups, the operators just assumed they were 'scratch tapes'. Second, after having solved that problem we found that a tape was giving us hundreds of thousands of duplicate records. When I visited the actual tape drive I could see that the drive found a bad spot on the tape, did 'auto-recovery' to reread the tape, rinse and repeat. The end result was a physically transparent section of tape. I used to carry it around in my wallet to explain that even backups fail.
- DrScump 9y ago
  Was that before tape catalogs and volume labels? That was well before my time. In the OS/MVS environment I was in in the 80s, the job's JCL would identify a data set and a relative version number (+1 when creating, 0 for current, -1 for its predecessor, etc.), then the tape catalog would look up the volume number, which was written on the tape case itself (and usually correct). If the operator mounted a wrong tape (either wrong dataset or version), the tape would be kicked off and you'd be prompted again ("you dummy humans can't even handle unpacked decimal?!") It was typical that backup copies be made via an additional DD pass.