5 ms·

> Holy crap. A tape backup solution that doesn't allow the tape to be read by any other PC? That's madness. What is needed is the backup catalog. This is fairl

> Holy crap. A tape backup solution that doesn't allow the tape to be read by any other PC? That's madness. What is needed is the backup catalog. This is fairly standard on a lot of tape-related software, even open source; see for example "Bacula Tape Restore Without Database": * http://www.dayaro.com/?p=122 http://www.dayaro.com/?p=122 When I was still doing tape backups the (commercial) backup software we were using would e-mail us the bootstrap information daily in case we had to do a from-scratch data centre restore. The first step would get a base OS going, then install the backup software, then import the catalog. From there you can restore everything else. (The software in question allowed restores even without a license (key?), so that even if you lost that, you could still get going.)

tinus_hn 3y ago
Obviously to know what to restore, you need to index the data on the tapes. Tape is not a random access medium, there is no way around this. This is only for a complete disaster scenario, if you’re restoring one PC or one file, you would still have the backup server and the database. But if you don’t, you need to run the command to reconstruct the database.
- ShadowBanThis01 3y ago
  There is a way around this: You allocate enough space at the beginning (or the end, or both) of the tape for a catalog. There are gigabytes on these tapes; they could have reserved enough space to store millions of filenames and indices.
  - beachy 3y ago
    If the catalog was at the start of the tape, how would you expand it when adding more files to the tape? And if the catalog was at the end of the tape, how would you add more files at all?
    - jonhohle 3y ago
      There are lots of append-only data structures that would support this, but would also require scanning the tape to reconstruct the catalog.
      - KMag 3y ago
        > but would also require scanning the tape to reconstruct the catalog. If the index consists of a b+-tree/b*-tree interleaved with the data (with the root of the index the last record in the archive), a single backward pass across the tape (including rapid seeks to skip irrelevant data) is sufficient to efficiently restore everything. This should be very close in throughput and latency to restoring with an index on random-access storage. (Though, if you're restoring to a filesystem that doesn't support sparse files, writing the data back in reverse order is going to involve twice as much write traffic. On a side note, I've heard HFS+ supports efficient prepending of data to files.) In other words, yes, you need to scan the tape to reconstruct the catalog, but since the tape isn't random access, you need to scan/seek through the entire tape anyway (even if you have a separate index on random-access media). If you're smart about your data structures, it can all be done in a single backward pass over the tape (with no forward pass). Keeping a second b+-tree/b*-tree interleaved with the data (keyed by archive write time) makes point-in-time snapshot backups just as easy, all with a single reverse pass over the tape and efficient seeks across unneeded sections of tape.
    - mmis1000 3y ago
      > And if the catalog was at the end of the tape, how would you add more files at all? modern zip softwares just remove the whole index at end, add file, reconstruct the index and append again.
      - larusso 3y ago
        But you had multiple backups on these tapes. If you rewrite the index how do you restore from a certain day?
        3y ago
  - tinus_hn 3y ago
    Then you would have to rewind the tape at the end, which is not what you want. You want to write the tape and keep it at that position and be done. If you write the catalog at the end, you have to rewind and read the whole tape to find it and read it. Which is not an improvement over reading the tape and reconstructing it. This is all either impossible or very difficult to fix when there is actually not a problem, if there is a disaster and the database is lost, you just read the tapes to reconstruct it.
IshKebab 3y ago
Wouldn't it make sense to also write the backup catalog to the tape though? Seems like a very obvious thing to do to me.
- fsckboy 3y ago
  you'd have to put the catalog at the end of the tape, but in that case you might as well rebuild the catalog by simply reading the tape on your way to the end (yeah, if the tape is partially unreadable blah blah backup of your backup...)
  - IshKebab 3y ago
    > you might as well rebuild the catalog by simply reading the tape on your way to the end Right but is that actually possible? From what people are saying it sounds like it isn't, but you rightly assumed that it is because anything else would be incredibly dumb.
- throw0101b 3y ago
  > Wouldn't it make sense to also write the backup catalog to the tape though? Seems like a very obvious thing to do to me. The catalog would be written to tape regularly: this is what would gets e-mailed out. But it wouldn't necessarily be written to every tape. Remember that the catalog changes every day: you'd have Version 3142 of the catalog at the beginning of Monday, but then you'd back a bunch of clients, so that catalog would now be out-of-date, so Version 3143 would have to be written out for disaster recovery purposes (and you'd get an e-mail telling you about the tape labels and offsets for it). In a DR situation you'd go through your e-mails and restore the catalog listed in the most recent e-mail.
  - blincoln 3y ago
    50GB was an enormous amount of space in the late 90s. Why wouldn't each file on the tape have something like begin/end sentinels and metadata about the file so that the current catalogue could be rebuilt in a DR scenario by just spinning through the whole tape? I'm with the OP - depending 100% on a file that's not on the tape to restore the tape is bonkers. It's fine as optimization, but there should have always been a way to restore from the tape alone in an emergency.
ilyt 3y ago
Right, the on-PC database act as index to data on the tape. That's pretty standard. But having format where you can't recreate the index from data easily is just abhorrently bad coding...
deleted 3y ago
[deleted]
hnlmorg 3y ago
Storing the catalogue on the PC is standard. But being able to rebuild that catalogue from scratch is also standard. I’ve not used any tapes before now where you couldn’t recover the catalogue.

tinus_hn 3y ago

Obviously to know what to restore, you need to index the data on the tapes. Tape is not a random access medium, there is no way around this. This is only for a complete disaster scenario, if you’re restoring one PC or one file, you would still have the backup server and the database. But if you don’t, you need to run the command to reconstruct the database.

ShadowBanThis01 3y ago
There is a way around this: You allocate enough space at the beginning (or the end, or both) of the tape for a catalog. There are gigabytes on these tapes; they could have reserved enough space to store millions of filenames and indices.
- beachy 3y ago
  If the catalog was at the start of the tape, how would you expand it when adding more files to the tape? And if the catalog was at the end of the tape, how would you add more files at all?
  - jonhohle 3y ago
    There are lots of append-only data structures that would support this, but would also require scanning the tape to reconstruct the catalog.
    - KMag 3y ago
      > but would also require scanning the tape to reconstruct the catalog. If the index consists of a b+-tree/b*-tree interleaved with the data (with the root of the index the last record in the archive), a single backward pass across the tape (including rapid seeks to skip irrelevant data) is sufficient to efficiently restore everything. This should be very close in throughput and latency to restoring with an index on random-access storage. (Though, if you're restoring to a filesystem that doesn't support sparse files, writing the data back in reverse order is going to involve twice as much write traffic. On a side note, I've heard HFS+ supports efficient prepending of data to files.) In other words, yes, you need to scan the tape to reconstruct the catalog, but since the tape isn't random access, you need to scan/seek through the entire tape anyway (even if you have a separate index on random-access media). If you're smart about your data structures, it can all be done in a single backward pass over the tape (with no forward pass). Keeping a second b+-tree/b*-tree interleaved with the data (keyed by archive write time) makes point-in-time snapshot backups just as easy, all with a single reverse pass over the tape and efficient seeks across unneeded sections of tape.
  - mmis1000 3y ago
    > And if the catalog was at the end of the tape, how would you add more files at all? modern zip softwares just remove the whole index at end, add file, reconstruct the index and append again.
    - larusso 3y ago
      But you had multiple backups on these tapes. If you rewrite the index how do you restore from a certain day?
      - 3y ago
- tinus_hn 3y ago
  Then you would have to rewind the tape at the end, which is not what you want. You want to write the tape and keep it at that position and be done. If you write the catalog at the end, you have to rewind and read the whole tape to find it and read it. Which is not an improvement over reading the tape and reconstructing it. This is all either impossible or very difficult to fix when there is actually not a problem, if there is a disaster and the database is lost, you just read the tapes to reconstruct it.

IshKebab 3y ago

Wouldn't it make sense to also write the backup catalog to the tape though? Seems like a very obvious thing to do to me.

fsckboy 3y ago
you'd have to put the catalog at the end of the tape, but in that case you might as well rebuild the catalog by simply reading the tape on your way to the end (yeah, if the tape is partially unreadable blah blah backup of your backup...)
- IshKebab 3y ago
  > you might as well rebuild the catalog by simply reading the tape on your way to the end Right but is that actually possible? From what people are saying it sounds like it isn't, but you rightly assumed that it is because anything else would be incredibly dumb.
throw0101b 3y ago
> Wouldn't it make sense to also write the backup catalog to the tape though? Seems like a very obvious thing to do to me. The catalog would be written to tape regularly: this is what would gets e-mailed out. But it wouldn't necessarily be written to every tape. Remember that the catalog changes every day: you'd have Version 3142 of the catalog at the beginning of Monday, but then you'd back a bunch of clients, so that catalog would now be out-of-date, so Version 3143 would have to be written out for disaster recovery purposes (and you'd get an e-mail telling you about the tape labels and offsets for it). In a DR situation you'd go through your e-mails and restore the catalog listed in the most recent e-mail.
- blincoln 3y ago
  50GB was an enormous amount of space in the late 90s. Why wouldn't each file on the tape have something like begin/end sentinels and metadata about the file so that the current catalogue could be rebuilt in a DR scenario by just spinning through the whole tape? I'm with the OP - depending 100% on a file that's not on the tape to restore the tape is bonkers. It's fine as optimization, but there should have always been a way to restore from the tape alone in an emergency.

ilyt 3y ago

Right, the on-PC database act as index to data on the tape. That's pretty standard. But having format where you can't recreate the index from data easily is just abhorrently bad coding...

deleted 3y ago

[deleted]

hnlmorg 3y ago

Storing the catalogue on the PC is standard. But being able to rebuild that catalogue from scratch is also standard. I’ve not used any tapes before now where you couldn’t recover the catalogue.