5 ms·

> Tar is pretty unusual for an archive file format. There's no archive header, no index of files to fascilitate seeking, no magic bytes to help file and its ilk

> Tar is pretty unusual for an archive file format. There's no archive header, no index of files to fascilitate seeking, no magic bytes to help file and its ilk detect whether a file is a tar archive, no footer, no archive-wide metadata. The only kind of thing in a tar file is a file object. tar = tape archive. It was designed around streaming files to a tape drive in a serial fashion. Tape bandwidth has almost always been slow compared to source data devices. You need to start writing to the target device as fast as possible. Taking time to construct a full index just delays starting the slowest stage of things, while providing only relatively minimal benefit. Generating an index is fairly straightforward, the file headers give you the information you need, including what you need to know to get to the next file header. Your only bottleneck is just how fast random reads are. If the file is on disk, it should be relatively trivial. It's even possible to do this relatively cheaply with Range headers if you're looking at a tar file stored on an HTTP endpoint, though you'll likely want to think about some smart pre-fetching behaviour.

vetinari 4y ago
The issue with tapes is not the bandwidth, it is the seek latency, could be around ~2 minutes end to end (with the correct tape already loaded). So tapes are not used as random-access, thus no central index on tape archive. The backup software systems keep their catalogs elsewhere.
- Waterluvian 4y ago
  When I was a beginner I once accidentally created an Amazon RDS Postgres database backed by tape. I was so flustered by how slow it was until I saw the setting.
  - ufo 4y ago
    Lol! I'm curious how slow it got... How slow are we talking about here?
    - Waterluvian 4y ago
      A non-cached GET for a single database record was like 7 seconds. It was especially scary because I was in the “I have no idea what I’m doing, and I’m at a tech startup and am a team of one.”
      - lazide 4y ago
        Still too fast for it to have been tape hah. If it was on tape, it would have taken minutes
  - emmjay_ 4y ago
    I'm not sure how this is possible. You can only choose EBS backed storage for RDS Postgres. Do you mean 'magnetic' storage? That's not tape.
    - Waterluvian 4y ago
      It could be tape. But perhaps you’re right that it’s HDDs I guess? I assumed most of their normal storage is hard disk and not SSD. They’ve got a lot. Edit: I checked. You’re 100% right. Well now my anecdote sucks! :) And now, seven years later, I’m back to start on wondering why it was so brutally slow.
      - emmjay_ 4y ago
        Heh. It's not tape. :) If you provisioned a really small amount of GP2, you can run out of I/O credits pretty quickly and get throttled to baseline performance (and it used to be 7 times the storage provisioned, or some such, and now it's 100 IOPS). That's brutally slow for a working database.
    - oogali 4y ago
      It's not possible. GP was likely confused by the different types of hard drives (standard, sc1, st1), and assumed it was tape or was incorrectly told by someone else that it was tape. If I squint hard enough, the only viable explanation I could come up with is GP said "I created the database using the st1 storage type" and someone responded "st!? As in a UNIX SCSI tape drive[0]?" 0: https://man7.org/linux/man-pages/man4/st.4.html https://man7.org/linux/man-pages/man4/st.4.html
  - SoftTalker 4y ago
    When I was in school, the professor of our operating systems class told us about a virtual memory project implementation that had the swapfile on tape. It worked, but it wasn't anything you'd actually ever use.
    - Waterluvian 4y ago
      When I was 9 or 10 I once set virtual memory on my Macintosh to live in a 44MB SCSI cartridge drive. It was brutally slow.
- Twirrim 4y ago
  Latency is absolutely killer with tapes, but bandwidth is still a major issue. The current generation LTO-9 is only capable of 400MB/s in throughput. That's still slower than a SATA3 drive. Last time I used tape in earnest was a while back though (2005ish), LTO-2 ruled the roost at a whopping 40MB/s, with LTO-3 just making an appearance. Around the time I left the company they were building out a disk storage array as a backup target, and shifting tape further and further away from the main backup path. Data production and storage needs across the industry had long been growing at a greater rate than tape capacity and performance.
  - vetinari 4y ago
    > The current generation LTO-9 is only capable of 400MB/s in throughput. That's still slower than a SATA3 drive. True, but that's the speed of SATA3 interface, or SSD drives. Single spinning rust drive can realistically achieve 150-250 MB/s.
forrestthewoods 4y ago
- mort96 4y ago
  You don't loathe tar files due to a deficiency in the tar format. You loathe tar files because your operating system vendor doesn't build in support for one of the most widely used archive formats in the world. Complain to Microsoft, not to people who use tar. Tar might've been designed for use with tape, but no part of its design makes the HDD/SSD use-case worse just to work better with tape.
  - guipsp 4y ago
    Well, the lack of index for one
    - spicybright 4y ago
      It seems fairly trivial to scan the whole file and build an in memory index to speed things up, and I'm sure most viewers do. I can't say I've ever had significant speed issues with most archives to see this as a problem.
      - lazide 4y ago
        Good luck doing that with a multi-terabyte compressed tar file!
      - nrclark 4y ago
        I see tar files used with compression pretty often, usually gzip or xz. That can mess up the ability to seek around in the tar archive and build a good index. I use tar.gz as my go-to archive format, don't get me wrong. It's easy, well-supported, handles POSIX concepts well, and is pretty universal. But seekability isn't it's strong-suit.
      - jcrawfordor 4y ago
        It's actually pretty terrible in practice. Keep in mind that most common tar files have also been run through a stream compression algorithm, which means scanning the archive for file headers requires decompressing the entire thing. Some software does index tar files automatically (and of course the typical CLI tools will if you want) but for a large archive, say 20gb, I have had to stare at a progress bar for 30s+ while the tool indexed the archive. Extracting a single file required another 30s+ wait while it scanned the archive again to get back to that file, since with gzip compression it's not trivial to store enough state to be able to decompress at a point without having to decompress everything up to there.
jcrawfordor 4y ago
I wouldn't view this as being about write speed... keep in mind that the common ZIP implements a very simple optimization to address this exact issue, which is writing the index at the end of the file instead of at the beginning. Actually there were tools that did this when writing to tape as well, but tar wasn't one of them. In fact a lot of common tape archiving solutions would just put the index on its own dedicated tape, which of course makes sense when you consider that a computer operator would probably need to find out which tape a record of interest was on. Similarly it is possible to build an "out-of-band" index for a tar file but I'm not aware of a tool that does so... probably mostly because this is way less practical if the tarball has been compressed (compressing the files in the tarball instead fixes this issue and I've written tools that do it that way before, but it doesn't seem to be common either). The issue is much more about seek speed. Putting the master header at the end of ZIP files requires initially reading them backwards to locate the beginning of that data structure. This works fine on random access devices but is of course pretty untenable on tape. Even if the tape drive could actually read backwards without seeking to read each successive block in the normal direction (I'm sure there's at least one weird tape drive out there that could but not the common models, although if you read very small blocks you could get pretty good performance doing this as long as you kept the seeks within the buffer columns), the seek time to get to the end of the tape and then back to the beginning could be minutes.
- bluedino 4y ago
  Backup programs (at least as early as the 90's) would get around this by creating index or catalog files for the backup tape, and keeping those locally. You could re-generate the catalog file by re-indexing the backup tape, but of course that took a considerable amount of time. If you were restoring to a new system and no longer had the catalog files, for example. Another benefit was that you could quickly browse the contents of your backups without having to read or even insert the tape.
wglb 4y ago
> Tape bandwidth has almost always been slow compared to source data devices This is not the case.
iforgotpassword 4y ago
It's only really seekable if you don't compress it further. That's because a tar file gets compressed after the fact. Even with compression formats that support starting decompression somewhere in the middle, you now either need an additional file index that tells you which files there are and where in the compressed stream to start reading, or you have to try and scan the while archive as you would with an uncompressed tar by seeking along the headers and hope the files are large enough that you can actually skip large enough chunks to matter. It is indeed really odd that the *nix world never came up with their own, agreed upon, indexed or at least efficiency seekable archive format. I think if one of the extensions would have added per-file compression that would have had the highest chance of success.
IshKebab 4y ago
> Your only bottleneck is how fast random reads are. If the file is on disk, it should be relatively trivial. Ah yes, how fast is a random read to the middle of a 1TB file... that has been compressed as a single gzip stream...

vetinari 4y ago

The issue with tapes is not the bandwidth, it is the seek latency, could be around ~2 minutes end to end (with the correct tape already loaded). So tapes are not used as random-access, thus no central index on tape archive. The backup software systems keep their catalogs elsewhere.

Waterluvian 4y ago
When I was a beginner I once accidentally created an Amazon RDS Postgres database backed by tape. I was so flustered by how slow it was until I saw the setting.
- ufo 4y ago
  Lol! I'm curious how slow it got... How slow are we talking about here?
  - Waterluvian 4y ago
    A non-cached GET for a single database record was like 7 seconds. It was especially scary because I was in the “I have no idea what I’m doing, and I’m at a tech startup and am a team of one.”
    - lazide 4y ago
      Still too fast for it to have been tape hah. If it was on tape, it would have taken minutes
- emmjay_ 4y ago
  I'm not sure how this is possible. You can only choose EBS backed storage for RDS Postgres. Do you mean 'magnetic' storage? That's not tape.
  - Waterluvian 4y ago
    It could be tape. But perhaps you’re right that it’s HDDs I guess? I assumed most of their normal storage is hard disk and not SSD. They’ve got a lot. Edit: I checked. You’re 100% right. Well now my anecdote sucks! :) And now, seven years later, I’m back to start on wondering why it was so brutally slow.
    - emmjay_ 4y ago
      Heh. It's not tape. :) If you provisioned a really small amount of GP2, you can run out of I/O credits pretty quickly and get throttled to baseline performance (and it used to be 7 times the storage provisioned, or some such, and now it's 100 IOPS). That's brutally slow for a working database.
  - oogali 4y ago
    It's not possible. GP was likely confused by the different types of hard drives (standard, sc1, st1), and assumed it was tape or was incorrectly told by someone else that it was tape. If I squint hard enough, the only viable explanation I could come up with is GP said "I created the database using the st1 storage type" and someone responded "st!? As in a UNIX SCSI tape drive[0]?" 0: https://man7.org/linux/man-pages/man4/st.4.html https://man7.org/linux/man-pages/man4/st.4.html
- SoftTalker 4y ago
  When I was in school, the professor of our operating systems class told us about a virtual memory project implementation that had the swapfile on tape. It worked, but it wasn't anything you'd actually ever use.
  - Waterluvian 4y ago
    When I was 9 or 10 I once set virtual memory on my Macintosh to live in a 44MB SCSI cartridge drive. It was brutally slow.
Twirrim 4y ago
Latency is absolutely killer with tapes, but bandwidth is still a major issue. The current generation LTO-9 is only capable of 400MB/s in throughput. That's still slower than a SATA3 drive. Last time I used tape in earnest was a while back though (2005ish), LTO-2 ruled the roost at a whopping 40MB/s, with LTO-3 just making an appearance. Around the time I left the company they were building out a disk storage array as a backup target, and shifting tape further and further away from the main backup path. Data production and storage needs across the industry had long been growing at a greater rate than tape capacity and performance.
- vetinari 4y ago
  > The current generation LTO-9 is only capable of 400MB/s in throughput. That's still slower than a SATA3 drive. True, but that's the speed of SATA3 interface, or SSD drives. Single spinning rust drive can realistically achieve 150-250 MB/s.

forrestthewoods 4y ago

mort96 4y ago
You don't loathe tar files due to a deficiency in the tar format. You loathe tar files because your operating system vendor doesn't build in support for one of the most widely used archive formats in the world. Complain to Microsoft, not to people who use tar. Tar might've been designed for use with tape, but no part of its design makes the HDD/SSD use-case worse just to work better with tape.
- guipsp 4y ago
  Well, the lack of index for one
  - spicybright 4y ago
    It seems fairly trivial to scan the whole file and build an in memory index to speed things up, and I'm sure most viewers do. I can't say I've ever had significant speed issues with most archives to see this as a problem.
    - lazide 4y ago
      Good luck doing that with a multi-terabyte compressed tar file!
    - nrclark 4y ago
      I see tar files used with compression pretty often, usually gzip or xz. That can mess up the ability to seek around in the tar archive and build a good index. I use tar.gz as my go-to archive format, don't get me wrong. It's easy, well-supported, handles POSIX concepts well, and is pretty universal. But seekability isn't it's strong-suit.
    - jcrawfordor 4y ago
      It's actually pretty terrible in practice. Keep in mind that most common tar files have also been run through a stream compression algorithm, which means scanning the archive for file headers requires decompressing the entire thing. Some software does index tar files automatically (and of course the typical CLI tools will if you want) but for a large archive, say 20gb, I have had to stare at a progress bar for 30s+ while the tool indexed the archive. Extracting a single file required another 30s+ wait while it scanned the archive again to get back to that file, since with gzip compression it's not trivial to store enough state to be able to decompress at a point without having to decompress everything up to there.

jcrawfordor 4y ago

I wouldn't view this as being about write speed... keep in mind that the common ZIP implements a very simple optimization to address this exact issue, which is writing the index at the end of the file instead of at the beginning. Actually there were tools that did this when writing to tape as well, but tar wasn't one of them. In fact a lot of common tape archiving solutions would just put the index on its own dedicated tape, which of course makes sense when you consider that a computer operator would probably need to find out which tape a record of interest was on. Similarly it is possible to build an "out-of-band" index for a tar file but I'm not aware of a tool that does so... probably mostly because this is way less practical if the tarball has been compressed (compressing the files in the tarball instead fixes this issue and I've written tools that do it that way before, but it doesn't seem to be common either). The issue is much more about seek speed. Putting the master header at the end of ZIP files requires initially reading them backwards to locate the beginning of that data structure. This works fine on random access devices but is of course pretty untenable on tape. Even if the tape drive could actually read backwards without seeking to read each successive block in the normal direction (I'm sure there's at least one weird tape drive out there that could but not the common models, although if you read very small blocks you could get pretty good performance doing this as long as you kept the seeks within the buffer columns), the seek time to get to the end of the tape and then back to the beginning could be minutes.

bluedino 4y ago
Backup programs (at least as early as the 90's) would get around this by creating index or catalog files for the backup tape, and keeping those locally. You could re-generate the catalog file by re-indexing the backup tape, but of course that took a considerable amount of time. If you were restoring to a new system and no longer had the catalog files, for example. Another benefit was that you could quickly browse the contents of your backups without having to read or even insert the tape.

wglb 4y ago

> Tape bandwidth has almost always been slow compared to source data devices This is not the case.

iforgotpassword 4y ago

It's only really seekable if you don't compress it further. That's because a tar file gets compressed after the fact. Even with compression formats that support starting decompression somewhere in the middle, you now either need an additional file index that tells you which files there are and where in the compressed stream to start reading, or you have to try and scan the while archive as you would with an uncompressed tar by seeking along the headers and hope the files are large enough that you can actually skip large enough chunks to matter. It is indeed really odd that the *nix world never came up with their own, agreed upon, indexed or at least efficiency seekable archive format. I think if one of the extensions would have added per-file compression that would have had the highest chance of success.

IshKebab 4y ago

> Your only bottleneck is how fast random reads are. If the file is on disk, it should be relatively trivial. Ah yes, how fast is a random read to the middle of a 1TB file... that has been compressed as a single gzip stream...