4 ms·

After having helped on a lot of tape storage implementations, I get why it's useful for quick estimates on backup solutions. Compression really is a game chang

by csydas 3y ago

After having helped on a lot of tape storage implementations, I get why it's useful for quick estimates on backup solutions. Compression really is a game changer when you're over a certain amount of data you need to protect, and backup technology does some really clever space savings at basically all stages, making it hard to find a good and easy metric to help sell to management. It's a lot easier to tell management: "These tapes hold 50 TB of production data, and we have about 45 TB in production. Backups and restores will be faster, we'll have fewer tapes to manage for backups" than it is to explain the different types of data you need to move to tape, the expected compression ratio, etc. That being said, I do agree they market it too much and it gets confusing, as it should only be a reference for implementation and quick calculations.

ta1243 3y ago
Surely how well it compresses depends on the material you're backing up It seems they went all in on the "compressed" value after missing the doubling of the real space in LTO6.
- sebazzz 3y ago
  Especially if you backup encrypted data (which isn't unusual - to have data encrypted at rest) you will have a compression ratio of 100% or even more (metadata of the compression needs to be stored too)
  - monocasa 3y ago
    The drives will do encryption too so that you don't have to pick between encryption at rest and compression.
    - sebazzz 3y ago
      From the SSL world we know that encrypting compressed data can lead to vulnerabilities in the encryption. Does that nog apply here? https://en.wikipedia.org/wiki/CRIME?wprov=sfti1 https://en.wikipedia.org/wiki/CRIME?wprov=sfti1
      - monocasa 3y ago
        It doesn't really apply as both CRIME and BREACH rely on small N on the payload size, and heavy use of repeated chosen plaintexts to iterate through the secret state space.
lovasoa 3y ago
Isn't it harder when you have to later report to management that actually, your 45TB of production data do not fit on the 50TB tape ? I suppose compression ratios vary a lot between users of these things, right ? And if you are backing up user-generated data, you don't know in advance how well it will compress.
- sumtechguy 3y ago
  In this case "The latest TS1170 drive supports 50TB JF media cartridges with 150TB compressed capacity through 3:1 compression" So looks like 50TB raw if I am reading this right. For these sorts of solutions you are usually not using just 1 tape. The tape is the cheap part. The drives cost a decent amount though. So in your case if the data does not fit. You buy another tape and pop it into the existing drive and finish the backup. Also you would have some sort of rotation schedule and at least a few dozen tapes. If you are doing it right offsite and onsite rotation too. An extra tape in the mix of that is not something they would worry about. When I did this for a small org I had the weekends to do a full backup of all of our computers. Then during the week I would do incrementals. At the end of the week I would ship off the prev week to an offsite and start the next full backup. Not perfect but gave us a onsite 1 day return offsite (building gone) 1 week out. The offsite would keep about 6 months of tapes and return them about once a month.
  - klodolph 3y ago
    Yeah. For most operations, you pay for drive throughput & redundancy, and the actual storage itself is “too cheap to meter”. Except when it isn’t, but those customers know who they are.
    - dylan604 3y ago
      These are the same customers that only make 1 tape of the data. One facility I was at required 3 copies written by 3 different writers. These were in large robot operated units, so it was all automated by our front end. One copy stayed locally, the other 2 went to 2 different offsite locations in different parts of the country with the closest being in another town across state. Each tape had to be removed from the writer and placed into a different drive to do a read back for comparison. In the LTO4 days, we would catch lots of write errors like this. I wonder if the newer formats are any better about these errors? How I miss the old days of tar/mt commands, er, no, who am I kidding!
Johnny555 3y ago
>After having helped on a lot of tape storage implementations, I get why it's useful for quick estimates on backup solutions. Can't you just do that estimate in your head? If you know your data mostly text and has a 3:1 compression ratio, when you buy your tapes you can multiply the raw capacity by 3, if it's mostly compressed images (i.e. gif, jpg, etc), you're probably not going see much compression at all. Does going with some tape drive manufacturer's estimate really help you?
Dylan16807 3y ago
> "These tapes hold 50 TB of production data, and we have about 45 TB in production. Backups and restores will be faster, we'll have fewer tapes to manage for backups" Sure, if "production" means database. Which is sometimes sensible and sometimes extremely far off. > than it is to explain the different types of data you need to move to tape, the expected compression ratio, etc. To some extent, but you mostly just need two numbers. Database capacity and photo/video/audio capacity.
dylan604 3y ago
Not once in my days of using tape storage have I ever enabled compression, so it always seems like a waste to my specific use case of storing large files of media formats, specifically video. Unless the video was in some sort of RAW uncompressed format, which is close to never, and video in a CoDec would not benefit from this compression.
- leptons 3y ago
  Tape compression is probably not meant for your use cases, it's likely meant for large government institutions or corporations that generate petabytes of uncompressed text data a week - so instead of needing to manage compression as a separate process, the tape drive just does it without any other processing needed. The data can go from collection/creation to long-term storage without any other steps in between. For some organizations that's a big win. For the average home user backing up their plex, it might seem unnecessary.
  - dylan604 3y ago
    Lots of home users backing up their Plex with an LTO5 or greater are there?
    - leptons 3y ago
      I am one of those home users with an LTO5, but I do not use tape compression and I have no illusions about what it was made for. I write tapes without any compression, because the stuff I write can't be compressed. Tape compression wasn't made for my use case, it was made for organizations that generate petabytes of text data.
  - dredmorbius 3y ago
    There's still a fair bit of mainframe-format data (zoned decimal, etc.) out there. These are typically in defined-data formats, a/k/a hierarchical , which is sort of a hybrid between data files and text files ("records" are lines, but often with multiple formats within a single file). These can achieve compression ratios ~= 90%. Standard written text typically runs about 50%. Compressed audio, image, and video data of course compress very little, if any.

by csydas 3y ago

ta1243 3y ago

Surely how well it compresses depends on the material you're backing up It seems they went all in on the "compressed" value after missing the doubling of the real space in LTO6.

sebazzz 3y ago
Especially if you backup encrypted data (which isn't unusual - to have data encrypted at rest) you will have a compression ratio of 100% or even more (metadata of the compression needs to be stored too)
- monocasa 3y ago
  The drives will do encryption too so that you don't have to pick between encryption at rest and compression.
  - sebazzz 3y ago
    From the SSL world we know that encrypting compressed data can lead to vulnerabilities in the encryption. Does that nog apply here? https://en.wikipedia.org/wiki/CRIME?wprov=sfti1 https://en.wikipedia.org/wiki/CRIME?wprov=sfti1
    - monocasa 3y ago
      It doesn't really apply as both CRIME and BREACH rely on small N on the payload size, and heavy use of repeated chosen plaintexts to iterate through the secret state space.

lovasoa 3y ago

Isn't it harder when you have to later report to management that actually, your 45TB of production data do not fit on the 50TB tape ? I suppose compression ratios vary a lot between users of these things, right ? And if you are backing up user-generated data, you don't know in advance how well it will compress.

sumtechguy 3y ago
In this case "The latest TS1170 drive supports 50TB JF media cartridges with 150TB compressed capacity through 3:1 compression" So looks like 50TB raw if I am reading this right. For these sorts of solutions you are usually not using just 1 tape. The tape is the cheap part. The drives cost a decent amount though. So in your case if the data does not fit. You buy another tape and pop it into the existing drive and finish the backup. Also you would have some sort of rotation schedule and at least a few dozen tapes. If you are doing it right offsite and onsite rotation too. An extra tape in the mix of that is not something they would worry about. When I did this for a small org I had the weekends to do a full backup of all of our computers. Then during the week I would do incrementals. At the end of the week I would ship off the prev week to an offsite and start the next full backup. Not perfect but gave us a onsite 1 day return offsite (building gone) 1 week out. The offsite would keep about 6 months of tapes and return them about once a month.
- klodolph 3y ago
  Yeah. For most operations, you pay for drive throughput & redundancy, and the actual storage itself is “too cheap to meter”. Except when it isn’t, but those customers know who they are.
  - dylan604 3y ago
    These are the same customers that only make 1 tape of the data. One facility I was at required 3 copies written by 3 different writers. These were in large robot operated units, so it was all automated by our front end. One copy stayed locally, the other 2 went to 2 different offsite locations in different parts of the country with the closest being in another town across state. Each tape had to be removed from the writer and placed into a different drive to do a read back for comparison. In the LTO4 days, we would catch lots of write errors like this. I wonder if the newer formats are any better about these errors? How I miss the old days of tar/mt commands, er, no, who am I kidding!

Johnny555 3y ago

>After having helped on a lot of tape storage implementations, I get why it's useful for quick estimates on backup solutions. Can't you just do that estimate in your head? If you know your data mostly text and has a 3:1 compression ratio, when you buy your tapes you can multiply the raw capacity by 3, if it's mostly compressed images (i.e. gif, jpg, etc), you're probably not going see much compression at all. Does going with some tape drive manufacturer's estimate really help you?

Dylan16807 3y ago

> "These tapes hold 50 TB of production data, and we have about 45 TB in production. Backups and restores will be faster, we'll have fewer tapes to manage for backups" Sure, if "production" means database. Which is sometimes sensible and sometimes extremely far off. > than it is to explain the different types of data you need to move to tape, the expected compression ratio, etc. To some extent, but you mostly just need two numbers. Database capacity and photo/video/audio capacity.

dylan604 3y ago

Not once in my days of using tape storage have I ever enabled compression, so it always seems like a waste to my specific use case of storing large files of media formats, specifically video. Unless the video was in some sort of RAW uncompressed format, which is close to never, and video in a CoDec would not benefit from this compression.

leptons 3y ago
Tape compression is probably not meant for your use cases, it's likely meant for large government institutions or corporations that generate petabytes of uncompressed text data a week - so instead of needing to manage compression as a separate process, the tape drive just does it without any other processing needed. The data can go from collection/creation to long-term storage without any other steps in between. For some organizations that's a big win. For the average home user backing up their plex, it might seem unnecessary.
- dylan604 3y ago
  Lots of home users backing up their Plex with an LTO5 or greater are there?
  - leptons 3y ago
    I am one of those home users with an LTO5, but I do not use tape compression and I have no illusions about what it was made for. I write tapes without any compression, because the stuff I write can't be compressed. Tape compression wasn't made for my use case, it was made for organizations that generate petabytes of text data.
- dredmorbius 3y ago
  There's still a fair bit of mainframe-format data (zoned decimal, etc.) out there. These are typically in defined-data formats, a/k/a hierarchical , which is sort of a hybrid between data files and text files ("records" are lines, but often with multiple formats within a single file). These can achieve compression ratios ~= 90%. Standard written text typically runs about 50%. Compressed audio, image, and video data of course compress very little, if any.