5 ms·

Ptar: Replacing .tgz for petabyte-scale S3 archives

57 points by vcoisne 1y ago

nemothekid 1y ago
>By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 backup" if (1) I had an existing system (e.g. tapes), or (2) I need multi-cloud redundancy. Other than that, once I assume something is in S3, I confident it's safe. Obviously this was built over years (decades?) or reliability, and if your DRP requires alternatives, you should do them, but is anyone realistically paranoid about S3?
- SteveNuts 1y ago
  Yes, I am paranoid of S3. Not only could a once in a lifetime event happen, an attacker could get in and delete all my data. Data could be accidentally deleted. Corrupted data could be written...
  - burnt-resistor 1y ago
    Then 3 steps. 1. Use tarsnap so there's an encryption and a management layer. 2. Use a second service so there's redundancy and no SPoF. 3. Keep cryptographic signatures (not hashes) of each backup job in something like a WORM blockchain KVS.
  - nemothekid 1y ago
    >Data could be accidentally deleted. Corrupted data could be written... You guys should really have versioning enabled. Now if someone deleted your data and all the versions, that could be possible, but that would take real effort and would like be malicious.
    - imglorp 1y ago
      Nobody mentioned the case where you get locked out of your cloud provider with no humans to speak to, or your account gets deleted by the algorithm. Both happen routinely and we only hear about it when the victim takes to the socials.
  - mrflop 1y ago
    That’s basically one of the reasons that led us to build Plakar.
- tecleandor 1y ago
  But don't make the same mistake people make with RAID. "More durable" doesn't mean "backup". What if somebody deletes the file? What if it got corrupted for a problem in one of your processes? What if your API key falls in the wrong hands?
  - nemothekid 1y ago
    Yes - backups also protect against someone doing a `rm -rf /*` by accident. However, I don't think I've created an S3 bucket without versioning enabled for years. If someone deletes the file, or the file gets corrupted - I just restore a pervious version. I don't want to suggest that people should place all their eggs in one basket - it's obviously irresponsible. However, S3 (and versioning) has been the "final storage" for years now. I can only imagine a catastrophic situation like an entire s3 region blowing up. And I'm sure a disgruntled employee could do a lot of damage as well.
- joshka 1y ago
  Backups don't just protect you from durability issues. They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc. The context that this article suggests is that if your S3 bucket is your primary storage, then it's possible that you're not thinking about where the second copy of your data should belong.
  - nemothekid 1y ago
    >They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc. S3 with versioning enabled provides this. I'm not being naive when I say S3 really provides everything you might need. Its my observation over the last 13 years, dealing with tons of fires, that there has every been a situation where I couldn't retrieve something from S3. Legally you might need an alternative. Going multi-cloud doesn't hurt - after all I do it. But practically? I don't think I would lose sleep if someone told me they only back up to S3.
    - icedchai 1y ago
      What if someone deletes a bucket? Then all your versioning is gone...
      - charcircuit 1y ago
        It doesn't let you.
        icedchai 1y ago
        It can be done if you delete the versions. You’ll need to use the aws cli.
        fpoling 1y ago
        It cannot be done if S3 objects use the object lock in compliance mode. Such objects cannot be altered in any way and the bucket cannot be deleted until the lock expires .
- treve 1y ago
  We can get everything back except data. It feels silly to take the risk _not_ to if you're somewhat established.
- firesteelrain 1y ago
  My HOA uses a SmartNAS in addition to S3. And we aren’t a huge operation.
- zzo38computer 1y ago
  I prefer to store backups on "write once read many" media, such as DVDs. However, having multiple backups would be helpful.
- kjellsbells 1y ago
  Perhaps reframe the problem not as data loss because S3's technical infrastructure failed but because of one of the many other ways that data can get zapped or that you might need it. For example: - Employee goes rogue and nukes buckets. - Code fault quietly deletes data, or doesnt store it like you thought. - State entity demands access to data, and you'd rather give them a tape than your S3 keys. I agree that with eleven-nines or whatever it is of availability, a write to S3 is not going to disappoint you, but most data losses are more about policy and personnel than infrastructure failures.
  - coredog64 1y ago
    A fun one I've seen before: Your encrypted content reused a KMS key that was provisioned by a temporary CloudFormation stack and got torn down months ago.
    - foota 1y ago
      Accidental crypto shredding? Oof.
  - toomuchtodo 1y ago
    This is solved for using versioning with MFA for delete or corruption risk, S3 export if required to provide a copy. Data can also be replicated to a write only bucket in another account, with only the ability to replicate. https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiFactorAuthenticationDelete.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiF... https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/object...
    - xyzzy123 1y ago
      Yep, for many applications, versioning is the lightweight solve. But.. aws backup is still nice, if a bit heavy. I like common workflows to restore all stuff (ddbs, managed dbs, buckets etc) to a common point in time. Also, one of the under-appreciated causes of massive data loss is subtly incorrect lifecycle policies. Backup can save you here even when other techniques may not.
- hxtk 1y ago
  I’ve worked on a project with strict legal record-keeping requirements that had a plan for the primary AWS region literally getting nuked. But that was the only contingency in our book of plans that really required the S3 backup. We generally assumed that as long as the region still existed, S3 still had everything we put in it. Of course, since we had the backups, restoration of individual objects would’ve been possible, but we would’ve needed to do it by hand.
- Spooky23 1y ago
  AWS is an incredible company and S3 a best in class service. Blindly trust my business to their SLA? To every thing with write access to data? Hell, no.
- jamesfinlayson 1y ago
  I worked at a place that uses AWS Backup - which I assume under the hood uses S3. The backups themselves were off-limits to regular employees though - only the team that managed AWS could edit or delete the backups.
winrid 1y ago
If you zoom in on your site before the cookies banner pops up you are stuck with just "Hi, we're cookies!" stuck on the screen and can't zoom out out
- msgodel 1y ago
  You don't even need a banner like this unless you have third party cookies which there are no good reasons for.
chungy 1y ago
Another similar archive format is WIM, the thing created by Microsoft for the Windows Vista (and newer) installer; an open source implementation is at: https://wimlib.net/ https://wimlib.net/ It offers similar deduplication, indexing, per-file compression, and versioning advantages
- mrflop 1y ago
  But it works only for Windows, right?
  - chungy 1y ago
    No, it works on many OSes. That's the point of linking to wimlib :) It even supports Unix metadata!
gcr 1y ago
How does this differ from zpaq and dwarFS? Zpaq is quite mature and also handles deduplication, versioning, etc.
- jauntywundrkind 1y ago
  Or pax. https://linux.die.net/man/1/pax https://linux.die.net/man/1/pax Or eStargz. https://github.com/containerd/stargz-snapshotter https://github.com/containerd/stargz-snapshotter Or Nydus RAFS. https://github.com/dragonflyoss/nydus https://github.com/dragonflyoss/nydus Links for your mentioned zpaq and dwarFS https://www.mattmahoney.net/dc/zpaq.html https://www.mattmahoney.net/dc/zpaq.html https://github.com/mhx/dwarfs https://github.com/mhx/dwarfs
tux1968 1y ago
They mention in the article that some people don't want to install the full Plakar backup software just to read and write ptar archives; so a dedicated open-source tool is offered for download as of yesterday: https://plakar.io/posts/2025-07-07/kapsul-a-tool-to-create-and-manage-deduplicated-compressed-and-encrypted-ptar-vaults/ https://plakar.io/posts/2025-07-07/kapsul-a-tool-to-create-a...
- throwaway127482 1y ago
  Direct link to GitHub: https://github.com/PlakarKorp/kapsul https://github.com/PlakarKorp/kapsul
ac29 1y ago
Are people really using gzip in 2025 for new projects? Zstd has been widely available for a long time. Debian, which is pretty conservative with new software, has shipped zstd since at least stretch (released 2017).
- kazinator 1y ago
  I integrated gzip into TXR Lisp in 2022. I evaluated all the choices and went with that one because of: - tiny code size; - widely used standard; - fast compression and decompression. And it also beat Zstandard on compressing TXR Lisp .tlo files by a non-negligible margin. I can reproduce that today: $ zstd -o compiler.tlo.zstd stdlib/compiler.tlo stdlib/compiler.tlo : 25.60% (250146 => 64037 bytes, compiler.tlo.zstd) $ gzip -c > compiler.tlo.gzip stdlib/compiler.tlo $ ls -l compiler.tlo.* -rw-rw-r-- 1 kaz kaz 60455 Jul 8 21:17 compiler.tlo.gzip -rw-rw-r-- 1 kaz kaz 64037 Jul 8 17:43 compiler.tlo.zstd The .gzip file is 0.944 as large as the .zstd file. So for this use case, gzip is faster (zstd has only decompression that is fast), compresses better and has way smaller code footprint.
  - jonas21 1y ago
    zstd uses a fairly low compression level by default. If you run with `zstd -19 -o compiler.tlo.zstd stdlib/compiler.tlo` you will probably get much better compression than gzip, even at its highest setting. That said, the tiny code footprint of gzip can be a real benefit. And you can usually count on gzip being available as a system library on whatever platform you're targeting, while that's often not the case for zstd (on iOS, for example).
    - kazinator 1y ago
      Additional datapoints: Tne Zopfli gzip-compatible compressor gets the file down to 54343. But zstd with level -19 beats that: -rw-rw-r-- 1 kaz kaz 54373 Jul 8 22:59 compiler.tlo.zopfli -rw-rw-r-- 1 kaz kaz 50102 Jul 8 17:43 compiler.tlo.zstd.19 I have no idea which is more CPU/memory intensive. For applications in which compression speed is not important (data is being prepared once to be decompressed many times), if you want the best compression and stick with gzip, Zopfli is the ticket.
Scaevolus 1y ago
Having the entire backup as a single file is interesting, but does it matter? Restic has a similar featureset (deduplicated encrypted backups), but almost certainly has better incremental performance for complex use cases like storing X daily backups, Y weekly backups, etc. At the same time, it struggles with RAM usage when handling even 1TB of data, and presumably ptar has better scaling at that size.
- mkroman 1y ago
  > At the same time, it struggles with RAM usage when handling even 1TB of data, and presumably ptar has better scaling at that size. There's also rustic, which supposedly is optimized for memory: https://rustic.cli.rs/docs/ https://rustic.cli.rs/docs/
throwaway127482 1y ago
Does this support content-defined chunking (CDC)?
- mrflop 1y ago
  Yes, both ptar and plakar. If you want to read more about the internal: https://www.plakar.io/posts/2025-04-29/kloset-the-immutable-data-store/ https://www.plakar.io/posts/2025-04-29/kloset-the-immutable-...
ahofmann 1y ago
I'm trying to evaluate what plakar is. Is it like restic, Borgbackup, Kopia?
- mrflop 1y ago
  Yes Plakar works much like Restic and Kopia: it takes content-addressed, encrypted and deduplicated snapshots and offers efficient incremental backups via a simple CLI. Under the hood, its Kloset engine splits data into encrypted, compressed chunks. Plakar main strengths: UI: In addition to a simple Unix-style CLI, Plakar provides an web interface and API for monitoring, browsing snapshots Data-agnostic snapshots: Plakar’s Kloset engine captures any structured data—filesystems, databases, applications—not just files, by organizing them into self-describing snapshots Source/target decoupling: You can back up from one system (e.g. a local filesystem) and restore to another (e.g. an S3 bucket) using pluggable source and target connectors Universal storage backends: Storage connectors let you persist encrypted, compressed chunks to local filesystems, SFTP servers or S3-compatible object stores (and more)—all via a unified interface Extreme scale with low RAM: A virtual filesystem with lazy loading and backpressure-aware parallelism keeps memory use minimal, even on very large datasets Network- and egress-optimized: Advanced client-side deduplication and compression dramatically cut storage and network transfer costs—ideal for inter-cloud or cross-provider migrations Online maintenance: you don't need to stop you backup to free some space ptar...
  - ahofmann 1y ago
    Thank you!

57 points by vcoisne 1y ago

>By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 backup" if (1) I had an existing system (e.g. tapes), or (2) I need multi-cloud redundancy. Other than that, once I assume something is in S3, I confident it's safe. Obviously this was built over years (decades?) or reliability, and if your DRP requires alternatives, you should do them, but is anyone realistically paranoid about S3?

SteveNuts 1y ago
Yes, I am paranoid of S3. Not only could a once in a lifetime event happen, an attacker could get in and delete all my data. Data could be accidentally deleted. Corrupted data could be written...
- burnt-resistor 1y ago
  Then 3 steps. 1. Use tarsnap so there's an encryption and a management layer. 2. Use a second service so there's redundancy and no SPoF. 3. Keep cryptographic signatures (not hashes) of each backup job in something like a WORM blockchain KVS.
- nemothekid 1y ago
  >Data could be accidentally deleted. Corrupted data could be written... You guys should really have versioning enabled. Now if someone deleted your data and all the versions, that could be possible, but that would take real effort and would like be malicious.
  - imglorp 1y ago
    Nobody mentioned the case where you get locked out of your cloud provider with no humans to speak to, or your account gets deleted by the algorithm. Both happen routinely and we only hear about it when the victim takes to the socials.
- mrflop 1y ago
  That’s basically one of the reasons that led us to build Plakar.
tecleandor 1y ago
But don't make the same mistake people make with RAID. "More durable" doesn't mean "backup". What if somebody deletes the file? What if it got corrupted for a problem in one of your processes? What if your API key falls in the wrong hands?
- nemothekid 1y ago
  Yes - backups also protect against someone doing a `rm -rf /*` by accident. However, I don't think I've created an S3 bucket without versioning enabled for years. If someone deletes the file, or the file gets corrupted - I just restore a pervious version. I don't want to suggest that people should place all their eggs in one basket - it's obviously irresponsible. However, S3 (and versioning) has been the "final storage" for years now. I can only imagine a catastrophic situation like an entire s3 region blowing up. And I'm sure a disgruntled employee could do a lot of damage as well.
joshka 1y ago
Backups don't just protect you from durability issues. They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc. The context that this article suggests is that if your S3 bucket is your primary storage, then it's possible that you're not thinking about where the second copy of your data should belong.
- nemothekid 1y ago
  >They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc. S3 with versioning enabled provides this. I'm not being naive when I say S3 really provides everything you might need. Its my observation over the last 13 years, dealing with tons of fires, that there has every been a situation where I couldn't retrieve something from S3. Legally you might need an alternative. Going multi-cloud doesn't hurt - after all I do it. But practically? I don't think I would lose sleep if someone told me they only back up to S3.
  - icedchai 1y ago
    What if someone deletes a bucket? Then all your versioning is gone...
    - charcircuit 1y ago
      It doesn't let you.
      - icedchai 1y ago
        It can be done if you delete the versions. You’ll need to use the aws cli.
        fpoling 1y ago
        It cannot be done if S3 objects use the object lock in compliance mode. Such objects cannot be altered in any way and the bucket cannot be deleted until the lock expires .
treve 1y ago
We can get everything back except data. It feels silly to take the risk _not_ to if you're somewhat established.
firesteelrain 1y ago
My HOA uses a SmartNAS in addition to S3. And we aren’t a huge operation.
zzo38computer 1y ago
I prefer to store backups on "write once read many" media, such as DVDs. However, having multiple backups would be helpful.
kjellsbells 1y ago
Perhaps reframe the problem not as data loss because S3's technical infrastructure failed but because of one of the many other ways that data can get zapped or that you might need it. For example: - Employee goes rogue and nukes buckets. - Code fault quietly deletes data, or doesnt store it like you thought. - State entity demands access to data, and you'd rather give them a tape than your S3 keys. I agree that with eleven-nines or whatever it is of availability, a write to S3 is not going to disappoint you, but most data losses are more about policy and personnel than infrastructure failures.
- coredog64 1y ago
  A fun one I've seen before: Your encrypted content reused a KMS key that was provisioned by a temporary CloudFormation stack and got torn down months ago.
  - foota 1y ago
    Accidental crypto shredding? Oof.
- toomuchtodo 1y ago
  This is solved for using versioning with MFA for delete or corruption risk, S3 export if required to provide a copy. Data can also be replicated to a write only bucket in another account, with only the ability to replicate. https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiFactorAuthenticationDelete.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiF... https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/object...
  - xyzzy123 1y ago
    Yep, for many applications, versioning is the lightweight solve. But.. aws backup is still nice, if a bit heavy. I like common workflows to restore all stuff (ddbs, managed dbs, buckets etc) to a common point in time. Also, one of the under-appreciated causes of massive data loss is subtly incorrect lifecycle policies. Backup can save you here even when other techniques may not.
hxtk 1y ago
I’ve worked on a project with strict legal record-keeping requirements that had a plan for the primary AWS region literally getting nuked. But that was the only contingency in our book of plans that really required the S3 backup. We generally assumed that as long as the region still existed, S3 still had everything we put in it. Of course, since we had the backups, restoration of individual objects would’ve been possible, but we would’ve needed to do it by hand.
Spooky23 1y ago
AWS is an incredible company and S3 a best in class service. Blindly trust my business to their SLA? To every thing with write access to data? Hell, no.
jamesfinlayson 1y ago
I worked at a place that uses AWS Backup - which I assume under the hood uses S3. The backups themselves were off-limits to regular employees though - only the team that managed AWS could edit or delete the backups.

winrid 1y ago

If you zoom in on your site before the cookies banner pops up you are stuck with just "Hi, we're cookies!" stuck on the screen and can't zoom out out

msgodel 1y ago
You don't even need a banner like this unless you have third party cookies which there are no good reasons for.

chungy 1y ago

Another similar archive format is WIM, the thing created by Microsoft for the Windows Vista (and newer) installer; an open source implementation is at: https://wimlib.net/ https://wimlib.net/ It offers similar deduplication, indexing, per-file compression, and versioning advantages

mrflop 1y ago
But it works only for Windows, right?
- chungy 1y ago
  No, it works on many OSes. That's the point of linking to wimlib :) It even supports Unix metadata!

gcr 1y ago

How does this differ from zpaq and dwarFS? Zpaq is quite mature and also handles deduplication, versioning, etc.

jauntywundrkind 1y ago
Or pax. https://linux.die.net/man/1/pax https://linux.die.net/man/1/pax Or eStargz. https://github.com/containerd/stargz-snapshotter https://github.com/containerd/stargz-snapshotter Or Nydus RAFS. https://github.com/dragonflyoss/nydus https://github.com/dragonflyoss/nydus Links for your mentioned zpaq and dwarFS https://www.mattmahoney.net/dc/zpaq.html https://www.mattmahoney.net/dc/zpaq.html https://github.com/mhx/dwarfs https://github.com/mhx/dwarfs

tux1968 1y ago

They mention in the article that some people don't want to install the full Plakar backup software just to read and write ptar archives; so a dedicated open-source tool is offered for download as of yesterday: https://plakar.io/posts/2025-07-07/kapsul-a-tool-to-create-and-manage-deduplicated-compressed-and-encrypted-ptar-vaults/ https://plakar.io/posts/2025-07-07/kapsul-a-tool-to-create-a...

throwaway127482 1y ago
Direct link to GitHub: https://github.com/PlakarKorp/kapsul https://github.com/PlakarKorp/kapsul

ac29 1y ago

Are people really using gzip in 2025 for new projects? Zstd has been widely available for a long time. Debian, which is pretty conservative with new software, has shipped zstd since at least stretch (released 2017).

kazinator 1y ago
I integrated gzip into TXR Lisp in 2022. I evaluated all the choices and went with that one because of: - tiny code size; - widely used standard; - fast compression and decompression. And it also beat Zstandard on compressing TXR Lisp .tlo files by a non-negligible margin. I can reproduce that today: $ zstd -o compiler.tlo.zstd stdlib/compiler.tlo stdlib/compiler.tlo : 25.60% (250146 => 64037 bytes, compiler.tlo.zstd) $ gzip -c > compiler.tlo.gzip stdlib/compiler.tlo $ ls -l compiler.tlo.* -rw-rw-r-- 1 kaz kaz 60455 Jul 8 21:17 compiler.tlo.gzip -rw-rw-r-- 1 kaz kaz 64037 Jul 8 17:43 compiler.tlo.zstd The .gzip file is 0.944 as large as the .zstd file. So for this use case, gzip is faster (zstd has only decompression that is fast), compresses better and has way smaller code footprint.
- jonas21 1y ago
  zstd uses a fairly low compression level by default. If you run with `zstd -19 -o compiler.tlo.zstd stdlib/compiler.tlo` you will probably get much better compression than gzip, even at its highest setting. That said, the tiny code footprint of gzip can be a real benefit. And you can usually count on gzip being available as a system library on whatever platform you're targeting, while that's often not the case for zstd (on iOS, for example).
  - kazinator 1y ago
    Additional datapoints: Tne Zopfli gzip-compatible compressor gets the file down to 54343. But zstd with level -19 beats that: -rw-rw-r-- 1 kaz kaz 54373 Jul 8 22:59 compiler.tlo.zopfli -rw-rw-r-- 1 kaz kaz 50102 Jul 8 17:43 compiler.tlo.zstd.19 I have no idea which is more CPU/memory intensive. For applications in which compression speed is not important (data is being prepared once to be decompressed many times), if you want the best compression and stick with gzip, Zopfli is the ticket.

Scaevolus 1y ago

Having the entire backup as a single file is interesting, but does it matter? Restic has a similar featureset (deduplicated encrypted backups), but almost certainly has better incremental performance for complex use cases like storing X daily backups, Y weekly backups, etc. At the same time, it struggles with RAM usage when handling even 1TB of data, and presumably ptar has better scaling at that size.

mkroman 1y ago
> At the same time, it struggles with RAM usage when handling even 1TB of data, and presumably ptar has better scaling at that size. There's also rustic, which supposedly is optimized for memory: https://rustic.cli.rs/docs/ https://rustic.cli.rs/docs/

throwaway127482 1y ago

Does this support content-defined chunking (CDC)?

mrflop 1y ago
Yes, both ptar and plakar. If you want to read more about the internal: https://www.plakar.io/posts/2025-04-29/kloset-the-immutable-data-store/ https://www.plakar.io/posts/2025-04-29/kloset-the-immutable-...

ahofmann 1y ago

I'm trying to evaluate what plakar is. Is it like restic, Borgbackup, Kopia?

mrflop 1y ago
Yes Plakar works much like Restic and Kopia: it takes content-addressed, encrypted and deduplicated snapshots and offers efficient incremental backups via a simple CLI. Under the hood, its Kloset engine splits data into encrypted, compressed chunks. Plakar main strengths: UI: In addition to a simple Unix-style CLI, Plakar provides an web interface and API for monitoring, browsing snapshots Data-agnostic snapshots: Plakar’s Kloset engine captures any structured data—filesystems, databases, applications—not just files, by organizing them into self-describing snapshots Source/target decoupling: You can back up from one system (e.g. a local filesystem) and restore to another (e.g. an S3 bucket) using pluggable source and target connectors Universal storage backends: Storage connectors let you persist encrypted, compressed chunks to local filesystems, SFTP servers or S3-compatible object stores (and more)—all via a unified interface Extreme scale with low RAM: A virtual filesystem with lazy loading and backpressure-aware parallelism keeps memory use minimal, even on very large datasets Network- and egress-optimized: Advanced client-side deduplication and compression dramatically cut storage and network transfer costs—ideal for inter-cloud or cross-provider migrations Online maintenance: you don't need to stop you backup to free some space ptar...
- ahofmann 1y ago
  Thank you!