4 ms·

Sort of. Not really. spiped operates at the level of individual stream connections, so you can e.g. make one end a local socket in a filesystem and use UNIX p

Sort of. Not really. spiped operates at the level of individual stream connections, so you can e.g. make one end a local socket in a filesystem and use UNIX permissions to control access to it. In fact that's exactly why I wrote it -- so I could have a set of daemons designed to communicate via local sockets and transparently (aside from performance) have them running on different systems.

tiffanyh 5y ago
Thanks Colin! Love all the things you create :)
sillysaurusx 5y ago
Is it possible to use tarsnap's deduplication code on my own server? We're setting up an ML dataset distribution box, and I was hoping to avoid storing e.g. imagenet as a tarball + untar'd (so that nginx can serve each photo individually) + imagenet in TFDS format. https://github.com/xolox/dedupfs https://github.com/xolox/dedupfs was the closest I found, but it has a lot of downsides. Has anyone made an interface to tarsnap's tarball dedup code? A python wrapper around the block dedup code would be ideal, but I doubt it exists. (Sorry for the random question -- I was just hoping for a standalone library along the lines of tarsnap's "filesystem block database" APIs. I thought about emailing this to you instead, but I'm crossing my fingers that some random HN'er might know. I'm sort of surprised that filesystems don't make it effortless. In fact, I delayed posting this for an hour to go research whether ZFS is the actual solution -- apparently "no, not unless you have specific brands of SSDs: https://www.truenas.com/community/resources/my-experiments-in-building-a-home-server-capable-of-handling-fast-consistent-deduplication.148/ https://www.truenas.com/community/resources/my-experiments-i..." which rules out my non-SSD 64TB Hetzner server. But like, dropbox solved this problem a decade ago -- isn't there something similar by now?) EDIT: How timely -- Wyng (https://news.ycombinator.com/item?id=28537761 https://news.ycombinator.com/item?id=28537761) was just submitted a few hours ago. It seems to support "Data deduplication," though I wonder if it's block-level or file-level dedup. Tarsnap's block dedup is basically flawless, so I'm keen to find something that closely matches it.
- gperciva 5y ago
  > Has anyone made an interface to tarsnap's tarball dedup code? Tarsnap's deduplication code is not available under an open-source license.
  - sillysaurusx 5y ago
    True, but a couple years ago I ported most of the Tarsnap dedup algorithms to Python. It wasn't too hard, just time consuming. I was hoping someone else did that in a thorough way, but I guess the intersection of "I love tarsnap's design!" and "I have the time to port it from C!" might not be too large.
    - akerl_ 5y ago
      This may be a foolish question, but if the dedup code isn’t open source, how did you port it to Python?
      - sillysaurusx 5y ago
        Not foolish! The Tarsnap client code is open source, but the license file prohibits anyone from using the code: https://github.com/Tarsnap/tarsnap/blob/master/COPYING https://github.com/Tarsnap/tarsnap/blob/master/COPYING > Redistribution and use in source and binary forms, without modification, is permitted for the sole purpose of using the "tarsnap" backup service provided by Tarsnap Backup Inc. The codebase is a jewel. I love the design, the way it's organized, the coding style, the algorithms, everything. My process was to skim Colin's thesis: http://www.daemonology.net/papers/thesis.pdf http://www.daemonology.net/papers/thesis.pdf Along with the rsync thesis: https://www.samba.org/~tridge/phd_thesis.pdf https://www.samba.org/~tridge/phd_thesis.pdf Then I started making a mental map of tarsnap: How does it build its deduplication index? How does it decide where block boundaries start within a file? Etc. Eventually I started coding the algorithms in Python, mostly as a way of understanding the code. It's not actually as hard as it sounds, but you have to be rigorous. (It's a C -> Python conversion, after all, so there's not much room for error.) My process was basically: Copy the C code into a Python file; comment out the code; for each line, write the corresponding Python; try to get something running as quickly as possible. It worked pretty well, but I eventually lost interest. Over the years, I've wanted a deduplication library, and 2021 is no exception. Someday I'll just roll up my sleeves and finish porting it.

tiffanyh 5y ago

Thanks Colin! Love all the things you create :)

sillysaurusx 5y ago

Is it possible to use tarsnap's deduplication code on my own server? We're setting up an ML dataset distribution box, and I was hoping to avoid storing e.g. imagenet as a tarball + untar'd (so that nginx can serve each photo individually) + imagenet in TFDS format. https://github.com/xolox/dedupfs https://github.com/xolox/dedupfs was the closest I found, but it has a lot of downsides. Has anyone made an interface to tarsnap's tarball dedup code? A python wrapper around the block dedup code would be ideal, but I doubt it exists. (Sorry for the random question -- I was just hoping for a standalone library along the lines of tarsnap's "filesystem block database" APIs. I thought about emailing this to you instead, but I'm crossing my fingers that some random HN'er might know. I'm sort of surprised that filesystems don't make it effortless. In fact, I delayed posting this for an hour to go research whether ZFS is the actual solution -- apparently "no, not unless you have specific brands of SSDs: https://www.truenas.com/community/resources/my-experiments-in-building-a-home-server-capable-of-handling-fast-consistent-deduplication.148/ https://www.truenas.com/community/resources/my-experiments-i..." which rules out my non-SSD 64TB Hetzner server. But like, dropbox solved this problem a decade ago -- isn't there something similar by now?) EDIT: How timely -- Wyng (https://news.ycombinator.com/item?id=28537761 https://news.ycombinator.com/item?id=28537761) was just submitted a few hours ago. It seems to support "Data deduplication," though I wonder if it's block-level or file-level dedup. Tarsnap's block dedup is basically flawless, so I'm keen to find something that closely matches it.

gperciva 5y ago
> Has anyone made an interface to tarsnap's tarball dedup code? Tarsnap's deduplication code is not available under an open-source license.
- sillysaurusx 5y ago
  True, but a couple years ago I ported most of the Tarsnap dedup algorithms to Python. It wasn't too hard, just time consuming. I was hoping someone else did that in a thorough way, but I guess the intersection of "I love tarsnap's design!" and "I have the time to port it from C!" might not be too large.
  - akerl_ 5y ago
    This may be a foolish question, but if the dedup code isn’t open source, how did you port it to Python?
    - sillysaurusx 5y ago
      Not foolish! The Tarsnap client code is open source, but the license file prohibits anyone from using the code: https://github.com/Tarsnap/tarsnap/blob/master/COPYING https://github.com/Tarsnap/tarsnap/blob/master/COPYING > Redistribution and use in source and binary forms, without modification, is permitted for the sole purpose of using the "tarsnap" backup service provided by Tarsnap Backup Inc. The codebase is a jewel. I love the design, the way it's organized, the coding style, the algorithms, everything. My process was to skim Colin's thesis: http://www.daemonology.net/papers/thesis.pdf http://www.daemonology.net/papers/thesis.pdf Along with the rsync thesis: https://www.samba.org/~tridge/phd_thesis.pdf https://www.samba.org/~tridge/phd_thesis.pdf Then I started making a mental map of tarsnap: How does it build its deduplication index? How does it decide where block boundaries start within a file? Etc. Eventually I started coding the algorithms in Python, mostly as a way of understanding the code. It's not actually as hard as it sounds, but you have to be rigorous. (It's a C -> Python conversion, after all, so there's not much room for error.) My process was basically: Copy the C code into a Python file; comment out the code; for each line, write the corresponding Python; try to get something running as quickly as possible. It worked pretty well, but I eventually lost interest. Over the years, I've wanted a deduplication library, and 2021 is no exception. Someday I'll just roll up my sleeves and finish porting it.