4 ms·

This a feature I was completely unaware of. Why would you choose to use a sparse file instead of multiple files?

• vlovich123 3y ago
  The number of file descriptors you can have open by a single program is limited and eats up kernel resources.
  • ajross 3y ago
    Even for a torrent client, the number of active file descriptors is a function of the number of peer connections (e.g. a few dozen). It doesn't scale with the size of the output file.
    • vlovich123 3y ago
      Think databases like RocksDB not torrents.
• wolf550e 3y ago
  Imagine a torrent client (or http client downloading a file in parallel using HTTP range requests). It creates an empty file and then it has downloaded a 1MB of data to write at offset 100GB and wants to write it to disk. It does not want to pay the price of waiting for 100GB of zeroes to be written. The other blocks will all be downloaded and written eventually, all out of order. If the filesystem had an atomic operation to transform a bunch of (block aligned) files into a single file (like AWS S3 Multipart Upload), then sparse files would not be needed for this case.
  • myself248 3y ago
    Out of curiosity, do you know off-hand how torrent clients do it on filesystems that don't support sparse files? There must be either a preallocate-the-whole-thing step, or a gather-the-pieces-together-and-write-out-the-large-file step. The latter would seem to briefly double the disk space needed at the end of the download, so I suspect they do the former.
    • ajross 3y ago
      Chunk it up and resassemble, one assumes. Things aren't nearly as clear in the modern world of gigabit pipes into suburban households[1], but when these things were written the filesystem was 100x faster than the link to those peer connections from which the data was fetched. A final copy was only a small overhead. [1] Which is why all the stuff we used to torrent is in the cloud now.
    • cesarb 3y ago
      AFAIK, they preallocate; and even on filesystems which support sparse files, most bittorrent clients have an option to always preallocate (to both reserve space and reduce fragmentation).
  • loeg 3y ago
    Fallocate is a much better interface for this than sparse files. The torrent client does not care how the underlying filesystem provides the ability to randomly write a large file. And fallocate is a much clearer signal to the filesystem than a sparse file.
    • avianlyric 3y ago
      fallocate is just an interface to create sparse files. The result of using `fallocate` is a sparse file.
      • loeg 3y ago
        You should read my comment in the context of the one it is replying to. That comment suggested a torrent client using seeks + writes to randomly insert chunks as they were downloaded. I have summarized this approach in my comment as "sparse files," expecting charitable readers to be familiar with the context. This method of creating sparse files does not tell the filesystem anything about the intent of the application and usually creates a bunch of fragmentation under torrent-like workloads.
        • epcoa 3y ago
          It’s not about being charitable. For those unfamiliar with the terminology this is just confusing, and for those that are familiar this discussion is all fundamental and well known anyway. Unfortunately for COW filesystems including zfs and btrfs fallocate doesn’t do anything useful for preallocation. You’re still going to get fragmentation. The two methods outlined are essentially equivalent.
          • loeg 3y ago
            > For those unfamiliar with the terminology this is just confusing, and for those that are familiar this discussion is all fundamental and well known anyway. Eh, agree to disagree. > Unfortunately for COW filesystems including zfs and btrfs fallocate doesn’t do anything useful for preallocation. Both ZFS and BtrFS have "nocow" modes that are probably more suitable to this type of use case. And other filesystems are widely used.
• axus 3y ago
  A memory map of a gigabyte/whatever, that uses a bunch of large addresses but typically only uses 1% of the available space. Saves someone the trouble of managing the map or compressing it. It does feel like a weird decision from long time ago that we're stuck with. I thought it was some quirky Linux feature but it's been around https://en.wikipedia.org/wiki/Sparse_file https://en.wikipedia.org/wiki/Sparse_file
• andoma 3y ago
  Somewhat related is that a few filesystem types on Linux allows you to remove / insert bytes "within" a file. But it needs alignment to filesystem block size. This uses the same syscall, fallocate(2), which can be used to punch new sparse holes in a file where it previously had data. See https://man7.org/linux/man-pages/man2/fallocate.2.html https://man7.org/linux/man-pages/man2/fallocate.2.html