5 ms·

Well, if the rest of the system treats the spare file as regular file[1], then it doesn't matter it is sparse, the space still can't be used. [1] for example s

by takeda 6y ago

Well, if the rest of the system treats the spare file as regular file[1], then it doesn't matter it is sparse, the space still can't be used. [1] for example someone there mentioned that Apple Software Update app refuses to update system, claiming there's not enough space.

tgv 6y ago
It is a sparse file: copying it is very fast when it's empty. I don't know if the OS recognizes the space claims, but it would be logical if it did so.
oefrha 6y ago
I took a brief look at the issue and it's very strange. People reporting the issue and piling on clearly don't understand sparse files (which is fine in and of itself), and their claims that "the rest of the system treats the spare file as regular file" are just not very credible to me. The thing is, if you want to look at the physical size of a file/directory, you don't use ls, you use du. Similarly, if you want to look at disk usage and free space on a volume, you use df, not summing up ls reported size of all files. Using df for free space is especially important on APFS since you have APFS volumes sharing the space in a container. And of course, du and df correctly report actual disk usage. If you check Docker.raw in Finder, it says something like "63,999,836,160 bytes (10.25 GB on disk)", too. Btw, even `ls -l` reports correct total disk usage: $ ls -lh ~/Library/Containers/com.docker.docker/Data/vms/0/data total 9.6G -rw-r--r-- 1 <omitted> staff 60G Jun 17 02:01 Docker.raw There's just no way "the rest of the system treats the spare file as regular file". I suspect people actually have disk space tied up in APFS snapshots (Time Machine takes a lot of those and I have to clear them constantly with tmutil on machines with meager SSDs), but they're not aware of that, instead they ran some crappy script or something found online, located this "huge file", then just blamed it for the disk usage. Any proper tool, e.g. Disk Utility (builtin), DaisyDisk won't make this obvious mistake. I thought maybe people were using Finder search to find large files and that's the problem, but I just tried and Finder search also makes a distinction between logical size and physical size, so it's not even that. Edit: In case it's not clear, sparse files don't "reserve" their space. Try this: for i in {000..999}; do dd if=/dev/zero of=/tmp/sparse-$i bs=1024 count=0 seek=104857600; done You just created a thousand 100GiB files totaling zero bytes in disk usage (not counting metadata).
- sdflhasjd 6y ago
  > Edit: In case it's not clear, sparse files don't "reserve" their space. Try this: > for i in {000..999}; do dd if=/dev/zero of=/tmp/sparse-$i bs=1024 count=0 seek=104857600; done On an APFS volume with 18GB of free space, that bombed out with "No space left on device"
  - oefrha 6y ago
    I have no trouble creating a 1TiB file this way on a 512GB max volume. Trying to create a 2TiB file this way does error out. MacBookPro11,5, macOS 10.15.5.
- brigandish 6y ago
  I understand what a sparse file is and so do many of the people in that thread. The important point is, do all the apps on the Mac understand that sparse files don’t take up all that space? Some clearly don’t, which means their developers either don’t or didn’t account for them. It’s not the fault of the users in that thread.
  - oefrha 6y ago
    They don't need to. If an application needs to know how much free disk space there is, they use an API like statvfs(3) that queries the filesystem, which clearly knows how to compute the disk usage of itself. No application is gonna ask for the logical size of every single file on the disk; why the hell would they even care about this particular Docker.raw file? As I said, reports like "macOS installer says there's not enough disk space because of a sparsefile somewhere" and "macOS tells me disk space is low every two minutes because of a sparsefile somewhere" just aren't credible if you understand how it works. Otherwise, how would they behave if you create a thousand "100GB files" like I demonstrated? Going all crazy? Your free disk space is negative?
    - brigandish 6y ago
      > just aren't credible if you understand how it works. Yes, your theoretical knowledge, which several of the people in that thread obviously have, beats actual experience. Not only is that not credible, it's about as useful and insightful as "it works for me" - and who on Earth is using statvfs on a Mac? Is the installer using statvfs? You've looked, right? And you believe every developer uses the correct APIs and never introduces a bug… Try reading and not skimming, it's especially important when dealing with bug reports.

by takeda 6y ago

tgv 6y ago

It is a sparse file: copying it is very fast when it's empty. I don't know if the OS recognizes the space claims, but it would be logical if it did so.

oefrha 6y ago

I took a brief look at the issue and it's very strange. People reporting the issue and piling on clearly don't understand sparse files (which is fine in and of itself), and their claims that "the rest of the system treats the spare file as regular file" are just not very credible to me. The thing is, if you want to look at the physical size of a file/directory, you don't use ls, you use du. Similarly, if you want to look at disk usage and free space on a volume, you use df, not summing up ls reported size of all files. Using df for free space is especially important on APFS since you have APFS volumes sharing the space in a container. And of course, du and df correctly report actual disk usage. If you check Docker.raw in Finder, it says something like "63,999,836,160 bytes (10.25 GB on disk)", too. Btw, even `ls -l` reports correct total disk usage: $ ls -lh ~/Library/Containers/com.docker.docker/Data/vms/0/data total 9.6G -rw-r--r-- 1 <omitted> staff 60G Jun 17 02:01 Docker.raw There's just no way "the rest of the system treats the spare file as regular file". I suspect people actually have disk space tied up in APFS snapshots (Time Machine takes a lot of those and I have to clear them constantly with tmutil on machines with meager SSDs), but they're not aware of that, instead they ran some crappy script or something found online, located this "huge file", then just blamed it for the disk usage. Any proper tool, e.g. Disk Utility (builtin), DaisyDisk won't make this obvious mistake. I thought maybe people were using Finder search to find large files and that's the problem, but I just tried and Finder search also makes a distinction between logical size and physical size, so it's not even that. Edit: In case it's not clear, sparse files don't "reserve" their space. Try this: for i in {000..999}; do dd if=/dev/zero of=/tmp/sparse-$i bs=1024 count=0 seek=104857600; done You just created a thousand 100GiB files totaling zero bytes in disk usage (not counting metadata).

sdflhasjd 6y ago
> Edit: In case it's not clear, sparse files don't "reserve" their space. Try this: > for i in {000..999}; do dd if=/dev/zero of=/tmp/sparse-$i bs=1024 count=0 seek=104857600; done On an APFS volume with 18GB of free space, that bombed out with "No space left on device"
- oefrha 6y ago
  I have no trouble creating a 1TiB file this way on a 512GB max volume. Trying to create a 2TiB file this way does error out. MacBookPro11,5, macOS 10.15.5.
brigandish 6y ago
I understand what a sparse file is and so do many of the people in that thread. The important point is, do all the apps on the Mac understand that sparse files don’t take up all that space? Some clearly don’t, which means their developers either don’t or didn’t account for them. It’s not the fault of the users in that thread.
- oefrha 6y ago
  They don't need to. If an application needs to know how much free disk space there is, they use an API like statvfs(3) that queries the filesystem, which clearly knows how to compute the disk usage of itself. No application is gonna ask for the logical size of every single file on the disk; why the hell would they even care about this particular Docker.raw file? As I said, reports like "macOS installer says there's not enough disk space because of a sparsefile somewhere" and "macOS tells me disk space is low every two minutes because of a sparsefile somewhere" just aren't credible if you understand how it works. Otherwise, how would they behave if you create a thousand "100GB files" like I demonstrated? Going all crazy? Your free disk space is negative?
  - brigandish 6y ago
    > just aren't credible if you understand how it works. Yes, your theoretical knowledge, which several of the people in that thread obviously have, beats actual experience. Not only is that not credible, it's about as useful and insightful as "it works for me" - and who on Earth is using statvfs on a Mac? Is the installer using statvfs? You've looked, right? And you believe every developer uses the correct APIs and never introduces a bug… Try reading and not skimming, it's especially important when dealing with bug reports.