4 ms·

Dataless files are interesting! They are a special type of file with a bit set (and an xattr) in the file system and no actual file contents. They all live insi

by Dig1t 3y ago

Dataless files are interesting! They are a special type of file with a bit set (and an xattr) in the file system and no actual file contents. They all live inside a “sync root” in APFS, where each sync root belongs to a specific file provider like Box, Dropbox, iCloud Drive etc. Whenever the kernel detects access to one of these files it will synchronously “upcall” into the current user context and XPC directly to fileproviderd which will in turn communicate with an installed file provider to download and provide the file (changing it from a dataless file into a normal file). This is called “materializing” the file. It will also emit an event using the FSEvents API. fileproviderd (a launch agent on macOS and a daemon on iOS) watches the FSEvents stream and saves the events in its database. fileproviderd maintains its own sync engine for each provider on the system and will forward events to/from each provider. Dataless files are an interesting example of Apple using its vertical integration to create an end to end solution that others can’t quite make. Since they own the filesystem, kernel, and user space daemons they can make something that works like this. This is part of Apple’s years-long effort to kick 3rd parties out of the kernel. As apart of the M1 transition, kernel extensions were no longer aloud and file provider apps were using kernel extensions to intercept file system events at a low level before. Now that they are no longer able to do that, Apple has provided this replacement.

tambourine_man 3y ago
You comment is better than the linked article. Also, I think you meant allowed not aloud
- Dig1t 3y ago
  Thank you! D'oh! Damnit I hate when I do that, thanks for letting me know. I wish I could still edit the comment..
shrubble 3y ago
It sounds like a very much better integrated, more foolproof version of say, the FUSE filesystem with SSH or FTP as the remote side. You would start with a listing of files but then upon access the remote file will be accessed.
- abwizz 3y ago
  it'll be a day of celebration when apple allows fuse as a backend.
- TheNewsIsHere 3y ago
  My experience using applications built around the FUSE stack or the File Provider API suggest to me that while your take is generally true, it’s a “grass is always greener” problem. The File Provider API provides a much better admin and user experience, but the storage providers tend to really easily make a confusing mess of it. OneDrive took the time to figure out how to use sync roots in a way that enables you to store your actual data-containing files to an external drive. Dropbox didn’t care about supporting that use case on Macs, so what was once possible with both is now just possible with one. That has been a source of problems for many. On the other hand the migration to OneDrive’s File Provider implementation was a data-thrashing horror show for many. And iCloud Drive doesn’t truly let you decide a persistent sync state for everything. The security implications are better but it’s just the same old mess file syncing has been for a long time :)
  - xp84 3y ago
    Re: External drive with OneDrive. Can you share details? In my experimentation I found it would refuse to place my OneDrive root on any disk marked as removable. Since I don’t own a $6000 computer, with actual drive bays or nVme slots, that limits me to keeping it on my anemic internal drive. I’d love to find out I’m wrong. (I keep several SSDs connected via USB through a USB-C dock, if that matters to the answer)
    - TheNewsIsHere 3y ago
      Hey! I didn’t see this before today, sorry :) macOS enforces that the sync root remain local, but OneDrive does a little bit of slight of hand to be able to store files on an external drive. There is a write-up from the OneDrive team at [0] that details some caveats on how this works. OneDrive technical documentation can be difficult to follow since they like to disclose version/dated changes as blogs instead of something more formal and managed. I believe this is still possible, but experimentation with non-critical data might be advisable. [0] https://techcommunity.microsoft.com/t5/microsoft-onedrive-blog/inside-the-new-files-on-demand-experience-on-macos/ba-p/3058922 https://techcommunity.microsoft.com/t5/microsoft-onedrive-bl...
Kwpolska 3y ago
This isn't unique to Apple, OneDrive on Windows can also download files on-demand.

by Dig1t 3y ago

tambourine_man 3y ago

You comment is better than the linked article. Also, I think you meant allowed not aloud

Dig1t 3y ago
Thank you! D'oh! Damnit I hate when I do that, thanks for letting me know. I wish I could still edit the comment..

shrubble 3y ago

It sounds like a very much better integrated, more foolproof version of say, the FUSE filesystem with SSH or FTP as the remote side. You would start with a listing of files but then upon access the remote file will be accessed.

abwizz 3y ago
it'll be a day of celebration when apple allows fuse as a backend.
TheNewsIsHere 3y ago
My experience using applications built around the FUSE stack or the File Provider API suggest to me that while your take is generally true, it’s a “grass is always greener” problem. The File Provider API provides a much better admin and user experience, but the storage providers tend to really easily make a confusing mess of it. OneDrive took the time to figure out how to use sync roots in a way that enables you to store your actual data-containing files to an external drive. Dropbox didn’t care about supporting that use case on Macs, so what was once possible with both is now just possible with one. That has been a source of problems for many. On the other hand the migration to OneDrive’s File Provider implementation was a data-thrashing horror show for many. And iCloud Drive doesn’t truly let you decide a persistent sync state for everything. The security implications are better but it’s just the same old mess file syncing has been for a long time :)
- xp84 3y ago
  Re: External drive with OneDrive. Can you share details? In my experimentation I found it would refuse to place my OneDrive root on any disk marked as removable. Since I don’t own a $6000 computer, with actual drive bays or nVme slots, that limits me to keeping it on my anemic internal drive. I’d love to find out I’m wrong. (I keep several SSDs connected via USB through a USB-C dock, if that matters to the answer)
  - TheNewsIsHere 3y ago
    Hey! I didn’t see this before today, sorry :) macOS enforces that the sync root remain local, but OneDrive does a little bit of slight of hand to be able to store files on an external drive. There is a write-up from the OneDrive team at [0] that details some caveats on how this works. OneDrive technical documentation can be difficult to follow since they like to disclose version/dated changes as blogs instead of something more formal and managed. I believe this is still possible, but experimentation with non-critical data might be advisable. [0] https://techcommunity.microsoft.com/t5/microsoft-onedrive-blog/inside-the-new-files-on-demand-experience-on-macos/ba-p/3058922 https://techcommunity.microsoft.com/t5/microsoft-onedrive-bl...

Kwpolska 3y ago

This isn't unique to Apple, OneDrive on Windows can also download files on-demand.