5 ms·

I worked on the original (long since superseded) implementation of the metadata store for Google Drive, i.e. the system which was responsible for tracking file

I worked on the original (long since superseded) implementation of the metadata store for Google Drive, i.e. the system which was responsible for tracking file / folder relationships. The requirement to allow an item to appear in multiple locations was a huge complication, in part because of the way it interacted with permissions being inherited from a folder to the items in that folder. I imagine this change may be motivated by a desire to move away from that complex model, and whatever team owns that system now may be very happy to see it going away. (IIRC, the requirement stemmed from the need to support the various applications that were being folded into / integrated with Google Drive, such as Photos which of course allows a photo to appear in multiple albums.)

zmj 5y ago
I worked on another sync client's representation of filesystem structure, and came to the same conclusion. Hard links enable some cool behavior, but in retrospect added more complexity than anyone expected. Migrating to shortcuts / soft links seems very reasonable - I wish I had started there.
- djsavvy 5y ago
  What kinds of cool behavior do hard links enable?
  - mcoliver 5y ago
    Every file is essentially the master pointing at the same inode. So the file remains as long as any of the hard links exist. With soft links if you delete the master file all the soft links stop working.
  - deleted 5y ago
    [deleted]
  - dtgriscom 5y ago
    Apple's Time Machine uses hard-linked directories to replicate your home directory again and again, but with only the changed directories/files actually taking up space. The unchanged directories are just hard-linked to the previous versions, so any one version looks like your entire drive, but it takes much less space than full copies.
    - saagarjha 5y ago
      Used to, the new APFS implementation uses snapshots.
andybak 5y ago
> various applications that were being folded into / integrated with Google Drive The Photos/Drive integration was removed a long time ago. What other integrations were behind the original requirement? I'm curious to know if the extra complication was worth it in the long run and how long the integrations that needed this feature hung around for.
- snewman 5y ago
  I don't specifically remember whether there were other motivations for that requirement. Possibly a desire to support "tag" style interfaces, where an item can have multiple tags applied. But it may have mostly been the Photos integration. And no, definitely the complication was not worth it in the long run. :-) That project got way more complicated than anyone wanted (not that there's anything unusual about that).
  - Andrew_nenakhov 5y ago
    I actually remember early Google Drive to have only tags for files, and no folders. Apparently, the idea was to distance from the file tree paradigm, but eventually the traditional approach proved to be more user friendly.
layer8 5y ago
So is this basically a switch from hardlinks to softlinks?
- pishpash 5y ago
  It's a switch from inference to direct collection of intent from the user.
  - lupire 5y ago
    How so? Are hard links still allowed?
- lupire 5y ago
  Yes.
kyrra 5y ago
Googler, opinions are my own. This was my understanding as well. The original Drive was built effectively as a directed graph (with cycles allowed). Any file or folder could be stored in multiple locations. And permissions were at a per-file basis, so 2 people viewing the same folder may see different sets of files. And permissions were definitely a hard part of it, as if you applied new permissions to a folder and all children, it had to walk the entire graph to update the permissions. This is the advantage of the Team Drive style structure that the Drive team put out. It follows the classic filesystem design of a tree, which allows for easier permissions modeling, among other things. It's also why all "hard links" are now becoming shortcuts / Soft-links.
- wunderlust 5y ago
  Forgive my ignorance (and there's a lot of it), but navigating a graph and setting permissions doesn't seem like a terribly difficult problem, especially for Google - the king of graphs, in a word. I would think maybe the issue has to do with how permissions of a file/folder apply to different users, but then isn't that just exactly the raison d'etre of permission systems? This isn't an area I'm qualified to have a technical opinion on but if you'd care to elaborate I'd be interested to learn more.
  - kyrra 5y ago
    Googler opinions are my own. I've never worked on Drive. It's not hard, but expensive. If you have a file system where it is a proper tree in any given node within that tree only has One parent, and walking up all the parents you always only have one parent up to the root node, you gain some useful properties from that. You can build a permission model where you just have to check the permissions of all of your parents to see if someone has access to a given file deep within the system. This means that if you Grant Reed access on any node within that line of parents, all children implicitly get access. Now imagine a directed graph, and you set permission on a node and you want to say all connected nodes also should get this permission. You could definitely walk all the nodes and apply that permission, but now you have to write said permission to every node, rather than being able to set it on a parent and let all children inherit it. Even if you know all of your parents, then all of those nodes have multiple parents. And you can have cycles. Checking parents for permissions becomes way more complicated for users to understand, which is why Drive likely didn't do this.

zmj 5y ago

I worked on another sync client's representation of filesystem structure, and came to the same conclusion. Hard links enable some cool behavior, but in retrospect added more complexity than anyone expected. Migrating to shortcuts / soft links seems very reasonable - I wish I had started there.

djsavvy 5y ago
What kinds of cool behavior do hard links enable?
- mcoliver 5y ago
  Every file is essentially the master pointing at the same inode. So the file remains as long as any of the hard links exist. With soft links if you delete the master file all the soft links stop working.
- deleted 5y ago
  [deleted]
- dtgriscom 5y ago
  Apple's Time Machine uses hard-linked directories to replicate your home directory again and again, but with only the changed directories/files actually taking up space. The unchanged directories are just hard-linked to the previous versions, so any one version looks like your entire drive, but it takes much less space than full copies.
  - saagarjha 5y ago
    Used to, the new APFS implementation uses snapshots.

andybak 5y ago

> various applications that were being folded into / integrated with Google Drive The Photos/Drive integration was removed a long time ago. What other integrations were behind the original requirement? I'm curious to know if the extra complication was worth it in the long run and how long the integrations that needed this feature hung around for.

snewman 5y ago
I don't specifically remember whether there were other motivations for that requirement. Possibly a desire to support "tag" style interfaces, where an item can have multiple tags applied. But it may have mostly been the Photos integration. And no, definitely the complication was not worth it in the long run. :-) That project got way more complicated than anyone wanted (not that there's anything unusual about that).
- Andrew_nenakhov 5y ago
  I actually remember early Google Drive to have only tags for files, and no folders. Apparently, the idea was to distance from the file tree paradigm, but eventually the traditional approach proved to be more user friendly.

layer8 5y ago

So is this basically a switch from hardlinks to softlinks?

pishpash 5y ago
It's a switch from inference to direct collection of intent from the user.
- lupire 5y ago
  How so? Are hard links still allowed?
lupire 5y ago
Yes.

kyrra 5y ago

Googler, opinions are my own. This was my understanding as well. The original Drive was built effectively as a directed graph (with cycles allowed). Any file or folder could be stored in multiple locations. And permissions were at a per-file basis, so 2 people viewing the same folder may see different sets of files. And permissions were definitely a hard part of it, as if you applied new permissions to a folder and all children, it had to walk the entire graph to update the permissions. This is the advantage of the Team Drive style structure that the Drive team put out. It follows the classic filesystem design of a tree, which allows for easier permissions modeling, among other things. It's also why all "hard links" are now becoming shortcuts / Soft-links.

wunderlust 5y ago
Forgive my ignorance (and there's a lot of it), but navigating a graph and setting permissions doesn't seem like a terribly difficult problem, especially for Google - the king of graphs, in a word. I would think maybe the issue has to do with how permissions of a file/folder apply to different users, but then isn't that just exactly the raison d'etre of permission systems? This isn't an area I'm qualified to have a technical opinion on but if you'd care to elaborate I'd be interested to learn more.
- kyrra 5y ago
  Googler opinions are my own. I've never worked on Drive. It's not hard, but expensive. If you have a file system where it is a proper tree in any given node within that tree only has One parent, and walking up all the parents you always only have one parent up to the root node, you gain some useful properties from that. You can build a permission model where you just have to check the permissions of all of your parents to see if someone has access to a given file deep within the system. This means that if you Grant Reed access on any node within that line of parents, all children implicitly get access. Now imagine a directed graph, and you set permission on a node and you want to say all connected nodes also should get this permission. You could definitely walk all the nodes and apply that permission, but now you have to write said permission to every node, rather than being able to set it on a parent and let all children inherit it. Even if you know all of your parents, then all of those nodes have multiple parents. And you can have cycles. Checking parents for permissions becomes way more complicated for users to understand, which is why Drive likely didn't do this.