3 ms·

Why isn't there a syncing filesystem that works like this?

14 points by AndrewDucker 13y ago

samwillis 13y ago
As mentioned in the article Fuse is a good tool for building something like this but it is only supported on Unix/Linux and OSX. There is however a Windows project that does something very similar called Dokan (http://dokan-dev.net/ http://dokan-dev.net/). I recently discoverd a python libarary called pyfilesystem (https://code.google.com/p/pyfilesystem/ https://code.google.com/p/pyfilesystem/) that provides a unified API for both Fuse and Doken so that you can write the fs once and run it everywhere. What's also quite nice is that you actually create your fs as a python API first and then mount it. This results in you creating a good API that could be used inside a phone/tablet/web app, if you stick with python... The problem the OP is trying to solve is very similar to one I was experimenting with a few weeks ago but abandoned in order to concentrate on my main project that is beginning to really take off. If anyone is interested in starting a OS project around this sort of idea I would love to get involved.
lmm 13y ago
Resolving conflicts is hard, and if you try to do it yourself you will get it wrong. AFS is the Right Solution for this, but is complicated because it's a complicated problem. They have really worked hard to get the primitives correct though, so if your end goal is something reliable that everyone can use easily, I think the best approach would be to improve the admin tools for AFS. Like crypto or databases, filesystems are an area where you really don't want to write your own.
charlysisto 13y ago
Excellent point. Why doesn't dropbox have an "upload only" or "on demand" feature, I don't get it. Apple's iCloud solved this quit well : if you delete an aac file, it asks you if you want to delete it as well on the cloud. If you don't, the music is still available for streaming or downloading & you've saved space on your HD.
rlpb 13y ago
I don't understand why git-annex is dismissed. The only reason given is "But for that reason, it's impossible to fit all my 200Gb of Stuff[tm]into the 120Gb of SSD on my work laptop.". Except that with git-annex, you don't have to keep all the files in all places. Files not present are represented as broken symlinks exactly as described. You can even specify pattern matches to determine what files are kept where by default (and override as necessary).
seclorum 13y ago
I'd like to answer the question. The answer to the question is: "because OS vendors have stopped doing their job". Have you not noticed that the OS no longer provides the features of an Operating System? Wouldn't a "true OS vendor" have added peer-to-peer and file-sharing-across-multiple-networks/nodes to their feature set by now? What I think has happened is that the OS vendors: Microsoft, Apple, and to a much lesser extent, the Linux crowd - have forgotten that they, themselves, define what the "OS" is, but it is the users who require it. Why on Earth some OS vendor hasn't solved this mid-80's problem of network file sync, is simply this: there is no profit in it. In short, the bugs and holes in featuresets provided by the current OS vendors have been filled in by an industry far too furiously devouring itself to care that they've gone off-task. The fact is, we should've had DropBox -like features integrated into our OS's, already. We should definitely have had Bittorrent-sync like features, implemented already. We should definitely be demanding these sorts of features from the OS vendors - but do we? The answer is, no. We don't. Because, every 5 years or so, newcomers to the computer scene, the teenagers of the now who will be the next-bright-new-thing of the morrow, fail to keep up with the state of the art just well enough that they end up re-inventing the very thing that their much-hated "prior generations" have been, literally, on the cusp of inventing .. for years. So yeah, we have this situation because there is no longer a 'true OS' out there in the world, but rather 'front-ends to a business model'. Maybe that will change, who knows, but I bet that before it does, the words "operating" and "system" will have to suffer a bit of re-definition, here and there. Or wait, maybe that is exactly the problem ..
- rlpb 13y ago
  > and to a much lesser extent, the Linux crowd The Linux crowd needs to write the tool, release it under a suitable licence, and only then can OS vendors adopt it. Note that Canonical did it with Ubuntu One years ago (I'm not sure if the server side is open though). It's a Dropbox-alike, but doesn't address the situations the original article described. Joey Hess is doing it with git-annex. I think it does describe the situations the original article described. If it proves itself in the general case (rather than just as a specialist tool, which I think it already has), then I expect that Linux distributions will eventually bring it to the forefront. So the OS is evolving. But in the Free Software world, you can't demand that the vendors do their job. It's up to people who want it to get it done. If that's not fast enough for you, get involved.
eterps 13y ago
CodaFS was also an interesting effort. But it was too difficult to get working. http://en.wikipedia.org/wiki/Coda_(file_system) http://en.wikipedia.org/wiki/Coda_(file_system)

14 points by AndrewDucker 13y ago

As mentioned in the article Fuse is a good tool for building something like this but it is only supported on Unix/Linux and OSX. There is however a Windows project that does something very similar called Dokan (http://dokan-dev.net/ http://dokan-dev.net/). I recently discoverd a python libarary called pyfilesystem (https://code.google.com/p/pyfilesystem/ https://code.google.com/p/pyfilesystem/) that provides a unified API for both Fuse and Doken so that you can write the fs once and run it everywhere. What's also quite nice is that you actually create your fs as a python API first and then mount it. This results in you creating a good API that could be used inside a phone/tablet/web app, if you stick with python... The problem the OP is trying to solve is very similar to one I was experimenting with a few weeks ago but abandoned in order to concentrate on my main project that is beginning to really take off. If anyone is interested in starting a OS project around this sort of idea I would love to get involved.

lmm 13y ago

Resolving conflicts is hard, and if you try to do it yourself you will get it wrong. AFS is the Right Solution for this, but is complicated because it's a complicated problem. They have really worked hard to get the primitives correct though, so if your end goal is something reliable that everyone can use easily, I think the best approach would be to improve the admin tools for AFS. Like crypto or databases, filesystems are an area where you really don't want to write your own.

charlysisto 13y ago

Excellent point. Why doesn't dropbox have an "upload only" or "on demand" feature, I don't get it. Apple's iCloud solved this quit well : if you delete an aac file, it asks you if you want to delete it as well on the cloud. If you don't, the music is still available for streaming or downloading & you've saved space on your HD.

rlpb 13y ago

I don't understand why git-annex is dismissed. The only reason given is "But for that reason, it's impossible to fit all my 200Gb of Stuff[tm]into the 120Gb of SSD on my work laptop.". Except that with git-annex, you don't have to keep all the files in all places. Files not present are represented as broken symlinks exactly as described. You can even specify pattern matches to determine what files are kept where by default (and override as necessary).

seclorum 13y ago

I'd like to answer the question. The answer to the question is: "because OS vendors have stopped doing their job". Have you not noticed that the OS no longer provides the features of an Operating System? Wouldn't a "true OS vendor" have added peer-to-peer and file-sharing-across-multiple-networks/nodes to their feature set by now? What I think has happened is that the OS vendors: Microsoft, Apple, and to a much lesser extent, the Linux crowd - have forgotten that they, themselves, define what the "OS" is, but it is the users who require it. Why on Earth some OS vendor hasn't solved this mid-80's problem of network file sync, is simply this: there is no profit in it. In short, the bugs and holes in featuresets provided by the current OS vendors have been filled in by an industry far too furiously devouring itself to care that they've gone off-task. The fact is, we should've had DropBox -like features integrated into our OS's, already. We should definitely have had Bittorrent-sync like features, implemented already. We should definitely be demanding these sorts of features from the OS vendors - but do we? The answer is, no. We don't. Because, every 5 years or so, newcomers to the computer scene, the teenagers of the now who will be the next-bright-new-thing of the morrow, fail to keep up with the state of the art just well enough that they end up re-inventing the very thing that their much-hated "prior generations" have been, literally, on the cusp of inventing .. for years. So yeah, we have this situation because there is no longer a 'true OS' out there in the world, but rather 'front-ends to a business model'. Maybe that will change, who knows, but I bet that before it does, the words "operating" and "system" will have to suffer a bit of re-definition, here and there. Or wait, maybe that is exactly the problem ..

rlpb 13y ago
> and to a much lesser extent, the Linux crowd The Linux crowd needs to write the tool, release it under a suitable licence, and only then can OS vendors adopt it. Note that Canonical did it with Ubuntu One years ago (I'm not sure if the server side is open though). It's a Dropbox-alike, but doesn't address the situations the original article described. Joey Hess is doing it with git-annex. I think it does describe the situations the original article described. If it proves itself in the general case (rather than just as a specialist tool, which I think it already has), then I expect that Linux distributions will eventually bring it to the forefront. So the OS is evolving. But in the Free Software world, you can't demand that the vendors do their job. It's up to people who want it to get it done. If that's not fast enough for you, get involved.

eterps 13y ago

CodaFS was also an interesting effort. But it was too difficult to get working. http://en.wikipedia.org/wiki/Coda_(file_system) http://en.wikipedia.org/wiki/Coda_(file_system)