10 ms·

A quick search for "google drive fuse" reveals a better approach to this... https://github.com/astrada/google-drive-ocamlfuse/ https://github.com/astrada/googl

by zx2c4 14y ago

A quick search for "google drive fuse" reveals a better approach to this... https://github.com/astrada/google-drive-ocamlfuse/ https://github.com/astrada/google-drive-ocamlfuse/ https://github.com/jcline/fuse-google-drive https://github.com/jcline/fuse-google-drive Didn't get much attention when it was posted here 267 days ago: http://news.ycombinator.com/item?id=3887308 http://news.ycombinator.com/item?id=3887308

VMG 14y ago
What makes you think the FUSE approach is better?
- hnriot 14y ago
  FUSE is the way to make a source look like a file system. It's the right architecture for this.
  - dfc 14y ago
    "Umm. 'userspace filesystem'? The problem is right there. Always has been. People who think that userspace filesystems are realistic for anything but toys are just misguided." --LT[1] [1] http://www.spinics.net/lists/linux-fsdevel/msg46078.html http://www.spinics.net/lists/linux-fsdevel/msg46078.html
    - notacoward 14y ago
      Nice appeal to authority you've got there. I'd hate to see anything happen to it . . . like realizing that when it comes to storage Linus is no expert. He hasn't worked on non-trivial storage systems, and some of the world's largest systems make good use of user-space filesystems like PVFS. Is Intrepid a toy? Nope. He's simply wrong, as he usually is about storage. On to the facts. The fact is that without being in the I/O path you can't detect and propagate updates as they happen - even with inotify, which doesn't scale well to very large filesystems anyway. There will always be some lag, and some reordering. Whether those matter to you depends a lot on your specific needs, but I for one don't like to rely on storage software that don't even try to guarantee correct behavior and might fail me at the most inopportune moment. FUSE is the way to get in the I/O path without having to become a kernel developer (which I am BTW, since before Linus, since you happen to like appeals to authority). A FUSE-based result will give better functional guarantees and better performance than a hacked rsync. Whether it might have some disadvantages compared to kernel code isn't even the right question. Compared to what this thread is about, FUSE actually is a better answer.
      - dfc 14y ago
        I thought that the canonical way to use pvfs was with the kernel module. Isn't the pvfs2fuse interface an afterthought?
        notacoward 14y ago
        The kernel module is essentially an interceptor, much like FUSE; the serious logic is still all in user space. When I talked to the developers about four years ago, they had considered using FUSE on several occasions but it had never performed as well - for them - as what they had already developed themselves. When Linus says that user-space filesystems are just toys, he's simply wrong. If he were to say the same thing about FUSE in particular he'd be closer to mark, but a large part of the reason FUSE has lagged is Linus. You don't get credit for predicting an outcome you control, in my book. It's worse than it needs to be because Al Viro and others actively resist any changes that would make it better. Linus, who delegated this part of the kernel to others nearly twenty years ago, backs them up. Making the VFS layer look more exactly like what XFS or even NFS needs is par for the course. Making the smallest concession to what FUSE needs is verboten. I guess some folks are afraid of losing their niche if those artificial obstacles were removed so that more people could build filesystems without begging for keys from the gatekeepers.
      - andrewflnr 14y ago
        Ah, git is not a non-trivial storage system?
        notacoward 14y ago
        Git is not a storage system at all. I was talking about actual physical storage systems that drive requirements for storage software, and there don't seem to be many petabyte-level git repositories. Git doesn't even qualify as storage software, as it delegates all of the real storage tasks to filesystems and such. It's an application that uses storage, much like every other application from emacs to Excel. Also, have you noticed that git is in user space? Must be a toy, then. Just ask Linus.
    - 1amzave 14y ago
      Wow -- did you read even a single paragraph beyond that line? Immediately after that he goes on to say: > fuse works fine if the thing being exported is some random low-use interface to a fundamentally slow device. But for something like your root filesystem? Nope. Not going to happen. "Random low-use interface to a fundamentally slow device" (like say a remote server in some faraway "cloud") sounds exactly like the situation at hand. Compared to a "real" filesystem (like something you'd mount on /), GDrive is very much a toy, and seems to me like an ideal candidate for a FUSE filesystem.
      - dfc 14y ago
        Is Dropbox a toy too? Or is gdrive just a toy because you have decided that it will be a low use storage device?
        1amzave 14y ago
        > Is Dropbox a toy too? In this context, yes, for exactly the same reasons as GDrive. No one with any sense is going to be running a relational database or MapReduce out of either one. Note that when I say "toy" in this context it's not really anything pejorative -- I use such "toy" filesystems all the time (sshfs most often, but also encfs and s3ql), and they're awesome and highly useful. But they're the kind of thing where, say, lock contention (or hell, even being single-threaded) isn't much of a big deal because they're not performance-critical.
- fuzzix 14y ago
  My first thought on seeing Grive was "Oh, it's not FUSE" FUSE is a natural choice if you want fully integrated and transparent access to so-called "cloud storage" services. Performance overhead isn't so much of an issue, you'll probably manage to max out most home broadband services even with the several layers of syscalls. You'll have less to remember, throw a line in fstab (or mount it in your shell's rc) and it's all just there, no sync commands to remember, just grab your favourite application and start hammering away at your files.
plywoodtrees 14y ago
Canonical's first cloud filesystem for Ubuntu One used Fuse, then it was later rewritten away from it, and became much more reliable. Correlation, or causation? It's complicated. But part of the attraction for Drive, Dropbox, and Ubuntu One is that the data is all on your local disk, so should still be accessible even if you're offline or there is a problem with the service. Making access to the local copy of the files depend on the client program being up is inherently more fragile. Now if you want to _not_ have them local, but rather fetch them lazily on demand, Fuse would be a good choice. It does open up a fair bit more complexity to do with scheduling and abandoning requests when programs assume the filesystem is as fast and reliable as a local one, similar to NFS. I think a read-through client could be good for v2.0, but getting local sync to a real filesystem working well would be better for 1.0.
- zx2c4 14y ago
  Good point; I'm compelled. Dropbox style local syncers vs remote file systems. I see.
  - zurn 14y ago
    I've had a Ubuntu laptop in development with root fs top of fuse (ntfs-3g) for 2+ years without a hitch. It's easily fast enough too.
    - zx2c4 14y ago
      This. Oh god no. No. Don't put your root fs on fuse. Or put ntfs on root. Please, for your own good? Or am I being trolled? I'm probably being trolled. I'm being trolled, right guys? Right?
      - zurn 14y ago
        That's what I thought when I discovered how it worked. The "WUBI" install type is (was?) a supported install type in Ubuntu. But it's been working fine and I can't think any fundamental reason why it would be unsound. I should say there have been non-fuse related hitches with the integration with Ubuntu (grub problems).

by zx2c4 14y ago

VMG 14y ago

What makes you think the FUSE approach is better?

hnriot 14y ago
FUSE is the way to make a source look like a file system. It's the right architecture for this.
- dfc 14y ago
  "Umm. 'userspace filesystem'? The problem is right there. Always has been. People who think that userspace filesystems are realistic for anything but toys are just misguided." --LT[1] [1] http://www.spinics.net/lists/linux-fsdevel/msg46078.html http://www.spinics.net/lists/linux-fsdevel/msg46078.html
  - notacoward 14y ago
    Nice appeal to authority you've got there. I'd hate to see anything happen to it . . . like realizing that when it comes to storage Linus is no expert. He hasn't worked on non-trivial storage systems, and some of the world's largest systems make good use of user-space filesystems like PVFS. Is Intrepid a toy? Nope. He's simply wrong, as he usually is about storage. On to the facts. The fact is that without being in the I/O path you can't detect and propagate updates as they happen - even with inotify, which doesn't scale well to very large filesystems anyway. There will always be some lag, and some reordering. Whether those matter to you depends a lot on your specific needs, but I for one don't like to rely on storage software that don't even try to guarantee correct behavior and might fail me at the most inopportune moment. FUSE is the way to get in the I/O path without having to become a kernel developer (which I am BTW, since before Linus, since you happen to like appeals to authority). A FUSE-based result will give better functional guarantees and better performance than a hacked rsync. Whether it might have some disadvantages compared to kernel code isn't even the right question. Compared to what this thread is about, FUSE actually is a better answer.
    - dfc 14y ago
      I thought that the canonical way to use pvfs was with the kernel module. Isn't the pvfs2fuse interface an afterthought?
      - notacoward 14y ago
        The kernel module is essentially an interceptor, much like FUSE; the serious logic is still all in user space. When I talked to the developers about four years ago, they had considered using FUSE on several occasions but it had never performed as well - for them - as what they had already developed themselves. When Linus says that user-space filesystems are just toys, he's simply wrong. If he were to say the same thing about FUSE in particular he'd be closer to mark, but a large part of the reason FUSE has lagged is Linus. You don't get credit for predicting an outcome you control, in my book. It's worse than it needs to be because Al Viro and others actively resist any changes that would make it better. Linus, who delegated this part of the kernel to others nearly twenty years ago, backs them up. Making the VFS layer look more exactly like what XFS or even NFS needs is par for the course. Making the smallest concession to what FUSE needs is verboten. I guess some folks are afraid of losing their niche if those artificial obstacles were removed so that more people could build filesystems without begging for keys from the gatekeepers.
    - andrewflnr 14y ago
      Ah, git is not a non-trivial storage system?
      - notacoward 14y ago
        Git is not a storage system at all. I was talking about actual physical storage systems that drive requirements for storage software, and there don't seem to be many petabyte-level git repositories. Git doesn't even qualify as storage software, as it delegates all of the real storage tasks to filesystems and such. It's an application that uses storage, much like every other application from emacs to Excel. Also, have you noticed that git is in user space? Must be a toy, then. Just ask Linus.
  - 1amzave 14y ago
    Wow -- did you read even a single paragraph beyond that line? Immediately after that he goes on to say: > fuse works fine if the thing being exported is some random low-use interface to a fundamentally slow device. But for something like your root filesystem? Nope. Not going to happen. "Random low-use interface to a fundamentally slow device" (like say a remote server in some faraway "cloud") sounds exactly like the situation at hand. Compared to a "real" filesystem (like something you'd mount on /), GDrive is very much a toy, and seems to me like an ideal candidate for a FUSE filesystem.
    - dfc 14y ago
      Is Dropbox a toy too? Or is gdrive just a toy because you have decided that it will be a low use storage device?
      - 1amzave 14y ago
        > Is Dropbox a toy too? In this context, yes, for exactly the same reasons as GDrive. No one with any sense is going to be running a relational database or MapReduce out of either one. Note that when I say "toy" in this context it's not really anything pejorative -- I use such "toy" filesystems all the time (sshfs most often, but also encfs and s3ql), and they're awesome and highly useful. But they're the kind of thing where, say, lock contention (or hell, even being single-threaded) isn't much of a big deal because they're not performance-critical.
fuzzix 14y ago
My first thought on seeing Grive was "Oh, it's not FUSE" FUSE is a natural choice if you want fully integrated and transparent access to so-called "cloud storage" services. Performance overhead isn't so much of an issue, you'll probably manage to max out most home broadband services even with the several layers of syscalls. You'll have less to remember, throw a line in fstab (or mount it in your shell's rc) and it's all just there, no sync commands to remember, just grab your favourite application and start hammering away at your files.

plywoodtrees 14y ago

Canonical's first cloud filesystem for Ubuntu One used Fuse, then it was later rewritten away from it, and became much more reliable. Correlation, or causation? It's complicated. But part of the attraction for Drive, Dropbox, and Ubuntu One is that the data is all on your local disk, so should still be accessible even if you're offline or there is a problem with the service. Making access to the local copy of the files depend on the client program being up is inherently more fragile. Now if you want to _not_ have them local, but rather fetch them lazily on demand, Fuse would be a good choice. It does open up a fair bit more complexity to do with scheduling and abandoning requests when programs assume the filesystem is as fast and reliable as a local one, similar to NFS. I think a read-through client could be good for v2.0, but getting local sync to a real filesystem working well would be better for 1.0.

zx2c4 14y ago
Good point; I'm compelled. Dropbox style local syncers vs remote file systems. I see.
- zurn 14y ago
  I've had a Ubuntu laptop in development with root fs top of fuse (ntfs-3g) for 2+ years without a hitch. It's easily fast enough too.
  - zx2c4 14y ago
    This. Oh god no. No. Don't put your root fs on fuse. Or put ntfs on root. Please, for your own good? Or am I being trolled? I'm probably being trolled. I'm being trolled, right guys? Right?
    - zurn 14y ago
      That's what I thought when I discovered how it worked. The "WUBI" install type is (was?) a supported install type in Ubuntu. But it's been working fine and I can't think any fundamental reason why it would be unsound. I should say there have been non-fuse related hitches with the integration with Ubuntu (grub problems).