4 ms·

Hello everyone! I have built a tool that allows Linux and macOS users to mount their Google Drive account locally as a virtual file system. The file system sup

Hello everyone! I have built a tool that allows Linux and macOS users to mount their Google Drive account locally as a virtual file system. The file system supports most typical operations (creating/deleting/moving/renaming files and directories, reading/writing to them). It can also export special Drive files as OpenOffice documents and has an in-memory cache which improves the speed of navigation and file access. Changes performed on other clients (e.g. on the web or mobile interface) are usually detected shortly and applied locally as well. I also wrote a paper[0] on it, as I am using the project for my bachelor thesis. It is still rough around the edges and lacks some functionality, but for the moment it is good enough for my personal use. I am looking forward to hearing your comments and feedback on how to improve it. [0] https://sergiu.ml/~sergiu/thesis.pdf https://sergiu.ml/~sergiu/thesis.pdf

Improvotter 8y ago
This is neat! Can I use this for multiple accounts? I've been looking for a MacOS and Linux solution for this exact problem but I'd like to use it for 3-4 Google accounts. Another question related to implementation. How easy is it to use a language like Rust for some web connection stuff like what is being used here? I've never used the language, but I've always been interested in it.
- harababurel 8y ago
  > This is neat! Can I use this for multiple accounts? I've been looking for a MacOS and Linux solution for this exact problem but I'd like to use it for 3-4 Google accounts. The current release allows you to mount a single account for each local user. For the moment, you could work around this limitation by creating another user on your machine and running a separate instance of GCSF as the new user. I created an issue [0] and will work on adding support for multiple accounts in a future release. > How easy is it to use a language like Rust for some web connection stuff like what is being used here? I've never used the language, but I've always been interested in it. You can make HTTP requests relatively easily with the help of hyper [1]. In the case of this project, I was lucky to have some useful libraries readily available: yup-oauth2 [2], google-drive3 [3]. I would place Rust somewhere below Python in terms of existing tools and support (for instance, Google doesn't provide any official client libraries for Rust, but it does for Python), but making relatively simple applications is completely achievable (and fun as well) in Rust. [0] https://github.com/harababurel/gcsf/issues/10 https://github.com/harababurel/gcsf/issues/10 [1] https://hyper.rs/ https://hyper.rs/ [2] https://github.com/dermesser/yup-oauth2 https://github.com/dermesser/yup-oauth2 [3] https://crates.io/crates/google-drive3 https://crates.io/crates/google-drive3
- CPAhem 8y ago
  This looks very useful! We've been using Syncdocs [0] to sync multiple accounts concurrently to Google Drive. Syncdocs does end-to-end encryption. Would it be possible to put in a PGP layer that automatically encrypts into your solution? [0] https://syncdocs.com https://syncdocs.com
  - stewbrew 8y ago
    This looks more like a contender to rclone, doesn't it?
marcoperaza 8y ago
Cool project. Just FYI: footnote 68, which is supposed to have the Github link, has a link to something else instead.
- harababurel 8y ago
  Thank you for pointing this out. The link should now be fixed.
brian_herman 8y ago
This is great Linux support finally! And in rust! Awesome work man!
- harababurel 8y ago
  Thank you!
dvfjsdhgfv 8y ago
Are multiple accounts supported?
- harababurel 8y ago
  Not at the present moment. I plan to add support for this in the near future.
giancarlostoro 8y ago
Guessing you're basically building a FUSE filesystem with a Google Drive backend? I know KeyBase.IO did something similar accross platforms. Not sure if thats how you did it but might interest you to check out FUSE too.
kbumsik 8y ago
As an undergrad thesis? Wow you are amazing! I'm wondering how it differ from the existing Google Drive FUSE project? [1] [1]: https://github.com/astrada/google-drive-ocamlfuse https://github.com/astrada/google-drive-ocamlfuse
- harababurel 8y ago
  Thank you! I have covered this question in a different comment: > In short, GCSF tends to be faster in several cases (listing files recursively, reading large files from Drive). The caching strategy it uses also leads to very fast reads (x4-7 improvement compared to google-drive-ocamlfuse) for files that have been cached, at the cost of using more RAM. > On the other hand, I arrived at the conclusion that google-drive-ocamlfuse provides a better overall experience, as it already has an active community behind it. My goal with GCSF is to more or less close the gap between the two projects and reach the same level of functionality.
- cheez 8y ago
  I did one as well. It's a no brainer: a semester to do whatever you want as long as you get someone to agree to mark it.
  - chatmasta 8y ago
    It was required for us, one semester of “independent study” where you must select a research adviser (professor) and produce a final report. I did mine on “TorCoin” and it was an awesome experience, the only time I legitimately enjoyed school.
    - cheez 8y ago
      Definitely not the only time I enjoyed school, but definitely my favourite part.
smaudet 8y ago
Not sure what is unique about this...no offense. In addition to FUSE which is a wrapper that must be then given calls just like this, there are I suspect tens if not hundreds of non fuse implementations - the drive api is practically written for people to do this. I say this not out of malice, I've done it myself before when integrating an application written in python which wad expecting a filesystem. Props for writing it in rust though - I've been looking for an application in rust to cut my teeth on, perhaps I'll use yours as an intro. :)
unholiness 8y ago
Awesome project! I'm curious, how does it compare to google's own similar product, Backup and Sync[0]? It seems like the primary difference (other than being FUSE-based) is that, for data where google has a collaborative interface (docs, sheet, etc), Backup and Sync will place into the folder a "file" which is simply a link to open up the docs/sheets/slides app. Your app, on the other hand, seems to translate this data into open office format when changes are detected (i.e. .ofd for docs). At first glance this strategy seems less good for real-time collaboration and less performant, but there may be advantages to it as well. Do you find that strategy to be practical in most cases? Are there other features that distinguish this from Backup and Sync? EDIT: Aha, looking into it more, it seems like mounting drive folder with this project doesn't trigger the downloading of any data, and rather lazily loads the data when the file system requests it, which would be a very significant difference. Is that right? [0]https://www.google.com/drive/download/backup-and-sync/ https://www.google.com/drive/download/backup-and-sync/
- harababurel 8y ago
  Thank you for your response! You are correct in saying that GCSF doesn't download any data upfront. It constructs the file tree at mount time using only file metadata and downloads the actual file content only when it encounters a `read` call. This is an advantage if you're running low on local space (the file system essentially adds 15 GB of "free" additional storage). Real time collaboration is indeed a shortcoming. I would still use the online interface of docs/sheets/slides for this purpose. I haven't personally used Backup and Sync, as there is no Linux version of it. From what I gather, it seems that it uploads local files to a new category on Drive instead of the 'My Drive' directory. This can be useful for automatic backup. You simply set it up once and forget about it. However, GCSF might be a better choice for the additional control it provides. Whereas with Backup and Sync you have to inspect a file manually in order to check whether it was synced or not, GCSF ensures that a pending write operation will only return once the file transfer is effectively complete. For instance, when copying a file to Drive, the execution of the command will take as long as the upload process itself. Once finished, an exit status of 0 will indicate precisely that the upload was successful and the file is certainly on Drive. I imagine this sort of strategy is a better fit for use cases which require high confidence and predictable behavior.

Improvotter 8y ago

This is neat! Can I use this for multiple accounts? I've been looking for a MacOS and Linux solution for this exact problem but I'd like to use it for 3-4 Google accounts. Another question related to implementation. How easy is it to use a language like Rust for some web connection stuff like what is being used here? I've never used the language, but I've always been interested in it.

harababurel 8y ago
> This is neat! Can I use this for multiple accounts? I've been looking for a MacOS and Linux solution for this exact problem but I'd like to use it for 3-4 Google accounts. The current release allows you to mount a single account for each local user. For the moment, you could work around this limitation by creating another user on your machine and running a separate instance of GCSF as the new user. I created an issue [0] and will work on adding support for multiple accounts in a future release. > How easy is it to use a language like Rust for some web connection stuff like what is being used here? I've never used the language, but I've always been interested in it. You can make HTTP requests relatively easily with the help of hyper [1]. In the case of this project, I was lucky to have some useful libraries readily available: yup-oauth2 [2], google-drive3 [3]. I would place Rust somewhere below Python in terms of existing tools and support (for instance, Google doesn't provide any official client libraries for Rust, but it does for Python), but making relatively simple applications is completely achievable (and fun as well) in Rust. [0] https://github.com/harababurel/gcsf/issues/10 https://github.com/harababurel/gcsf/issues/10 [1] https://hyper.rs/ https://hyper.rs/ [2] https://github.com/dermesser/yup-oauth2 https://github.com/dermesser/yup-oauth2 [3] https://crates.io/crates/google-drive3 https://crates.io/crates/google-drive3
CPAhem 8y ago
This looks very useful! We've been using Syncdocs [0] to sync multiple accounts concurrently to Google Drive. Syncdocs does end-to-end encryption. Would it be possible to put in a PGP layer that automatically encrypts into your solution? [0] https://syncdocs.com https://syncdocs.com
- stewbrew 8y ago
  This looks more like a contender to rclone, doesn't it?

marcoperaza 8y ago

Cool project. Just FYI: footnote 68, which is supposed to have the Github link, has a link to something else instead.

harababurel 8y ago
Thank you for pointing this out. The link should now be fixed.

brian_herman 8y ago

This is great Linux support finally! And in rust! Awesome work man!

harababurel 8y ago
Thank you!

dvfjsdhgfv 8y ago

Are multiple accounts supported?

harababurel 8y ago
Not at the present moment. I plan to add support for this in the near future.

giancarlostoro 8y ago

Guessing you're basically building a FUSE filesystem with a Google Drive backend? I know KeyBase.IO did something similar accross platforms. Not sure if thats how you did it but might interest you to check out FUSE too.

kbumsik 8y ago

As an undergrad thesis? Wow you are amazing! I'm wondering how it differ from the existing Google Drive FUSE project? [1] [1]: https://github.com/astrada/google-drive-ocamlfuse https://github.com/astrada/google-drive-ocamlfuse

harababurel 8y ago
Thank you! I have covered this question in a different comment: > In short, GCSF tends to be faster in several cases (listing files recursively, reading large files from Drive). The caching strategy it uses also leads to very fast reads (x4-7 improvement compared to google-drive-ocamlfuse) for files that have been cached, at the cost of using more RAM. > On the other hand, I arrived at the conclusion that google-drive-ocamlfuse provides a better overall experience, as it already has an active community behind it. My goal with GCSF is to more or less close the gap between the two projects and reach the same level of functionality.
cheez 8y ago
I did one as well. It's a no brainer: a semester to do whatever you want as long as you get someone to agree to mark it.
- chatmasta 8y ago
  It was required for us, one semester of “independent study” where you must select a research adviser (professor) and produce a final report. I did mine on “TorCoin” and it was an awesome experience, the only time I legitimately enjoyed school.
  - cheez 8y ago
    Definitely not the only time I enjoyed school, but definitely my favourite part.

smaudet 8y ago

Not sure what is unique about this...no offense. In addition to FUSE which is a wrapper that must be then given calls just like this, there are I suspect tens if not hundreds of non fuse implementations - the drive api is practically written for people to do this. I say this not out of malice, I've done it myself before when integrating an application written in python which wad expecting a filesystem. Props for writing it in rust though - I've been looking for an application in rust to cut my teeth on, perhaps I'll use yours as an intro. :)

unholiness 8y ago

Awesome project! I'm curious, how does it compare to google's own similar product, Backup and Sync[0]? It seems like the primary difference (other than being FUSE-based) is that, for data where google has a collaborative interface (docs, sheet, etc), Backup and Sync will place into the folder a "file" which is simply a link to open up the docs/sheets/slides app. Your app, on the other hand, seems to translate this data into open office format when changes are detected (i.e. .ofd for docs). At first glance this strategy seems less good for real-time collaboration and less performant, but there may be advantages to it as well. Do you find that strategy to be practical in most cases? Are there other features that distinguish this from Backup and Sync? EDIT: Aha, looking into it more, it seems like mounting drive folder with this project doesn't trigger the downloading of any data, and rather lazily loads the data when the file system requests it, which would be a very significant difference. Is that right? [0]https://www.google.com/drive/download/backup-and-sync/ https://www.google.com/drive/download/backup-and-sync/

harababurel 8y ago
Thank you for your response! You are correct in saying that GCSF doesn't download any data upfront. It constructs the file tree at mount time using only file metadata and downloads the actual file content only when it encounters a `read` call. This is an advantage if you're running low on local space (the file system essentially adds 15 GB of "free" additional storage). Real time collaboration is indeed a shortcoming. I would still use the online interface of docs/sheets/slides for this purpose. I haven't personally used Backup and Sync, as there is no Linux version of it. From what I gather, it seems that it uploads local files to a new category on Drive instead of the 'My Drive' directory. This can be useful for automatic backup. You simply set it up once and forget about it. However, GCSF might be a better choice for the additional control it provides. Whereas with Backup and Sync you have to inspect a file manually in order to check whether it was synced or not, GCSF ensures that a pending write operation will only return once the file transfer is effectively complete. For instance, when copying a file to Drive, the execution of the command will take as long as the upload process itself. Once finished, an exit status of 0 will indicate precisely that the upload was successful and the file is certainly on Drive. I imagine this sort of strategy is a better fit for use cases which require high confidence and predictable behavior.