7 ms·

Show HN: Git-remote-dropbox – use Dropbox as a true Git server

174 points by anishathalye 11y ago

dr4g0n 11y ago
Looks good, I had previously used Dropbox for personal repositories, but now I might try using them for shared ones. It seems from a quick read of the design document [1] that the important thing here is the updating of refs using a compare-and-swap, ensuring that the synchronisation issues that typically occur with using Dropbox as a remote cannot occur. Objects are always* safe from this since the file names are SHA1 hashes. Edit: The only downside I can see with this is that the remote will never have garbage collection take place, so unreachable objects will continue to take up space indefinitely. [1]: https://github.com/anishathalye/git-remote-dropbox/blob/master/DESIGN.md https://github.com/anishathalye/git-remote-dropbox/blob/mast...
- phs2501 11y ago
  Also since packs are not used (the design document says it stores all objects as loose objects) there is no delta compression. This won't be usable for large repos with lots of history.
  - anishathalye 11y ago
    Yep, no packing and no delta compression. I haven't yet thought of a good way of doing that entirely client side. (If you can think of anything, please let me know!) The target use case is small personal and group projects. With the current design, it won't scale that well to large repos with lots of history. The initial clone will take a while. However, individual fetches or pushes after that shouldn't take that long unless there is a ton that changes.
- anishathalye 11y ago
  Yes, you're right, there is no garbage collection on the remote. (I should probably add that to the design doc.) I don't know a way of doing it client side that is concurrency-safe. Perhaps I'll write a separate script that will do remote GC, but will require that you manually ensure that nobody is modifying the repository concurrently.
tlarkworthy 11y ago
bare git on dropbox is not dangerous in a multi user setup. I tested it a few years ago, the conflicts are minor annoyances but never destructive nor silent. http://edinburghhacklab.com/2012/11/when-git-on-dropbox-conflicts-no-problem/ http://edinburghhacklab.com/2012/11/when-git-on-dropbox-conf...
- leni536 11y ago
  I think it's mostly not dangerous. The object store shouldn't be dangerous at all. Updating refs concurrently however... I think it is dangerous.
  - tlarkworthy 11y ago
    It's not, you get a conflict, so someone loses their last ref manipulation and you have to clear the conflict and update the ref again. You have a backup of the repo anyway as you cloned from it. So basically someone has to push again after the repo breaks (this is a bare repo in Dropbox with a local clone setup)
    - tacos 11y ago
      Right. And if that happens twice a year and it costs someone 10 minutes to fix, you could've just paid $12/yr to AWS CodeCommit and made money on the deal. Why would you half-ass such a critical piece of infrastructure when there are free and super inexpensive solutions that work properly?
      - tlarkworthy 11y ago
        I would not dream of doing it in a commercial environment. I do it in things like hackathons when you might be working with a random group, no-one wants to pony up to host it privately indefinitely, and messing around with permissions on AWS and registering everyone is a pain in the ass. Share folder, git init --bare, done. Most people have Dropbox so its super easy.
- InclinedPlane 11y ago
viraptor 11y ago
Just curious... Given that we have gitlab, bitbucket, assembla, etc. which provide free private git hosting and even more services if you don't need private repo - why would anyone host on dropbox?
- ics 11y ago
  Some ideas... - Mobile apps: ability to browse/create super easily from your iPhone or whatever using any app that integrates with Dropbox - Storage space: though not always written in stone, most git hosts expect code and prefer you keep your media, if its heavy, elsewhere. Users of git-annex, largefile, etc. benefit from this. - Collaboration: lots of people use Dropbox, not all of them need to use git (or at least, not all the time). See above. Unfortunately, after reading the readme I see this: "Do not directly interact with Git repositories in your Dropbox folder - always use git-remote-dropbox. If you're using the Dropbox client to sync files, it's a good idea to use selective sync and disable syncing of the folder containing the repository to avoid any unexpected conflicts, just in case." So it's not as rosy as my above points hoped for.
amalag 11y ago
This is also really useful How to Use S3 as a Private Git Repository http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a-private-git-repository/ http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a...
- tacos 11y ago
  For a few pennies more (or less) a month you can use AWS CodeCommit. https://aws.amazon.com/codecommit/pricing/ https://aws.amazon.com/codecommit/pricing/
  - bchallenor 11y ago
    Apparently CodeCommit is a new feature (last couple of months?). I had no idea it existed - thanks!
zkhalique 11y ago
How about the other way around ... using git + node watch plugin to make a owncloud dropbox?
andreasklinger 11y ago
I am a bit confused. This only represents server side right? You are still not expected to leave the working directory in the Dropbox folder? Reason i ask - Dropbox's client is very inefficient when it comes to small files.
- anishathalye 11y ago
  This is a remote helper meant to interact with a git repo hosted on Dropbox - it runs entirely on the client side - there's no special git-specific code that the Dropbox servers run. Correct, you're not expected to leave the working directory in a Dropbox folder. In fact, that's a bad idea (for the reasons explained in the blog post / readme). It's also recommended to selective sync out the repository directory so the desktop client doesn't touch the files at all.
- objectivefs 11y ago
  As anishathalye said you probably don't want to keep your git working directory on Dropbox. If you want to share a git working directory between machines, you can use a shared filesystem such as our ObjectiveFS (https://objectivefs.com https://objectivefs.com). Note that if you use git with an OSXFUSE filesystem you currently need to add the "--quiet" option to work around an issue with signals (https://github.com/osxfuse/osxfuse/issues/213 https://github.com/osxfuse/osxfuse/issues/213).
dkarapetyan 11y ago
Or you know pay $5 and host your git repositories at Digital Ocean or some other VPS provider. I honestly don't get this obsession of tacking on git on top of things like Dropbox. If you're technical enough to understand git then you're technical enough to set up your own hosting server. Just skip the middleman.
- Argorak 11y ago
  > If you're technical enough to understand git then you're technical enough to set up your own hosting server. I don't agree. Many people use GIT without having any knowledge of the work necessary to keep a server secure all year around. Amateur git usage is definitely a thing and not necessarily a small one.
snnn 11y ago
Why not just use visual studio online, it's free for unlimited private repos.
lclemente 11y ago
Shameless plug: If you want the data in dropbox encrypted, try https://github.com/lucas-clemente/git-cr https://github.com/lucas-clemente/git-cr :) I think I've also solved most of the concurrency problems, but haven't tested it yet.
forgotmypassw 11y ago
>We keep all of our files in Dropbox. Wrong.
cryptograph 11y ago
Why not just use BitBucket? It's free and reliable and is meant as a repository manager. And trusted like GitHub, but BitBucket also gives you unlimited public AND private repos. Did I mention free?
- mbel 11y ago
  Considering superior alternatives like mentioned BitBucket, I don't think I will use this project any time soon. But I'm very glad that some has created it. For me it demonstrates that git is flexible enough to support hosting/sharing solutions that are not standard today. I take it as a sign of git being a least a bit futureproof. But aside from that, do all hacks have to be practical?
- jesseendahl 11y ago
  BitBucket has no 2FA, both GitHub and Dropbox do.
  - fjarlq 11y ago
    ETA for BitBucket's 2FA support is less than 6 weeks: https://bitbucket.org/site/master/issues/5811/support-two-factor-authentication-bb-7016#comment-20910749 https://bitbucket.org/site/master/issues/5811/support-two-fa...
- xeromal 11y ago
  I thought it gave you only 5 private repos?
  - xasos 11y ago
    No, that's what you get with the GitHub student account (free for any student with a .edu/school email). BitBucket offers unlimited repositories for free.
    - sytse 11y ago
      You might also consider our GitLab.com, unlimited repo's and unlimited collaborators, unlike BitBucket.
tcdent 11y ago
This seems like something Dropbox should be aware of, and actively trying to fix.
- tcdent 11y ago
  My point being, if there is such demand from your customers for a particular feature that they hack it together themselves, or endure what commenters have expressed to be "tolerable strange behavior", then you should probably think about offering a solution. Is it right to host your git repositories in a Dropbox folder? Probably not. Is it their fault? No. Is there sufficient evidence that quite a few people are trying to do it? Yes: https://www.google.com/search?q=dropbox+git https://www.google.com/search?q=dropbox+git Unless, of course, they've become too big to care.
ossreality 11y ago
It's going to blow peoples' mind when they find out they can host a bare git repository and access it normally over http, ssh, file system, and more! Oh wait, oh no! I told the secret!
Vecrios 11y ago
I'm still not sold on the idea of using Dropbox and Git on the same folder. Would anyone mind explaining the benefits of having such setup?
gingerlime 11y ago
Seeing all those "why not use X instead" makes me wonder why it appears to be a binary option? Git lets you use ALL of them. You can push to multiple remotes and even automate things with hooks if you need to. Perfect for backup and redundancy in case any single remote is unavailable...
rcarmo 11y ago
Hmmm. I've been storing Git repos in Dropbox for... ages. Haven't tried sharing them, but for a single user I see no need to have an additional piece of software. I am, however, quite interested in the potential for business-related repos - I guess it all depends on the authentication and transport mechanisms (I assume this will be using Dropbox's auth and TLS API)?
- hellofunk 11y ago
  Have you used DB for your repos where you work on different machines that are not always on, and therefore there is a sync delay occasionally? And then get conflicted syncs on DB's end as a result when you try to interact with the repo before syncing is done across all machines? There are many headaches to just putting a git repo in DB and forgetting about it -- the one possible exception if you are only working on one machine all the time, with zero collaborators.
  - rcarmo 11y ago
    I use DB across many machines, some "permanently" on, two laptops and a (sporadically on) home desktop. Never had a conflict I couldn't recover from - and the 5-10 minutes some of the machines take to finish syncing are usually best spent catching up on e-mail rather than coding.
    - tacos 11y ago
      This is bad engineering. You are asking for trouble. Don't use DropBox for this use case.
      - rcarmo 11y ago
        I've been doing this since Dropbox 1.0. Haven't had any serious issues since 2.x, and the benefits of having my complete working state available everywhere far outweigh the risks.
        11y ago
stonewhite 11y ago
I used to mock people for using Dropbox for deployment (by configuring the document root as the dropbox folder). I hope they won't discover if I start using this.

174 points by anishathalye 11y ago

Looks good, I had previously used Dropbox for personal repositories, but now I might try using them for shared ones. It seems from a quick read of the design document [1] that the important thing here is the updating of refs using a compare-and-swap, ensuring that the synchronisation issues that typically occur with using Dropbox as a remote cannot occur. Objects are always* safe from this since the file names are SHA1 hashes. Edit: The only downside I can see with this is that the remote will never have garbage collection take place, so unreachable objects will continue to take up space indefinitely. [1]: https://github.com/anishathalye/git-remote-dropbox/blob/master/DESIGN.md https://github.com/anishathalye/git-remote-dropbox/blob/mast...

phs2501 11y ago
Also since packs are not used (the design document says it stores all objects as loose objects) there is no delta compression. This won't be usable for large repos with lots of history.
- anishathalye 11y ago
  Yep, no packing and no delta compression. I haven't yet thought of a good way of doing that entirely client side. (If you can think of anything, please let me know!) The target use case is small personal and group projects. With the current design, it won't scale that well to large repos with lots of history. The initial clone will take a while. However, individual fetches or pushes after that shouldn't take that long unless there is a ton that changes.
anishathalye 11y ago
Yes, you're right, there is no garbage collection on the remote. (I should probably add that to the design doc.) I don't know a way of doing it client side that is concurrency-safe. Perhaps I'll write a separate script that will do remote GC, but will require that you manually ensure that nobody is modifying the repository concurrently.

tlarkworthy 11y ago

bare git on dropbox is not dangerous in a multi user setup. I tested it a few years ago, the conflicts are minor annoyances but never destructive nor silent. http://edinburghhacklab.com/2012/11/when-git-on-dropbox-conflicts-no-problem/ http://edinburghhacklab.com/2012/11/when-git-on-dropbox-conf...

leni536 11y ago
I think it's mostly not dangerous. The object store shouldn't be dangerous at all. Updating refs concurrently however... I think it is dangerous.
- tlarkworthy 11y ago
  It's not, you get a conflict, so someone loses their last ref manipulation and you have to clear the conflict and update the ref again. You have a backup of the repo anyway as you cloned from it. So basically someone has to push again after the repo breaks (this is a bare repo in Dropbox with a local clone setup)
  - tacos 11y ago
    Right. And if that happens twice a year and it costs someone 10 minutes to fix, you could've just paid $12/yr to AWS CodeCommit and made money on the deal. Why would you half-ass such a critical piece of infrastructure when there are free and super inexpensive solutions that work properly?
    - tlarkworthy 11y ago
      I would not dream of doing it in a commercial environment. I do it in things like hackathons when you might be working with a random group, no-one wants to pony up to host it privately indefinitely, and messing around with permissions on AWS and registering everyone is a pain in the ass. Share folder, git init --bare, done. Most people have Dropbox so its super easy.
InclinedPlane 11y ago

viraptor 11y ago

Just curious... Given that we have gitlab, bitbucket, assembla, etc. which provide free private git hosting and even more services if you don't need private repo - why would anyone host on dropbox?

ics 11y ago
Some ideas... - Mobile apps: ability to browse/create super easily from your iPhone or whatever using any app that integrates with Dropbox - Storage space: though not always written in stone, most git hosts expect code and prefer you keep your media, if its heavy, elsewhere. Users of git-annex, largefile, etc. benefit from this. - Collaboration: lots of people use Dropbox, not all of them need to use git (or at least, not all the time). See above. Unfortunately, after reading the readme I see this: "Do not directly interact with Git repositories in your Dropbox folder - always use git-remote-dropbox. If you're using the Dropbox client to sync files, it's a good idea to use selective sync and disable syncing of the folder containing the repository to avoid any unexpected conflicts, just in case." So it's not as rosy as my above points hoped for.

amalag 11y ago

This is also really useful How to Use S3 as a Private Git Repository http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a-private-git-repository/ http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a...

tacos 11y ago
For a few pennies more (or less) a month you can use AWS CodeCommit. https://aws.amazon.com/codecommit/pricing/ https://aws.amazon.com/codecommit/pricing/
- bchallenor 11y ago
  Apparently CodeCommit is a new feature (last couple of months?). I had no idea it existed - thanks!

zkhalique 11y ago

How about the other way around ... using git + node watch plugin to make a owncloud dropbox?

andreasklinger 11y ago

I am a bit confused. This only represents server side right? You are still not expected to leave the working directory in the Dropbox folder? Reason i ask - Dropbox's client is very inefficient when it comes to small files.

anishathalye 11y ago
This is a remote helper meant to interact with a git repo hosted on Dropbox - it runs entirely on the client side - there's no special git-specific code that the Dropbox servers run. Correct, you're not expected to leave the working directory in a Dropbox folder. In fact, that's a bad idea (for the reasons explained in the blog post / readme). It's also recommended to selective sync out the repository directory so the desktop client doesn't touch the files at all.
objectivefs 11y ago
As anishathalye said you probably don't want to keep your git working directory on Dropbox. If you want to share a git working directory between machines, you can use a shared filesystem such as our ObjectiveFS (https://objectivefs.com https://objectivefs.com). Note that if you use git with an OSXFUSE filesystem you currently need to add the "--quiet" option to work around an issue with signals (https://github.com/osxfuse/osxfuse/issues/213 https://github.com/osxfuse/osxfuse/issues/213).

dkarapetyan 11y ago

Or you know pay $5 and host your git repositories at Digital Ocean or some other VPS provider. I honestly don't get this obsession of tacking on git on top of things like Dropbox. If you're technical enough to understand git then you're technical enough to set up your own hosting server. Just skip the middleman.

Argorak 11y ago
> If you're technical enough to understand git then you're technical enough to set up your own hosting server. I don't agree. Many people use GIT without having any knowledge of the work necessary to keep a server secure all year around. Amateur git usage is definitely a thing and not necessarily a small one.

snnn 11y ago

Why not just use visual studio online, it's free for unlimited private repos.

lclemente 11y ago

Shameless plug: If you want the data in dropbox encrypted, try https://github.com/lucas-clemente/git-cr https://github.com/lucas-clemente/git-cr :) I think I've also solved most of the concurrency problems, but haven't tested it yet.

forgotmypassw 11y ago

>We keep all of our files in Dropbox. Wrong.

cryptograph 11y ago

Why not just use BitBucket? It's free and reliable and is meant as a repository manager. And trusted like GitHub, but BitBucket also gives you unlimited public AND private repos. Did I mention free?

mbel 11y ago
Considering superior alternatives like mentioned BitBucket, I don't think I will use this project any time soon. But I'm very glad that some has created it. For me it demonstrates that git is flexible enough to support hosting/sharing solutions that are not standard today. I take it as a sign of git being a least a bit futureproof. But aside from that, do all hacks have to be practical?
jesseendahl 11y ago
BitBucket has no 2FA, both GitHub and Dropbox do.
- fjarlq 11y ago
  ETA for BitBucket's 2FA support is less than 6 weeks: https://bitbucket.org/site/master/issues/5811/support-two-factor-authentication-bb-7016#comment-20910749 https://bitbucket.org/site/master/issues/5811/support-two-fa...
xeromal 11y ago
I thought it gave you only 5 private repos?
- xasos 11y ago
  No, that's what you get with the GitHub student account (free for any student with a .edu/school email). BitBucket offers unlimited repositories for free.
  - sytse 11y ago
    You might also consider our GitLab.com, unlimited repo's and unlimited collaborators, unlike BitBucket.

tcdent 11y ago

This seems like something Dropbox should be aware of, and actively trying to fix.

tcdent 11y ago
My point being, if there is such demand from your customers for a particular feature that they hack it together themselves, or endure what commenters have expressed to be "tolerable strange behavior", then you should probably think about offering a solution. Is it right to host your git repositories in a Dropbox folder? Probably not. Is it their fault? No. Is there sufficient evidence that quite a few people are trying to do it? Yes: https://www.google.com/search?q=dropbox+git https://www.google.com/search?q=dropbox+git Unless, of course, they've become too big to care.

ossreality 11y ago

It's going to blow peoples' mind when they find out they can host a bare git repository and access it normally over http, ssh, file system, and more! Oh wait, oh no! I told the secret!

Vecrios 11y ago

I'm still not sold on the idea of using Dropbox and Git on the same folder. Would anyone mind explaining the benefits of having such setup?

gingerlime 11y ago

Seeing all those "why not use X instead" makes me wonder why it appears to be a binary option? Git lets you use ALL of them. You can push to multiple remotes and even automate things with hooks if you need to. Perfect for backup and redundancy in case any single remote is unavailable...

rcarmo 11y ago

Hmmm. I've been storing Git repos in Dropbox for... ages. Haven't tried sharing them, but for a single user I see no need to have an additional piece of software. I am, however, quite interested in the potential for business-related repos - I guess it all depends on the authentication and transport mechanisms (I assume this will be using Dropbox's auth and TLS API)?

hellofunk 11y ago
Have you used DB for your repos where you work on different machines that are not always on, and therefore there is a sync delay occasionally? And then get conflicted syncs on DB's end as a result when you try to interact with the repo before syncing is done across all machines? There are many headaches to just putting a git repo in DB and forgetting about it -- the one possible exception if you are only working on one machine all the time, with zero collaborators.
- rcarmo 11y ago
  I use DB across many machines, some "permanently" on, two laptops and a (sporadically on) home desktop. Never had a conflict I couldn't recover from - and the 5-10 minutes some of the machines take to finish syncing are usually best spent catching up on e-mail rather than coding.
  - tacos 11y ago
    This is bad engineering. You are asking for trouble. Don't use DropBox for this use case.
    - rcarmo 11y ago
      I've been doing this since Dropbox 1.0. Haven't had any serious issues since 2.x, and the benefits of having my complete working state available everywhere far outweigh the risks.
      - 11y ago

stonewhite 11y ago

I used to mock people for using Dropbox for deployment (by configuring the document root as the dropbox folder). I hope they won't discover if I start using this.