5 ms·

How we use BitTorrent to improve our design sharing workflow

63 points by sgottit 13y ago

GBiT 13y ago
Congrats on using bitorrent for syncing files. You need at least RAID1 setup to don't lose information. Never use only one hdd.
robmil 13y ago
I was hoping this was going to be something where the files themselves were distributed so that they didn't take up space on everyone's machines, a la MogileFS[1], but alas. This is always a point of pain for us: in our studios we work from a central file server, so our shiny iMacs with their 1TB drives all have about 90% capacity on them — but we don't have any easy way to pool all that extra capacity and use it even as overflow for our central fileserver. The search continues... [1]: https://code.google.com/p/mogilefs/ https://code.google.com/p/mogilefs/
- icebraining 13y ago
  That's what git-annex is for. Everyone has a list of all the files, and then if you want to access one, you can just do "git annex get [file]" and it'll get it from any of the machines that happen to have that file. If you want to delete it, "git annex drop [file]" will ensure at least N machines keep a copy of the file before deleting it. I use it to manage files between my VPS, desktop, laptop and NAS, with each machine having a different subset of the whole pool, and it works great.
  - robmil 13y ago
    Is there any way to get around the fact that adding a new machine involves adding N*2 remotes? (That is, one remote on each of the current machines to represent the new one; and then on the new machine, one remote for each existing machine). This seems like it would be the sticking point with Git annex.
    - icebraining 13y ago
      That's what the new Assistant (now part of git-annex) is for! Tons of pairing work, which culminated today in pairing fully working for the very first time. And it works great! Type something like "my hovercraft is full of eels" into two git annex webapps on the same LAN and the two will find each other, automatically set up ssh keys, and sync up, like magic. Magic based on math. http://www.kickstarter.com/projects/joeyh/git-annex-assistant-like-dropbox-but-with-your-own/ http://www.kickstarter.com/projects/joeyh/git-annex-assistan... http://git-annex.branchable.com/assistant/local_pairing_walkthrough/ http://git-annex.branchable.com/assistant/local_pairing_walk...
skimmas 13y ago
IMHO btSync is a great ideia but the use of random ports makes hard to use in some setups.
- babas 13y ago
  You can set the port in the configuration: http://labs.bittorrent.com/experiments/sync/get-started.html#app-preferences http://labs.bittorrent.com/experiments/sync/get-started.html... The closed source nature of Bitorrent Sync is biggest drawback IMHO. But I guess FOSS clones are on their way. The bit torrent protocol in itself is very well known after all.
livnev 13y ago
Note that the author seems to misunderstand a few things about BitTorrent sync: 1. Firstly, Btsync is not open-source (unfortunately), as he claims. [1] 2. 24-hour secrets don't give someone read access for 24 hours. They must be redeemed within 24 hours after creation, but if they are redeemed on time they become permanent read-only secrets. [1] - Answered in these threads: http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-source/ http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-... http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/ http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-...
- reycharles 13y ago
  I'm curious why this is being downvoted. Is the information incorrect?
- sgottit 13y ago
  Ah my bad – I've corrected the open source claim and I'll reword about the 24 hr secrets. Thanks.
peterejhamilton 13y ago
How does this handle a situation where multiple designers edit large files at roughly the same time?
- freijus 13y ago
  I also always wondered if there was some version control system for pictures.
  - icebraining 13y ago
    Github supports some basic features: https://github.com/blog/817-behold-image-view-modes https://github.com/blog/817-behold-image-view-modes
  - digitalengineer 13y ago
    There are a few, one by adobe that comes free with the Creative Suite. Some are integrated in Photoshop with a clever UI: http://sixrevisions.com/project-management/the-ultimate-guide-to-version-control-for-designers/ http://sixrevisions.com/project-management/the-ultimate-guid...
tuananh 13y ago
" not to mention it means everyone who wants access needs to commit to losing 1TB of space on their hard drive." Dropbox has selective sync for years already, doesn't it?
- dagw 13y ago
  Yup, but for some reason most people don't seem to know about it. They could set up a shared top level project directory with sub directories for each aspect of the project and then people only need to sync the stuff relevant to them.
rodolphoarruda 13y ago
OP mentions this app "Arq" for AWS S3/Glacier backups. Does anyone know a similar app for Linux?
- deleted 13y ago
  [deleted]
- dschep 13y ago
  Duplicity[0] supports it through a boto[1] backend. [0] http://duplicity.nongnu.org http://duplicity.nongnu.org [1] https://github.com/boto/boto https://github.com/boto/boto
- bryanlarsen 13y ago
  dschep is dead for some reason, so copying & pasting his answer so all can see: Duplicity[0] supports it through a boto[1] backend. [0] http://duplicity.nongnu.org http://duplicity.nongnu.org [1] https://github.com/boto/boto https://github.com/boto/boto
- adamnemecek 13y ago
  Depending on what functionality you need, one alternative could be Tarsnap (http://www.tarsnap.com/ http://www.tarsnap.com/) by HN's own cperciva https://news.ycombinator.com/user?id=cperciva https://news.ycombinator.com/user?id=cperciva.
terhechte 13y ago
There's git-annex for git with big files: http://git-annex.branchable.com/ http://git-annex.branchable.com/ "git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, time, or disk space." and "The git-annex assistant creates a folder on each of your computers, Android devices, removable drives, and cloud services, which it keeps synchronised, so its contents are the same everywhere. It's very easy to use, and has all the power of git and git-annex."
shacharz 13y ago
If you want to transfer a big file easily without the need to upload it to a server http://www.sharefest.me http://www.sharefest.me does the trick. bit-torrent like service but instant and without any client.
- StavrosK 13y ago
  That looks pretty slick. There was a static HTML version I could host on my site, do you know if this does that? I don't want my files going through any third parties.
  - shacharz 13y ago
    Not sure what you mean there, but if you want to share files you can use this platform. If you want to hose your own version of sharefest the open source is in github.com/peer5/sharefest
    - StavrosK 13y ago
      Thanks, I meant I didn't want to have to run the node.js server, but I see it's only used for STUN/negotiation. Thanks for your reply.

63 points by sgottit 13y ago

GBiT 13y ago

Congrats on using bitorrent for syncing files. You need at least RAID1 setup to don't lose information. Never use only one hdd.

robmil 13y ago

I was hoping this was going to be something where the files themselves were distributed so that they didn't take up space on everyone's machines, a la MogileFS[1], but alas. This is always a point of pain for us: in our studios we work from a central file server, so our shiny iMacs with their 1TB drives all have about 90% capacity on them — but we don't have any easy way to pool all that extra capacity and use it even as overflow for our central fileserver. The search continues... [1]: https://code.google.com/p/mogilefs/ https://code.google.com/p/mogilefs/

icebraining 13y ago
That's what git-annex is for. Everyone has a list of all the files, and then if you want to access one, you can just do "git annex get [file]" and it'll get it from any of the machines that happen to have that file. If you want to delete it, "git annex drop [file]" will ensure at least N machines keep a copy of the file before deleting it. I use it to manage files between my VPS, desktop, laptop and NAS, with each machine having a different subset of the whole pool, and it works great.
- robmil 13y ago
  Is there any way to get around the fact that adding a new machine involves adding N*2 remotes? (That is, one remote on each of the current machines to represent the new one; and then on the new machine, one remote for each existing machine). This seems like it would be the sticking point with Git annex.
  - icebraining 13y ago
    That's what the new Assistant (now part of git-annex) is for! Tons of pairing work, which culminated today in pairing fully working for the very first time. And it works great! Type something like "my hovercraft is full of eels" into two git annex webapps on the same LAN and the two will find each other, automatically set up ssh keys, and sync up, like magic. Magic based on math. http://www.kickstarter.com/projects/joeyh/git-annex-assistant-like-dropbox-but-with-your-own/ http://www.kickstarter.com/projects/joeyh/git-annex-assistan... http://git-annex.branchable.com/assistant/local_pairing_walkthrough/ http://git-annex.branchable.com/assistant/local_pairing_walk...

skimmas 13y ago

IMHO btSync is a great ideia but the use of random ports makes hard to use in some setups.

babas 13y ago
You can set the port in the configuration: http://labs.bittorrent.com/experiments/sync/get-started.html#app-preferences http://labs.bittorrent.com/experiments/sync/get-started.html... The closed source nature of Bitorrent Sync is biggest drawback IMHO. But I guess FOSS clones are on their way. The bit torrent protocol in itself is very well known after all.

livnev 13y ago

Note that the author seems to misunderstand a few things about BitTorrent sync: 1. Firstly, Btsync is not open-source (unfortunately), as he claims. [1] 2. 24-hour secrets don't give someone read access for 24 hours. They must be redeemed within 24 hours after creation, but if they are redeemed on time they become permanent read-only secrets. [1] - Answered in these threads: http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-source/ http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-... http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/ http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-...

reycharles 13y ago
I'm curious why this is being downvoted. Is the information incorrect?
sgottit 13y ago
Ah my bad – I've corrected the open source claim and I'll reword about the 24 hr secrets. Thanks.

peterejhamilton 13y ago

How does this handle a situation where multiple designers edit large files at roughly the same time?

freijus 13y ago
I also always wondered if there was some version control system for pictures.
- icebraining 13y ago
  Github supports some basic features: https://github.com/blog/817-behold-image-view-modes https://github.com/blog/817-behold-image-view-modes
- digitalengineer 13y ago
  There are a few, one by adobe that comes free with the Creative Suite. Some are integrated in Photoshop with a clever UI: http://sixrevisions.com/project-management/the-ultimate-guide-to-version-control-for-designers/ http://sixrevisions.com/project-management/the-ultimate-guid...

tuananh 13y ago

" not to mention it means everyone who wants access needs to commit to losing 1TB of space on their hard drive." Dropbox has selective sync for years already, doesn't it?

dagw 13y ago
Yup, but for some reason most people don't seem to know about it. They could set up a shared top level project directory with sub directories for each aspect of the project and then people only need to sync the stuff relevant to them.

rodolphoarruda 13y ago

OP mentions this app "Arq" for AWS S3/Glacier backups. Does anyone know a similar app for Linux?

deleted 13y ago
[deleted]
dschep 13y ago
Duplicity[0] supports it through a boto[1] backend. [0] http://duplicity.nongnu.org http://duplicity.nongnu.org [1] https://github.com/boto/boto https://github.com/boto/boto
bryanlarsen 13y ago
dschep is dead for some reason, so copying & pasting his answer so all can see: Duplicity[0] supports it through a boto[1] backend. [0] http://duplicity.nongnu.org http://duplicity.nongnu.org [1] https://github.com/boto/boto https://github.com/boto/boto
adamnemecek 13y ago
Depending on what functionality you need, one alternative could be Tarsnap (http://www.tarsnap.com/ http://www.tarsnap.com/) by HN's own cperciva https://news.ycombinator.com/user?id=cperciva https://news.ycombinator.com/user?id=cperciva.

terhechte 13y ago

There's git-annex for git with big files: http://git-annex.branchable.com/ http://git-annex.branchable.com/ "git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, time, or disk space." and "The git-annex assistant creates a folder on each of your computers, Android devices, removable drives, and cloud services, which it keeps synchronised, so its contents are the same everywhere. It's very easy to use, and has all the power of git and git-annex."

shacharz 13y ago

If you want to transfer a big file easily without the need to upload it to a server http://www.sharefest.me http://www.sharefest.me does the trick. bit-torrent like service but instant and without any client.

StavrosK 13y ago
That looks pretty slick. There was a static HTML version I could host on my site, do you know if this does that? I don't want my files going through any third parties.
- shacharz 13y ago
  Not sure what you mean there, but if you want to share files you can use this platform. If you want to hose your own version of sharefest the open source is in github.com/peer5/sharefest
  - StavrosK 13y ago
    Thanks, I meant I didn't want to have to run the node.js server, but I see it's only used for STUN/negotiation. Thanks for your reply.