5 ms·

Introducing streaming sync: supercharged sync for large files

27 points by lalwanivikas 12y ago

gmturbo 12y ago
wonder what's on the backend? :P
rcthompson 12y ago
The technical details of what they changed are here: https://tech.dropbox.com/2014/07/streaming-file-synchronization/ https://tech.dropbox.com/2014/07/streaming-file-synchronizat...
giovannibajo1 12y ago
Looks like it still doesn't fix the LAN scenario, where one has to wait for a server upload before the file is synced through the LAN. With this new version, it seems like the LAN sync will happen in parallel, but the file is still confirmed only after a full upload to the remote server and thus local peers don't see it appearing immediately in the shared folder.
- sp332 12y ago
  That seems like a policy decision, not a technical limitation. If Computer 1 uploads a file to Computer 2 and then Computer 2 deletes the file before it gets uploaded to the server, it's going to be difficult to sort that out. Letting the server be the final authority on a file's state is a lot simpler.
  - icebraining 12y ago
    Why? The server already knows about the file, who created it, etc. Why would they need its contents?
    - sp332 12y ago
      If the server doesn't know that Computer 2 had the file earlier, it might re-send it when the upload is complete and it tries to sync other computers. Also, what if Computer 2 modified the contents?
      - icebraining 12y ago
        If the server doesn't know that Computer 2 had the file earlier My point is that you could keep the server as the master, without having to upload the contents before pushing to other machines. Imagine this exchange: Computer 1 > server: I have a new file X, created on timestamp Y, with hash Z Server > computer 1: ack Server > computer 2: there's a new file X, etc. Computers [1] have it. Computer 1 > server: ack. I'm requesting it from computer 1. The server doesn't actually need the contents, just the metadata. By the way, this is all fairly archaic. Nowadays with DVCSs, you can do much niftier things. I have a network of five git-annex nodes, some of which have two degrees of separation (my Android tablet syncs with my personal laptop, which syncs with my VPS, which syncs with my work laptop) and it all runs smoothly.
- kasperset 12y ago
  This may be be related: https://www.dropbox.com/developers/blog/48/how-the-datastore-api-handles-conflicts-part-1-basics-of-offline-conflict-handling https://www.dropbox.com/developers/blog/48/how-the-datastore...
deleted 12y ago
[deleted]

27 points by lalwanivikas 12y ago

wonder what's on the backend? :P

The technical details of what they changed are here: https://tech.dropbox.com/2014/07/streaming-file-synchronization/ https://tech.dropbox.com/2014/07/streaming-file-synchronizat...

giovannibajo1 12y ago

Looks like it still doesn't fix the LAN scenario, where one has to wait for a server upload before the file is synced through the LAN. With this new version, it seems like the LAN sync will happen in parallel, but the file is still confirmed only after a full upload to the remote server and thus local peers don't see it appearing immediately in the shared folder.

sp332 12y ago
That seems like a policy decision, not a technical limitation. If Computer 1 uploads a file to Computer 2 and then Computer 2 deletes the file before it gets uploaded to the server, it's going to be difficult to sort that out. Letting the server be the final authority on a file's state is a lot simpler.
- icebraining 12y ago
  Why? The server already knows about the file, who created it, etc. Why would they need its contents?
  - sp332 12y ago
    If the server doesn't know that Computer 2 had the file earlier, it might re-send it when the upload is complete and it tries to sync other computers. Also, what if Computer 2 modified the contents?
    - icebraining 12y ago
      If the server doesn't know that Computer 2 had the file earlier My point is that you could keep the server as the master, without having to upload the contents before pushing to other machines. Imagine this exchange: Computer 1 > server: I have a new file X, created on timestamp Y, with hash Z Server > computer 1: ack Server > computer 2: there's a new file X, etc. Computers [1] have it. Computer 1 > server: ack. I'm requesting it from computer 1. The server doesn't actually need the contents, just the metadata. By the way, this is all fairly archaic. Nowadays with DVCSs, you can do much niftier things. I have a network of five git-annex nodes, some of which have two degrees of separation (my Android tablet syncs with my personal laptop, which syncs with my VPS, which syncs with my work laptop) and it all runs smoothly.
kasperset 12y ago
This may be be related: https://www.dropbox.com/developers/blog/48/how-the-datastore-api-handles-conflicts-part-1-basics-of-offline-conflict-handling https://www.dropbox.com/developers/blog/48/how-the-datastore...

deleted 12y ago

[deleted]