5 ms·

Testing Sync at Dropbox

180 points by rbtying 6y ago

sujayakar 6y ago
hey all, author of the previous post (https://news.ycombinator.com/item?id=22595782 https://news.ycombinator.com/item?id=22595782) here, and I'm happy to answer any questions about the system! I think the author isaac is here too.
- 0xy 6y ago
  What are some of the trickiest edge cases that you try to account for? I can only assume that this kind of system is enormously complicated, given the high expectations from users and the complexity of the underlying environments.
  - sujayakar 6y ago
    here's some of the greatest hits from memory: 1) when we upload a file, we first ask the server which blocks we need to upload. this allows us to deduplicate blocks with other files or previous revisions of files you have. I had written some code like... let blocks_needed: HashSet<BlockHash> = ...; let file_blocks: Vec<BlockHash> = ...; let to_upload: Vec<_> = file_blocks .into_iter() .filter(|b| blocks_needed.contains(b)) .enumerate(); for (offset, block_hash) in to_upload { ... } the bug here is that we're computing the offset into the file after filtering, so we'd be uploading incorrect contents to the server. we have protections elsewhere that would prevent file corruption, but finding this bug pre-commit with trinity was pretty awesome. 2) nontrivial interactions between different components in the system are hard to test manually and come for free with trinity. trinity will simulate cancelling file transfers, crashing the system, and even dropping the local databases (to simulate disk corruption). these are all hard enough to test in isolation, but knowing how to combine them is almost impossible. so, having "automated test generation" for these cases is really useful. 3) the "theory of trees" seems like it'd be pretty simple, but canopycheck has found some really interesting cases, especially around moves. when we started, we hadn't really thought about how to handle move cycles, concurrent moves that cross ancestor chains, ...
- dralley 6y ago
  No question, just wanted to say that these blog posts are fantastic! Thank you for writing them.
- OskarS 6y ago
  How smart is the sync engine's "merge" system? Can it cleverly merge files, or if it sees a single file that has diverged, does it just create a conflict copy and let the user sort it out? Like, if both Alice and Bob has changed somefile.txt, but Alice has added a few lines in the beginning, and Bob a few lines at the end (i.e. something that would "git merge" cleanly), will the sync engine merge that into one file? I'm very curious about what thoughts you guys had on this issue, because I can see arguments for both sides.
  - catmanjan 6y ago
    I don't think Dropbox does merge at all does it? The file object model wouldn't work for merging Word documents and such
  - sujayakar 6y ago
    yeah, we don't merge file contents. it's really hard to do this well, since you'd need to implement something like operational transforms for every different file type, and the cost of getting wrong is really high, since we'd be corrupting users' files. I can see us perhaps doing this in the future for more targeted file types where the "rebase" behavior is well-defined, though.
- tasubotadas 6y ago
  Whenever I start Dropbox (boot my laptop) it takes ~30min of maxed out single-core work to "catch-up" since the "last-update". It's just really annoying as it makes my laptop quite loud. Why is that? Google Drive doesn't seem to need to struggle that much.
  - bluedino 6y ago
    How much data (total size/count of files) do you have?
    - tasubotadas 6y ago
      According to windows: 371790 files and 15.6GB.
  - dmd 6y ago
    So... that’s weird. I have a Macbook Air - not the most powerful machine on the planet - and a nearly full 1 TB dropbox with ~3 million files. Initial resync on boot takes under a minute.
  - sujayakar 6y ago
    hey, can you file a bug from the app? it will collect a report that we can use to diagnose. there's a "Report Bug" option adjacent to "Preferences..."
    - tasubotadas 6y ago
      I'll do that.
      - sujayakar 6y ago
        if you can email me (my HN username @dropbox.com) the email address associated with your account once you've sent the bug, I can take a look!
- zmj 6y ago
  Thanks for sharing! I've worked on similar applications - it's always cool to see other successful architectures.
- bmichel 6y ago
  Thanks for sharing your experience on this rewrite. I have a few questions: 1. I'm a bit surprised that you don't persist the scenarios of the tests after a failure, only its seed. Does it mean that when you want to replay it, you have to redo the minimization phase? Or, do you have a way to find a seed for the minimized scenario? 2. Do you have some tests where you generate a set of operations and play them twice: one time with the mocks and one time on the real servers to check that they have the same results? 3. The article says "Note also the importance of the commit hash, as another type of “test input” alongside the seed: if the code changes, the course of execution may change too!". How are you ensuring that a commit really fixes a bug, and not just change the execution path to a happy path where the conditions of the bug are not met? By playing again a lot of tests, or do you write a new unit test that exhibit the bug to ensure the reproductability? 4. Do you think we can say that CanopyCheck is applying randomized testing at the unit tests level and Trinity is applying it at the integration tests level?
  - rbtying 6y ago
    I'm not Sujay, but I worked on this system once upon a time too! (not anymore) 1. It does redo the minimization phase, but the actual execution is extremely fast, so this cost is minimal. Storing test outputs gets pretty expensive when you are running millions of tests, and since there are very few failures, recomputing this is worthwhile 2. Yes and no! the article talks about this a little, but the "heirloom" system does essentially this, and the "native" filesystem variant of Trinity runs the same Trinity tester code against a real filesystem. The "no" is due to the issues with randomized testing -- since any operation that you do can affect the RNG, the exact operation that is run for a particular seed can change if you swap any part of the system. For regression tests, the operation sequence can be put into a separate, non-randomized test. 3. both 4. Testing in Nucleus is a sliding scale from "unit-test-like" to "integration-test-like" -- Trinity is mocking plenty of functionality; CanopyCheck is simultaneously testing many different components. It would probably be more accurate to say that CanopyCheck is testing a smaller subset of components with much greater fidelity, and Trinity is testing as much of the sync engine is practical.
    - bmichel 6y ago
rakoo 6y ago
Interesting, I learned this technique of having a third, synchronization intermediary with offlineimap and this helpful blog post: http://blog.ezyang.com/2012/08/how-offlineimap-works/ http://blog.ezyang.com/2012/08/how-offlineimap-works/
- Xoltus 6y ago
  Thanks for sharing. This is indeed a very helpful blog post
saagarjha 6y ago
> To cover this layer of our codebase, we also run Trinity in a “native” mode, targeting the platform’s actual filesystem. However, running against the native filesystem incurs a huge performance penalty (roughly 10x), which in turn means Trinity Native can’t test as many different seeds. What platforms do you test in “native” mode? What hardware backs it?
- sujayakar 6y ago
  we just use our regular CI infrastructure for running linux, macOS, and windows. we have infrastructure for managing VMs for our different supported platforms, setting up filesystems, and so on. here's a talk from one of our engineers on our macOS CI infrastructure: https://blog.macstadium.com/blog/virtualizing-mac-infrastructure-at-scale-a-ci-case-study-with-dropbox https://blog.macstadium.com/blog/virtualizing-mac-infrastruc...
amq 6y ago
I'm sure Dropbox is first-class behind the scenes, but the desktop app has become simply awful with time, so that I almost feel sorry for them and embarrassed towards other people when I use Dropbox.
hilmart 6y ago
I find the capitalization of the title here on HN to be rather misleading. I thought Dropbox was actually testing Sync (https://www.sync.com/ https://www.sync.com/) in-house and doing a public comparison.

180 points by rbtying 6y ago

hey all, author of the previous post (https://news.ycombinator.com/item?id=22595782 https://news.ycombinator.com/item?id=22595782) here, and I'm happy to answer any questions about the system! I think the author isaac is here too.

0xy 6y ago
What are some of the trickiest edge cases that you try to account for? I can only assume that this kind of system is enormously complicated, given the high expectations from users and the complexity of the underlying environments.
- sujayakar 6y ago
  here's some of the greatest hits from memory: 1) when we upload a file, we first ask the server which blocks we need to upload. this allows us to deduplicate blocks with other files or previous revisions of files you have. I had written some code like... let blocks_needed: HashSet<BlockHash> = ...; let file_blocks: Vec<BlockHash> = ...; let to_upload: Vec<_> = file_blocks .into_iter() .filter(|b| blocks_needed.contains(b)) .enumerate(); for (offset, block_hash) in to_upload { ... } the bug here is that we're computing the offset into the file after filtering, so we'd be uploading incorrect contents to the server. we have protections elsewhere that would prevent file corruption, but finding this bug pre-commit with trinity was pretty awesome. 2) nontrivial interactions between different components in the system are hard to test manually and come for free with trinity. trinity will simulate cancelling file transfers, crashing the system, and even dropping the local databases (to simulate disk corruption). these are all hard enough to test in isolation, but knowing how to combine them is almost impossible. so, having "automated test generation" for these cases is really useful. 3) the "theory of trees" seems like it'd be pretty simple, but canopycheck has found some really interesting cases, especially around moves. when we started, we hadn't really thought about how to handle move cycles, concurrent moves that cross ancestor chains, ...
dralley 6y ago
No question, just wanted to say that these blog posts are fantastic! Thank you for writing them.
OskarS 6y ago
How smart is the sync engine's "merge" system? Can it cleverly merge files, or if it sees a single file that has diverged, does it just create a conflict copy and let the user sort it out? Like, if both Alice and Bob has changed somefile.txt, but Alice has added a few lines in the beginning, and Bob a few lines at the end (i.e. something that would "git merge" cleanly), will the sync engine merge that into one file? I'm very curious about what thoughts you guys had on this issue, because I can see arguments for both sides.
- catmanjan 6y ago
  I don't think Dropbox does merge at all does it? The file object model wouldn't work for merging Word documents and such
- sujayakar 6y ago
  yeah, we don't merge file contents. it's really hard to do this well, since you'd need to implement something like operational transforms for every different file type, and the cost of getting wrong is really high, since we'd be corrupting users' files. I can see us perhaps doing this in the future for more targeted file types where the "rebase" behavior is well-defined, though.
tasubotadas 6y ago
Whenever I start Dropbox (boot my laptop) it takes ~30min of maxed out single-core work to "catch-up" since the "last-update". It's just really annoying as it makes my laptop quite loud. Why is that? Google Drive doesn't seem to need to struggle that much.
- bluedino 6y ago
  How much data (total size/count of files) do you have?
  - tasubotadas 6y ago
    According to windows: 371790 files and 15.6GB.
- dmd 6y ago
  So... that’s weird. I have a Macbook Air - not the most powerful machine on the planet - and a nearly full 1 TB dropbox with ~3 million files. Initial resync on boot takes under a minute.
- sujayakar 6y ago
  hey, can you file a bug from the app? it will collect a report that we can use to diagnose. there's a "Report Bug" option adjacent to "Preferences..."
  - tasubotadas 6y ago
    I'll do that.
    - sujayakar 6y ago
      if you can email me (my HN username @dropbox.com) the email address associated with your account once you've sent the bug, I can take a look!
zmj 6y ago
Thanks for sharing! I've worked on similar applications - it's always cool to see other successful architectures.
bmichel 6y ago
Thanks for sharing your experience on this rewrite. I have a few questions: 1. I'm a bit surprised that you don't persist the scenarios of the tests after a failure, only its seed. Does it mean that when you want to replay it, you have to redo the minimization phase? Or, do you have a way to find a seed for the minimized scenario? 2. Do you have some tests where you generate a set of operations and play them twice: one time with the mocks and one time on the real servers to check that they have the same results? 3. The article says "Note also the importance of the commit hash, as another type of “test input” alongside the seed: if the code changes, the course of execution may change too!". How are you ensuring that a commit really fixes a bug, and not just change the execution path to a happy path where the conditions of the bug are not met? By playing again a lot of tests, or do you write a new unit test that exhibit the bug to ensure the reproductability? 4. Do you think we can say that CanopyCheck is applying randomized testing at the unit tests level and Trinity is applying it at the integration tests level?
- rbtying 6y ago
  I'm not Sujay, but I worked on this system once upon a time too! (not anymore) 1. It does redo the minimization phase, but the actual execution is extremely fast, so this cost is minimal. Storing test outputs gets pretty expensive when you are running millions of tests, and since there are very few failures, recomputing this is worthwhile 2. Yes and no! the article talks about this a little, but the "heirloom" system does essentially this, and the "native" filesystem variant of Trinity runs the same Trinity tester code against a real filesystem. The "no" is due to the issues with randomized testing -- since any operation that you do can affect the RNG, the exact operation that is run for a particular seed can change if you swap any part of the system. For regression tests, the operation sequence can be put into a separate, non-randomized test. 3. both 4. Testing in Nucleus is a sliding scale from "unit-test-like" to "integration-test-like" -- Trinity is mocking plenty of functionality; CanopyCheck is simultaneously testing many different components. It would probably be more accurate to say that CanopyCheck is testing a smaller subset of components with much greater fidelity, and Trinity is testing as much of the sync engine is practical.
  - bmichel 6y ago

rakoo 6y ago

Interesting, I learned this technique of having a third, synchronization intermediary with offlineimap and this helpful blog post: http://blog.ezyang.com/2012/08/how-offlineimap-works/ http://blog.ezyang.com/2012/08/how-offlineimap-works/

Xoltus 6y ago
Thanks for sharing. This is indeed a very helpful blog post

saagarjha 6y ago

> To cover this layer of our codebase, we also run Trinity in a “native” mode, targeting the platform’s actual filesystem. However, running against the native filesystem incurs a huge performance penalty (roughly 10x), which in turn means Trinity Native can’t test as many different seeds. What platforms do you test in “native” mode? What hardware backs it?

sujayakar 6y ago
we just use our regular CI infrastructure for running linux, macOS, and windows. we have infrastructure for managing VMs for our different supported platforms, setting up filesystems, and so on. here's a talk from one of our engineers on our macOS CI infrastructure: https://blog.macstadium.com/blog/virtualizing-mac-infrastructure-at-scale-a-ci-case-study-with-dropbox https://blog.macstadium.com/blog/virtualizing-mac-infrastruc...

amq 6y ago

I'm sure Dropbox is first-class behind the scenes, but the desktop app has become simply awful with time, so that I almost feel sorry for them and embarrassed towards other people when I use Dropbox.

hilmart 6y ago

I find the capitalization of the title here on HN to be rather misleading. I thought Dropbox was actually testing Sync (https://www.sync.com/ https://www.sync.com/) in-house and doing a public comparison.