5 ms·

Testing sync at Dropbox (2020)

68 points by sh_tomer 2y ago

sujayakar 2y ago
former area tech lead for dropbox (and frequent collaborator with isaac) here! happy to answer any questions.
- brcmthrowaway 2y ago
  Why does Dropbox have horrendous dark patterns that should be illegal?
  - wk_end 2y ago
    Not OP, and I'm not sure specifically about which dark patterns you're referring to, but I can take a stab: It's because Dropbox doesn't have a good business model (anymore). Microsoft, Apple, Google, etc. are happy enough to give storage away for free, or almost, and their solutions come pre-installed. For the average user, it'd be an uphill battle to get people to install Dropbox even if it were comparable price-wise to OneDrive, iCloud, Google Drive - and it's not, because Dropbox actually needs to make money off cloud storage and these companies don't. It'd need to be leagues better than those other products (think Firefox overtaking IE, for a time) - and it's not. I think its sync is quite good, but I don't know if it's so much better than the alternatives that it'd persuade significant numbers to switch. And, as others in this thread have pointed out, its desktop client kind of sucks. Moreover, "files" themselves are becoming an antiquated concept. They're definitely not dead yet, but they're almost a power user thing at this point. Documents are stored on the cloud, embedded inside the specific web app (Google Docs, Notion, Figma...) you used to create it. It sucks, but that's the direction we're heading. File sync is becoming less-and-less essential, to fewer-and-fewer users. Dropbox is a public company: they can't just shrug their shoulders, tend to their core user base, and iterate on their product. That's not going to produce the kind of growth they need to keep investors happy; you need a good business model to produce that kind of growth organically. Hence you need to produce it inorganically: dark patterns.
  - zinodaur 2y ago
    Do you have some examples? I'm curious
eslaught 2y ago
> decoupling the global PRNG into several independent ones Given this is 2020, I'm guessing you probably already solved this one, but this is something I had to do for a concurrent fuzzer I wrote recently. Basically my solution is to make SipHash the PRNG. The way this works is you make a tuple of your seed and any other information you want to hash (at minimum, a "channel" number, so you get different streams of random numbers, and a sequence number). You hash the tuple and increment the sequence number and return. Because it's literally just a hash function, it's easy to reason about its properties: same input in, same output out. As long as the hash function is high enough quality, you avoid bias in the output even when the input has low entropy (thus why I chose SipHash). There's an Apache-licensed implementation here for anyone who cares. My implementation has streams inside each channel associated with hashable objects, just to make it easier to subdivide the channels. https://github.com/StanfordLegion/fuzzer/blob/master/src/deterministic_random.h https://github.com/StanfordLegion/fuzzer/blob/master/src/det... https://github.com/StanfordLegion/fuzzer/blob/master/src/deterministic_random.cc https://github.com/StanfordLegion/fuzzer/blob/master/src/det... https://github.com/StanfordLegion/fuzzer/blob/master/src/deterministic_random.inl https://github.com/StanfordLegion/fuzzer/blob/master/src/det...
- AlotOfReading 2y ago
  Hash based PRNGs can be surprisingly tricky to reason about. For example, the other common way to turn hash functions into PRNGs (state = hash{state}) often leads to severe flaws from state space collapse, among other issues.
  - rendaw 2y ago
    Do you know anywhere I could read more about this?
    - AlotOfReading 2y ago
      For the mathematical background (i.e. the theory and behavior in idealized models), standard cryptography books are excellent. I recall that both Serious Cryptography and Handbook of Applied Cryptography have excellent introductions to the topic of rho structures that underlie the specific issue I mentioned. Where I've found them lacking is getting a practical understanding of how important these issues are in the specific situations we see in the real world. For example, if you were to run Blake2b in 8 bit mode with the above construction, what would your PRNG period actually be? In the ideal case it would 2^(8). In practice, the best you'll get is 26 or 10% of the ideal case. There's a significant probability you won't get that either for a given seed, as much of the state space actually drops into a lower period cycle. The issue is that a real hash isn't the correct construction here, even though a mathematically ideal hash would be fine. I'm not aware of any actual reference material for that kind of stuff and the people I've met who are knowledgeable learned from seeing other's mistakes.
bentt 2y ago
Just Dropped Dropbox this month after many years. Their quest to grow and offer more and more unwanted services has finally made their core advantage of easy sync, share, and backup not worth the cost, complexity, and user-unfriendliness. Google Drive is now Fine.
antman 2y ago
I was a customer, at some point they activated multi factor auth using an old email and I lost all my files
blutfink 2y ago
(2020)*
- dang 2y ago
  Added above. Thanks!
simon_acca 2y ago
Seems like a great use case for https://antithesis.com https://antithesis.com
- sujayakar 2y ago
  yeah! we were talking to the antithesis folks during this project, but it wasn't far enough along when we started the project in 2016. we were heavily influenced by the original foundationdb testing talk from strange loop (wwilson, one of the antithesis founders): https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc
jtriangle 2y ago
I was an early adopter at dropbox, got all my friends/family/etc on it. Maxed out the free affiliate space when I added my .edu email to it for a grand total of 16.125GB, technically over the max, but, whatever I guess. I eventually got into a 'make decisions' IT roll, and, Dropbox was an easy choice, did everything we needed it to, price was fair, and we used it for years. Everyone had access to the webUI that needed it, and, everyone else had access to an SMB share where dropbox actually ran. All was good until suddenly nothing would sync. Tech support was useless for about a month until I finally got a call with some kind of higher level tech, who told me that what we were doing wouldn't work, and never should have worked. Mind you, we'd been doing this for years, and stuck with dropbox because of it. So I dropped them like a bad habit and never looked back. Our portal system had half-ass but usable remote file sharing, so, we used that while we rolled our own system. Was a rocky 6 months, but, now, we have extremely tight, easy, automated, simple to manage file sharing. So really I should thank them for the impetus, and I would if they weren't so abundantly unhelpful. Personally, well, my personal account continued to slowly lose features over time, and it finally got to the point where the messaging from them was "Pay or Leave", so, I left and did my own thing, which is a combo of Syncthing and Nextcloud that works extremely well, and I own 100% of my data. So dropbox might have some glitz, but I wouldn't touch it with a 10ft poll, they're too big for their britches, and have been for awhile, and worse, they seem to want you to believe that file synchronization is hard, cloud storage is hard, and file sharing is hard. None of that is true anymore, so what you're buying is a name and a logo with them with a free side of smoke blown up your ass. No thanks, I'm allergic.
- sureIy 2y ago
  Could you explain the setup? Who would access the SMB and who’d access Dropbox’ web interface? If it’s running on a server with a network-shared folder, I don’t see how that’s an issue for Dropbox. If you were passing around Dropbox credentials for a single user, that’s definitely a breach of ToS.
  - jtriangle 2y ago
    Had a business account, multiple users in charge of their little zones. Worker bees do work, drop it in a dropbox folder as they go. When they're finished, they tell the person in charge of their zone it's ready, and that person ships it off to the client. Nothing was TOS breaking there, just that it couldn't detect changes to the folders, so, it would never sync. Triggering a manual re-sync would just hang indefinitely and never finish.
dang 2y ago
Discussed at the time: Testing Sync at Dropbox - https://news.ycombinator.com/item?id=22928726 https://news.ycombinator.com/item?id=22928726 - April 2020 (25 comments)

68 points by sh_tomer 2y ago

sujayakar 2y ago

former area tech lead for dropbox (and frequent collaborator with isaac) here! happy to answer any questions.

brcmthrowaway 2y ago
Why does Dropbox have horrendous dark patterns that should be illegal?
- wk_end 2y ago
  Not OP, and I'm not sure specifically about which dark patterns you're referring to, but I can take a stab: It's because Dropbox doesn't have a good business model (anymore). Microsoft, Apple, Google, etc. are happy enough to give storage away for free, or almost, and their solutions come pre-installed. For the average user, it'd be an uphill battle to get people to install Dropbox even if it were comparable price-wise to OneDrive, iCloud, Google Drive - and it's not, because Dropbox actually needs to make money off cloud storage and these companies don't. It'd need to be leagues better than those other products (think Firefox overtaking IE, for a time) - and it's not. I think its sync is quite good, but I don't know if it's so much better than the alternatives that it'd persuade significant numbers to switch. And, as others in this thread have pointed out, its desktop client kind of sucks. Moreover, "files" themselves are becoming an antiquated concept. They're definitely not dead yet, but they're almost a power user thing at this point. Documents are stored on the cloud, embedded inside the specific web app (Google Docs, Notion, Figma...) you used to create it. It sucks, but that's the direction we're heading. File sync is becoming less-and-less essential, to fewer-and-fewer users. Dropbox is a public company: they can't just shrug their shoulders, tend to their core user base, and iterate on their product. That's not going to produce the kind of growth they need to keep investors happy; you need a good business model to produce that kind of growth organically. Hence you need to produce it inorganically: dark patterns.
- zinodaur 2y ago
  Do you have some examples? I'm curious

eslaught 2y ago

> decoupling the global PRNG into several independent ones Given this is 2020, I'm guessing you probably already solved this one, but this is something I had to do for a concurrent fuzzer I wrote recently. Basically my solution is to make SipHash the PRNG. The way this works is you make a tuple of your seed and any other information you want to hash (at minimum, a "channel" number, so you get different streams of random numbers, and a sequence number). You hash the tuple and increment the sequence number and return. Because it's literally just a hash function, it's easy to reason about its properties: same input in, same output out. As long as the hash function is high enough quality, you avoid bias in the output even when the input has low entropy (thus why I chose SipHash). There's an Apache-licensed implementation here for anyone who cares. My implementation has streams inside each channel associated with hashable objects, just to make it easier to subdivide the channels. https://github.com/StanfordLegion/fuzzer/blob/master/src/deterministic_random.h https://github.com/StanfordLegion/fuzzer/blob/master/src/det... https://github.com/StanfordLegion/fuzzer/blob/master/src/deterministic_random.cc https://github.com/StanfordLegion/fuzzer/blob/master/src/det... https://github.com/StanfordLegion/fuzzer/blob/master/src/deterministic_random.inl https://github.com/StanfordLegion/fuzzer/blob/master/src/det...

AlotOfReading 2y ago
Hash based PRNGs can be surprisingly tricky to reason about. For example, the other common way to turn hash functions into PRNGs (state = hash{state}) often leads to severe flaws from state space collapse, among other issues.
- rendaw 2y ago
  Do you know anywhere I could read more about this?
  - AlotOfReading 2y ago
    For the mathematical background (i.e. the theory and behavior in idealized models), standard cryptography books are excellent. I recall that both Serious Cryptography and Handbook of Applied Cryptography have excellent introductions to the topic of rho structures that underlie the specific issue I mentioned. Where I've found them lacking is getting a practical understanding of how important these issues are in the specific situations we see in the real world. For example, if you were to run Blake2b in 8 bit mode with the above construction, what would your PRNG period actually be? In the ideal case it would 2^(8). In practice, the best you'll get is 26 or 10% of the ideal case. There's a significant probability you won't get that either for a given seed, as much of the state space actually drops into a lower period cycle. The issue is that a real hash isn't the correct construction here, even though a mathematically ideal hash would be fine. I'm not aware of any actual reference material for that kind of stuff and the people I've met who are knowledgeable learned from seeing other's mistakes.

bentt 2y ago

Just Dropped Dropbox this month after many years. Their quest to grow and offer more and more unwanted services has finally made their core advantage of easy sync, share, and backup not worth the cost, complexity, and user-unfriendliness. Google Drive is now Fine.

antman 2y ago

I was a customer, at some point they activated multi factor auth using an old email and I lost all my files

blutfink 2y ago

(2020)*

dang 2y ago
Added above. Thanks!

simon_acca 2y ago

Seems like a great use case for https://antithesis.com https://antithesis.com

sujayakar 2y ago
yeah! we were talking to the antithesis folks during this project, but it wasn't far enough along when we started the project in 2016. we were heavily influenced by the original foundationdb testing talk from strange loop (wwilson, one of the antithesis founders): https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc

jtriangle 2y ago

I was an early adopter at dropbox, got all my friends/family/etc on it. Maxed out the free affiliate space when I added my .edu email to it for a grand total of 16.125GB, technically over the max, but, whatever I guess. I eventually got into a 'make decisions' IT roll, and, Dropbox was an easy choice, did everything we needed it to, price was fair, and we used it for years. Everyone had access to the webUI that needed it, and, everyone else had access to an SMB share where dropbox actually ran. All was good until suddenly nothing would sync. Tech support was useless for about a month until I finally got a call with some kind of higher level tech, who told me that what we were doing wouldn't work, and never should have worked. Mind you, we'd been doing this for years, and stuck with dropbox because of it. So I dropped them like a bad habit and never looked back. Our portal system had half-ass but usable remote file sharing, so, we used that while we rolled our own system. Was a rocky 6 months, but, now, we have extremely tight, easy, automated, simple to manage file sharing. So really I should thank them for the impetus, and I would if they weren't so abundantly unhelpful. Personally, well, my personal account continued to slowly lose features over time, and it finally got to the point where the messaging from them was "Pay or Leave", so, I left and did my own thing, which is a combo of Syncthing and Nextcloud that works extremely well, and I own 100% of my data. So dropbox might have some glitz, but I wouldn't touch it with a 10ft poll, they're too big for their britches, and have been for awhile, and worse, they seem to want you to believe that file synchronization is hard, cloud storage is hard, and file sharing is hard. None of that is true anymore, so what you're buying is a name and a logo with them with a free side of smoke blown up your ass. No thanks, I'm allergic.

sureIy 2y ago
Could you explain the setup? Who would access the SMB and who’d access Dropbox’ web interface? If it’s running on a server with a network-shared folder, I don’t see how that’s an issue for Dropbox. If you were passing around Dropbox credentials for a single user, that’s definitely a breach of ToS.
- jtriangle 2y ago
  Had a business account, multiple users in charge of their little zones. Worker bees do work, drop it in a dropbox folder as they go. When they're finished, they tell the person in charge of their zone it's ready, and that person ships it off to the client. Nothing was TOS breaking there, just that it couldn't detect changes to the folders, so, it would never sync. Triggering a manual re-sync would just hang indefinitely and never finish.

dang 2y ago

Discussed at the time: Testing Sync at Dropbox - https://news.ycombinator.com/item?id=22928726 https://news.ycombinator.com/item?id=22928726 - April 2020 (25 comments)