7 ms·

Rewriting the heart of our sync engine

158 points by ublaze 7y ago

zanellato19 7y ago
I really liked this article because they spoke more of their decision making process than tech. Rewrite It In Rust is a meme by this point, but the blog actually goes through every point that could be discussed in such a strategy and why it made sense for Dropbox at this moment. Really good blog post, imo.
- Recursing 7y ago
  The author (and some of his coworkers) gave a lot of interesting answers on reddit as well: https://www.reddit.com/r/rust/comments/fjt4q3/rewriting_the_heart_of_our_sync_engine_in_rust/ https://www.reddit.com/r/rust/comments/fjt4q3/rewriting_the_...
  - sujayakar 7y ago
    and on lobste.rs too :) https://lobste.rs/s/ob6a8z/rewriting_heart_our_sync_engine https://lobste.rs/s/ob6a8z/rewriting_heart_our_sync_engine
aw1621107 7y ago
A bit light on details, but it seems more are incoming. Can't wait for those blog posts!
sujayakar 7y ago
hey everyone, author here! happy to answer any questions about the post, our system, or our usage of rust :)
- quantified 7y ago
  How did the decision to use Rust go from idea to acceptance/adoption? Management can be risk-averse in choice of dev platform, and if Rust was adopted more than 18 months ago it was maybe even before the “rewrite it Rust” meme. Lessons learned and preconditions for success on this decision are especially welcomed.
  - rbtying 7y ago
    disclaimer: was on the team, but didn't write the article / don't work on this anymore Rust was adopted at Dropbox for some serving infrastructure use cases more than a year before the sync rewrite was started, which was about four years ago. I'd say we solidly predated the "rewrite it in Rust" meme. I believe that this rewrite was only successful because of Rust's ability to both interact safely/efficiently with underlying OS APIs (they're pretty much all C-like) and to encode complex concepts into the type system and the compiler. Rust isn't the only language with these properties, but it is one of the few -- and it's one that we really enjoyed using.
    - palerdot 7y ago
      > Rust isn't the only language with these properties, but it is one of the few Just curious, what are the other languages that competes with Rust in terms of correctness, safety, ergonomics and efficiency? Go is more on the ergonomics side and Haskell on the correctness side, but are there any serious alternatives for Rust that checks all the boxes?
      - The_rationalist 7y ago
        The only box that rust has over others is runtime performance. One would consider languages such as Kotlin/modern C# to be the modern sweet spot as they tick all the boxes, and for performance they are ~2 time slower than rust which is just fast enough for anything that isn't HPC/AI/very specific need
      - rbtying 7y ago
        It depends on what you mean by checking all the boxes: Go is tricky if you run on diverse platforms and want to do a lot of FFI, since cgo overhead is significant. The type system in Go is also not very powerful, which is both good and bad. Haskell tends to hit performance walls that are very difficult to debug, and has a pretty similar learning curve to Rust (most people you hire onto the team won't know the language already). The predominant competitor in this space is probably a high-level dynamic language combined with C/C++ library code. With good tooling and good practices to mitigate footguns, the extensive library support in C++ has a lot to offer. Of course, I think Rust makes a better trade-off there, but early in the project it was not at all obvious that the good parts outweighed the fact that we would probably have been the biggest user of any library we depended on. We had some fun adventures in stress-testing HTTP/2 support here :)
- steveklabnik 7y ago
  Very psyched to see even more Rust at Dropbox!
- CodeWriter23 7y ago
  I have a question, is hooking filesystem write notifications on macOS hard? As soon as I paid for the Plus Plan, I started having to wait several minutes for a file to upload. Even from one computer to the other with LAN Sync enabled. The day before that, my files would sync faster than I could hit refresh in the browser I was developing in.
  - saagarjha 7y ago
    There’s API for it, which I’m sure the Dropbox team is aware of. There’s probably another reason why it didn’t sync immediately.
  - sujayakar 7y ago
    for mac it's FSEveents: https://developer.apple.com/documentation/coreservices/file_system_events https://developer.apple.com/documentation/coreservices/file_.... > As soon as I paid for the Plus Plan, I started having to wait several minutes for a file to upload. Even from one computer to the other with LAN Sync enabled. The day before that, my files would sync faster than I could hit refresh in the browser I was developing in. that definitely seems like a bug, could you report it? click on the dropbox tray, click on the dropdown in the top right, and then click "Report Bug." this will collect some information about your client to help us debug.
    - saagarjha 7y ago
      I'm curious if Dropbox could replace their kernel extension and FSEvents usage with an EndpointSecurity client. Would you happen to know if there's any work being done in this area?
- asplake 7y ago
  Interesting to see Rust praised for its ergonomics, possibly not the first word that springs to mind (to an interested observer, not yet a user). Could you say a bit more about that?
  - sujayakar 7y ago
    sure! the rust team has done a fantastic job with the language design, and once a programmer gets over the initial learning curve, the ergonomics of "pair programming" with the compiler are really great. they've put a lot of work into error messages, and the language definitely steers the programmer towards writing correct, efficient code. also, a lot of the standard library APIs are really well thought out, and it's great to then use the type system to build great internal APIs as well. the ability to design APIs to make correct use easy and incorrect use difficult feels like good "ergonomics" to me :)
  - _bxg1 7y ago
    Rust is funny because in one sense it's hard and clunky. However, it's only ever precisely as hard and clunky as it needs to be. Everywhere something can be made more concise, or readable, or convenient, without sacrificing any control, it has been. Anytime something is hard or inconvenient, it's because the underlying domain really is exactly that hard or inconvenient. Contrast this with other languages, which are often clunky when they don't need to be and/or "easy" when they shouldn't be.
    - ssokolow 7y ago
      You're the second person I've seen approach that "Anytime something is hard or inconvenient, it's because the underlying domain really is exactly that hard or inconvenient." point in the last little while. Have you read https://fasterthanli.me/blog/2020/i-want-off-mr-golangs-wild-ride/ https://fasterthanli.me/blog/2020/i-want-off-mr-golangs-wild...?
- jhayward 7y ago
  Very interesting writeup. Did you consider using formal verification tools such as the TLA family to ensure that your sync protocol is well-specified and consistent?
  - sujayakar 7y ago
    yeah, I'm familiar with those tools, and other teams at dropbox have used TLA+ for verifying their protocols, but we haven't done it for our system yet. we've spent most of our time on testing working on better randomized testing, with the idea that covering more surface area there has higher ROI. but I can definitely see value there, especially once the protocol "settles down" and sees less change over time.
- justicezyx 7y ago
  Before the rewriting, how many SWE hours were devoted to the maintenance of this piece of code? Did the maintainer, if any, got meaningful recognition which correctly translate the importance of the work?
  - sujayakar 7y ago
    IIRC, we had a team of about six engineers working on the core system itself plus a few product teams building features on top. and yes, the team did get recognition for their work, which definitely helped when we decided to begin rewriting!
    - justicezyx 7y ago
      Were the six engineers taking part in the rewriting at all?
      - sujayakar 7y ago
        yep! it was 100% critical to have the experts on sync engine classic involved with nucleus. we shared the oncall rotation of maintaining the old system among the entire rewriting team.
- FullyFunctional 7y ago
  It is a great article and I really enjoyed it. I fear that people might latch on too quickly to "Rust" and not appreciate that the preciser (formally defined?) data model was key, and that languages like Rust (and Haskell) are merely very suited for this, but not the key. [Reworded to make my point clearer] OT: I really really hate the top bar that drops down and covers what I'm reading when I'm trying to scroll up. It's such a frequent pattern on the modern web and I can't wait for people to get rid of it. (It violates the symmetry of scrolling up and down)
- GordonS 7y ago
  Years ago when I used Dropbox on Windows, if would poll every single file on startup, which in the days of spinning disks rendered the machine unusable for several minutes - does Dropbox on Windows still do this? Also, after the startup phase, IIRC Dropbox used the `FindFirstChangeNotification` Windows API function, which can miss changes under heavy load - do you still use this method, or have you moved to something else, such as using the NTFS USN journal, or a file system minifilter driver?
  - sujayakar 7y ago
    we still have to check every file on startup, since we use the `ReadDirectoryChangesW` call on windows, but it should be less expensive on nucleus. but, now that we've rolled out nucleus, we can start exploring going beyond sync engine classic feature parity, like using the USN journal on windows.
    - GordonS 7y ago
      For NTFS volumes (basically all Windows volumes), you could use the change journal to eliminate that painfully expensive startup check of all files - that would be huge for Windows users. Also, have you previously explored the possibility of using the NTFS change journal, and if so, what challenges did you face? (I don't mean to come across negative; I'm genuinely interested to learn why you didn't use this a decade ago, since there must be a reason).
      - rbtying 7y ago
        As I recall, this was investigated at one point on Sync Engine Classic and is definitely a thing that folks were thinking about for Nucleus, for pretty much exactly this reason. The biggest challenges with this kind of work (not specific to the USN journal) are in the lack of reliable cross-platform support for features which can be shimmed to look similar. The more unique the code path, the harder it is to test en masse. It's also the case that the execution environment on Windows machines tends to be very diverse due to a long history of backwards compatibility and the ability to set complex domain policies -- Dropbox strives for a good user experience, and "go talk to IT to have them change this setting" is rarely one of those. disclaimer: worked on sync at Dropbox; don't work on it anymore; don't have current context
_bxg1 7y ago
This is a wonderful examination of the "why" (which usually gets glossed over in these kinds of posts). Looking forward to the subsequent posts on the "how".
saagarjha 7y ago
Anyone know what the hero image is supposed to be?
- okramcivokram 7y ago
  Looks like a construction crane arm to me. Filename is `Infrastructure-3-1440x305px-light.png` so most likely a stock or free image, just ran through ascii-art filter. Hint, open the file directly, zoom out a bit, and squint.
- sogen 7y ago
  construction, pillars
justicezyx 7y ago
You rewrite because it has to be done, despite the fact that it is no doubt the inferior solution both in terms of technical complexity and economic benefits, as Joel articulated well many years ago (and many others even earlier as well). The code goes into this state because no one appreciates piece size enhancement. And those enhancement has no "impact". Everyone is waiting for the point that everyone's suffering from the bitrotten code is beyond most people's tolerance. And everyone knows well that some really bad things happen during the process, and they'll wisely refrain from doing anything meaningful. That's basically one fact of how software engineering is done in practice. Poor humans. And this phenomenon is well known even in ancient time in other areas of society. Like even in Qin dynasty cicra 300BC, a Chinese physician Bian Que once claimed that his brothers are better than him [1], because they prevent illness before they occur. And of course, everyone only knows Bian Que.
j88439h84 7y ago
Its unfortunate that everybody's rewriting stuff with futures only to realize five years from now that structured concurrency is much safer and easier.
- ayberk 7y ago
  Citation on that?
  - ofrzeta 7y ago
    Well, it's an opinion, however I'd like to know what exactly "structured concurrency" is supposed to mean.
    - j88439h84 7y ago
      Good intro here https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ https://vorpus.org/blog/notes-on-structured-concurrency-or-g...
  - The_rationalist 7y ago
    It's at least the opinion of the latest languages effort on concurrency: Kotlin, project loom, etc
- sujayakar 7y ago
  we actually extensively use structured concurrency patterns with futures in rust. it's easy to have "nested" futures in rust, where one top-level future (our control thread) owns its subcomponents (like a protocol component, scheduler component, and so on), and the `poll` method for the top-level future then `poll`s all of its subcomponents. in this case we manually write our own `poll` implementations, but it's also easy to express these patterns with combinators like `FuturesUnordered`. then, we have really good cancellation properties in our system. if we decide to cancel a file, we just drop its future, and we can be sure that all of the resources held by its "subtree" of futures will be released.
xiaodai 7y ago
I wonder what's the incentive to write such a thing on company blog? Any ideas? To help with recruitment?
- ublaze 7y ago
  As someone who's written a Dropbox blog post - yes. A tech blog is often cited by candidates as something they were intrigued by.

158 points by ublaze 7y ago

I really liked this article because they spoke more of their decision making process than tech. Rewrite It In Rust is a meme by this point, but the blog actually goes through every point that could be discussed in such a strategy and why it made sense for Dropbox at this moment. Really good blog post, imo.

Recursing 7y ago
The author (and some of his coworkers) gave a lot of interesting answers on reddit as well: https://www.reddit.com/r/rust/comments/fjt4q3/rewriting_the_heart_of_our_sync_engine_in_rust/ https://www.reddit.com/r/rust/comments/fjt4q3/rewriting_the_...
- sujayakar 7y ago
  and on lobste.rs too :) https://lobste.rs/s/ob6a8z/rewriting_heart_our_sync_engine https://lobste.rs/s/ob6a8z/rewriting_heart_our_sync_engine

aw1621107 7y ago

A bit light on details, but it seems more are incoming. Can't wait for those blog posts!

sujayakar 7y ago

hey everyone, author here! happy to answer any questions about the post, our system, or our usage of rust :)

quantified 7y ago
How did the decision to use Rust go from idea to acceptance/adoption? Management can be risk-averse in choice of dev platform, and if Rust was adopted more than 18 months ago it was maybe even before the “rewrite it Rust” meme. Lessons learned and preconditions for success on this decision are especially welcomed.
- rbtying 7y ago
  disclaimer: was on the team, but didn't write the article / don't work on this anymore Rust was adopted at Dropbox for some serving infrastructure use cases more than a year before the sync rewrite was started, which was about four years ago. I'd say we solidly predated the "rewrite it in Rust" meme. I believe that this rewrite was only successful because of Rust's ability to both interact safely/efficiently with underlying OS APIs (they're pretty much all C-like) and to encode complex concepts into the type system and the compiler. Rust isn't the only language with these properties, but it is one of the few -- and it's one that we really enjoyed using.
  - palerdot 7y ago
    > Rust isn't the only language with these properties, but it is one of the few Just curious, what are the other languages that competes with Rust in terms of correctness, safety, ergonomics and efficiency? Go is more on the ergonomics side and Haskell on the correctness side, but are there any serious alternatives for Rust that checks all the boxes?
    - The_rationalist 7y ago
      The only box that rust has over others is runtime performance. One would consider languages such as Kotlin/modern C# to be the modern sweet spot as they tick all the boxes, and for performance they are ~2 time slower than rust which is just fast enough for anything that isn't HPC/AI/very specific need
    - rbtying 7y ago
      It depends on what you mean by checking all the boxes: Go is tricky if you run on diverse platforms and want to do a lot of FFI, since cgo overhead is significant. The type system in Go is also not very powerful, which is both good and bad. Haskell tends to hit performance walls that are very difficult to debug, and has a pretty similar learning curve to Rust (most people you hire onto the team won't know the language already). The predominant competitor in this space is probably a high-level dynamic language combined with C/C++ library code. With good tooling and good practices to mitigate footguns, the extensive library support in C++ has a lot to offer. Of course, I think Rust makes a better trade-off there, but early in the project it was not at all obvious that the good parts outweighed the fact that we would probably have been the biggest user of any library we depended on. We had some fun adventures in stress-testing HTTP/2 support here :)
steveklabnik 7y ago
Very psyched to see even more Rust at Dropbox!
CodeWriter23 7y ago
I have a question, is hooking filesystem write notifications on macOS hard? As soon as I paid for the Plus Plan, I started having to wait several minutes for a file to upload. Even from one computer to the other with LAN Sync enabled. The day before that, my files would sync faster than I could hit refresh in the browser I was developing in.
- saagarjha 7y ago
  There’s API for it, which I’m sure the Dropbox team is aware of. There’s probably another reason why it didn’t sync immediately.
- sujayakar 7y ago
  for mac it's FSEveents: https://developer.apple.com/documentation/coreservices/file_system_events https://developer.apple.com/documentation/coreservices/file_.... > As soon as I paid for the Plus Plan, I started having to wait several minutes for a file to upload. Even from one computer to the other with LAN Sync enabled. The day before that, my files would sync faster than I could hit refresh in the browser I was developing in. that definitely seems like a bug, could you report it? click on the dropbox tray, click on the dropdown in the top right, and then click "Report Bug." this will collect some information about your client to help us debug.
  - saagarjha 7y ago
    I'm curious if Dropbox could replace their kernel extension and FSEvents usage with an EndpointSecurity client. Would you happen to know if there's any work being done in this area?
asplake 7y ago
Interesting to see Rust praised for its ergonomics, possibly not the first word that springs to mind (to an interested observer, not yet a user). Could you say a bit more about that?
- sujayakar 7y ago
  sure! the rust team has done a fantastic job with the language design, and once a programmer gets over the initial learning curve, the ergonomics of "pair programming" with the compiler are really great. they've put a lot of work into error messages, and the language definitely steers the programmer towards writing correct, efficient code. also, a lot of the standard library APIs are really well thought out, and it's great to then use the type system to build great internal APIs as well. the ability to design APIs to make correct use easy and incorrect use difficult feels like good "ergonomics" to me :)
- _bxg1 7y ago
  Rust is funny because in one sense it's hard and clunky. However, it's only ever precisely as hard and clunky as it needs to be. Everywhere something can be made more concise, or readable, or convenient, without sacrificing any control, it has been. Anytime something is hard or inconvenient, it's because the underlying domain really is exactly that hard or inconvenient. Contrast this with other languages, which are often clunky when they don't need to be and/or "easy" when they shouldn't be.
  - ssokolow 7y ago
    You're the second person I've seen approach that "Anytime something is hard or inconvenient, it's because the underlying domain really is exactly that hard or inconvenient." point in the last little while. Have you read https://fasterthanli.me/blog/2020/i-want-off-mr-golangs-wild-ride/ https://fasterthanli.me/blog/2020/i-want-off-mr-golangs-wild...?
jhayward 7y ago
Very interesting writeup. Did you consider using formal verification tools such as the TLA family to ensure that your sync protocol is well-specified and consistent?
- sujayakar 7y ago
  yeah, I'm familiar with those tools, and other teams at dropbox have used TLA+ for verifying their protocols, but we haven't done it for our system yet. we've spent most of our time on testing working on better randomized testing, with the idea that covering more surface area there has higher ROI. but I can definitely see value there, especially once the protocol "settles down" and sees less change over time.
justicezyx 7y ago
Before the rewriting, how many SWE hours were devoted to the maintenance of this piece of code? Did the maintainer, if any, got meaningful recognition which correctly translate the importance of the work?
- sujayakar 7y ago
  IIRC, we had a team of about six engineers working on the core system itself plus a few product teams building features on top. and yes, the team did get recognition for their work, which definitely helped when we decided to begin rewriting!
  - justicezyx 7y ago
    Were the six engineers taking part in the rewriting at all?
    - sujayakar 7y ago
      yep! it was 100% critical to have the experts on sync engine classic involved with nucleus. we shared the oncall rotation of maintaining the old system among the entire rewriting team.
FullyFunctional 7y ago
It is a great article and I really enjoyed it. I fear that people might latch on too quickly to "Rust" and not appreciate that the preciser (formally defined?) data model was key, and that languages like Rust (and Haskell) are merely very suited for this, but not the key. [Reworded to make my point clearer] OT: I really really hate the top bar that drops down and covers what I'm reading when I'm trying to scroll up. It's such a frequent pattern on the modern web and I can't wait for people to get rid of it. (It violates the symmetry of scrolling up and down)
GordonS 7y ago
Years ago when I used Dropbox on Windows, if would poll every single file on startup, which in the days of spinning disks rendered the machine unusable for several minutes - does Dropbox on Windows still do this? Also, after the startup phase, IIRC Dropbox used the `FindFirstChangeNotification` Windows API function, which can miss changes under heavy load - do you still use this method, or have you moved to something else, such as using the NTFS USN journal, or a file system minifilter driver?
- sujayakar 7y ago
  we still have to check every file on startup, since we use the `ReadDirectoryChangesW` call on windows, but it should be less expensive on nucleus. but, now that we've rolled out nucleus, we can start exploring going beyond sync engine classic feature parity, like using the USN journal on windows.
  - GordonS 7y ago
    For NTFS volumes (basically all Windows volumes), you could use the change journal to eliminate that painfully expensive startup check of all files - that would be huge for Windows users. Also, have you previously explored the possibility of using the NTFS change journal, and if so, what challenges did you face? (I don't mean to come across negative; I'm genuinely interested to learn why you didn't use this a decade ago, since there must be a reason).
    - rbtying 7y ago
      As I recall, this was investigated at one point on Sync Engine Classic and is definitely a thing that folks were thinking about for Nucleus, for pretty much exactly this reason. The biggest challenges with this kind of work (not specific to the USN journal) are in the lack of reliable cross-platform support for features which can be shimmed to look similar. The more unique the code path, the harder it is to test en masse. It's also the case that the execution environment on Windows machines tends to be very diverse due to a long history of backwards compatibility and the ability to set complex domain policies -- Dropbox strives for a good user experience, and "go talk to IT to have them change this setting" is rarely one of those. disclaimer: worked on sync at Dropbox; don't work on it anymore; don't have current context

_bxg1 7y ago

This is a wonderful examination of the "why" (which usually gets glossed over in these kinds of posts). Looking forward to the subsequent posts on the "how".

saagarjha 7y ago

Anyone know what the hero image is supposed to be?

okramcivokram 7y ago
Looks like a construction crane arm to me. Filename is `Infrastructure-3-1440x305px-light.png` so most likely a stock or free image, just ran through ascii-art filter. Hint, open the file directly, zoom out a bit, and squint.
sogen 7y ago
construction, pillars

justicezyx 7y ago

You rewrite because it has to be done, despite the fact that it is no doubt the inferior solution both in terms of technical complexity and economic benefits, as Joel articulated well many years ago (and many others even earlier as well). The code goes into this state because no one appreciates piece size enhancement. And those enhancement has no "impact". Everyone is waiting for the point that everyone's suffering from the bitrotten code is beyond most people's tolerance. And everyone knows well that some really bad things happen during the process, and they'll wisely refrain from doing anything meaningful. That's basically one fact of how software engineering is done in practice. Poor humans. And this phenomenon is well known even in ancient time in other areas of society. Like even in Qin dynasty cicra 300BC, a Chinese physician Bian Que once claimed that his brothers are better than him [1], because they prevent illness before they occur. And of course, everyone only knows Bian Que.

j88439h84 7y ago

Its unfortunate that everybody's rewriting stuff with futures only to realize five years from now that structured concurrency is much safer and easier.

ayberk 7y ago
Citation on that?
- ofrzeta 7y ago
  Well, it's an opinion, however I'd like to know what exactly "structured concurrency" is supposed to mean.
  - j88439h84 7y ago
    Good intro here https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ https://vorpus.org/blog/notes-on-structured-concurrency-or-g...
- The_rationalist 7y ago
  It's at least the opinion of the latest languages effort on concurrency: Kotlin, project loom, etc
sujayakar 7y ago
we actually extensively use structured concurrency patterns with futures in rust. it's easy to have "nested" futures in rust, where one top-level future (our control thread) owns its subcomponents (like a protocol component, scheduler component, and so on), and the `poll` method for the top-level future then `poll`s all of its subcomponents. in this case we manually write our own `poll` implementations, but it's also easy to express these patterns with combinators like `FuturesUnordered`. then, we have really good cancellation properties in our system. if we decide to cancel a file, we just drop its future, and we can be sure that all of the resources held by its "subtree" of futures will be released.

xiaodai 7y ago

I wonder what's the incentive to write such a thing on company blog? Any ideas? To help with recruitment?

ublaze 7y ago
As someone who's written a Dropbox blog post - yes. A tech blog is often cited by candidates as something they were intrigued by.