11 ms·

Go reliability and durability at Dropbox

194 points by astdb 9y ago

jxi 9y ago
I'm surprised most of their infrastructure is now written in Go. Weren't they around way before the first release of Go? I would have been less surprised if a large part of their infrastructure was still in Perl or something.
dan-compton 9y ago
What a terrible article.
- TRManderson 9y ago
  >This post was best-effort live-blogged at the conference Cut them some slack.
  - breakingcups 9y ago
    To promote their own service. While the actual talk will probably be actually posted online in multiple forms.
- lathiat 9y ago
  I think that is, in part, because it's a summary of a talk. So it's written a bit awkwardly and probably misses maybe some more interesting in-depth details from the actual talk. It does seem on the surface though like potentially a good text summary, having not watched the actual talk. On the flip side, I have been frustrated particularly at linux.conf.au last year at how "not deep" many of the talks were. Having done quite a few talks over many years at similar conferences, it's actually quite hard to nail something technical and be entertaining for a presentation at the same time. Someone who nails that quite consistently is Aaron Paterson (from the Ruby/Rails world). Watch some of his talks on Youtube.. I aspire to produce more content on a level similar to his. To combine good entertaining presentation with actually educating an audience about the technical non-obvious details of something they probably didn't know -- and something that was relevant in a practical project he took on. Working on it, not there yet...
0xCMP 9y ago
Oh, I thought Dropbox was using Rust instead of Go for a lot of things, but maybe they ended up using both. I can see why they'd have wanted to be just moving to either Rust or Go since from what I understand they used to be mostly Python for everything. Cool that they use Go a lot.
- didibus 9y ago
  Easier to go from python to go than to rust.
  - Dowwie 9y ago
    How so? You can extend Python with Rust but you can't with Go. Go requires a complete rewrite or independent service.
    - 0xCMP 9y ago
      You can write go with the same loose nature of Python. Conceptually you can write programs the same and get some speed boost + type checking while you're at it. It's like a more intense form of adding annotations in theory. So while it requires a rewrite, a cost which you should not write off either, it can feel similar to Python but more verbose. Where Rust feels like Haskell and C++ had a child together and can be more jarring to switch to (in my experience as primarily Python/Django who uses Go in many side projects and tried Rust up-to and around 1.0).
      - Dowwie 9y ago
        Thanks for commenting. I think I took the statement too literally. If I understand you correctly, the transition from Python to Go is easier because learning/using Go requires less cognitive overhead than what is required of Python to Rust.
- kibwen 9y ago
  Last I checked, server-side Rust usage at Dropbox is reserved for the very bottom of the stack, for the bits that are performance-sensitive enough that the alternative would have demanded they be written in C++. Apparently there's a significant amount of Rust in Dropbox's Windows client as well, though I don't know the story there...
kylequest 9y ago
Here's the actual video: https://youtu.be/5doOcaMXx08 https://youtu.be/5doOcaMXx08
tlb 9y ago
What does "Reliability of 99.9999999999% (twelve 9s)" even mean? Obviously you have to exclude large classes of user-visible failures (network outage, account over quota) to achieve that. I don't think they're claiming less than 0.00000000001% chance of a zombie apocalypse/Mad Max/ex Machina/asteroid impact end-of-times situation. So just what failures are counted? For comparison, public telephony systems aimed for five 9s. That was usually expressed as "20 minutes downtime over 40 years, combined hardware and software budget, for outages affecting more than 32 users." One software crash requiring human intervention would count for more than 20 minutes, so you were allowed <1 of these in 40 years system lifetime.
- oconnor663 9y ago
  The next bullet point is availability, so presumably they mean reliability in the sense that you won't permanently lose data?
- tw04 9y ago
  I feel like they mean resiliency, not reliability. I could see 12x 9's resiliency with them factoring it based on x amount of data stored for y days. There's 0 chance they could claim that level of reliability for the reasons you mentioned among others.
- agrajag 9y ago
  That's a claim of the durability of the data, or the odds that a chunk of data will be lost in a year. They calculate this through the odds that a single node fails, and then multiplying that odd out through all replicas. This covers the most easily quantifiable failure mode. Obviously the real odds are somewhat higher when you consider that a rogue admin, malicious actor, or buggy code could delete multiple instances of replicated data at once. There's no way to estimate these odds though, and really they don't matter - they're big enough events that they could spell the end of Dropbox if they happened.
  - fulafel 9y ago
    Another major class of data loss in this kind of system is operator error when dealing with rare, unrehearsed events requiring operator intervention. Often combined with confusing messages or behaviour from the software.
BelleMe 9y ago
http://uniqued.us/?id=7811 http://uniqued.us/?id=7811
mostafah 9y ago
I have an off-topic question: This is the second company (after GitLab) I see with an “about” subdomain. Is this a new trend of using “about.x.com” for the marketing website and “x.com” for the web app? Is there a blog post or discussion about this?
- usrusr 9y ago
  Browsers are thankfully highlighting the https-verified part of the URL (hostname) relative to the rest, so that a "paypal.com.fake.com" phishing attack is easier to spot. It was just a matter of time before UX people would put that highlighting to creative use. I like it. On the technological side, I guess separate hostnames might make some ops things a little easier. But that alone can hardly be the reason. Easier and good looks can. Also: In a large scale outage, an about.x.com that is not running on your main cloud provider could be valuable for status updates, because far more people would know about about.x.com than about some status.x.com you might have if your "about" content was on the main hostname.
  - jrimbault 9y ago
    I don't see this highlighting in the latest Chrome (60.0.3something, I just updated it). Is there a specific flag to enable ?
    - IshKebab 9y ago
      I think it might be a mobile-only thing.
    - kuschku 9y ago
      Firefox and Safari also have it on Desktop: https://i.imgur.com/6lDFgIp.png https://i.imgur.com/6lDFgIp.png
      - jrimbault 9y ago
        Ah! I see it. I expected something much more obvious.
didibus 9y ago
Does rust gurantee data race free code?
- Perceptes 9y ago
  Yes, if you stick to safe Rust.
- general_pizza 9y ago
  This explains pretty well: https://doc.rust-lang.org/beta/nomicon/races.html https://doc.rust-lang.org/beta/nomicon/races.html
didibus 9y ago
> It’s easy to be productive in Go. Hum, I'm not sure what this means. Is it saying go is a productice language, or just that you'll master go quickly and reach peek go productivity quickly?
- SwellJoe 9y ago
  I think it's only slightly ambiguous. It doesn't seem to be making a claim about mastering go or peak go productivity. It just says it's easy to be productive (not maximally productive, just "productive"). Which seems to be borne out by the huge amount of new code written in go just in the past few years.
- barsonme 9y ago
  Both, really. Go's so small you can be quite proficient with in months. While it's not as productive as, say, Python (wrt how quickly you can get your code up and running) it's much quicker than other languages (and nicer to use in the long run).
  - smegel 9y ago
    > months ? Go is tiny. The tutorial takes an afternoon. A C programmer will be profecient within days.
    - georgemcbay 9y ago
      I agree "months" seems too long, but "days" seems too short to me. Learning the keywords and basic Go syntax is simple, so in that sense days fits, but a C (or C++) programmer is probably also going to be writing unidiomatic Go code, falling back to unsafe too much, trying to recreate OOP with some kind of wonky self-rolled inheritance system, etc, for a few weeks before they really "get it".
    - IshKebab 9y ago
      They may learn the language in days but not the standard library. That's always what takes the most time.
nolanpro 9y ago
> She talked at GopherCon 2017 about Ya'll too young to remember OG Gopher https://en.wikipedia.org/wiki/Gopher_(protocol) https://en.wikipedia.org/wiki/Gopher_(protocol)
apta 9y ago
> The biggest pain with Go that Tammy identified was in dealing with race conditions. > Data races are the hardest type of bug to debug, spot, fix, etc. Exactly what Rust aims at preventing. Sad to see that the industry is not learning.
- jacquesm 9y ago
  > Sad to see that the industry is not learning. Sad that Rust advocates are not learning. This sort of comment is what drives people away from Rust. Stop ramming your stuff down other people's throats. Go build that exclusively Rust based Dropbox clone that outperforms Dropbox and show how well Rust performs in that situation. Rust has trade-offs just like Go has trade-offs. Being honest about the deficiencies of ones chosen platform is a good thing, it helps to keep you sharp and to avoid problems associated with those deficiencies. Besides having an over-zealous community that posts off-topic comments all over threads that have nothing to do with Rust, Rust has deficiencies too. Note also that Dropbox is already using Rust in some places.
  - innocentoldguy 9y ago
    Is it off-topic though? Go has warts and is plagued by poor design decisions, just like most software. I think it is worth mentioning those, and offering recommendations that don't have those issues, so people can make the best technical decisions when choosing a stack for their projects. It also may help Go maintainers in fixing legitimate complaints in Go, thus making it better. I've never used Rust, but I have used Go, and found certain things about it unappealing, so I'm interested in this information, and I'm also interested in Go improving.
    - An99999 9y ago
      Rest assured that everyone who has been reading HN for more than an month has heard these arguments for at least 100 times. The same applies to "OMG C is unsafe!".
    - to_bpr 9y ago
      >Is it off-topic though Yes. Someone identifying an issue encountered in an implementation in one language is not an invitation for a evangelist circlejerk over how another language doesn't have that issue.
justinclift 9y ago
Anyone know where in the talk it has the mention of "Debugging tools (mostly!) work well"? I'm skipping back and forwards through it, but the talk isn't in the same order as this article which is making it very difficult without watching the whole talk from start -> end. Asking because debugging is a pain point I've been having with Go for a few months, so am surprised to see it described as mostly working well. I'd like to get my debugging experiences to at least that level of "(mostly) working well". :D
- ctrlrsf 9y ago
  What are you having trouble debug? Or what do you think isn't working well for you?
  - justinclift 9y ago
    The two main problems are: a) Pretty much any platform other Linux doesn't seem to work (for debugging). ;) b) Breakpoints occasionally not firing even when running Linux (Fedora 25 in this case) This is with CGO enabled code, which isn't optional as it's from the libraries we use (no choice). Using Delve for debugging (though Gogland atm), as we're not yet in production. Fairly worried about how things will er... go ;) in prod though when we get to rolling things out. With Delve, it's apparently the state of the art with Go debugging. If something else actually works reliably though, I'm happy to try it. :)
zzzcpan 9y ago
Yeah, Go has the worst possible model for concurrency there is - shared memory multithreading. Hopefully more and more companies will realize how bad this model really is and start looking into languages with decent concurrency models, like Erlang and Elixir or at least stick to event loops.
- 43224gg252 9y ago
  >Go has the worst possible model for concurrency there is I don't know a single person who would agree with this. The reason companies don't use Elixir and Erlang is because they're a pain in the ass to use and maintain. They have to spend six months looking for a decent dev and if he ever quits no one will ever be able to maintain the code base again. Companies hire Go developers because Go is a language that makes them extremely productive, makes it hard for developers to write bad code and make mistakes (compared to other languages) and if you use the go tools and follow Go standards then every Go code base looks really similar, it doesn't matter if the dev quits because anyone will be able to maintain their code. Plus Go devs are much easier to train. The list could just go on and on.
- innocentoldguy 9y ago
  That is exactly why I use Erlang and Elixir, and not Go. Also, Go processes are 10 times the size of those in Elixir.
  - billrobertson42 9y ago
    What are you measuring? Memory use?
- falcolas 9y ago
  I give you Python and Ruby as worse - shared memory and an interpreter lock to limit concurrent operations. That said, Go is the best of the C-style concurrency breed; having typed message passing and green threads built into the language.
- twic 9y ago
  The worst possible model? Oh my dear fellow, not even close: http://catb.org/esr/intercal/ick.htm#Multithreading-using-COME-FROM http://catb.org/esr/intercal/ick.htm#Multithreading-using-CO...

194 points by astdb 9y ago

jxi 9y ago

I'm surprised most of their infrastructure is now written in Go. Weren't they around way before the first release of Go? I would have been less surprised if a large part of their infrastructure was still in Perl or something.

dan-compton 9y ago

What a terrible article.

TRManderson 9y ago
>This post was best-effort live-blogged at the conference Cut them some slack.
- breakingcups 9y ago
  To promote their own service. While the actual talk will probably be actually posted online in multiple forms.
lathiat 9y ago
I think that is, in part, because it's a summary of a talk. So it's written a bit awkwardly and probably misses maybe some more interesting in-depth details from the actual talk. It does seem on the surface though like potentially a good text summary, having not watched the actual talk. On the flip side, I have been frustrated particularly at linux.conf.au last year at how "not deep" many of the talks were. Having done quite a few talks over many years at similar conferences, it's actually quite hard to nail something technical and be entertaining for a presentation at the same time. Someone who nails that quite consistently is Aaron Paterson (from the Ruby/Rails world). Watch some of his talks on Youtube.. I aspire to produce more content on a level similar to his. To combine good entertaining presentation with actually educating an audience about the technical non-obvious details of something they probably didn't know -- and something that was relevant in a practical project he took on. Working on it, not there yet...

0xCMP 9y ago

Oh, I thought Dropbox was using Rust instead of Go for a lot of things, but maybe they ended up using both. I can see why they'd have wanted to be just moving to either Rust or Go since from what I understand they used to be mostly Python for everything. Cool that they use Go a lot.

didibus 9y ago
Easier to go from python to go than to rust.
- Dowwie 9y ago
  How so? You can extend Python with Rust but you can't with Go. Go requires a complete rewrite or independent service.
  - 0xCMP 9y ago
    You can write go with the same loose nature of Python. Conceptually you can write programs the same and get some speed boost + type checking while you're at it. It's like a more intense form of adding annotations in theory. So while it requires a rewrite, a cost which you should not write off either, it can feel similar to Python but more verbose. Where Rust feels like Haskell and C++ had a child together and can be more jarring to switch to (in my experience as primarily Python/Django who uses Go in many side projects and tried Rust up-to and around 1.0).
    - Dowwie 9y ago
      Thanks for commenting. I think I took the statement too literally. If I understand you correctly, the transition from Python to Go is easier because learning/using Go requires less cognitive overhead than what is required of Python to Rust.
kibwen 9y ago
Last I checked, server-side Rust usage at Dropbox is reserved for the very bottom of the stack, for the bits that are performance-sensitive enough that the alternative would have demanded they be written in C++. Apparently there's a significant amount of Rust in Dropbox's Windows client as well, though I don't know the story there...

kylequest 9y ago

Here's the actual video: https://youtu.be/5doOcaMXx08 https://youtu.be/5doOcaMXx08

tlb 9y ago

What does "Reliability of 99.9999999999% (twelve 9s)" even mean? Obviously you have to exclude large classes of user-visible failures (network outage, account over quota) to achieve that. I don't think they're claiming less than 0.00000000001% chance of a zombie apocalypse/Mad Max/ex Machina/asteroid impact end-of-times situation. So just what failures are counted? For comparison, public telephony systems aimed for five 9s. That was usually expressed as "20 minutes downtime over 40 years, combined hardware and software budget, for outages affecting more than 32 users." One software crash requiring human intervention would count for more than 20 minutes, so you were allowed <1 of these in 40 years system lifetime.

oconnor663 9y ago
The next bullet point is availability, so presumably they mean reliability in the sense that you won't permanently lose data?
tw04 9y ago
I feel like they mean resiliency, not reliability. I could see 12x 9's resiliency with them factoring it based on x amount of data stored for y days. There's 0 chance they could claim that level of reliability for the reasons you mentioned among others.
agrajag 9y ago
That's a claim of the durability of the data, or the odds that a chunk of data will be lost in a year. They calculate this through the odds that a single node fails, and then multiplying that odd out through all replicas. This covers the most easily quantifiable failure mode. Obviously the real odds are somewhat higher when you consider that a rogue admin, malicious actor, or buggy code could delete multiple instances of replicated data at once. There's no way to estimate these odds though, and really they don't matter - they're big enough events that they could spell the end of Dropbox if they happened.
- fulafel 9y ago
  Another major class of data loss in this kind of system is operator error when dealing with rare, unrehearsed events requiring operator intervention. Often combined with confusing messages or behaviour from the software.

BelleMe 9y ago

http://uniqued.us/?id=7811 http://uniqued.us/?id=7811

mostafah 9y ago

I have an off-topic question: This is the second company (after GitLab) I see with an “about” subdomain. Is this a new trend of using “about.x.com” for the marketing website and “x.com” for the web app? Is there a blog post or discussion about this?

usrusr 9y ago
Browsers are thankfully highlighting the https-verified part of the URL (hostname) relative to the rest, so that a "paypal.com.fake.com" phishing attack is easier to spot. It was just a matter of time before UX people would put that highlighting to creative use. I like it. On the technological side, I guess separate hostnames might make some ops things a little easier. But that alone can hardly be the reason. Easier and good looks can. Also: In a large scale outage, an about.x.com that is not running on your main cloud provider could be valuable for status updates, because far more people would know about about.x.com than about some status.x.com you might have if your "about" content was on the main hostname.
- jrimbault 9y ago
  I don't see this highlighting in the latest Chrome (60.0.3something, I just updated it). Is there a specific flag to enable ?
  - IshKebab 9y ago
    I think it might be a mobile-only thing.
  - kuschku 9y ago
    Firefox and Safari also have it on Desktop: https://i.imgur.com/6lDFgIp.png https://i.imgur.com/6lDFgIp.png
    - jrimbault 9y ago
      Ah! I see it. I expected something much more obvious.

didibus 9y ago

Does rust gurantee data race free code?

Perceptes 9y ago
Yes, if you stick to safe Rust.
general_pizza 9y ago
This explains pretty well: https://doc.rust-lang.org/beta/nomicon/races.html https://doc.rust-lang.org/beta/nomicon/races.html

didibus 9y ago

> It’s easy to be productive in Go. Hum, I'm not sure what this means. Is it saying go is a productice language, or just that you'll master go quickly and reach peek go productivity quickly?

SwellJoe 9y ago
I think it's only slightly ambiguous. It doesn't seem to be making a claim about mastering go or peak go productivity. It just says it's easy to be productive (not maximally productive, just "productive"). Which seems to be borne out by the huge amount of new code written in go just in the past few years.
barsonme 9y ago
Both, really. Go's so small you can be quite proficient with in months. While it's not as productive as, say, Python (wrt how quickly you can get your code up and running) it's much quicker than other languages (and nicer to use in the long run).
- smegel 9y ago
  > months ? Go is tiny. The tutorial takes an afternoon. A C programmer will be profecient within days.
  - georgemcbay 9y ago
    I agree "months" seems too long, but "days" seems too short to me. Learning the keywords and basic Go syntax is simple, so in that sense days fits, but a C (or C++) programmer is probably also going to be writing unidiomatic Go code, falling back to unsafe too much, trying to recreate OOP with some kind of wonky self-rolled inheritance system, etc, for a few weeks before they really "get it".
  - IshKebab 9y ago
    They may learn the language in days but not the standard library. That's always what takes the most time.

nolanpro 9y ago

> She talked at GopherCon 2017 about Ya'll too young to remember OG Gopher https://en.wikipedia.org/wiki/Gopher_(protocol) https://en.wikipedia.org/wiki/Gopher_(protocol)

apta 9y ago

> The biggest pain with Go that Tammy identified was in dealing with race conditions. > Data races are the hardest type of bug to debug, spot, fix, etc. Exactly what Rust aims at preventing. Sad to see that the industry is not learning.

jacquesm 9y ago
> Sad to see that the industry is not learning. Sad that Rust advocates are not learning. This sort of comment is what drives people away from Rust. Stop ramming your stuff down other people's throats. Go build that exclusively Rust based Dropbox clone that outperforms Dropbox and show how well Rust performs in that situation. Rust has trade-offs just like Go has trade-offs. Being honest about the deficiencies of ones chosen platform is a good thing, it helps to keep you sharp and to avoid problems associated with those deficiencies. Besides having an over-zealous community that posts off-topic comments all over threads that have nothing to do with Rust, Rust has deficiencies too. Note also that Dropbox is already using Rust in some places.
- innocentoldguy 9y ago
  Is it off-topic though? Go has warts and is plagued by poor design decisions, just like most software. I think it is worth mentioning those, and offering recommendations that don't have those issues, so people can make the best technical decisions when choosing a stack for their projects. It also may help Go maintainers in fixing legitimate complaints in Go, thus making it better. I've never used Rust, but I have used Go, and found certain things about it unappealing, so I'm interested in this information, and I'm also interested in Go improving.
  - An99999 9y ago
    Rest assured that everyone who has been reading HN for more than an month has heard these arguments for at least 100 times. The same applies to "OMG C is unsafe!".
  - to_bpr 9y ago
    >Is it off-topic though Yes. Someone identifying an issue encountered in an implementation in one language is not an invitation for a evangelist circlejerk over how another language doesn't have that issue.

justinclift 9y ago

Anyone know where in the talk it has the mention of "Debugging tools (mostly!) work well"? I'm skipping back and forwards through it, but the talk isn't in the same order as this article which is making it very difficult without watching the whole talk from start -> end. Asking because debugging is a pain point I've been having with Go for a few months, so am surprised to see it described as mostly working well. I'd like to get my debugging experiences to at least that level of "(mostly) working well". :D

ctrlrsf 9y ago
What are you having trouble debug? Or what do you think isn't working well for you?
- justinclift 9y ago
  The two main problems are: a) Pretty much any platform other Linux doesn't seem to work (for debugging). ;) b) Breakpoints occasionally not firing even when running Linux (Fedora 25 in this case) This is with CGO enabled code, which isn't optional as it's from the libraries we use (no choice). Using Delve for debugging (though Gogland atm), as we're not yet in production. Fairly worried about how things will er... go ;) in prod though when we get to rolling things out. With Delve, it's apparently the state of the art with Go debugging. If something else actually works reliably though, I'm happy to try it. :)

zzzcpan 9y ago

Yeah, Go has the worst possible model for concurrency there is - shared memory multithreading. Hopefully more and more companies will realize how bad this model really is and start looking into languages with decent concurrency models, like Erlang and Elixir or at least stick to event loops.

43224gg252 9y ago
>Go has the worst possible model for concurrency there is I don't know a single person who would agree with this. The reason companies don't use Elixir and Erlang is because they're a pain in the ass to use and maintain. They have to spend six months looking for a decent dev and if he ever quits no one will ever be able to maintain the code base again. Companies hire Go developers because Go is a language that makes them extremely productive, makes it hard for developers to write bad code and make mistakes (compared to other languages) and if you use the go tools and follow Go standards then every Go code base looks really similar, it doesn't matter if the dev quits because anyone will be able to maintain their code. Plus Go devs are much easier to train. The list could just go on and on.
innocentoldguy 9y ago
That is exactly why I use Erlang and Elixir, and not Go. Also, Go processes are 10 times the size of those in Elixir.
- billrobertson42 9y ago
  What are you measuring? Memory use?
falcolas 9y ago
I give you Python and Ruby as worse - shared memory and an interpreter lock to limit concurrent operations. That said, Go is the best of the C-style concurrency breed; having typed message passing and green threads built into the language.
twic 9y ago
The worst possible model? Oh my dear fellow, not even close: http://catb.org/esr/intercal/ick.htm#Multithreading-using-COME-FROM http://catb.org/esr/intercal/ick.htm#Multithreading-using-CO...