6 ms·

Automating Datacenter Operations at Dropbox

94 points by dmicher 8y ago

nemothekid 8y ago
A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) - which is interesting given most providers are writing new infrastructure code largely in Go (Spinnaker is another Python exception). I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebases at their scale?
- amanzi 8y ago
  Dropbox hired the creator of Python to help migrate their huge Python 2 codebase to Python 3. It would make sense for them to continue with their investment in Python.
- lwf 8y ago
  Dropbox sponsors most of the developers of http://mypy-lang.org/ http://mypy-lang.org/ , an optional static typing system for Python. I've found it to be hugely useful in safely working in a large Python codebase :) (NB: I work at Dropbox)
  - mroche 8y ago
    Curious if you’ve ever tried Cython out. Not quite the same, but a semi-similar end goal to a degree. I started looking into the other day and it might provide some nice improvements to heavier applications.
- jedberg 8y ago
  Unless I'm mistaken, Guido still works for Dropbox. And as far as I know they still mostly do Python.
- jensvdh 8y ago
  Spinnaker is written in Java
- jabl 8y ago
  > A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) Sounds like a reasonable guess. > which is interesting given most providers are writing new infrastructure code largely in Go Dropbox apparently uses Go a lot, see e.g. https://about.sourcegraph.com/go/go-reliability-and-durability-at-dropbox-tammy-butow https://about.sourcegraph.com/go/go-reliability-and-durabili... They also use Rust for some performance-critical stuff, e.g. https://news.ycombinator.com/item?id=11282948 https://news.ycombinator.com/item?id=11282948 That being said, AFAIK originally Dropbox was written mostly in python. Probably there's still lots of that left.
  - talonx 8y ago
    This is also borne out by the fact that they mention Celery.
- danpalmer 8y ago
  > which is interesting given most providers are writing new infrastructure code largely in Go I'm not sure this is true. There's a lot of Java around, Scala is pretty popular too, in many industries C/C++ are the norm still for this sort of code. Go might be the trend in open source infrastructure projects right now, but a significant amount of that is likely to be inertia from Docker and Kubernetes.
peterwwillis 8y ago
"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs." I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tools that support the software developers writing the business products. "Using the network database and configuration tool developed by our Network Reliability Engineering (NRE) team," Another custom tool? Network inventory and config management tools do exist already... "Rather than having engineers manually running tests using playbooks, Pirlo performed an automated sequential battery of tests that reduced the need for hands-on attention and concurrently increased diagnostic accuracy." Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system.
- mrbanks 8y ago
  Developers at these big companies get bored & assume they know better than a mature open source solution so like to reinvent a perfectly good wheel basically.
  - inferiorhuman 8y ago
    Or they've actually used Jenkins (or any of these other suggested alternatives). I've used Jenkins personally and professionally. Most recently I've been rebuilding my own CI stack and never really gave much thought to going back to Jenkins. So I've been asking around and one of the only common complaints I've heard so far is that getting the initial configuration done is painful and generally orthogonal to automation. Plus the documentation is atrocious. No off-the-shelf product will be a perfect fit, but with CI software I was truly surprised at just how large the gaps were. So, sure, if you're Dropbox and you want to automate everything Jenkins is almost certainly not the right tool for the job. If Dropbox already had a supported, mature in-house job queue system, why not use it? Conversely at megacorp, they spent 4+ years claiming to work on deploying a Jenkins (CloudBees) cluster and still came up with bupkis. Our own internal job and message queuing systems were astoundingly bad (and support for internal tools was almost entirely forbidden). At megacorp I absolutely decried any sort of home grown solution. But if Dropbox were to actually tackle the problems of internal testing and support and come up with mature solutions, why shouldn't they use them?
PostThisTooFast 8y ago
How does Dropbox manage to LOSE customers' files? That's a more pressing topic, I think.

94 points by dmicher 8y ago

A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) - which is interesting given most providers are writing new infrastructure code largely in Go (Spinnaker is another Python exception). I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebases at their scale?

amanzi 8y ago
Dropbox hired the creator of Python to help migrate their huge Python 2 codebase to Python 3. It would make sense for them to continue with their investment in Python.
lwf 8y ago
Dropbox sponsors most of the developers of http://mypy-lang.org/ http://mypy-lang.org/ , an optional static typing system for Python. I've found it to be hugely useful in safely working in a large Python codebase :) (NB: I work at Dropbox)
- mroche 8y ago
  Curious if you’ve ever tried Cython out. Not quite the same, but a semi-similar end goal to a degree. I started looking into the other day and it might provide some nice improvements to heavier applications.
jedberg 8y ago
Unless I'm mistaken, Guido still works for Dropbox. And as far as I know they still mostly do Python.
jensvdh 8y ago
Spinnaker is written in Java
jabl 8y ago
> A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) Sounds like a reasonable guess. > which is interesting given most providers are writing new infrastructure code largely in Go Dropbox apparently uses Go a lot, see e.g. https://about.sourcegraph.com/go/go-reliability-and-durability-at-dropbox-tammy-butow https://about.sourcegraph.com/go/go-reliability-and-durabili... They also use Rust for some performance-critical stuff, e.g. https://news.ycombinator.com/item?id=11282948 https://news.ycombinator.com/item?id=11282948 That being said, AFAIK originally Dropbox was written mostly in python. Probably there's still lots of that left.
- talonx 8y ago
  This is also borne out by the fact that they mention Celery.
danpalmer 8y ago
> which is interesting given most providers are writing new infrastructure code largely in Go I'm not sure this is true. There's a lot of Java around, Scala is pretty popular too, in many industries C/C++ are the norm still for this sort of code. Go might be the trend in open source infrastructure projects right now, but a significant amount of that is likely to be inertia from Docker and Kubernetes.

peterwwillis 8y ago

"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs." I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tools that support the software developers writing the business products. "Using the network database and configuration tool developed by our Network Reliability Engineering (NRE) team," Another custom tool? Network inventory and config management tools do exist already... "Rather than having engineers manually running tests using playbooks, Pirlo performed an automated sequential battery of tests that reduced the need for hands-on attention and concurrently increased diagnostic accuracy." Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system.

mrbanks 8y ago
Developers at these big companies get bored & assume they know better than a mature open source solution so like to reinvent a perfectly good wheel basically.
- inferiorhuman 8y ago
  Or they've actually used Jenkins (or any of these other suggested alternatives). I've used Jenkins personally and professionally. Most recently I've been rebuilding my own CI stack and never really gave much thought to going back to Jenkins. So I've been asking around and one of the only common complaints I've heard so far is that getting the initial configuration done is painful and generally orthogonal to automation. Plus the documentation is atrocious. No off-the-shelf product will be a perfect fit, but with CI software I was truly surprised at just how large the gaps were. So, sure, if you're Dropbox and you want to automate everything Jenkins is almost certainly not the right tool for the job. If Dropbox already had a supported, mature in-house job queue system, why not use it? Conversely at megacorp, they spent 4+ years claiming to work on deploying a Jenkins (CloudBees) cluster and still came up with bupkis. Our own internal job and message queuing systems were astoundingly bad (and support for internal tools was almost entirely forbidden). At megacorp I absolutely decried any sort of home grown solution. But if Dropbox were to actually tackle the problems of internal testing and support and come up with mature solutions, why shouldn't they use them?

PostThisTooFast 8y ago

How does Dropbox manage to LOSE customers' files? That's a more pressing topic, I think.