4 ms·

"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-

"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs." I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tools that support the software developers writing the business products. "Using the network database and configuration tool developed by our Network Reliability Engineering (NRE) team," Another custom tool? Network inventory and config management tools do exist already... "Rather than having engineers manually running tests using playbooks, Pirlo performed an automated sequential battery of tests that reduced the need for hands-on attention and concurrently increased diagnostic accuracy." Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system.

mrbanks 8y ago
Developers at these big companies get bored & assume they know better than a mature open source solution so like to reinvent a perfectly good wheel basically.
- inferiorhuman 8y ago
  Or they've actually used Jenkins (or any of these other suggested alternatives). I've used Jenkins personally and professionally. Most recently I've been rebuilding my own CI stack and never really gave much thought to going back to Jenkins. So I've been asking around and one of the only common complaints I've heard so far is that getting the initial configuration done is painful and generally orthogonal to automation. Plus the documentation is atrocious. No off-the-shelf product will be a perfect fit, but with CI software I was truly surprised at just how large the gaps were. So, sure, if you're Dropbox and you want to automate everything Jenkins is almost certainly not the right tool for the job. If Dropbox already had a supported, mature in-house job queue system, why not use it? Conversely at megacorp, they spent 4+ years claiming to work on deploying a Jenkins (CloudBees) cluster and still came up with bupkis. Our own internal job and message queuing systems were astoundingly bad (and support for internal tools was almost entirely forbidden). At megacorp I absolutely decried any sort of home grown solution. But if Dropbox were to actually tackle the problems of internal testing and support and come up with mature solutions, why shouldn't they use them?
  - glenngillen 8y ago
    Have you looked at https://buildkite.com/ https://buildkite.com/? A great UX and workflow for orchestrating tests, but the tests still run entirely in/on your own infra.
    - inferiorhuman 8y ago
      I have not. The continuous integration infrastructure at megacorp was merely a symptom of some major cultural problems that over the course of a few years just weren't going to be addressed to my satisfaction, so I left. Paying for Cloudbees was a painful exercise, and I think the guy that was pushing for it ended up getting shitcanned. After all, the whole point of deploying Jenkins was to avoid paying for TeamCity. For my personal stuff, I haven't looked at buildkite primarily because I'm looking for free, open source solutions. The secondary (but no less important) issue is that I'm also looking for coverage for *BSD. Buildkite lists FreeBSD support so, assuming it's not a Linux binary expected to run under the translation layer, they could presumably offer binaries for the other BSD variants. Right now I've got some buildbot workers setup on some bhyve VMs with df/net/openbsd and some jails with FreeBSD. The big thing I've noticed is that lots of tools will list FreeBSD support at least, but few of those claims are actually tested. Even fewer mention any of the other BSDs. For example: Elixir deployments on FreeBSD have been broken for months; Hashicorp put out a FreeBSD build of their vault ssh helper that didn't work by design -- but they had no idea because they never tried to validate the build; rust's libc crate lists FreeBSD, NetBSD, and OpenBSD as tier 1 platforms, but they're only testing the crate on an old version of FreeBSD (where they've disabled some key tests due to a misunderstanding rendering the tests not super useful)... plus they're not in a good position to deal with the C ABI changing between releases.
  - deleted 8y ago
    [deleted]
  - kohsuke 8y ago
    Hi, I'm Kohsuke, the creator of Jenkins. I'm sorry to hear that you had a bad experience. Would you be willing to letting me interview you so that I can learn where it failed your expectation? I'm honestly trying to learn where we can do things better, and often what's obvious to one person is completely incomprehensible to another. So I think this is a great opportunity for me to learn a fresh perspective. My contact information is in my personal profile.
- swish_bob 8y ago
  The irony of people saying essentially "Dropbox could have done this just by using open source tools" on hackernews, where people famously announced the Dropbox would fail because it was easy to reproduce using open source tools entertains me. Maybe, just maybe, the engineers in question considered mature open source tools and decided they didn't serve their needs.
deleted 8y ago
[deleted]
danpalmer 8y ago
> I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tools that support the software developers writing the business products. I think the part "Leveraging in-house primitives" is the important bit here. It sounds like they have an in-house task queue, so introducing another external one like Celery might add a lot of operational complexity rather than piggy backing on existing infrastructure. As for the reason for the existing infrastructure not being Celery, I suspect at Dropbox's scale tools like Celery start to break down. > Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system. They likely haven't written their own job queue, they've just used an existing one. Again, the operational overhead of Jenkins is unfortunately very non trivial.
- swish_bob 8y ago
  > Again, the operational overhead of Jenkins is unfortunately very non trivial. Not to mention the overhead of running Jenkins at the kind of scale they're talking about here. We're not talking about small numbers here.
  - peterwwillis 8y ago
    I don't know other people's experience with Jenkins, but I didn't find it to be much overhead. It's got all the capabilities you need to run at high scale, assuming you use it properly, like using the DSL and containers, and managing the configuration and deployment as code. Jenkins X, for example, is inherently cloud-based and dynamically scaling. If you need a generic job queue to grow with the organization, that's what it does. And you can even pay someone to set you up, reducing lead time. But there's also a dozen similar systems out there, so you're not even limited to this one. There's lots of choices and really no need to make a new one. Re-using an existing component to make a new tool is still an independent software development project, which will almost never be as cheap as integrating a finished tool.
chrisweekly 8y ago
>> "in-house primitives" ^ perfect band name

mrbanks 8y ago

Developers at these big companies get bored & assume they know better than a mature open source solution so like to reinvent a perfectly good wheel basically.

inferiorhuman 8y ago
Or they've actually used Jenkins (or any of these other suggested alternatives). I've used Jenkins personally and professionally. Most recently I've been rebuilding my own CI stack and never really gave much thought to going back to Jenkins. So I've been asking around and one of the only common complaints I've heard so far is that getting the initial configuration done is painful and generally orthogonal to automation. Plus the documentation is atrocious. No off-the-shelf product will be a perfect fit, but with CI software I was truly surprised at just how large the gaps were. So, sure, if you're Dropbox and you want to automate everything Jenkins is almost certainly not the right tool for the job. If Dropbox already had a supported, mature in-house job queue system, why not use it? Conversely at megacorp, they spent 4+ years claiming to work on deploying a Jenkins (CloudBees) cluster and still came up with bupkis. Our own internal job and message queuing systems were astoundingly bad (and support for internal tools was almost entirely forbidden). At megacorp I absolutely decried any sort of home grown solution. But if Dropbox were to actually tackle the problems of internal testing and support and come up with mature solutions, why shouldn't they use them?
- glenngillen 8y ago
  Have you looked at https://buildkite.com/ https://buildkite.com/? A great UX and workflow for orchestrating tests, but the tests still run entirely in/on your own infra.
  - inferiorhuman 8y ago
    I have not. The continuous integration infrastructure at megacorp was merely a symptom of some major cultural problems that over the course of a few years just weren't going to be addressed to my satisfaction, so I left. Paying for Cloudbees was a painful exercise, and I think the guy that was pushing for it ended up getting shitcanned. After all, the whole point of deploying Jenkins was to avoid paying for TeamCity. For my personal stuff, I haven't looked at buildkite primarily because I'm looking for free, open source solutions. The secondary (but no less important) issue is that I'm also looking for coverage for *BSD. Buildkite lists FreeBSD support so, assuming it's not a Linux binary expected to run under the translation layer, they could presumably offer binaries for the other BSD variants. Right now I've got some buildbot workers setup on some bhyve VMs with df/net/openbsd and some jails with FreeBSD. The big thing I've noticed is that lots of tools will list FreeBSD support at least, but few of those claims are actually tested. Even fewer mention any of the other BSDs. For example: Elixir deployments on FreeBSD have been broken for months; Hashicorp put out a FreeBSD build of their vault ssh helper that didn't work by design -- but they had no idea because they never tried to validate the build; rust's libc crate lists FreeBSD, NetBSD, and OpenBSD as tier 1 platforms, but they're only testing the crate on an old version of FreeBSD (where they've disabled some key tests due to a misunderstanding rendering the tests not super useful)... plus they're not in a good position to deal with the C ABI changing between releases.
- deleted 8y ago
  [deleted]
- kohsuke 8y ago
  Hi, I'm Kohsuke, the creator of Jenkins. I'm sorry to hear that you had a bad experience. Would you be willing to letting me interview you so that I can learn where it failed your expectation? I'm honestly trying to learn where we can do things better, and often what's obvious to one person is completely incomprehensible to another. So I think this is a great opportunity for me to learn a fresh perspective. My contact information is in my personal profile.
swish_bob 8y ago
The irony of people saying essentially "Dropbox could have done this just by using open source tools" on hackernews, where people famously announced the Dropbox would fail because it was easy to reproduce using open source tools entertains me. Maybe, just maybe, the engineers in question considered mature open source tools and decided they didn't serve their needs.

deleted 8y ago

[deleted]

danpalmer 8y ago

> I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tools that support the software developers writing the business products. I think the part "Leveraging in-house primitives" is the important bit here. It sounds like they have an in-house task queue, so introducing another external one like Celery might add a lot of operational complexity rather than piggy backing on existing infrastructure. As for the reason for the existing infrastructure not being Celery, I suspect at Dropbox's scale tools like Celery start to break down. > Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system. They likely haven't written their own job queue, they've just used an existing one. Again, the operational overhead of Jenkins is unfortunately very non trivial.

swish_bob 8y ago
> Again, the operational overhead of Jenkins is unfortunately very non trivial. Not to mention the overhead of running Jenkins at the kind of scale they're talking about here. We're not talking about small numbers here.
- peterwwillis 8y ago
  I don't know other people's experience with Jenkins, but I didn't find it to be much overhead. It's got all the capabilities you need to run at high scale, assuming you use it properly, like using the DSL and containers, and managing the configuration and deployment as code. Jenkins X, for example, is inherently cloud-based and dynamically scaling. If you need a generic job queue to grow with the organization, that's what it does. And you can even pay someone to set you up, reducing lead time. But there's also a dozen similar systems out there, so you're not even limited to this one. There's lots of choices and really no need to make a new one. Re-using an existing component to make a new tool is still an independent software development project, which will almost never be as cheap as integrating a finished tool.

chrisweekly 8y ago

>> "in-house primitives" ^ perfect band name