3 ms·

So what's the end game here, is Dropbox going to keep building out an internal Kubernetes-like platform with Atlas, or do they plan to eventually just move to k

So what's the end game here, is Dropbox going to keep building out an internal Kubernetes-like platform with Atlas, or do they plan to eventually just move to k8s? I noticed this line in particular: "We evaluated using off-the-shelf solutions to run the platform. But in order to de-risk our migration and ensure low engineering costs, it made sense for us to continue hosting services on the same deployment orchestration platform used by the rest of Dropbox." It sounds like they acknowledge they're reinventing a lot of stuff but for now are sticking to their internal platform. Perhaps Atlas is a half-step then to get teams used to owning and running their code as isolated services. But everything I read that they built in Atlas--isolated orchestrated services, gRPC load balancing, canary deployments, horizontal scaling, etc.--are bog standard features of Kubernetes today. I'd be very leery of maintaining a bespoke Kubernetes-like platform in 2021 and beyond--in some ways it seems like it's just shifting the monolith technical debt into an internal Atlas platform team's technical debt. What's the plan to get rid of that debt for good I wonder? This hurdle shows there's already some cracks in the idea of long-term Atlas too: "While splitting up Metaserver had wins in production, it was infeasible to spin up 200+ Python processes in our integration testing framework. We decided to merge the processes back into a monolith for local development and testing purposes. We also built heavy integration with our Bazel rules, so that the merging happens behind the scene and developers can reference Atlasservlets as regular services." If I read that right does it really mean the first time a developer's code is run like it will run in production is when it goes out to canary deployment? I.e. integration tests are done in a local monolith instead of setting up a mini-prod cluster. That seems a bit nerve-racking as a dev to have no way to really test the service until bits are hitting user requests. In the k8s world a ton of work has been put into tooling and processes to make setting up local clusters easily. It's a shame to not have something similar for Atlas.

hayst4ck 6y ago
Counter considerations would be: What is the delta between out of box solutions and current solutions? What is the cost of the migration? For what period will two services be supported simultaneously? Will development effort continue on the previous service while the new one is created? Will the new service successfully solve problems the old one didn't? What happens when Kubernetes is insufficient for a task or has a critical bug that only appears at scale? How will people be onboarded into the new system? Will the team handling how services run perform the migration or will the teams who own services perform the migration? How much time should be spent experimenting with new things compared to fixing bugs/adding requested features? > in some ways it seems like it's just shifting the monolith technical debt into an internal Atlas platform team's technical debt. This is a key insight into the monolith problem. How does a monolith become poor and unmaintainable? A monolith becomes poor quality and unmaintainable when there is no entity enforcing architectural simplicity. It becomes unmaintainable when there is no team focusing solely on how the monolith functions. It becomes unmaintainable if there is no entity capable of saying "no" to a product engineer. A monolith in a company with weak leadership is a tragedy of the commons where everyone takes from the commons by adding complexity and there is no governing entity to ensure that the commons remains viable. The exact statement you made is the key strength of this approach. Where there was a vacuum of responsibility before (monolith technical debt), a team has been created with direct responsibility and authority creating a governing force over that technical debt/overall complexity and therefore an entity directly responsible for improving it. This is a key first step. Atlas appears to be a compromise solution rather than an ideal end state.
- spondyl 6y ago
  > A monolith becomes poor quality and unmaintainable when there is no entity enforcing architectural simplicity. Having worked in a company where no single team "owned" the monolith, the term "communally owned" tended to come up. It was generally understood within the platform teams that if everyone owns it then in reality, no one owns it :)
  - peterhunt 6y ago
    yes, this is why you need a dedicated team who owns the quality and scalability of the monolith once you reach a certain size. many companies call this team "product infrastructure".
  - hayst4ck 6y ago
    Exactly. The greater problem is that the ways to fail monolith specific work vastly outnumbers the way to succeed. Furthermore, the primary work involved is months to years worth of horrible unattractive refactoring. Then this is compounded by it being very hard to measure success. So you run into a situation that makes principal engineers run or look for the door. Rather than working on an actually hard problem, they would rather pad their resumes with increasingly complex novel work, further detracting from the commons. So not only is it an un-owned common, but senior engineer alignment is directly against work involving the root problem because no one in their right mind would be responsible or accountable for something with so few avenues to success. Not only is senior engineering alignment individually against being directly responsible for the monolith, but managerial leadership is as well. You have infrastructure with infrastructure concerns, and product worried about shipping product, and then you have to ask where this monolith responsible team is. Pushing code, monitoring, capacity management, oncall, etc. are all traditionally under the infra umbrella. Business logic is traditionally under the product umbrella. So who takes responsibility for the monolith, the chief repository of business logic? A monolith is a marriage between the two entities. Each individual product team doesn't want to do monolith work, because it makes it harder to ship the features they promised. Infra doesn't want to do monolith work because it is business logic code that needs to be refactored. So no infra team and no product team will directly feel responsible. Senior managerial leadership is at the mercy of senior engineering leadership, and therefore also cannot claim responsibility for the monolith. If you travel up the management hierarchy, that leaves roughly 4 people capable of doing anything about this most critical problem. The CEO, who has ultimate responsibility for all problems, the VP of infra, who could choose to claim responsibility for this area, the VP of product, who could choose to claim responsibility for this area, or the CTO who is the most directly responsible person for solving this problem as the de facto mediator of product and infra concerns and responsibilities. The answer is probably inventing a 3rd VP responsible for the area between product and infra and giving them proper authority.

hayst4ck 6y ago

Counter considerations would be: What is the delta between out of box solutions and current solutions? What is the cost of the migration? For what period will two services be supported simultaneously? Will development effort continue on the previous service while the new one is created? Will the new service successfully solve problems the old one didn't? What happens when Kubernetes is insufficient for a task or has a critical bug that only appears at scale? How will people be onboarded into the new system? Will the team handling how services run perform the migration or will the teams who own services perform the migration? How much time should be spent experimenting with new things compared to fixing bugs/adding requested features? > in some ways it seems like it's just shifting the monolith technical debt into an internal Atlas platform team's technical debt. This is a key insight into the monolith problem. How does a monolith become poor and unmaintainable? A monolith becomes poor quality and unmaintainable when there is no entity enforcing architectural simplicity. It becomes unmaintainable when there is no team focusing solely on how the monolith functions. It becomes unmaintainable if there is no entity capable of saying "no" to a product engineer. A monolith in a company with weak leadership is a tragedy of the commons where everyone takes from the commons by adding complexity and there is no governing entity to ensure that the commons remains viable. The exact statement you made is the key strength of this approach. Where there was a vacuum of responsibility before (monolith technical debt), a team has been created with direct responsibility and authority creating a governing force over that technical debt/overall complexity and therefore an entity directly responsible for improving it. This is a key first step. Atlas appears to be a compromise solution rather than an ideal end state.

spondyl 6y ago
> A monolith becomes poor quality and unmaintainable when there is no entity enforcing architectural simplicity. Having worked in a company where no single team "owned" the monolith, the term "communally owned" tended to come up. It was generally understood within the platform teams that if everyone owns it then in reality, no one owns it :)
- peterhunt 6y ago
  yes, this is why you need a dedicated team who owns the quality and scalability of the monolith once you reach a certain size. many companies call this team "product infrastructure".
- hayst4ck 6y ago
  Exactly. The greater problem is that the ways to fail monolith specific work vastly outnumbers the way to succeed. Furthermore, the primary work involved is months to years worth of horrible unattractive refactoring. Then this is compounded by it being very hard to measure success. So you run into a situation that makes principal engineers run or look for the door. Rather than working on an actually hard problem, they would rather pad their resumes with increasingly complex novel work, further detracting from the commons. So not only is it an un-owned common, but senior engineer alignment is directly against work involving the root problem because no one in their right mind would be responsible or accountable for something with so few avenues to success. Not only is senior engineering alignment individually against being directly responsible for the monolith, but managerial leadership is as well. You have infrastructure with infrastructure concerns, and product worried about shipping product, and then you have to ask where this monolith responsible team is. Pushing code, monitoring, capacity management, oncall, etc. are all traditionally under the infra umbrella. Business logic is traditionally under the product umbrella. So who takes responsibility for the monolith, the chief repository of business logic? A monolith is a marriage between the two entities. Each individual product team doesn't want to do monolith work, because it makes it harder to ship the features they promised. Infra doesn't want to do monolith work because it is business logic code that needs to be refactored. So no infra team and no product team will directly feel responsible. Senior managerial leadership is at the mercy of senior engineering leadership, and therefore also cannot claim responsibility for the monolith. If you travel up the management hierarchy, that leaves roughly 4 people capable of doing anything about this most critical problem. The CEO, who has ultimate responsibility for all problems, the VP of infra, who could choose to claim responsibility for this area, the VP of product, who could choose to claim responsibility for this area, or the CTO who is the most directly responsible person for solving this problem as the de facto mediator of product and infra concerns and responsibilities. The answer is probably inventing a 3rd VP responsible for the area between product and infra and giving them proper authority.