7 ms·

I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I n

by aeyes 5y ago

I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because working on this changes absolutely nothing for the end user, it is usually some internal system related to data storage or processing which can't cope anymore. Imagine that you are Amazon and for some scalability reason you have to rewrite the storage layer of your product catalog. Immediately you have a million problems like data migration, reporting, data ingestion, making it work with all the related systems like search, recommendations, reviews and so on. And even if you get the ball rolling you have to work across dozens of different teams which can be hard because naturally people resist change. Why do large sites like Facebook, Amazon, Twitter and Instagram all essentially look the same after 10 years but some of them now have 10x the amount of engineers? I think they have so much data and so many dependencies between parts of the system that any fundamental change is extremely hard to pull off. They even cut back on features like API access. But I am pretty sure that most of them have rewritten the whole thing at least 3 times.

andai 5y ago
Heh, I wish they still looked the same. They added an order of magnitude of HTML and JS bloat while removing functionality.
ClumsyPilot 5y ago
"That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because working on this changes absolutely nothing for the end user" It seems to be the same story in fiels of Infrastructure maintenance, Aircraft design (boeing Max), and mortgage CDOs (2008). Was it always like this or the new management doesn not care untill something explodes?
- imachine1980_ 5y ago
  a manufacturing company is designed the ground up to works whit machine but isn't the same whit software, is hard to understand that triple data isn't only triple server but a totaly different software stack, and exponentially more complex is not only put more factories like textile.
  - fragmede 5y ago
    There's still order of magnitude change analogies to real world processes, if people are willing to listen (which is the hard part). Use something that everybody can understand, like making pancakes or waffles or an omelet. Going from making 1 by hand, every 4 minutes at home for your family, to 1,000 pancakes per minute at a factory is obviously going to take a better system. You can scale horizontally, and do the equivalent of putting more VMs behind the load balancer, and hire 4,000+ people to cook, but you still need to have/make that load balancer in the first place for even that to work. That's the tip of iceberg when going from 1 per 4 minutes to 1,000 per minute though. How do you make and distribute enough batter for that system, and plating and serving that is going to take a pub/sub bus, err, conveyor belt to support the cooks' output. Again though, you still gotta make that kafka queue, err, conveyor belt, plus the maintenance for that is going to a team of people if you need the conveyor belt to operate 24/7/52. If your standards are so high that the system can never go down for more than 52.6 minutes per year or 13.15 minutes per quarter, then that team needs to consist of highly-trained and smart (read: expensive) people to call when the system breaks in the middle of the night.
pojzon 5y ago
Had that issue in my previous job. Higher management decided to migrate our properitary vendor locked platform from one cloud provider to the other one. Majority of migration fell on a single platform team that was constantly struggling with attricion. Unfortunately I was not able (neither our architects) to explain the higherups that we need bigger team and overall way more resources to pull that off. Hope that someone that comes after me will be able to make the miracle happen.
ratww 5y ago
> Why do large sites like Facebook, Amazon, Twitter and Instagram all essentially look the same after 10 years but some of them now have 10x the amount of engineers? I think they have so much data and so many dependencies between parts of the system that any fundamental change is extremely hard to pull off. They even cut back on features like API access. But I am pretty sure that most of them have rewritten the whole thing at least 3 times. I used to work on a Unicorn a few years ago, and this hits close to home. From 2016 to 2020, the pages didn't change one single pixel, however there we had 400 more engineers working on the code and three stack iterations: full-stack PHP, PHP backend + React SSR frontend, Java backend + [redacted] SSR frontend (redacted because only two popular companies use this framework). All were rewrites, and those rewrites were justified because none of them was ever stable, the site was constantly going offline. However each rewrite just added more bloat and failure points. At some point the three of them were running in tandem: PHP for legacy customers, another as main and another on an A/B test. (Yeah, it was a dysfunctional environment and I obviously quit).
- axiosgunnar 5y ago
  > Yeah, it was a dysfunctional environment and I obviously quit What do you think could management have done better to make it not dysfunctional and have people quitting?
  - deleted 5y ago
    [deleted]
  - ratww 5y ago
    I think just common sense and less bullshit rationalisation would have been enough. They had a billion dollars in cash to burn, so they hired more than they needed. They should have hired as needed, not as requested by Masayoshi Son. They shouldn't be so dogmatic. Some teams were too overworked, most were underworked (which means over-engineering will ensue), but no mobility was allowed because "ideally teams have N people". They shouldn't be so dogmatic pt 2. Services were one-per-team, instead of one-per-subject. So yeah, our internal tool for putting balloons and clowns into images lived together with the authentication micro-service, because it's the same team. Rewriting everything twice without analysis was wrong. The rewrites were because previous versions were "too complex" and too custom-made but newer ones had an even more complex architecture, but "this time it's right, software sometimes need complexity". Believing that some things were terrible would have gone a long way. Launching the main node.js server locally would take 10 to 20 minutes to launch, while something of the same complexity would often take about 2 or 3 seconds. Of course it would blow up in production! Maybe try to fix instead of ordering another rewrite. They were good people, I miss the company and still use the product, but it didn't need to be like this.
notimetorelax 5y ago
I usually move on to a different project/team/company when it gets to this. E.g. my new team builds a new product that grows like crazy and has its own set of challenges. I prefer to be deliver immediate customer value vs. long term hard to sell and hard to project the value work.
fxtentacle 5y ago
My impression has always been that FAANG need lots of engineers because the 10xers refuse to work there. I've seen plenty of really scalable systems being built by a small core team of people who know what they are doing. FAANG instead seem to be more into chasing trends, inventing new frameworks, rewriting to another more hip language, etc. I would have no idea how to coordinate 200 engineers. But then again, I have never worked on a project that truly needed 50+ engineers. "Imagine that you are Amazon and for some scalability reason you have to rewrite the storage layer of your product catalog." Probably that's 4 friends in a basement, similar to the core Android team ;)
- danny_taco 5y ago
  Your impression comes from the fact that you have not worked at larger teams, as you said so yourself. It's relatively easy to build something scalable from the beginning if you know what you need to build and if you are not already handling large amounts of traffic and data. It's a whole different ballgame to build on top of an existing complex system already in production that was made to satisfy the needs at the time it was built but it now needs to support other features, bug fixes and supporting existing features but at scale while having 50+ engineers not step on each other and not break each others code in the process. 4 friends in the basement will not achieve more than 50+ engineers in this scenario, even when considering the inefficiencies of the difficulty in communication that come along with so many minds working on the same thing.
  - ratww 5y ago
    GP said they have never work on something that truly needed 50+ engineers. Truly being the keyword here IMO. I have worked on a 1000+ engineer project and another that was 500+, but I'm on the same boat as GP. Both of those didn't needed 50+, and the presence of the extra 950/450 caused several communication, organisational and architectural issues that became impossible to fix on the long term. So I can definitely see where they're coming from.
    - exikyut 5y ago
dasil003 5y ago
I don't know your specifics, but I have worked on some large scale architecture changes, and 200 engineers + 2 year feature freeze is generally not a reasonable ask. In practice you need to find an incremental path with validation and course correction along the way to limit the amount of concurrent change in flight at any moment. If you don't do this run a very high risk of the entire initiative collapsing under its own weight. Assuming your estimation is more or less correct and it really is a 400 eng-year project, then you also need political capital as well as technical leadership to make it happen. There are lots of companies where a smart engineer can see a potential path out of a local maximum, but the org structure and lack of technical leadership in the highest ranks means that the problem is effectively intractable.
trhway 5y ago
>I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future sounds like a typical massive rewrite project. They almost never succeed, many fail outright and most hardly even reach the functionality/performance/etc. level of the stuff the rewrite was supposed to replace. 2-4 years is typical for such glorious attempt before being closed or folded into something else. Management in general likes such projects, and they declare victory usually around 2 years mark and move on on the wave of the supposed success before reality hits the fan. >to convince anyone to take engineers out of product development. that means raiding someone's budget. Not happening :) New glorious effort needs new glorious budget - that is what management likes and not doing much more on the same budget as you're basically suggesting (i.e. i'm sure you'll get much more traction if you restate your proposal as "to hire 200 more engineers ..." because that way you'll be putting serious technical foundation for some mid-managers to grow :). You're approaching this as an engineer and thus failing in what is the management game (or as Sun Tzu was pointing out one has to understand the enemy).

by aeyes 5y ago

andai 5y ago

Heh, I wish they still looked the same. They added an order of magnitude of HTML and JS bloat while removing functionality.

ClumsyPilot 5y ago

"That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because working on this changes absolutely nothing for the end user" It seems to be the same story in fiels of Infrastructure maintenance, Aircraft design (boeing Max), and mortgage CDOs (2008). Was it always like this or the new management doesn not care untill something explodes?

imachine1980_ 5y ago
a manufacturing company is designed the ground up to works whit machine but isn't the same whit software, is hard to understand that triple data isn't only triple server but a totaly different software stack, and exponentially more complex is not only put more factories like textile.
- fragmede 5y ago
  There's still order of magnitude change analogies to real world processes, if people are willing to listen (which is the hard part). Use something that everybody can understand, like making pancakes or waffles or an omelet. Going from making 1 by hand, every 4 minutes at home for your family, to 1,000 pancakes per minute at a factory is obviously going to take a better system. You can scale horizontally, and do the equivalent of putting more VMs behind the load balancer, and hire 4,000+ people to cook, but you still need to have/make that load balancer in the first place for even that to work. That's the tip of iceberg when going from 1 per 4 minutes to 1,000 per minute though. How do you make and distribute enough batter for that system, and plating and serving that is going to take a pub/sub bus, err, conveyor belt to support the cooks' output. Again though, you still gotta make that kafka queue, err, conveyor belt, plus the maintenance for that is going to a team of people if you need the conveyor belt to operate 24/7/52. If your standards are so high that the system can never go down for more than 52.6 minutes per year or 13.15 minutes per quarter, then that team needs to consist of highly-trained and smart (read: expensive) people to call when the system breaks in the middle of the night.

pojzon 5y ago

Had that issue in my previous job. Higher management decided to migrate our properitary vendor locked platform from one cloud provider to the other one. Majority of migration fell on a single platform team that was constantly struggling with attricion. Unfortunately I was not able (neither our architects) to explain the higherups that we need bigger team and overall way more resources to pull that off. Hope that someone that comes after me will be able to make the miracle happen.

ratww 5y ago

> Why do large sites like Facebook, Amazon, Twitter and Instagram all essentially look the same after 10 years but some of them now have 10x the amount of engineers? I think they have so much data and so many dependencies between parts of the system that any fundamental change is extremely hard to pull off. They even cut back on features like API access. But I am pretty sure that most of them have rewritten the whole thing at least 3 times. I used to work on a Unicorn a few years ago, and this hits close to home. From 2016 to 2020, the pages didn't change one single pixel, however there we had 400 more engineers working on the code and three stack iterations: full-stack PHP, PHP backend + React SSR frontend, Java backend + [redacted] SSR frontend (redacted because only two popular companies use this framework). All were rewrites, and those rewrites were justified because none of them was ever stable, the site was constantly going offline. However each rewrite just added more bloat and failure points. At some point the three of them were running in tandem: PHP for legacy customers, another as main and another on an A/B test. (Yeah, it was a dysfunctional environment and I obviously quit).

axiosgunnar 5y ago
> Yeah, it was a dysfunctional environment and I obviously quit What do you think could management have done better to make it not dysfunctional and have people quitting?
- deleted 5y ago
  [deleted]
- ratww 5y ago
  I think just common sense and less bullshit rationalisation would have been enough. They had a billion dollars in cash to burn, so they hired more than they needed. They should have hired as needed, not as requested by Masayoshi Son. They shouldn't be so dogmatic. Some teams were too overworked, most were underworked (which means over-engineering will ensue), but no mobility was allowed because "ideally teams have N people". They shouldn't be so dogmatic pt 2. Services were one-per-team, instead of one-per-subject. So yeah, our internal tool for putting balloons and clowns into images lived together with the authentication micro-service, because it's the same team. Rewriting everything twice without analysis was wrong. The rewrites were because previous versions were "too complex" and too custom-made but newer ones had an even more complex architecture, but "this time it's right, software sometimes need complexity". Believing that some things were terrible would have gone a long way. Launching the main node.js server locally would take 10 to 20 minutes to launch, while something of the same complexity would often take about 2 or 3 seconds. Of course it would blow up in production! Maybe try to fix instead of ordering another rewrite. They were good people, I miss the company and still use the product, but it didn't need to be like this.

notimetorelax 5y ago

I usually move on to a different project/team/company when it gets to this. E.g. my new team builds a new product that grows like crazy and has its own set of challenges. I prefer to be deliver immediate customer value vs. long term hard to sell and hard to project the value work.

fxtentacle 5y ago

My impression has always been that FAANG need lots of engineers because the 10xers refuse to work there. I've seen plenty of really scalable systems being built by a small core team of people who know what they are doing. FAANG instead seem to be more into chasing trends, inventing new frameworks, rewriting to another more hip language, etc. I would have no idea how to coordinate 200 engineers. But then again, I have never worked on a project that truly needed 50+ engineers. "Imagine that you are Amazon and for some scalability reason you have to rewrite the storage layer of your product catalog." Probably that's 4 friends in a basement, similar to the core Android team ;)

danny_taco 5y ago
Your impression comes from the fact that you have not worked at larger teams, as you said so yourself. It's relatively easy to build something scalable from the beginning if you know what you need to build and if you are not already handling large amounts of traffic and data. It's a whole different ballgame to build on top of an existing complex system already in production that was made to satisfy the needs at the time it was built but it now needs to support other features, bug fixes and supporting existing features but at scale while having 50+ engineers not step on each other and not break each others code in the process. 4 friends in the basement will not achieve more than 50+ engineers in this scenario, even when considering the inefficiencies of the difficulty in communication that come along with so many minds working on the same thing.
- ratww 5y ago
  GP said they have never work on something that truly needed 50+ engineers. Truly being the keyword here IMO. I have worked on a 1000+ engineer project and another that was 500+, but I'm on the same boat as GP. Both of those didn't needed 50+, and the presence of the extra 950/450 caused several communication, organisational and architectural issues that became impossible to fix on the long term. So I can definitely see where they're coming from.
  - exikyut 5y ago

dasil003 5y ago

I don't know your specifics, but I have worked on some large scale architecture changes, and 200 engineers + 2 year feature freeze is generally not a reasonable ask. In practice you need to find an incremental path with validation and course correction along the way to limit the amount of concurrent change in flight at any moment. If you don't do this run a very high risk of the entire initiative collapsing under its own weight. Assuming your estimation is more or less correct and it really is a 400 eng-year project, then you also need political capital as well as technical leadership to make it happen. There are lots of companies where a smart engineer can see a potential path out of a local maximum, but the org structure and lack of technical leadership in the highest ranks means that the problem is effectively intractable.

trhway 5y ago

>I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future sounds like a typical massive rewrite project. They almost never succeed, many fail outright and most hardly even reach the functionality/performance/etc. level of the stuff the rewrite was supposed to replace. 2-4 years is typical for such glorious attempt before being closed or folded into something else. Management in general likes such projects, and they declare victory usually around 2 years mark and move on on the wave of the supposed success before reality hits the fan. >to convince anyone to take engineers out of product development. that means raiding someone's budget. Not happening :) New glorious effort needs new glorious budget - that is what management likes and not doing much more on the same budget as you're basically suggesting (i.e. i'm sure you'll get much more traction if you restate your proposal as "to hire 200 more engineers ..." because that way you'll be putting serious technical foundation for some mid-managers to grow :). You're approaching this as an engineer and thus failing in what is the management game (or as Sun Tzu was pointing out one has to understand the enemy).