4 ms·

I have one opinion why this happens. Imagine you're in charge of Google Search in like 2005. You've got a killer product that could probably be run realistical

I have one opinion why this happens. Imagine you're in charge of Google Search in like 2005. You've got a killer product that could probably be run realistically by 50 extremely good engineers working efficiently. Take out server costs, what's your profit margin, like 99%? Well if you're the board, and you've got a 99% profit company, what do you do? On the one hand you can get rich and call it a day, but then how long before a competitor builds the same exact product and you lose half? Or lose everything if everyone switches? So you take a huge chunk that money and reinvest it with the hope of cementing your position or finding another product. At this stage there's tons of money flying around. Internal projects are started and stopped constantly. Extreme inefficiencies are tolerated, after all, as the business grows, plenty of people just want to get paid and do as little work as possible.

prepend 3y ago
I think it’s probably basic accounting math. Let’s say you can make $100B with 50 engineers at 99% profit margin. You pay your employees $1B and they are the highest paid in the world and all ecstatically happy and the stockholders make $99B and they are pretty happy too. The board decides they’d rather make $300B at 50% margins because that’s $150B to investors. So it’s lower margin, but they’ve already invested and they just want more money. Of course, I know many programmers that have insanely high profit margins with 1-4 employees and will never go public and no one except for an IRS computer will know they exist. I think corporations are usually pretty efficient. It’s just not always clear what they are efficient toward.
- tcgv 3y ago
  > I think corporations are usually pretty efficient. It’s just not always clear what they are efficient toward. This. People often examine things on a small scale and lack the context to understand why certain aspects appear counterintuitive. Once we gain a broader perspective, things begin to make more sense.
Cthulhu_ 3y ago
Yup; any department with money in their budget left over at the end of the budget period will see their budget reduced, even if they need it. And (I'm theorizing here, I know little of management layers) I'm confident that a manager's wage is a percentage of a department's budget. Therefore, it's in their interest to use up and ask for more budget constantly.
- KineticLensman 3y ago
  > And (I'm theorizing here, I know little of management layers) I'm confident that a manager's wage is a percentage of a department's budget Unlikely in a conventional organisation with salaried staff split into multiple depts / divisions. Bonuses may depend on the performance of a specific dept but base salaries typically have company-wide bands.
- robotcapital 3y ago
  > Yup; any department with money in their budget left over at the end of the budget period will see their budget reduced, even if they need it. Anecdote here. I've been a senior manager at small, medium, and large (ie FAANG) tech companies over the past 20 years and I've never run into this. Teams make a case for the budgets they need and those are approved (or not) based on the finance team's overall guidance and the return on investment. I'd be curious if others have have actually seen this "use it or lose it" mentality for budgets in practice. > And (I'm theorizing here, I know little of management layers) I'm confident that a manager's wage is a percentage of a department's budget. Therefore, it's in their interest to use up and ask for more budget constantly. Again, not my experience. Like you say, this would create perverse incentives which would quickly become apparent to a company's overall finances. I don't doubt these sorts of policy mistakes have happened at other companies in the past, but I'm doubtful that they're pervasive, or even common, in the tech industry.
onlyrealcuzzo 3y ago
50 engineers couldn't even keep Borg running, let alone keep all the machines maintained, let alone procure new ones to keep up with growth, let alone do any product development to get any growth, let alone keep up with the regulatory changes constantly coming in necessary to even operate.
- ZeroSolstice 3y ago
  50 SRE's sounds like they could keep borg running. There really isn't that much details on what borg is made up of outside of a 2015 research paper[1]. Just to address the parts you laid out. 1. Keep Borg running What would entail running? There are plenty of small teams that manage hundreds of servers and virtual machines through monitoring, deployment, networking, decom, etc lifecycle. This seems counter to purpose of cloud computing, commodity hardware and abstracted compute resources that can be deployed on-demand. Not to mention redundant sites, availability zones, <insert your cloud providers name for High Availability(HA) feature here>. Is this the part that SRE's maintain? 2. Machine maintenance From documentaries[2] and other comparable data center operations this would seem to be handled by on-site datacenter staff. Disk replacement, physical replacement, network cabling, etc. Without any other operational info I would doubt resources would just be dedicated to borg as its a bit of a overall technician site work. As for OS and software updates and again without any other info or insight would seem to fairly automated after passing testing or at least passed to another team that just handles updates. 3. Machine procurement Again commodity hardware or if its all custom still that would seem to be more of a EE task for build out and accounting for purchasing. Otherwise at Google's scale you would just get pre-populated racks delivered and replace the entire rack when 51% of the machines have failed. This doesn't really seem SRE or developer specific. It would also be happening for other services if they are already aren't abstracted from the underlying hardware/network layer. 4. Regulatory changes I'm not sure what this would be in relation to a job processing system? Is this checking where you are saving data? Seems like a feature that is built once. I agree that if you have follow-the-sun and need to be up with 5 (9's) 50 people in one time zone would be hard to ensure that but once the foundation is setup its just (50) people in a different location or timezone there isn't really anything too different about what they are doing. If you have additional information or insight that would interesting to hear about. [1] https://research.google/pubs/pub43438/ https://research.google/pubs/pub43438/ [2] https://www.youtube.com/watch?v=XZmGGAbHqa0 https://www.youtube.com/watch?v=XZmGGAbHqa0
mplewis 3y ago
Why do people keep spouting stuff like “I bet 50 people could run golden era Google search?” There’s no reason to believe this.
- rasz 3y ago
  WhatsApp
- ZeroSolstice 3y ago
  If you could expand on which parts you think would take more than 50 people to work on for a search engine that would be interesting to hear about. With the current cloud offerings hardware, processing, storage, high availability, etc are not really barriers to entry anymore and are offered as services. There look to be a number of search engine startups[1] some big, some small. [1] https://startupsavant.com/startups-to-watch/search-engine https://startupsavant.com/startups-to-watch/search-engine
- deleted 3y ago
  [deleted]
EVa5I7bHFq9mnYK 3y ago
>> Take out server costs Server costs were actually bulk of the costs, not engineers salaries.

prepend 3y ago

I think it’s probably basic accounting math. Let’s say you can make $100B with 50 engineers at 99% profit margin. You pay your employees $1B and they are the highest paid in the world and all ecstatically happy and the stockholders make $99B and they are pretty happy too. The board decides they’d rather make $300B at 50% margins because that’s $150B to investors. So it’s lower margin, but they’ve already invested and they just want more money. Of course, I know many programmers that have insanely high profit margins with 1-4 employees and will never go public and no one except for an IRS computer will know they exist. I think corporations are usually pretty efficient. It’s just not always clear what they are efficient toward.

tcgv 3y ago
> I think corporations are usually pretty efficient. It’s just not always clear what they are efficient toward. This. People often examine things on a small scale and lack the context to understand why certain aspects appear counterintuitive. Once we gain a broader perspective, things begin to make more sense.

Cthulhu_ 3y ago

Yup; any department with money in their budget left over at the end of the budget period will see their budget reduced, even if they need it. And (I'm theorizing here, I know little of management layers) I'm confident that a manager's wage is a percentage of a department's budget. Therefore, it's in their interest to use up and ask for more budget constantly.

KineticLensman 3y ago
> And (I'm theorizing here, I know little of management layers) I'm confident that a manager's wage is a percentage of a department's budget Unlikely in a conventional organisation with salaried staff split into multiple depts / divisions. Bonuses may depend on the performance of a specific dept but base salaries typically have company-wide bands.
robotcapital 3y ago
> Yup; any department with money in their budget left over at the end of the budget period will see their budget reduced, even if they need it. Anecdote here. I've been a senior manager at small, medium, and large (ie FAANG) tech companies over the past 20 years and I've never run into this. Teams make a case for the budgets they need and those are approved (or not) based on the finance team's overall guidance and the return on investment. I'd be curious if others have have actually seen this "use it or lose it" mentality for budgets in practice. > And (I'm theorizing here, I know little of management layers) I'm confident that a manager's wage is a percentage of a department's budget. Therefore, it's in their interest to use up and ask for more budget constantly. Again, not my experience. Like you say, this would create perverse incentives which would quickly become apparent to a company's overall finances. I don't doubt these sorts of policy mistakes have happened at other companies in the past, but I'm doubtful that they're pervasive, or even common, in the tech industry.

onlyrealcuzzo 3y ago

50 engineers couldn't even keep Borg running, let alone keep all the machines maintained, let alone procure new ones to keep up with growth, let alone do any product development to get any growth, let alone keep up with the regulatory changes constantly coming in necessary to even operate.

ZeroSolstice 3y ago
50 SRE's sounds like they could keep borg running. There really isn't that much details on what borg is made up of outside of a 2015 research paper[1]. Just to address the parts you laid out. 1. Keep Borg running What would entail running? There are plenty of small teams that manage hundreds of servers and virtual machines through monitoring, deployment, networking, decom, etc lifecycle. This seems counter to purpose of cloud computing, commodity hardware and abstracted compute resources that can be deployed on-demand. Not to mention redundant sites, availability zones, <insert your cloud providers name for High Availability(HA) feature here>. Is this the part that SRE's maintain? 2. Machine maintenance From documentaries[2] and other comparable data center operations this would seem to be handled by on-site datacenter staff. Disk replacement, physical replacement, network cabling, etc. Without any other operational info I would doubt resources would just be dedicated to borg as its a bit of a overall technician site work. As for OS and software updates and again without any other info or insight would seem to fairly automated after passing testing or at least passed to another team that just handles updates. 3. Machine procurement Again commodity hardware or if its all custom still that would seem to be more of a EE task for build out and accounting for purchasing. Otherwise at Google's scale you would just get pre-populated racks delivered and replace the entire rack when 51% of the machines have failed. This doesn't really seem SRE or developer specific. It would also be happening for other services if they are already aren't abstracted from the underlying hardware/network layer. 4. Regulatory changes I'm not sure what this would be in relation to a job processing system? Is this checking where you are saving data? Seems like a feature that is built once. I agree that if you have follow-the-sun and need to be up with 5 (9's) 50 people in one time zone would be hard to ensure that but once the foundation is setup its just (50) people in a different location or timezone there isn't really anything too different about what they are doing. If you have additional information or insight that would interesting to hear about. [1] https://research.google/pubs/pub43438/ https://research.google/pubs/pub43438/ [2] https://www.youtube.com/watch?v=XZmGGAbHqa0 https://www.youtube.com/watch?v=XZmGGAbHqa0

mplewis 3y ago

Why do people keep spouting stuff like “I bet 50 people could run golden era Google search?” There’s no reason to believe this.

rasz 3y ago
WhatsApp
ZeroSolstice 3y ago
If you could expand on which parts you think would take more than 50 people to work on for a search engine that would be interesting to hear about. With the current cloud offerings hardware, processing, storage, high availability, etc are not really barriers to entry anymore and are offered as services. There look to be a number of search engine startups[1] some big, some small. [1] https://startupsavant.com/startups-to-watch/search-engine https://startupsavant.com/startups-to-watch/search-engine
deleted 3y ago
[deleted]

EVa5I7bHFq9mnYK 3y ago

>> Take out server costs Server costs were actually bulk of the costs, not engineers salaries.