6 ms·
There is something I don't understand about the way those big companies work. From what I can see they are providing software and services. Even the biggest s
by stephc_int13 4y ago
There is something I don't understand about the way those big companies work.
From what I can see they are providing software and services.
Even the biggest software teams I have seen were rarely larger than 50-80 people.
And they were working on large complex codebase for AAA games.
Of course you need support, admin, HR, and sales, and R&D on new projects, but still.
How can they manage to productively use the time of 10000+ people?
What are they doing?
- randomUserHere 4y agoSo many times I see this kind of comments were people totally ignore the complexities that comes with large scale projects, huge complex legacy codebases that need to be maintained. What they see is a simple website that can be build by 5 people startup team in a day.
- acatton 4y ago> Even the biggest software teams I have seen were rarely larger than 50-80 people. And they were working on large complex codebase for AAA games. […] How can they manage to productively use the time of 10000+ people? I work for a 100k-employees company. According to the internet, a third of it are engineers.[1] I think I can explain. First of all, I think the AAA game example is bad. Most of my acquaintances working in the gaming industry work long hours for a crappy salary compared to their skill, because most of them are passionate. Most business software is written by overpaid engineers who see the whole thing as a 9to5 job. So you can already divide the amount of hour worked per employees by at least 1.5 if not 2. Even then, if you take something which seems as simple as Shopify the system can be made complex for managers and directors to justify headcount. It's an e-commerce website, right? So you can split 100 engineers over 5 teams. 10 "production engineers" (or SRE/DevOps/SysAdmins) and 10 software engineers for each team: - Product search team; - Payment team; - Authentication team; - Admin UI team; - CRM team. Once you have 100 engineers, you can multiply your headcount by 10 by just splitting each of these teams into five more. If you take the Payment team, you can create a credit card team, a paypal team, an alipay team, … For product search, you can create a team to manage just the suggestions of the search bar, another team which optimizes term matching with products, another team which does the list of related products on the side bar, … I just found jobs for 1,000 engineers, right at the top of my head! Now you have so many engineers and so many customers, you also need internal tools. You need an internal tool to manage customer refunds, customer support, etc… You need HR tools for vacations, performance review, … Of course all of this comes with waste, but who cares as long as you're offsetting it with your revenue? ¯\_(ツ)_/¯ Anyway, this is why I will never be a business person :) . [1] https://earthweb.com/how-many-software-engineers-does-google-have/ https://earthweb.com/how-many-software-engineers-does-google...
- rekrsiv 4y agoI like this perspective a lot. I just got laid off from a ~1.5k-employees private company that applied the same method to split up a large monolithic cash cow (crafted from scratch by a small clique of hackers who met early in life) into microservice and microfrontend internal products because it looked like a good long term plan in Q2 of 2020 and we thought we could scale that forever and start selling our dog food as a service. Something went wrong, and we hit a ceiling very early. The amount of waste that was generated by the rigid management ecosystem that was created to sustain such a large scale up (for tracking purposes) became so strict that any and all attempts to use the scientific method to solve anything by the then-outnumbered engineering staff became impossible to justify to any non-engineering roles. If the task didn't fit in a 2 week sprint, it couldn't be planned. If it can't be planned to be shipped from scratch in a 2 week sprint, from analysis by both your BE and FE devs to both of them shipping at the same time by end of the sprint, it couldn't be done. If you needed to solve for "what do people wish they could buy?" instead of optimizing for "what are the most humans currently using?", it couldn't be done. We constantly regretted not sticking to our engineering principles. The system collapsed. A lot of nice-to-have-but-working-at-a-distance positions were eliminated, as well as a few individual contributors that management hadn't realized were important to their core functions but were hired too recently and didn't have time to adjust to the point system. I wish I could tell you what happened next, but the layoff happened. Most of us thought it would happen because management would finally realize the (human) system was designed poorly and without a good feedback loop while growth happened. But it turns out they might have only reverted to a previous commit of the organization structure and are still intent on trying again with the same rulebook. Slower but infinite growth is still the objective, excessive tracking is still the norm. Their marketing had been doing poorly in recent years, and the layoff wasn't even mentioned in the news, but it seemed like it affected a lot of devs in my neck of the woods despite the near complete lack of media coverage. Does this happen often? Is this what scaling up always looks like?
- specialist 4y agoParaphrasing, from memory: Happy marriages are boring; They all look the same. Whereas each unhappy marriage is uniquely unhappy.
- nostrademons 4y agoYour example is pretty illuminating. AAA games are one-and-done: you ship, and then the codebase is basically frozen, and you make money off expansions. The other area where team sizes top out at 50-100 people is desktop software, where (barring future versions) it's also one-and-done. You sell a software package, money comes in, that's it. The way the calculus works for Internet-based SaaS is: say you have revenues of $4B (like Shopify), and an engineer costs $400K. If the engineer works on a project that improves sales by 0.01%, it's profitable. If a team of 10 engineers works on a project that improves sales by 0.1%, or 100 engineers improving sales by 1%, it's profitable. How many 0.1% opportunities do you see in the product? How many 1% opportunities? What if you add risk? An engineer might fail to see any effect on 4 of their projects, but if they improve sales by 0.1% on the 5th, they've still made the company twice as much as they cost. It's hill-climbing. There are usually lots of 0.1% opportunities available with a big worldwide SaaS. Things like localization and entering new markets will generate way more than that; even things like moving around CTAs and changing colors will usually generate more in profit than the engineer costs. Also add to that diminishing marginal productivity of each engineer. In a team of 5 moving a button around takes an afternoon. In a team of 1000 it often takes a couple quarters, because you need to get sign-offs from all the teams that the code touches, run it through QA, and gather metrics to show you didn't regress anything. The engineer is still profitable, because the effect of their change multiplied across millions of users is way more than their salary costs. But they're not as profitable, because they work so slowly.
- stephc_int13 4y agoI've also seen this kind of operations in mobile games. At Supercell, they have some old titles generating around 400K of revenue per day, with a minimal team working on new features, fixing bugs, looking after numbers and trends, managing ads... All of that with less than 15 people. Of course this is not the same scale, but they have a LOT of daily users, and at their peak the company was about 200 people. And I've been there, I can tell you that there is ample slack, most of them are barely working.
- majani 4y ago
- echelon 4y ago> What are they doing? Microservices. I haven't worked there, but I've worked at incredibly similar companies with the same scale. Every single business function split into fine grained services, with teams of size 5-20 (larger teams have split services, oncall, etc.) Auth, API gateway. Layers of devops teams. Traffic, monitoring, data store, many different analytics support teams. Kubernetes maintenance. The payments platform is probably huge. The app teams too. Infosec of various flavors. Vendor integrations... Think about the sprawling functionality of a company at scale. Anyone else is sales or support. And other business functions, like various legal departments. 10,000 is the right size for an org like Shopify.
- ryanbrunner 4y agoNeeding to have a big team so that you can refactor into microservices, which you need because you have such a big team is a very "the bureaucracy is expanding to meet the needs of the bureaucracy" sort of thing.
- echelon 4y agoIt's a business expanding to fill the market sort of thing. The people on those teams undoubtedly have plenty to work on.
- ecshafer 4y agoFWIW we don't really use Microservices at Shopify.
- ibawt 4y agocardserver, storefront renderer, remember-me... the list goes on.
- dehrmann 4y ago> What are they doing? Scale is harder to manage as the userbase grows. A lot of popular services could run on one machine if you can live with 3 nines of reliability and never scale past 100,000 users. When you reach that scale, you also start doing lots of product work that's either exploratory and unlikely to matter, or micro optimizations trying to move the needle 1%. If you hit a win on one of these, because of your scale, the win can be huge.
- foobiekr 4y agoSee the previous discussion of manager incentives to hire here: https://news.ycombinator.com/item?id=32149937#32171559 https://news.ycombinator.com/item?id=32149937#32171559
- jokethrowaway 4y agoWasting time. I've seen large companies and it's astonishing how little work and how much process they have for minuscole changes. The only productive way of managing this is do what Google or Amazon do: split in sections and teams and get them to do something independently, with the hope that someone will struck gold. It's a bit like a venture incubator and - sometimes good things come out of it, most of the time dead products come out. The company where I'm currently in is ~200 engs and a few companies ago with 50 engs was easily 10x more productive and had a way bigger codebase. Management is either too useless to care / too attached to saying they have N underlings and the business makes so much money they can afford to spend x% on tech workers, just to be safe.
- taf2 4y agoLots of teams - working on lots of different products that all connect together.
- 40acres 4y agoI agree with you for the most part. The number of people who could run Shopify at the current scale is definitely more than 80 people but probably less than 10000. They took a big bet on global growth and this would've exploded HC related to sales, marketing, compliance, recruiting, etc. They pulled hiring forward on an expectation that the market advanced 5 years during COVID and now have to cull that new growth. Lots of late stage private companies and recent IPOs are going through the same thing.
- hef19898 4y agoAnd in that light ten percent isn't even that much. It really sucks for those being laid off, Shopify seems to take their responsibility for their people serious so (if they follow through with what they promised). And yes, pulling your timetable and hiring forward was bet, but a less risky one than it turned out to be in hindsight. More general, we had the 2008 financial crisis after one not too big investment bank went under. Covid did, despite all the real world damage from lost lives to the economy, not doing anything like it. I always wondered why, aparently more free money, on top of all the free money we had since 2008, helped. I just hope that the postponement of hitting a wall doesn't mean we ginna hit it much faster. We'll see...
- pc86 4y agoAAA gaming is probably the worst possible example you can look at for what a well-run, sane team would look like. So let's say if you only want your people working 40ish hours a week and not 70-90, you'll 2x or 2.5x that count. Shopify also has a lot of dev rel folks even if they don't call them that - people writing engineering blog posts, working on tutorials, documentation, etc. They're typically not the same people actually working on core code day to day. My guess is all the infrastructure and security people at Shopify are non-devs. You'll also have all the devops people who are managing all the pipelines and everything. My understanding is Shopify deploys to prod many many times a day (hundreds?). The pipelines to automate that, and automate or at least manage the rollbacks, would be at least as complicated as the code itself. And across all these different areas you have that amorphous "Staff/Principal Engineer" level where you're doing a lot of reviewing, pseudo-management, that sort of thing. Not to mention managers, directors, VPs, SVPs, etc. But you ask how they can productively use the time of 10k people and I think this post is sort of evidence that they couldn't.
- stephc_int13 4y agoI understand the need for support around a complex and constantly evolving codebase, but this is the scale of it that I can't grasp. Software is supposed to scale almost for free, they don't need people to actually make the transactions and push the bits by hand, this is all automated. My guess is they have a lot of sales and an enormous amount of overhead. According to Price's Law about 100 people are doing half of the work, but of course it is difficult to guess which ones and they still need the other half to be there.
- deleted 4y ago[deleted]
- conorcleary 4y agoCustomer support and new-build implementation tied to marketing is a large chunk of the people hours.
- wollsmoth 4y agoHard to say without knowing what they're working on. Maybe there's a lot to do with integrating other software/services. Maybe they've got a good devops org to help smooth the process for everyone. Sometimes having a few more people than you need on many levels makes the org a lot more resilient to individuals leaving so the org can keep on rolling without a hitch.
- ActionHank 4y agoMultiple products targeting multiple users, both internal and external.
- stephc_int13 4y agoAlright. Let's say that each product dev team is about 80 engineers. Let's double that number for support, HR, admin, etc. Let's add a team of 10 people for sales ( not needed for internal products but hey) Now let's round it up to 200 heads. That would make 50 large software products, is that right?
- woofcat 4y agoOn a large chunk of Enterprise Software you'd have teams of people working on specific features / subsets of a larger product. I.e. a team that is working on only Facebook Messenger back-end, a team working on Facebook Messenger Front-end, a team working on Friend Recommendation, etc. Each of these teams is let's say 10 Dev, 2-4 QA, 1 Product Documentation, then all the normal overhead of HR, etc.
- ggreenbe 4y agoIt's more than 50... I mean just look at the number of API endpoints across multiple APIs both internal and external. It's easily hundreds then add sys ops, dev ops, dev content writing (docs/guides), frontend, design, pm, support, etc. I feel like most of you all are unaware of the complexity and scope of Shopify...
- jokethrowaway 4y agoI've seen companies where a team was managing 10 services and companies where a team was managing one endpoint The former had some stressed principal engineers, the latter was terribly inefficient and people had 1% productivity. There is a happy medium, but nobody can manage a large headcounts without splitting in a lot of tiny products. The core product of shopify is not something that needs 10k engineers for maintenance, even if they can afford that.
- perlgeek 4y ago> How can they manage to productively use the time of 10000+ people? Sales and support basically scale arbitrarily large, as long as the market is there.
- xwolfi 4y agoNot much. They're now starting to fire a bunch to make the rest quit. I'm in a well run, immensely profitable, 100+ yo company, so big a country needs us to run, literaly, and a few others would be destroyed if we suddenly closed shop and ran with the money. We are 60 000 and feel too fat. I cannot understand what frigging shopify is doing with 10k people. We make 5bn a year profit these days, is shopify close to 1? Edit: waoh, they indeed were at 3 bn last year, 300mil the year before, -100 mil before, without reading the details. It's not regular profit but if it was, it'd just confirm we're too fat, not them :p
- cosmiccatnap 4y agoMarketing...
- moneywoes 4y agoI feel it’s a matter a growing faster than your competitors