13 ms·

FAANG promo committees are killing Kubernetes: A Short Thread

218 points by lolsoftware 5y ago

pjmlp 5y ago
Great, the sooner we get rid of that yaml spaghetti, and container everywhere, the better.
paxys 5y ago
While it is still a stretch, the title should be "Google's promo committee is killing Kubernetes", since no other FAANG-equivalent is using or contributing to Kubernetes in a meaningful way. The core point: > It's too indirect, fixing a bug in kube-apiserver might retain a GCP customer or avoid a costly Apple services outage, but can you put a dollar value on that? How much is CI stability worth? Or community happiness? While correct, the same is true for most projects at these companies. Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted? Or a new training model? A large-scale refactor of a legacy codebase? (All of these are examples of very senior promos at my own FAANG company). Kubernetes is no different. An engineer has to show how the work they did first and foremost aligns with their team's goals. If it doesn't, then well they shouldn't have been working on it in the first place. While working on open-source projects might be tolerated, even on company time, it makes sense that it won't put you on a path to promotion unless there is direct benefit to the company from it.
- azurezyq 5y ago
  I think the tweet is a bit too much. I worked for GCP for several years before. Impact in promo is defined within the same group and aligned with the goals of the group. If improving the stability of k8s is among the goals, then no problem at all. OSS projects are not built on top of anyone's mercy. For infra projects like k8s, companies just contribute what they need and Google is no exception.
  - fhrow4484 5y ago
    > I worked for GCP for several years before. Impact in promo is defined within the same group Did you personally go through promo from L6 to 7 or L7 to 8 since that's what the OP thread is about?(or have seen the process from sufficiently close). I can see how your description of impact & promo would work when there's enough people inside the group to compare from (i.e other L4s, L5s, L6s), but when going L7 & L8 there might not be enough people in the group. Plus the folks at that level would be the one defining the goals, so it seems logical that they can't get promoted on that criteria.
    - joshuamorton 5y ago
      Up through 7 is defined within the same group for Cloud, afaik, which is almost certainly the relevant portion of the company. > Plus the folks at that level would be the one defining the goals, so it seems logical that they can't get promoted on that criteria. Fwiw you start defining goals and L6 (or even 5), that's one of the defining characteristic of 6 vs. 5, you can't get to 6 from execution alone, you need to be setting roadmaps that impact beyond your team. And at 7 and 8, the scope of both people and technology get's larger. A strong-L7/just promo'd L8 would probably be expected to exert direct influence over an organization of around 100 fulltime engineers (caveat: I'm saying this from below, but it fits what I've seen). Which is to say that the positions in k8s that qualify for L7/L8 promo-worthy work are to a first approximation, just the steering committee. And that looks like it's working as intended: the steering committee is full of people who are Principal or Distinguished (or more in some cases) engineers and manager equivalents from a variety of companies.
      - webmaven 5y ago
        > Which is to say that the positions in k8s that qualify for L7/L8 promo-worthy work are to a first approximation, just the steering committee. So you are basically agreeing that an engineer who wants to advance to that level has to go and start a shiny new project (open source or not) so they are, by default, on the steering committee or the equivalent.
- ashtonkem 5y ago
  > While correct, the same is true for most projects at these companies. Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted? Or a new training model? A large-scale refactor of a legacy codebase? (All of these are examples of very senior promos at my own FAANG company). Which FAANG though? Google is usually singled out for this criticism more than the rest of FAANG combined. It’s possible that your experience at a non-Google FAANG might not be relevant for Google specifically. (Note: I wouldn’t know first hand, I’ve never worked at a FAANG).
- nightvoomer 5y ago
  Even if you can prove a revenue number, it doesn't matter if they don't want to promote you. I've seen it explicitly stated at more than one company, that revenue impact is irrelevant and that you need to show a "large enough design and execution" to be promoted as a reason to just not give it out.
- coredog64 5y ago
  Here it’s focused on k8s, but it’s not out of line if we generically apply it to AMZN. How many services look cool, have a great re:Invent kick-off, and then go KTLO a year later? I’m going to lay that at the feet of the AMZN promo process, which prioritizes creating something new over real and needed maintenance. I’m going to further reduce this to lazy management. It’s easier to point to splashy announcements to show impact than it is to put in the work and help your direct reports quantity unglamorous maintenance work.
  - eigen-vector 5y ago
    Not sure how long you've been at or spent at Amazon, but the promotion process doesn't prioritize building new things at all. It's true that people think complexity = build new things, but promotion criteria specifically encourage people to stick to simple things and build on top of what exists rather than invent something new for promotion's sake.
    - faangiq 5y ago
      Laughably false.
    - notyourwork 5y ago
      I'd argue this depends on what org and team and the culture surrounding. It can be vastly different in various nooks of the company.
    - nelsondev 5y ago
      When selling your work to a promo committee, building a new shiny thing lowers the communication hurdle. Personal anecdote, at Amazon, my wife led a very challenging refactor of legacy code and didn’t get promoted, I built a shiny thing barely anyone is using yet, and got promoted.
      - webmaven 5y ago
        > Personal anecdote, at Amazon, my wife led a very challenging refactor of legacy code and didn’t get promoted, I built a shiny thing barely anyone is using yet, and got promoted. How much of that is the effect of shiny vs. dull, and how much is male vs. female (apologies if my assuming you are male was mistaken, but it seemed the way to bet)?
- saagarjha 5y ago
  > While it is still a stretch, the title should be "Google's promo committee is killing Kubernetes", since no other FAANG-equivalent is using or contributing to Kubernetes in a meaningful way. Apple puts Kubernetes on https://opensource.apple.com https://opensource.apple.com as a project they "lead and contribute to"… > Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted? While I generally agree, I've made changes in our internal build tools that I can easily put a number on, although it's not a revenue number. Make build process 10s faster, 1000 engineers run this 10x a day, you're saving more than an engineer day per day of time. That's a definite impact right there, assuming your management is on board with accepting it :)
  - rektide 5y ago
    Apple is #32 in contributor commits[1]. Between SAP and Glassdoor, with ~7000 commits. You need to start adding zeroes and then some to get to Google's number of commits. We shouldn't just measure in commits. I'd be interested to hear the case made. Where has Apple lead in Kubernetes? What have been their major initiatives? That'd be interesting to hear. [1] https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1 https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1
    - saagarjha 5y ago
      I really have no idea, I just find it amusing that they put it front and center on their open source website.
    - quadrifoliate 5y ago
      If you don't want the pointy-haired bosses to start measuring productivity by number of commits, start with not doing so yourself. Commit count is largely meaningless and a low-effort way to fling around "contribution" numbers. For example, large commit numbers could be a reflection of a specific company's internal conventions around making a larger number of small changes as their own commit. Now it's certainly plausible to me that Google is a much more major contributor to Kubernetes than Apple, but you need better numbers for showing that. For example, which companies have contributed to designing and implementing the major features in the last 5 releases?
- whateveracct 5y ago
  > While working on open-source projects might be tolerated, even on company time, it makes sense that it won't put you on a path to promotion unless there is direct benefit to the company from it. The thing is - this is all theater. It's mostly made-up storytelling. An individual engineer's self-review doesn't actually speak to their direct impact on the company's success. It's fallacy to act like you can map commits to dollars. But there's a lot of money to be made by HNers for trying, so I'm sure they'll act like they can do the impossible.
- deleted 5y ago
  [deleted]
- jayd16 5y ago
  Azure and AWS k8s offerings aren't a meaningful usage? Asking earnestly.
- reissbaker 5y ago
  Not only that, but the "promo packet" and "promo committee" is fairly unique to Google. Other companies do things very differently, e.g. FB's twice-yearly (now yearly) manager-driven PSC+"calibration" cycle, which has fairly different incentives than Google-style committees of unknown engineering peers with arbitrary periodization. Google's system is well known (at least, in my circle of colleagues!) to not promote healthy long-term maintainership, although it does incentivize point-in-time brilliance and solving deep complexity.
  - DannyBee 5y ago
    The system you talk about is mostly dead. Google promos everywhere up to a certain level are in-org (IE not unknown engineering peers). They are done alongside rating calibrations. Google promos everywhere up to an even higher level are basically in-PA. For cloud, they are in-org to even a higher level. So the stuff talked about in this tweet would have been evaluated not just by cloud engineers, but almost certainly fairly local cloud engineers. My experiencing having been on promo committees for google for 15 years is that for every case in my org i see of google incentivizing the wrong thing (or disincentivizing the right thing), i see probably 5-6 cases of it doing the right thing. I see a lot more cases that are like: 1. Person builds new shiny thing for ... questionable reasons. 2. They make a mess of the world by not making it easy to migrate people to it and get rid of the old thing 3. Things are in a bad half-state, where both things have to be supported and the new thing does not provide obvious higher value. 4. Google doesn't promote them, they get pissed off about it and post on twitter about how Google didn't care about product excellence or something. Than cases like: 1. Person toils away on the right thing forever 2. Person tries to get promoted for doing the right thing and having impact 3. Google turns them down 4. They get pissed off and post on twitter about how Google didn't care about about product excellence or something. Externally, of course, people can't distinguish these two cases because they look the same (angry people on twitter). I have been promoted from SWE III to Senior Director at Google, working only on open source until i was an L7.
    - vechagup 4y ago
      Agreed. I work in the PP's PA and was promoted on my first try for doing unsexy but worthwhile maintenance work on an important tool. It can happen. In my estimation the best thing to do if you want to get promoted is 0) have a positive relationship with your manager 1) read the SWE ladder for the next level 2) give your manager a document that describes your work using the language of the ladder as closely as possible. Said ladder does not say you need to deliver a shiny new thing, despite what is commonly assumed in this forum.
- gjvc 4y ago
  > (All of these are examples of very senior promos at my own FAANG company). you have your own FAANG company? remember always that "your" company will drop you in a moment.
- thockingoog 4y ago
  Define "killing kubernetes"? It's still pretty successful and the adoption hasn't slowed in any way I can measure. I promise you that some site you used TODAY is running, at least part of it, on Kubernetes. Google employees regularly get promoted, at all levels, based on their OSS work - Kubernetes and other projects. We have dozens of people who work on Kubernetes, in one area or another, at varying degrees of depth. Is that "killing" the project? Of course it is never ENOUGH. I'd happily consume hundreds more people. :)
rdxm 5y ago
mdaniel 5y ago
https://threadreaderapp.com/thread/1511791378497384454.html https://threadreaderapp.com/thread/1511791378497384454.html
devy 5y ago
Brendan Burns, the Kubernetes co-founder who currently works at Microsoft Azure as a Corporate Vice President disagrees the "killing Kubernetes" part.[1][2] But yes, Big Tech's promo committee's short-sighted interests do NOT align well with multi-vendor sponsored open source projects' the long term prosperity. And it's no secrets that open source contributors are overworked. And it's not only affect K8s, but it affects across the board.[3] [1]: https://www.linkedin.com/in/brendan-burns-487aa590 https://www.linkedin.com/in/brendan-burns-487aa590 [2]: https://twitter.com/brendandburns/status/1511841189686771716 https://twitter.com/brendandburns/status/1511841189686771716 [3]: https://twitter.com/kantrn/status/1511791447091003395 https://twitter.com/kantrn/status/1511791447091003395
- klabb3 5y ago
  Well yes, what did you expect a corporate VP with skin in the game would say publicly on Twitter?
  - hyperpape 5y ago
    The OP then agrees that Microsoft stands out as supporting OSS more than competitors. https://twitter.com/kantrn/status/1511844921921077249 https://twitter.com/kantrn/status/1511844921921077249
zozbot234 5y ago
Kubernetes' biggest problem by far is its mess of accumulated technical debt. FAANG promo committees very likely undervalue FLOSS contributions, but this hits all of FLOSS pretty much equally; it hardly explains why k8s specifically would be in so much trouble.
- billllll 5y ago
  I would go as far as to say it's not just limited to FLOSS contributors. In my experience at Google, not every service is profitable or brings in revenue, so it can be hard to justify "impact" on those teams. I certainly remember beating my head on how to spin my contributions as having "impact."
throwawax 5y ago
- jthrowsitaway 5y ago
  As someone that's manually stitched together some of what Kubernetes provides (before it even existed), I very much appreciate that Kubernetes exists.
datalopers 5y ago
Google created Kubernetes so potential competitors are dragged down with immense technical debt, of course they’re not going to award internal advancement of it.
- busterarm 5y ago
  Having participated in full on rearchitectures to Kubernetes several times at this point I can say that it wasn't justified a single time, even at the 10^4 microservice scale. Now having participated in several fairly-effortless rollouts of Nomad and arrived at a better place, it's funny to watch the rest of the industry cargo cult. I actually don't believe that either of these solutions are going to be a long-term settled end state (though Nomad with Firecracker, or really any other Firecracker-centric solution that isn't k8s will have some legs) the industry falls on, but I agree wholeheartedly that Kubernetes is purely a distraction having seen the pain and the gnashing of teeth. Every painless Kubernetes story that I've seen is at a scale where Kubernetes wasn't even necessary/justifiable and another solution would have been even simpler. But at least it's good for the resume. And Helm charts are are akin K8s' own borgcfg. We're doomed to repeat our mistakes it seems.
  - streetcat1 5y ago
    Can you elaborate more? Why it was not justified? The more detailed the better (If possible, concrete issues that you faced) ?
    - busterarm 5y ago
      I've seen small startups waste their time with things like self-hosted Kubernetes in a quarter-rack of colo space for workloads they could have instead hosted with KVM or on cloud instances with 1/100ths of the management overhead. Scale that up to a half-dozen racks of servers and you're really still telling the same story. Even OpenStack of all things is easier to manage. This covers the scale of the actual operations of 99% of companies. Probably some more 9's there afterwards. Go and read about how StackOverflow's infrastructure has developed over the years and how damned simple and effective it is. If you aren't Fortune 100 or you don't have extremely specific performance needs for your (re)deployments, then it's highly likely that rolling out Kubernetes infrastructure is akin to driving screws with a sledgehammer. I think part of this is the downside of capital being far too cheap for too long. Companies have way overbuilt their infrastructure for reasons that aren't really moving the needle forward. Many barely make an effort to control their costs. At most companies my expectation going in is to look at their infrastructure and see somewhere between 0.02 and 10% hardware utilization across everything they have. Even the companies running Kubernetes hardly seem to be doing a better job because very rarely is Kubernetes running 100% of their infrastructure, if even 10%.
colinmhayes 5y ago
> going L6->7 at Google is worth ~200k/year, 7->8 is ~400. Similar patterns at other places. People have kids, mortgages, student loans Honestly this is a bit ridiculous. If making 500k as an L6 isn't good enough for you maybe you should be working on projects that provably bring your company revenue?
- adastra22 5y ago
  This is insane. I can't understand these numbers. No wonder the Bay Area has become completely unaffordable for long-term residents.
  - deleted 5y ago
    [deleted]
  - barry-cotter 5y ago
    Indeed. If lots of people with money want to live in a place and building more housing is illegal the price of existing housing will go up a lot. If you want the problem to go away you can either build more housing or make it so the people with money don't want to live there.
  - rchiang 5y ago
    Google makes $5.5 billion of profit per month. I think they can afford to pay some of their higher performers more.
  - vwoolf 5y ago
    The Bay Area is unaffordable because most cities haven't allowed anywhere near enough housing to be built by land owners over the last five decades: https://techcrunch.com/2014/04/14/sf-housing/ https://techcrunch.com/2014/04/14/sf-housing/. Return building rights to land owners and the problem will be solved. The insane thing is the way coalitions of existing owners conspire to reduce supply and thus increase the values of their assets, and the rest of us just go, "Oh, okay, that's cool, that you're conspiring to embeggar the rest of us."
  - mike_d 5y ago
hapless 5y ago
Go figure, contributions to a multi-vendor consortium that just barely makes a marginal profit for an already marginal business unit (GCP) are not as highly valued as things that can be directly connected to revenues. This is not surprising in the least. Perhaps obviously, Mr. Kantrowitz hasn't quit, or been poached by a competitor, so I would say Google's (cynical) comp and promo policy is working out just great for them. Whatever personal interest keeps Mr. Kantrowitz going is working out for them!
- hyperpape 5y ago
  He does not appear to work at any of the FAANG companies: https://www.linkedin.com/in/noahkantrowitz/ https://www.linkedin.com/in/noahkantrowitz/.
sydthrowaway 5y ago
As someone in Europe, I am so sick of the implied salary pissing contest in every thread.
- busterarm 5y ago
  Market forces are moving in your favor. Most European companies have completely saturated their target markets for outsourcing (Barcelona, Estonia, Czech Republic) and Russia & Ukraine are suddenly unavailable options due to a war. Also the local markets have much more demand for engineers than are available to fill it. There has been a large push in the last two years to recruit Americans and Canadians to fill their unmet needs and salaries have been climbing rapidly. Salaries for new job postings in Berlin are about where they were at in New York 3-4 years ago which is EXCELLENT for that market (seriously, two years ago salaries were maybe half of what they are now). If you're in Europe and you haven't been interviewing you should look around and maybe even use a recruiter. My own recruiter has been feeding me opportunities across Europe with pretty much equal money to what I'm getting here in the States. It's likely that if you haven't changed jobs in the last two years that you're leaving significant amounts of money on the table. That said, I know that chasing after money isn't fashionable or socially acceptable to most Europeans...but the new hires are going to get it -- why shouldn't you?
  - sydthrowaway 5y ago
    How can we get access to your recruiter?
  - int_19h 5y ago
    There's over 4 million (and counting) Ukrainian refugees in Europe right now, and quite a few of those people are software engineers. Then there's over 100k (and counting) Russians that rushed to leave the country before the new iron curtain goes down - and I wouldn't be surprised if most of those are software engineers.
    - busterarm 5y ago
      You do know that refugees legally aren't allowed to work, right? You must fully complete your asylum-seeking process before you are allowed to seek employment. That often takes years if not decades. And because of various financial embargos it's historically very hard to hire Russians in European countries unless they have bank accounts in those countries, which without EU citizenship or long-term residency are very, very hard to get. Neither of these countries' citizens are allowed to freely travel and work within the EU the way that EU members can.
eigen-vector 5y ago
This seems like a strictly Google only problem. And by extension this is the problem with promotion committees that are removed from day to day of the candidate getting promoted. People are ridiculously bad at measuring growth based on some obscure company wide criteria - which is abstract at best and useless at worst at a company large enough as Google. I'm at AWS and the promotion process here is largely contained within the candidate's org which is at best two levels higher but in the same business. So if your team's business relies on contributions to FOSS, its impact is measurable and leaders and the 'committee' can easily tell you if your contributions are at the next level.
- anonymouscabc1 5y ago
  Yes FB is entirely in org too. It's weird to go out of org.
- deleted 5y ago
  [deleted]
- grogenaut 5y ago
  L7+ tech generally has in and out of org reviewing the promo. One for context, one for keeping the bar even.
axg11 5y ago
This thread is ridiculous. At FAANG, and everywhere really, promos are decided based on impact and influence. FAANG are just more likely to be working on open source projects than employees at other companies. All impact is difficult to quantify, not just contributions to OSS. The only easily quantifiable achievements for an engineer are delivered projects that directly generate revenue. In the teams I’ve worked in, I’d say that covers perhaps a third of projects.
- qbasic_forever 5y ago
  Let's pretend Linus Torvalds worked for Google, would he be able to get promoted for creating and maintaining Linux? As far as the bottom line goes it brings in zero dollars and costs the company his total compensation or more to maintain. How would Linus quantify his revenue impact in his promotion packet and compete against another hypothetical engineer that improved ad targeting by 0.0X% and can directly measure an increase of revenue for the comapny in the millions of dollars?
  - rrdharan 5y ago
    Revenue is not the only way to quantify impact. Plenty of infrastructure teams promote plenty of people at Google and other companies despite not directly selling products.
  - igetspam 5y ago
    Guido got paid to work on python
  - busterarm 5y ago
    Also it's Google, so we should be looking to promote operating system authors who aren't white or male anyway. Expecting hate for this, but I've been on the receiving end of company notifications (not at Google) where it was stated explicitly that the newly-opened distinguished engineer role was exclusively available to not-men. (I'm generally-supportive of DEI efforts, but at sane companies those efforts wouldn't block qualified people from being promoted to their proper roles. Among the FAANGs, I have a lot more confidence in Facebook and Amazon's ability to operate sanely than I do Google. Prior evidence and all.)
hintymad 5y ago
This is not really just the problem in Google. We might as well claim that promotion processes in big companies promote promotion-oriented work (yes, I intentionally use the word promotion this many times). The result is title inflation, lots of good employees leaving, and superfluous yet mediocre projects. I wouldn't say only unqualified people get promoted, though. Big companies have very different dynamics than small ones. Some people are just good at navigating big companies and are capable of aligning multiple organizations, for good or for bad. Big companies need such talent as well. As for individuals, the reward of getting promoted is not worth the effort, at least statistically. A much better alternative, is focusing on solving truly interesting and meaning problems in a blow-out company. The financial return and title bumps will follow naturally.
faddypaddy34 5y ago
I mean you should really be demoted for continuing to push the k8s microservices crap.
likpok 5y ago
Kubernetes doesn't get patches from hyperscale companies because kubernetes is not really hyperscale software. If I recall correctly (probably don't), clusters can handle tens of thousands of container hosts. That's not manageable at large scale unless you're selling people individual kubernetes instances directly (which, to be fair, google and amazon are). In my experience the thing that kills open source at big companies is the divergence of needs: The company needs massive scale software, but the community wants ease-of-use and features. Most open source models are a big drag on productivity (you can't just refactor all the callsites to do a migration!), and it's hard to change directions when the company's need does (since the other contributors will likely want the original software). The tailscale people have been arguing this as well (with more data and better reasoning), but fundamentally you need different software when you're a scrappy startup or a medium size company or a giant behemoth.
- busterarm 5y ago
  Having operated all the way up to "hyperscale", I agree with your point completely.
  - zozbot234 5y ago
    What's preventing k8s from being useful in these larger scale deployments? GP was not very clear about that, they just said that it's not scaling enough.
    - busterarm 5y ago
      K8s doesn't do anything to help you with managing the fleet of k8s hosts. At 10^5 host machines you have a much bigger problem on your hands than the one that k8s is solving for. If you have and are solving that hyperscale infrastructure problem then whether or not you're running kubernetes doesn't actually make a damn bit of difference to how you allocate your human/financial resources. Or summarized, at the scale where Kubernetes is advertised to make the biggest difference, it's actually irrelevant/an implementation detail.
dbish 5y ago
I love open source but I’m happy if we take the credit for killing kubernetes :)
ChrisArchitect 5y ago
Threadrip to make 'this could have been a blog post' more readable: https://the.rip/@kantrn-1511791378497384454-121542889 https://the.rip/@kantrn-1511791378497384454-121542889 https://unrollthread.com/t/1511791378497384454/ https://unrollthread.com/t/1511791378497384454/
cletus 5y ago
Not sure why this says FAANG because it's talking about Google. Calibration, promo committees, feedback and the like are intended to create equivalent expectations on impact across different orgs. The dirty little secret however is that at best it has limited success. The best advice I can give anyone in such an organization is to be liked by your manager and their manager. If that's true, good things will tend to happen. If it's not, good things will be a lot less frequent. Put another way: you can take the exact same set of objective facts and use them to say a person did a good job or a bad job. There's a popular meme about feedback at Google that goes something like: > This project would've failed without this person. It failed anyway but it definitely would've failed without them. The difference ultimately boils down to whether or not they like you. Here are a few Google-specific tidbits worth knowing: 1. Ratings are fit to a curve across a sufficiently large pool, typically at the director level and usually over 100-150+ people. This means there will be a percentage range of people who get Meets All, Exceeds Expectations, Greatly Exceeds, etc. This is intended to stop ratings inflation; 2. A consequenc eof (1) is that ultimately you are competing with people in your org for those better ratings. This can create some perverse incentives and a toxic environment; 3. It is almost always better to let something blow up and come and fix it rather than preventing that from ever happening. The first will get you a lot of recognition. The latter will get you almost none; 4. Promos at Google are stack-ranked. Each committee gets 10-15 packets that will be for a particular level. The committee will rank those packets. After that the promotion target will come into play. This is set by management and was allegedly cut as a cost-saving measure when Ruth Porat came on board. If it's 20% then the top 20% from that ranking process be promoted. You will find people who serve on those committees who say this isn't how it works and they'll argue they're evaluating if someone is operating at the next level or not. This is partially true. Thes packets will be divided between promote, don't promote and on-the-bubble. The on-the-bubble group will be sufficiently large to allow for the promotion target to be met; 5. For SWEs. L5 is the "terminal" level, meaning there is an expectation of growth to that level. L3->L4 and L4->L5 once went through promo committee but now don't. Management within orgs decide this. These too have target percentages and there have been cases where the promot rate has been too "high" and orgs have been told to cut back on promotions to meet the targets; 6. There is a massive backup at L5->L6. Because of the low target percentages the impact required keeps going up and really you need your management to really push for this to happen. There are limited slots so you may be waiting eyars and again this is why them liking you matters so much. Google is full of L6s who got promoted 5-10+ years ago that would never make the grade by today's standard. For really old cases you can find archives of why there were promoted and you'll find cases like "promoted unit testing". I say all this because the author of this thread seems to fundamentally misunderstand how this process works.
deleted 5y ago
[deleted]
GauntletWizard 5y ago
I basically can't stand the comment section of Kubernetes threads. Lots of big teams are getting big things done with k8s, and you would not know it from here.
MuffinFlavored 5y ago
What's the alternative to k8s then?
marsven_422 5y ago
[dead]
lumost 5y ago
This is the case with most major products from most major tech companies. There is a reason you don't (often) see steady incremental progress on products, but rather big flashy releases that never quite get their bugs cleaned up unless they are wildly* successful. * Where wildly is defined as a success level which would leave any rational founder very wealthy, and very content.
avs733 5y ago
It sounds like FAANG has reinvented tenure and promotion processes from academia…and the resulting inherent pressures. A moment to reflect on Han’s critique of publish or perish academia?
henning 5y ago
Hopefully Kubernetes does die. Most companies write shit slow software and then think they're big smart boys because they have to buy $20,000/month of servers to make it not be unusable. In many shops it could and should be replaced by a handful number of efficient servers that do not suck shit.
reilly3000 5y ago
This is just the law of inertia at play. Big projects slow down. K8s isn’t just the K8s core, it’s a massive ecosystem, and when taken as a whole it’s unbelievably vast and growing at an amazing pace. As it does, the core of it moves slower, there are more stakeholders, more arguments. Small changes make for big problems and not everyone can be happy. Patches will keep coming, major changes will become eventually impossible. The death of Kubernetes makes for great clickbait, but most of us won’t see it in our lifetimes. Not when our banks run on 60 year old code. Kubernetes is too big to fail at this point. As far as FAANG promo committees go, let them value what they value. Kubernetes is a direct revenue driver for many companies; it’s health is tied to billions of dollars in investments. Just because one cohort of contributors age out, cash out, or fall in love with someone doesn’t mean there is no one to take their place. The new people won’t do it the same way and that is okay, even great. I’m grateful for the vision and effort that have make K8s the platform it is, and if I can translate that into a contribution in the future I will. Finally, I’ll say that people who really love working on the project may not get support to work on it full time, but then may find themselves able to retire sooner than many and have ample opportunity to contribute.
throwaway787544 4y ago
K8s is a great example of what murders open source: monolithic platforms that exist unto themselves, but have some seemingly benign model allowing arbitrary "plugins" to pretend it's not really a monolith in sheep's clothing. It looks like open source and has the right license, but it ends up just mirroring corporate interests rather than those of the Commons that the open source movement was created to support.
- zozbot234 4y ago
  If you're going to have monolithic platforms anyway (and you will, because designing for modularity is hard, whereas hacking together some random monolithic code is easier) they should at least be open source rather than proprietary.
  - throwaway787544 4y ago
    Actually I disagree. Any time there's an Open Source tool that does everything for free, it's extremely hard to justify paying for a proprietary product that does it much better, because companies are cheap. There's many open source projects like this that are just terrible but a company will always use them first because they're free. The answer isn't modularity, it's composeability, which is a significant difference. A modular program requires "integration" (tight coupling of APIs/ABIs) whereas a composeable program has loose interfaces which require virtually no "integration". Drone.io is one example; you can implement any "plugin" purely by creating a container with an initial CMD entry that reads environment variables, and that program can interface back with Drone a number of ways (STDIN/STDOUT, REST API, database). Another is any application that just reads in or spits out simple line-by-line instructions or a JSON blob. Unix pipe based programs are the penultimate example. The dumber the interface, the easier it is to compose.
wallfacer120 4y ago
Oh man, another rant that boils down to "big evil corporations are KILLING open source software by not supporting it in the specific ways and in the specific amounts that I want."
otabdeveloper4 4y ago
> killing Kubernetes Really? Finally some good news in these troubled times.
wink 4y ago
That's the most ridiculous bubble valley thing I read in a while, coming from a non-k8s using non-FAANG-employed European. Which isn't meant dismissive of the problem at hand, I can totally see how it's happening and how it might be bad for the people involved in the project. To phrase it differently - if the project is dying because it hinges on people working on a promotion that gives them a /bigger paycheck increase than my yearly income/ - well, good riddance. I'm staying on my side of the fence and continue to not buy a new Tesla every year.
- helen___keller 4y ago
  > if the project is dying because it hinges on people working on a promotion that gives them a /bigger paycheck increase than my yearly income/ - well, good riddance. I'm staying on my side of the fence and continue to not buy a new Tesla every year. An understandable position but this is simply how competitive environments work. It’s a lot like the college application mania. There’s tons of gifted high school students who clearly can succeed without needing to volunteer in a foreign country every summer while playing oboe and competing in lacrosse at the state level. But seats at Harvard are competitive, and those same talented and well-to-do high school seniors often only have college admissions as their ruler to measure their growth and self-worth. So they compete on every angle to try and get into Harvard rather than merely Dartmouth or heaven forbid BU. It’s reasonable to think working adults should have measures of personal growth that don’t rely on climbing the corporate ladder, but conversely all those who *do* obsess with climbing the ladder will naturally be clustered around places of wealth/prestige, because that’s what they are optimizing for. Much like the Harvard application pool, you’ll find these types doing what it takes to stand out and land that 700k/yr L7 position at Google. So I guess the answer is that the most stable and secure position for an open source project to be in, is one that does not rely on development being coupled to big tech. I imagine most open source folks could have told you that anyways :)
shiftoutbox 4y ago
cbushko 4y ago
There are certain classes of work that are completely ignored by most companies such as build, automation, developer tooling and infrastructure cost cutting. These are not flashy features that you can show off to customers but they can pay in dividends year over year. (Personally, I have saved millions a year in infrastructure costs and my promotions/compensation were not close to what a sales person would get if they brought in millions of recurring revenue)
thockingoog 4y ago
There's a perception that this is true, and like all perceptions it is based in reality, but it is not the reality itself. We have plenty of data that shows that people DO get promoted at all levels based on OSS work.

218 points by lolsoftware 5y ago

Great, the sooner we get rid of that yaml spaghetti, and container everywhere, the better.

While it is still a stretch, the title should be "Google's promo committee is killing Kubernetes", since no other FAANG-equivalent is using or contributing to Kubernetes in a meaningful way. The core point: > It's too indirect, fixing a bug in kube-apiserver might retain a GCP customer or avoid a costly Apple services outage, but can you put a dollar value on that? How much is CI stability worth? Or community happiness? While correct, the same is true for most projects at these companies. Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted? Or a new training model? A large-scale refactor of a legacy codebase? (All of these are examples of very senior promos at my own FAANG company). Kubernetes is no different. An engineer has to show how the work they did first and foremost aligns with their team's goals. If it doesn't, then well they shouldn't have been working on it in the first place. While working on open-source projects might be tolerated, even on company time, it makes sense that it won't put you on a path to promotion unless there is direct benefit to the company from it.

azurezyq 5y ago
I think the tweet is a bit too much. I worked for GCP for several years before. Impact in promo is defined within the same group and aligned with the goals of the group. If improving the stability of k8s is among the goals, then no problem at all. OSS projects are not built on top of anyone's mercy. For infra projects like k8s, companies just contribute what they need and Google is no exception.
- fhrow4484 5y ago
  > I worked for GCP for several years before. Impact in promo is defined within the same group Did you personally go through promo from L6 to 7 or L7 to 8 since that's what the OP thread is about?(or have seen the process from sufficiently close). I can see how your description of impact & promo would work when there's enough people inside the group to compare from (i.e other L4s, L5s, L6s), but when going L7 & L8 there might not be enough people in the group. Plus the folks at that level would be the one defining the goals, so it seems logical that they can't get promoted on that criteria.
  - joshuamorton 5y ago
    Up through 7 is defined within the same group for Cloud, afaik, which is almost certainly the relevant portion of the company. > Plus the folks at that level would be the one defining the goals, so it seems logical that they can't get promoted on that criteria. Fwiw you start defining goals and L6 (or even 5), that's one of the defining characteristic of 6 vs. 5, you can't get to 6 from execution alone, you need to be setting roadmaps that impact beyond your team. And at 7 and 8, the scope of both people and technology get's larger. A strong-L7/just promo'd L8 would probably be expected to exert direct influence over an organization of around 100 fulltime engineers (caveat: I'm saying this from below, but it fits what I've seen). Which is to say that the positions in k8s that qualify for L7/L8 promo-worthy work are to a first approximation, just the steering committee. And that looks like it's working as intended: the steering committee is full of people who are Principal or Distinguished (or more in some cases) engineers and manager equivalents from a variety of companies.
    - webmaven 5y ago
      > Which is to say that the positions in k8s that qualify for L7/L8 promo-worthy work are to a first approximation, just the steering committee. So you are basically agreeing that an engineer who wants to advance to that level has to go and start a shiny new project (open source or not) so they are, by default, on the steering committee or the equivalent.
ashtonkem 5y ago
> While correct, the same is true for most projects at these companies. Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted? Or a new training model? A large-scale refactor of a legacy codebase? (All of these are examples of very senior promos at my own FAANG company). Which FAANG though? Google is usually singled out for this criticism more than the rest of FAANG combined. It’s possible that your experience at a non-Google FAANG might not be relevant for Google specifically. (Note: I wouldn’t know first hand, I’ve never worked at a FAANG).
nightvoomer 5y ago
Even if you can prove a revenue number, it doesn't matter if they don't want to promote you. I've seen it explicitly stated at more than one company, that revenue impact is irrelevant and that you need to show a "large enough design and execution" to be promoted as a reason to just not give it out.
coredog64 5y ago
Here it’s focused on k8s, but it’s not out of line if we generically apply it to AMZN. How many services look cool, have a great re:Invent kick-off, and then go KTLO a year later? I’m going to lay that at the feet of the AMZN promo process, which prioritizes creating something new over real and needed maintenance. I’m going to further reduce this to lazy management. It’s easier to point to splashy announcements to show impact than it is to put in the work and help your direct reports quantity unglamorous maintenance work.
- eigen-vector 5y ago
  Not sure how long you've been at or spent at Amazon, but the promotion process doesn't prioritize building new things at all. It's true that people think complexity = build new things, but promotion criteria specifically encourage people to stick to simple things and build on top of what exists rather than invent something new for promotion's sake.
  - faangiq 5y ago
    Laughably false.
  - notyourwork 5y ago
    I'd argue this depends on what org and team and the culture surrounding. It can be vastly different in various nooks of the company.
  - nelsondev 5y ago
    When selling your work to a promo committee, building a new shiny thing lowers the communication hurdle. Personal anecdote, at Amazon, my wife led a very challenging refactor of legacy code and didn’t get promoted, I built a shiny thing barely anyone is using yet, and got promoted.
    - webmaven 5y ago
      > Personal anecdote, at Amazon, my wife led a very challenging refactor of legacy code and didn’t get promoted, I built a shiny thing barely anyone is using yet, and got promoted. How much of that is the effect of shiny vs. dull, and how much is male vs. female (apologies if my assuming you are male was mistaken, but it seemed the way to bet)?
saagarjha 5y ago
> While it is still a stretch, the title should be "Google's promo committee is killing Kubernetes", since no other FAANG-equivalent is using or contributing to Kubernetes in a meaningful way. Apple puts Kubernetes on https://opensource.apple.com https://opensource.apple.com as a project they "lead and contribute to"… > Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted? While I generally agree, I've made changes in our internal build tools that I can easily put a number on, although it's not a revenue number. Make build process 10s faster, 1000 engineers run this 10x a day, you're saving more than an engineer day per day of time. That's a definite impact right there, assuming your management is on board with accepting it :)
- rektide 5y ago
  Apple is #32 in contributor commits[1]. Between SAP and Glassdoor, with ~7000 commits. You need to start adding zeroes and then some to get to Google's number of commits. We shouldn't just measure in commits. I'd be interested to hear the case made. Where has Apple lead in Kubernetes? What have been their major initiatives? That'd be interesting to hear. [1] https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1 https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1
  - saagarjha 5y ago
    I really have no idea, I just find it amusing that they put it front and center on their open source website.
  - quadrifoliate 5y ago
    If you don't want the pointy-haired bosses to start measuring productivity by number of commits, start with not doing so yourself. Commit count is largely meaningless and a low-effort way to fling around "contribution" numbers. For example, large commit numbers could be a reflection of a specific company's internal conventions around making a larger number of small changes as their own commit. Now it's certainly plausible to me that Google is a much more major contributor to Kubernetes than Apple, but you need better numbers for showing that. For example, which companies have contributed to designing and implementing the major features in the last 5 releases?
whateveracct 5y ago
> While working on open-source projects might be tolerated, even on company time, it makes sense that it won't put you on a path to promotion unless there is direct benefit to the company from it. The thing is - this is all theater. It's mostly made-up storytelling. An individual engineer's self-review doesn't actually speak to their direct impact on the company's success. It's fallacy to act like you can map commits to dollars. But there's a lot of money to be made by HNers for trying, so I'm sure they'll act like they can do the impossible.
deleted 5y ago
[deleted]
jayd16 5y ago
Azure and AWS k8s offerings aren't a meaningful usage? Asking earnestly.
reissbaker 5y ago
Not only that, but the "promo packet" and "promo committee" is fairly unique to Google. Other companies do things very differently, e.g. FB's twice-yearly (now yearly) manager-driven PSC+"calibration" cycle, which has fairly different incentives than Google-style committees of unknown engineering peers with arbitrary periodization. Google's system is well known (at least, in my circle of colleagues!) to not promote healthy long-term maintainership, although it does incentivize point-in-time brilliance and solving deep complexity.
- DannyBee 5y ago
  The system you talk about is mostly dead. Google promos everywhere up to a certain level are in-org (IE not unknown engineering peers). They are done alongside rating calibrations. Google promos everywhere up to an even higher level are basically in-PA. For cloud, they are in-org to even a higher level. So the stuff talked about in this tweet would have been evaluated not just by cloud engineers, but almost certainly fairly local cloud engineers. My experiencing having been on promo committees for google for 15 years is that for every case in my org i see of google incentivizing the wrong thing (or disincentivizing the right thing), i see probably 5-6 cases of it doing the right thing. I see a lot more cases that are like: 1. Person builds new shiny thing for ... questionable reasons. 2. They make a mess of the world by not making it easy to migrate people to it and get rid of the old thing 3. Things are in a bad half-state, where both things have to be supported and the new thing does not provide obvious higher value. 4. Google doesn't promote them, they get pissed off about it and post on twitter about how Google didn't care about product excellence or something. Than cases like: 1. Person toils away on the right thing forever 2. Person tries to get promoted for doing the right thing and having impact 3. Google turns them down 4. They get pissed off and post on twitter about how Google didn't care about about product excellence or something. Externally, of course, people can't distinguish these two cases because they look the same (angry people on twitter). I have been promoted from SWE III to Senior Director at Google, working only on open source until i was an L7.
  - vechagup 4y ago
    Agreed. I work in the PP's PA and was promoted on my first try for doing unsexy but worthwhile maintenance work on an important tool. It can happen. In my estimation the best thing to do if you want to get promoted is 0) have a positive relationship with your manager 1) read the SWE ladder for the next level 2) give your manager a document that describes your work using the language of the ladder as closely as possible. Said ladder does not say you need to deliver a shiny new thing, despite what is commonly assumed in this forum.
gjvc 4y ago
> (All of these are examples of very senior promos at my own FAANG company). you have your own FAANG company? remember always that "your" company will drop you in a moment.
thockingoog 4y ago
Define "killing kubernetes"? It's still pretty successful and the adoption hasn't slowed in any way I can measure. I promise you that some site you used TODAY is running, at least part of it, on Kubernetes. Google employees regularly get promoted, at all levels, based on their OSS work - Kubernetes and other projects. We have dozens of people who work on Kubernetes, in one area or another, at varying degrees of depth. Is that "killing" the project? Of course it is never ENOUGH. I'd happily consume hundreds more people. :)

rdxm 5y ago

mdaniel 5y ago

https://threadreaderapp.com/thread/1511791378497384454.html https://threadreaderapp.com/thread/1511791378497384454.html

devy 5y ago

Brendan Burns, the Kubernetes co-founder who currently works at Microsoft Azure as a Corporate Vice President disagrees the "killing Kubernetes" part.[1][2] But yes, Big Tech's promo committee's short-sighted interests do NOT align well with multi-vendor sponsored open source projects' the long term prosperity. And it's no secrets that open source contributors are overworked. And it's not only affect K8s, but it affects across the board.[3] [1]: https://www.linkedin.com/in/brendan-burns-487aa590 https://www.linkedin.com/in/brendan-burns-487aa590 [2]: https://twitter.com/brendandburns/status/1511841189686771716 https://twitter.com/brendandburns/status/1511841189686771716 [3]: https://twitter.com/kantrn/status/1511791447091003395 https://twitter.com/kantrn/status/1511791447091003395

klabb3 5y ago
Well yes, what did you expect a corporate VP with skin in the game would say publicly on Twitter?
- hyperpape 5y ago
  The OP then agrees that Microsoft stands out as supporting OSS more than competitors. https://twitter.com/kantrn/status/1511844921921077249 https://twitter.com/kantrn/status/1511844921921077249

zozbot234 5y ago

Kubernetes' biggest problem by far is its mess of accumulated technical debt. FAANG promo committees very likely undervalue FLOSS contributions, but this hits all of FLOSS pretty much equally; it hardly explains why k8s specifically would be in so much trouble.

billllll 5y ago
I would go as far as to say it's not just limited to FLOSS contributors. In my experience at Google, not every service is profitable or brings in revenue, so it can be hard to justify "impact" on those teams. I certainly remember beating my head on how to spin my contributions as having "impact."

throwawax 5y ago

jthrowsitaway 5y ago
As someone that's manually stitched together some of what Kubernetes provides (before it even existed), I very much appreciate that Kubernetes exists.

datalopers 5y ago

Google created Kubernetes so potential competitors are dragged down with immense technical debt, of course they’re not going to award internal advancement of it.

busterarm 5y ago
Having participated in full on rearchitectures to Kubernetes several times at this point I can say that it wasn't justified a single time, even at the 10^4 microservice scale. Now having participated in several fairly-effortless rollouts of Nomad and arrived at a better place, it's funny to watch the rest of the industry cargo cult. I actually don't believe that either of these solutions are going to be a long-term settled end state (though Nomad with Firecracker, or really any other Firecracker-centric solution that isn't k8s will have some legs) the industry falls on, but I agree wholeheartedly that Kubernetes is purely a distraction having seen the pain and the gnashing of teeth. Every painless Kubernetes story that I've seen is at a scale where Kubernetes wasn't even necessary/justifiable and another solution would have been even simpler. But at least it's good for the resume. And Helm charts are are akin K8s' own borgcfg. We're doomed to repeat our mistakes it seems.
- streetcat1 5y ago
  Can you elaborate more? Why it was not justified? The more detailed the better (If possible, concrete issues that you faced) ?
  - busterarm 5y ago
    I've seen small startups waste their time with things like self-hosted Kubernetes in a quarter-rack of colo space for workloads they could have instead hosted with KVM or on cloud instances with 1/100ths of the management overhead. Scale that up to a half-dozen racks of servers and you're really still telling the same story. Even OpenStack of all things is easier to manage. This covers the scale of the actual operations of 99% of companies. Probably some more 9's there afterwards. Go and read about how StackOverflow's infrastructure has developed over the years and how damned simple and effective it is. If you aren't Fortune 100 or you don't have extremely specific performance needs for your (re)deployments, then it's highly likely that rolling out Kubernetes infrastructure is akin to driving screws with a sledgehammer. I think part of this is the downside of capital being far too cheap for too long. Companies have way overbuilt their infrastructure for reasons that aren't really moving the needle forward. Many barely make an effort to control their costs. At most companies my expectation going in is to look at their infrastructure and see somewhere between 0.02 and 10% hardware utilization across everything they have. Even the companies running Kubernetes hardly seem to be doing a better job because very rarely is Kubernetes running 100% of their infrastructure, if even 10%.

colinmhayes 5y ago

> going L6->7 at Google is worth ~200k/year, 7->8 is ~400. Similar patterns at other places. People have kids, mortgages, student loans Honestly this is a bit ridiculous. If making 500k as an L6 isn't good enough for you maybe you should be working on projects that provably bring your company revenue?

adastra22 5y ago
This is insane. I can't understand these numbers. No wonder the Bay Area has become completely unaffordable for long-term residents.
- deleted 5y ago
  [deleted]
- barry-cotter 5y ago
  Indeed. If lots of people with money want to live in a place and building more housing is illegal the price of existing housing will go up a lot. If you want the problem to go away you can either build more housing or make it so the people with money don't want to live there.
- rchiang 5y ago
  Google makes $5.5 billion of profit per month. I think they can afford to pay some of their higher performers more.
- vwoolf 5y ago
  The Bay Area is unaffordable because most cities haven't allowed anywhere near enough housing to be built by land owners over the last five decades: https://techcrunch.com/2014/04/14/sf-housing/ https://techcrunch.com/2014/04/14/sf-housing/. Return building rights to land owners and the problem will be solved. The insane thing is the way coalitions of existing owners conspire to reduce supply and thus increase the values of their assets, and the rest of us just go, "Oh, okay, that's cool, that you're conspiring to embeggar the rest of us."
- mike_d 5y ago

hapless 5y ago

Go figure, contributions to a multi-vendor consortium that just barely makes a marginal profit for an already marginal business unit (GCP) are not as highly valued as things that can be directly connected to revenues. This is not surprising in the least. Perhaps obviously, Mr. Kantrowitz hasn't quit, or been poached by a competitor, so I would say Google's (cynical) comp and promo policy is working out just great for them. Whatever personal interest keeps Mr. Kantrowitz going is working out for them!

hyperpape 5y ago
He does not appear to work at any of the FAANG companies: https://www.linkedin.com/in/noahkantrowitz/ https://www.linkedin.com/in/noahkantrowitz/.

sydthrowaway 5y ago

As someone in Europe, I am so sick of the implied salary pissing contest in every thread.

busterarm 5y ago
Market forces are moving in your favor. Most European companies have completely saturated their target markets for outsourcing (Barcelona, Estonia, Czech Republic) and Russia & Ukraine are suddenly unavailable options due to a war. Also the local markets have much more demand for engineers than are available to fill it. There has been a large push in the last two years to recruit Americans and Canadians to fill their unmet needs and salaries have been climbing rapidly. Salaries for new job postings in Berlin are about where they were at in New York 3-4 years ago which is EXCELLENT for that market (seriously, two years ago salaries were maybe half of what they are now). If you're in Europe and you haven't been interviewing you should look around and maybe even use a recruiter. My own recruiter has been feeding me opportunities across Europe with pretty much equal money to what I'm getting here in the States. It's likely that if you haven't changed jobs in the last two years that you're leaving significant amounts of money on the table. That said, I know that chasing after money isn't fashionable or socially acceptable to most Europeans...but the new hires are going to get it -- why shouldn't you?
- sydthrowaway 5y ago
  How can we get access to your recruiter?
- int_19h 5y ago
  There's over 4 million (and counting) Ukrainian refugees in Europe right now, and quite a few of those people are software engineers. Then there's over 100k (and counting) Russians that rushed to leave the country before the new iron curtain goes down - and I wouldn't be surprised if most of those are software engineers.
  - busterarm 5y ago
    You do know that refugees legally aren't allowed to work, right? You must fully complete your asylum-seeking process before you are allowed to seek employment. That often takes years if not decades. And because of various financial embargos it's historically very hard to hire Russians in European countries unless they have bank accounts in those countries, which without EU citizenship or long-term residency are very, very hard to get. Neither of these countries' citizens are allowed to freely travel and work within the EU the way that EU members can.

eigen-vector 5y ago

This seems like a strictly Google only problem. And by extension this is the problem with promotion committees that are removed from day to day of the candidate getting promoted. People are ridiculously bad at measuring growth based on some obscure company wide criteria - which is abstract at best and useless at worst at a company large enough as Google. I'm at AWS and the promotion process here is largely contained within the candidate's org which is at best two levels higher but in the same business. So if your team's business relies on contributions to FOSS, its impact is measurable and leaders and the 'committee' can easily tell you if your contributions are at the next level.

anonymouscabc1 5y ago
Yes FB is entirely in org too. It's weird to go out of org.
deleted 5y ago
[deleted]
grogenaut 5y ago
L7+ tech generally has in and out of org reviewing the promo. One for context, one for keeping the bar even.

axg11 5y ago

This thread is ridiculous. At FAANG, and everywhere really, promos are decided based on impact and influence. FAANG are just more likely to be working on open source projects than employees at other companies. All impact is difficult to quantify, not just contributions to OSS. The only easily quantifiable achievements for an engineer are delivered projects that directly generate revenue. In the teams I’ve worked in, I’d say that covers perhaps a third of projects.

qbasic_forever 5y ago
Let's pretend Linus Torvalds worked for Google, would he be able to get promoted for creating and maintaining Linux? As far as the bottom line goes it brings in zero dollars and costs the company his total compensation or more to maintain. How would Linus quantify his revenue impact in his promotion packet and compete against another hypothetical engineer that improved ad targeting by 0.0X% and can directly measure an increase of revenue for the comapny in the millions of dollars?
- rrdharan 5y ago
  Revenue is not the only way to quantify impact. Plenty of infrastructure teams promote plenty of people at Google and other companies despite not directly selling products.
- igetspam 5y ago
  Guido got paid to work on python
- busterarm 5y ago
  Also it's Google, so we should be looking to promote operating system authors who aren't white or male anyway. Expecting hate for this, but I've been on the receiving end of company notifications (not at Google) where it was stated explicitly that the newly-opened distinguished engineer role was exclusively available to not-men. (I'm generally-supportive of DEI efforts, but at sane companies those efforts wouldn't block qualified people from being promoted to their proper roles. Among the FAANGs, I have a lot more confidence in Facebook and Amazon's ability to operate sanely than I do Google. Prior evidence and all.)

hintymad 5y ago

This is not really just the problem in Google. We might as well claim that promotion processes in big companies promote promotion-oriented work (yes, I intentionally use the word promotion this many times). The result is title inflation, lots of good employees leaving, and superfluous yet mediocre projects. I wouldn't say only unqualified people get promoted, though. Big companies have very different dynamics than small ones. Some people are just good at navigating big companies and are capable of aligning multiple organizations, for good or for bad. Big companies need such talent as well. As for individuals, the reward of getting promoted is not worth the effort, at least statistically. A much better alternative, is focusing on solving truly interesting and meaning problems in a blow-out company. The financial return and title bumps will follow naturally.

faddypaddy34 5y ago

I mean you should really be demoted for continuing to push the k8s microservices crap.

likpok 5y ago

Kubernetes doesn't get patches from hyperscale companies because kubernetes is not really hyperscale software. If I recall correctly (probably don't), clusters can handle tens of thousands of container hosts. That's not manageable at large scale unless you're selling people individual kubernetes instances directly (which, to be fair, google and amazon are). In my experience the thing that kills open source at big companies is the divergence of needs: The company needs massive scale software, but the community wants ease-of-use and features. Most open source models are a big drag on productivity (you can't just refactor all the callsites to do a migration!), and it's hard to change directions when the company's need does (since the other contributors will likely want the original software). The tailscale people have been arguing this as well (with more data and better reasoning), but fundamentally you need different software when you're a scrappy startup or a medium size company or a giant behemoth.

busterarm 5y ago
Having operated all the way up to "hyperscale", I agree with your point completely.
- zozbot234 5y ago
  What's preventing k8s from being useful in these larger scale deployments? GP was not very clear about that, they just said that it's not scaling enough.
  - busterarm 5y ago
    K8s doesn't do anything to help you with managing the fleet of k8s hosts. At 10^5 host machines you have a much bigger problem on your hands than the one that k8s is solving for. If you have and are solving that hyperscale infrastructure problem then whether or not you're running kubernetes doesn't actually make a damn bit of difference to how you allocate your human/financial resources. Or summarized, at the scale where Kubernetes is advertised to make the biggest difference, it's actually irrelevant/an implementation detail.

dbish 5y ago

I love open source but I’m happy if we take the credit for killing kubernetes :)

ChrisArchitect 5y ago

Threadrip to make 'this could have been a blog post' more readable: https://the.rip/@kantrn-1511791378497384454-121542889 https://the.rip/@kantrn-1511791378497384454-121542889 https://unrollthread.com/t/1511791378497384454/ https://unrollthread.com/t/1511791378497384454/

cletus 5y ago

Not sure why this says FAANG because it's talking about Google. Calibration, promo committees, feedback and the like are intended to create equivalent expectations on impact across different orgs. The dirty little secret however is that at best it has limited success. The best advice I can give anyone in such an organization is to be liked by your manager and their manager. If that's true, good things will tend to happen. If it's not, good things will be a lot less frequent. Put another way: you can take the exact same set of objective facts and use them to say a person did a good job or a bad job. There's a popular meme about feedback at Google that goes something like: > This project would've failed without this person. It failed anyway but it definitely would've failed without them. The difference ultimately boils down to whether or not they like you. Here are a few Google-specific tidbits worth knowing: 1. Ratings are fit to a curve across a sufficiently large pool, typically at the director level and usually over 100-150+ people. This means there will be a percentage range of people who get Meets All, Exceeds Expectations, Greatly Exceeds, etc. This is intended to stop ratings inflation; 2. A consequenc eof (1) is that ultimately you are competing with people in your org for those better ratings. This can create some perverse incentives and a toxic environment; 3. It is almost always better to let something blow up and come and fix it rather than preventing that from ever happening. The first will get you a lot of recognition. The latter will get you almost none; 4. Promos at Google are stack-ranked. Each committee gets 10-15 packets that will be for a particular level. The committee will rank those packets. After that the promotion target will come into play. This is set by management and was allegedly cut as a cost-saving measure when Ruth Porat came on board. If it's 20% then the top 20% from that ranking process be promoted. You will find people who serve on those committees who say this isn't how it works and they'll argue they're evaluating if someone is operating at the next level or not. This is partially true. Thes packets will be divided between promote, don't promote and on-the-bubble. The on-the-bubble group will be sufficiently large to allow for the promotion target to be met; 5. For SWEs. L5 is the "terminal" level, meaning there is an expectation of growth to that level. L3->L4 and L4->L5 once went through promo committee but now don't. Management within orgs decide this. These too have target percentages and there have been cases where the promot rate has been too "high" and orgs have been told to cut back on promotions to meet the targets; 6. There is a massive backup at L5->L6. Because of the low target percentages the impact required keeps going up and really you need your management to really push for this to happen. There are limited slots so you may be waiting eyars and again this is why them liking you matters so much. Google is full of L6s who got promoted 5-10+ years ago that would never make the grade by today's standard. For really old cases you can find archives of why there were promoted and you'll find cases like "promoted unit testing". I say all this because the author of this thread seems to fundamentally misunderstand how this process works.

deleted 5y ago

[deleted]

GauntletWizard 5y ago

I basically can't stand the comment section of Kubernetes threads. Lots of big teams are getting big things done with k8s, and you would not know it from here.

MuffinFlavored 5y ago

What's the alternative to k8s then?

marsven_422 5y ago

[dead]

lumost 5y ago

This is the case with most major products from most major tech companies. There is a reason you don't (often) see steady incremental progress on products, but rather big flashy releases that never quite get their bugs cleaned up unless they are wildly* successful. * Where wildly is defined as a success level which would leave any rational founder very wealthy, and very content.

avs733 5y ago

It sounds like FAANG has reinvented tenure and promotion processes from academia…and the resulting inherent pressures. A moment to reflect on Han’s critique of publish or perish academia?

henning 5y ago

Hopefully Kubernetes does die. Most companies write shit slow software and then think they're big smart boys because they have to buy $20,000/month of servers to make it not be unusable. In many shops it could and should be replaced by a handful number of efficient servers that do not suck shit.

reilly3000 5y ago

This is just the law of inertia at play. Big projects slow down. K8s isn’t just the K8s core, it’s a massive ecosystem, and when taken as a whole it’s unbelievably vast and growing at an amazing pace. As it does, the core of it moves slower, there are more stakeholders, more arguments. Small changes make for big problems and not everyone can be happy. Patches will keep coming, major changes will become eventually impossible. The death of Kubernetes makes for great clickbait, but most of us won’t see it in our lifetimes. Not when our banks run on 60 year old code. Kubernetes is too big to fail at this point. As far as FAANG promo committees go, let them value what they value. Kubernetes is a direct revenue driver for many companies; it’s health is tied to billions of dollars in investments. Just because one cohort of contributors age out, cash out, or fall in love with someone doesn’t mean there is no one to take their place. The new people won’t do it the same way and that is okay, even great. I’m grateful for the vision and effort that have make K8s the platform it is, and if I can translate that into a contribution in the future I will. Finally, I’ll say that people who really love working on the project may not get support to work on it full time, but then may find themselves able to retire sooner than many and have ample opportunity to contribute.

throwaway787544 4y ago

K8s is a great example of what murders open source: monolithic platforms that exist unto themselves, but have some seemingly benign model allowing arbitrary "plugins" to pretend it's not really a monolith in sheep's clothing. It looks like open source and has the right license, but it ends up just mirroring corporate interests rather than those of the Commons that the open source movement was created to support.

zozbot234 4y ago
If you're going to have monolithic platforms anyway (and you will, because designing for modularity is hard, whereas hacking together some random monolithic code is easier) they should at least be open source rather than proprietary.
- throwaway787544 4y ago
  Actually I disagree. Any time there's an Open Source tool that does everything for free, it's extremely hard to justify paying for a proprietary product that does it much better, because companies are cheap. There's many open source projects like this that are just terrible but a company will always use them first because they're free. The answer isn't modularity, it's composeability, which is a significant difference. A modular program requires "integration" (tight coupling of APIs/ABIs) whereas a composeable program has loose interfaces which require virtually no "integration". Drone.io is one example; you can implement any "plugin" purely by creating a container with an initial CMD entry that reads environment variables, and that program can interface back with Drone a number of ways (STDIN/STDOUT, REST API, database). Another is any application that just reads in or spits out simple line-by-line instructions or a JSON blob. Unix pipe based programs are the penultimate example. The dumber the interface, the easier it is to compose.

wallfacer120 4y ago

Oh man, another rant that boils down to "big evil corporations are KILLING open source software by not supporting it in the specific ways and in the specific amounts that I want."

otabdeveloper4 4y ago

> killing Kubernetes Really? Finally some good news in these troubled times.

wink 4y ago

That's the most ridiculous bubble valley thing I read in a while, coming from a non-k8s using non-FAANG-employed European. Which isn't meant dismissive of the problem at hand, I can totally see how it's happening and how it might be bad for the people involved in the project. To phrase it differently - if the project is dying because it hinges on people working on a promotion that gives them a /bigger paycheck increase than my yearly income/ - well, good riddance. I'm staying on my side of the fence and continue to not buy a new Tesla every year.

helen___keller 4y ago
> if the project is dying because it hinges on people working on a promotion that gives them a /bigger paycheck increase than my yearly income/ - well, good riddance. I'm staying on my side of the fence and continue to not buy a new Tesla every year. An understandable position but this is simply how competitive environments work. It’s a lot like the college application mania. There’s tons of gifted high school students who clearly can succeed without needing to volunteer in a foreign country every summer while playing oboe and competing in lacrosse at the state level. But seats at Harvard are competitive, and those same talented and well-to-do high school seniors often only have college admissions as their ruler to measure their growth and self-worth. So they compete on every angle to try and get into Harvard rather than merely Dartmouth or heaven forbid BU. It’s reasonable to think working adults should have measures of personal growth that don’t rely on climbing the corporate ladder, but conversely all those who *do* obsess with climbing the ladder will naturally be clustered around places of wealth/prestige, because that’s what they are optimizing for. Much like the Harvard application pool, you’ll find these types doing what it takes to stand out and land that 700k/yr L7 position at Google. So I guess the answer is that the most stable and secure position for an open source project to be in, is one that does not rely on development being coupled to big tech. I imagine most open source folks could have told you that anyways :)

shiftoutbox 4y ago

cbushko 4y ago

There are certain classes of work that are completely ignored by most companies such as build, automation, developer tooling and infrastructure cost cutting. These are not flashy features that you can show off to customers but they can pay in dividends year over year. (Personally, I have saved millions a year in infrastructure costs and my promotions/compensation were not close to what a sales person would get if they brought in millions of recurring revenue)

thockingoog 4y ago

There's a perception that this is true, and like all perceptions it is based in reality, but it is not the reality itself. We have plenty of data that shows that people DO get promoted at all levels based on OSS work.