14 ms·

I work at Google so my perspective is to be biased, but that's not what I see. I work on infrastructure, and so a few years back, when I proposed a major proje

by tytso 4y ago

I work at Google so my perspective is to be biased, but that's not what I see. I work on infrastructure, and so a few years back, when I proposed a major project, I had to demonstrate how it would save *many* times the fully loaded cost of the engineers on the team, by reducing the Storage TCO for all of Google (for example). It was not enough for the project to "break even" --- the benefits had to do more than just exceed the "nominal" SWE cost. It had to be multiple times the cost of the SWE's, to account for the opportunity cost of those SWE's --- SWE's are a constrained resource, which is why a project needs to save $$$ (or increase profits) by many multiples the fully loaded SWE cost. (That project has since been completed, successfully, and I got a promotion to Sr Staff Engineer out of it.) The reason why SWE's are a constrained resource is becaused finding good SWE's is non-trivial. As a TL, I don't want to waste my precious approved headcount on people who just want to rest and vest, or people who believe in the crazy talk of only needing to work 30 minutes each day. I'm trying to find highly motivated, smart, and talented SWE's who can also be team players. And if they need to have domain expertise (say, be proficient kernel engineers), it's super-duper difficult. So I don't see any indication of people getting hired just to starve statups of talented engineers. We need every single talented engineer we can get for the projects that we want to accomplish. And in the time when we may need to slow down our growth, it may mean that we will need to slow, or shut down some projects. That may suck, especially if it's a project that we had invested a lot of passion into. But it's certainly no reason to panic. Slowing down growth is not the same as layoffs, and there is no shortage of work for us to do.

deleted 4y ago
[deleted]
saagarjha 4y ago
> As a TL, I don't want to waste my precious approved headcount on people who just want to rest and vest, or people who believe in the crazy talk of only needing to work 30 minutes each day. And yet these people exist and get hired, even at Google. Perhaps not on your team, but they’re definitely there.
- sbierwagen 4y ago
  21% of Alphabet was hired this quarter. >Alphabet, Google’s parent company, said its head count rose 21% in the second quarter of 2022 to 174,014 full-time employees from 144,056 the year prior. https://www.hcamag.com/us/specialization/corporate-wellness/google-chief-people-officer-addresses-potential-layoffs/415192 https://www.hcamag.com/us/specialization/corporate-wellness/... SV talks the big talk about hiring the best of the best of the best, then hires 30 thousand people in three months. There probably aren't that many 10x programmers in the world.
  - marshray 4y ago
    Quibble: A 21% rise results in 17% of the new group having come from the increase.
  - ghaff 4y ago
    Interview process gatekeeping notwithstanding, the idea that you can hire 30,000 people (admittedly a decent proportion is probably backfilling/growing administrative, etc. positions) who are all "the best and the brightest" doesn't pass the sniff test.
  - skinnymuch 4y ago
    That’s over 12 months. It’s year over year.
kragen 4y ago
You also wrote major parts of Linux, so it's possible your contact with Googlers might be biased towards the more functional parts of the organization. Your bar for "talented engineer" may be higher than average parts of Google.
vl 4y ago
As so many people learned over the years in storage org, essential modus operandi was “just do your current shit, this new shit you are proposing is too hard and we don’t want to do it”. I lost count of improvement proposals that ultimately were not getting supported (there was a funny one with (first) hosted NFS prototype, when they tried to turn it off, turned out there were pissed-off customers running business workloads already). Instead we tried to fit square pegs into the round holes “by using existing technology”.
- tytso 4y ago
  Oh, you can certainly do big projects. My project[1] spanned 3 departments, and involved dozens of engineers, and required that we work with multiple hard drive vendors (our first two partners for Hybrid SMR were Seagate and WDC) on an entirely new type of HDD, as well as the T10/T13 standards committees so we could standardize the commands that we need to send to these HDD's. So this was all a huge amount of "new shit" that was not only new to Google, it was new to the HDD industry. You just have to have a really strong business case that shows how you can save Google a large amount of money. [1] https://blog.google/products/google-cloud/dynamic-hybrid-smr-ocp-proposal-improve-data-center-disk-drives/ https://blog.google/products/google-cloud/dynamic-hybrid-smr... [2] https://www.t10.org/pipermail/t10/2018-September/018566.html https://www.t10.org/pipermail/t10/2018-September/018566.html On the production kernel team, colleagues of mine worked on some really cool and new shit: ghOSt, which delegates scheduling decisions to userspace in a highly efficient manner[3]. It was published in SOSP 2021/SIGOPS [4][5], so peer reviewers thought it was a pretty big deal. I wasn't involved in it, but I'm in awe this cool new work that my peers in the prodkernel team created, all of which was not only described in detail in peer-reviewed papers, but also published as Open Source. [3] https://research.google/pubs/pub50833/ https://research.google/pubs/pub50833/ [4] https://www.youtube.com/watch?v=j4ABe4dsbIY https://www.youtube.com/watch?v=j4ABe4dsbIY [5] https://dl.acm.org/doi/10.1145/3477132.3483542 https://dl.acm.org/doi/10.1145/3477132.3483542 We have some really top-notch engineers in our production kernel team, and I'm very proud to be part of an organization has this kind of talent.
  - vl 4y ago
    I'm not saying storage didn't do big projects. I'm saying that over time it got calcified and instead of doing proper stack refactoring and delivering features beneficial for customers, it continued to sadly chug along team boundaries. For example: RePD is at just wrong level at all. It should have been at CFS/chunk level and thus benefit other teams as well. BigStore stack is beyond bizarre. For years there were no object-level SLOs (not sure if there are now), which meant that sometimes your object disappeared and BigStore SREs were "la-la-la, we are fully within SLO for your project". Or you would delete something and your quota would not get back, and they would "or, Flume job got stuck in this cell, for a week...". Not a single cloud (or internal, for that matter) customer asked for a "block device", they all want just to store files. Which means that cloud posix/nfs/smb should have been worked on from the day 1 (of cloud), we all know how it went.
francisofascii 4y ago
> proficient kernel engineers How is the world does one become a proficient kernel engineer? Without already being at a FANNG.
- tytso 4y ago
  Here are the latest development statistics from the just-released 5.19 kernel. (Please consider supporting Linux Weekly News by subscribing if you find content like this useful; one of the benefits is you can help share subscriber-only content to friends and colleagues via Subscriber Links): https://lwn.net/SubscriberLink/902854/b788a6a3d77aba7a/ https://lwn.net/SubscriberLink/902854/b788a6a3d77aba7a/ If you scroll down to the Most active employers in 5.19 by commits you'll see: 1. Intel 10.9% 2. (Unknown) 7.5% 3. Linaro 5.7% 4. AMD 5.5% 5. Red Hat 5.2% 6. (None) 4.3% 7. Google 4.1% 8. Meta 3.5% 9. SUSE 3.1% 10. Huawei 2.9% The statistics are slightly different if you count by lines of codes changed, but either way, it's not all FANNG companies, not by a long shot. There are plenty of people who get started coding via kernelnewbies.org and other resources.

by tytso 4y ago

deleted 4y ago

[deleted]

saagarjha 4y ago

> As a TL, I don't want to waste my precious approved headcount on people who just want to rest and vest, or people who believe in the crazy talk of only needing to work 30 minutes each day. And yet these people exist and get hired, even at Google. Perhaps not on your team, but they’re definitely there.

sbierwagen 4y ago
21% of Alphabet was hired this quarter. >Alphabet, Google’s parent company, said its head count rose 21% in the second quarter of 2022 to 174,014 full-time employees from 144,056 the year prior. https://www.hcamag.com/us/specialization/corporate-wellness/google-chief-people-officer-addresses-potential-layoffs/415192 https://www.hcamag.com/us/specialization/corporate-wellness/... SV talks the big talk about hiring the best of the best of the best, then hires 30 thousand people in three months. There probably aren't that many 10x programmers in the world.
- marshray 4y ago
  Quibble: A 21% rise results in 17% of the new group having come from the increase.
- ghaff 4y ago
  Interview process gatekeeping notwithstanding, the idea that you can hire 30,000 people (admittedly a decent proportion is probably backfilling/growing administrative, etc. positions) who are all "the best and the brightest" doesn't pass the sniff test.
- skinnymuch 4y ago
  That’s over 12 months. It’s year over year.

kragen 4y ago

You also wrote major parts of Linux, so it's possible your contact with Googlers might be biased towards the more functional parts of the organization. Your bar for "talented engineer" may be higher than average parts of Google.

vl 4y ago

As so many people learned over the years in storage org, essential modus operandi was “just do your current shit, this new shit you are proposing is too hard and we don’t want to do it”. I lost count of improvement proposals that ultimately were not getting supported (there was a funny one with (first) hosted NFS prototype, when they tried to turn it off, turned out there were pissed-off customers running business workloads already). Instead we tried to fit square pegs into the round holes “by using existing technology”.

tytso 4y ago
Oh, you can certainly do big projects. My project[1] spanned 3 departments, and involved dozens of engineers, and required that we work with multiple hard drive vendors (our first two partners for Hybrid SMR were Seagate and WDC) on an entirely new type of HDD, as well as the T10/T13 standards committees so we could standardize the commands that we need to send to these HDD's. So this was all a huge amount of "new shit" that was not only new to Google, it was new to the HDD industry. You just have to have a really strong business case that shows how you can save Google a large amount of money. [1] https://blog.google/products/google-cloud/dynamic-hybrid-smr-ocp-proposal-improve-data-center-disk-drives/ https://blog.google/products/google-cloud/dynamic-hybrid-smr... [2] https://www.t10.org/pipermail/t10/2018-September/018566.html https://www.t10.org/pipermail/t10/2018-September/018566.html On the production kernel team, colleagues of mine worked on some really cool and new shit: ghOSt, which delegates scheduling decisions to userspace in a highly efficient manner[3]. It was published in SOSP 2021/SIGOPS [4][5], so peer reviewers thought it was a pretty big deal. I wasn't involved in it, but I'm in awe this cool new work that my peers in the prodkernel team created, all of which was not only described in detail in peer-reviewed papers, but also published as Open Source. [3] https://research.google/pubs/pub50833/ https://research.google/pubs/pub50833/ [4] https://www.youtube.com/watch?v=j4ABe4dsbIY https://www.youtube.com/watch?v=j4ABe4dsbIY [5] https://dl.acm.org/doi/10.1145/3477132.3483542 https://dl.acm.org/doi/10.1145/3477132.3483542 We have some really top-notch engineers in our production kernel team, and I'm very proud to be part of an organization has this kind of talent.
- vl 4y ago
  I'm not saying storage didn't do big projects. I'm saying that over time it got calcified and instead of doing proper stack refactoring and delivering features beneficial for customers, it continued to sadly chug along team boundaries. For example: RePD is at just wrong level at all. It should have been at CFS/chunk level and thus benefit other teams as well. BigStore stack is beyond bizarre. For years there were no object-level SLOs (not sure if there are now), which meant that sometimes your object disappeared and BigStore SREs were "la-la-la, we are fully within SLO for your project". Or you would delete something and your quota would not get back, and they would "or, Flume job got stuck in this cell, for a week...". Not a single cloud (or internal, for that matter) customer asked for a "block device", they all want just to store files. Which means that cloud posix/nfs/smb should have been worked on from the day 1 (of cloud), we all know how it went.

francisofascii 4y ago

> proficient kernel engineers How is the world does one become a proficient kernel engineer? Without already being at a FANNG.

tytso 4y ago
Here are the latest development statistics from the just-released 5.19 kernel. (Please consider supporting Linux Weekly News by subscribing if you find content like this useful; one of the benefits is you can help share subscriber-only content to friends and colleagues via Subscriber Links): https://lwn.net/SubscriberLink/902854/b788a6a3d77aba7a/ https://lwn.net/SubscriberLink/902854/b788a6a3d77aba7a/ If you scroll down to the Most active employers in 5.19 by commits you'll see: 1. Intel 10.9% 2. (Unknown) 7.5% 3. Linaro 5.7% 4. AMD 5.5% 5. Red Hat 5.2% 6. (None) 4.3% 7. Google 4.1% 8. Meta 3.5% 9. SUSE 3.1% 10. Huawei 2.9% The statistics are slightly different if you count by lines of codes changed, but either way, it's not all FANNG companies, not by a long shot. There are plenty of people who get started coding via kernelnewbies.org and other resources.