7 ms·

in addition to contributing to upstream python, we * maintained a stable version of python within google, and made sure that everything in the monorepo worked

by zem 3y ago

in addition to contributing to upstream python, we * maintained a stable version of python within google, and made sure that everything in the monorepo worked with it. in my time on the team we moved from 2.7 to 3.6, then incrementally to 3.11, each update taking months to over a year because the rule at google is if you check any code in, you are responsible for every single breakage it causes * maintained tools to keep thousands of third party packages constantly updated from their open source versions, with patch queues for the ones that needed google-specific changes * had highly customised versions of tools like pylint and black, targeted to google's style guide and overall codebase * contributed to pybind11, and maintained tools for c++ integration * developed and maintained build system rules for python, including a large effort to move python rules to pure starlark code rather than having them entangled in the blaze/bazel core engine * developed and maintained a typechecker (pytype) that would do inference on code without type annotations, and work over very large projects with a one-file-at-a-time architecture (this was my primary job at google, ama) * performed automated refactorings across hundreds of millions of lines of code and that was just the dev portion of our jobs. we also acted as a help desk of sorts for python users at google, helping troubleshoot tricky issues, and point newcomers in the right direction. plus we worked with a lot of other teams, including the machine learning and AI teams, the colaboratory and IDE teams, teams like protobuf that integrated with and generated python bindings, teams like google cloud who wanted to offer python runtimes to their customers, teams like youtube who had an unusually large system built in python and needed to do extraordinary things to keep it performant and maintainable. and we did all this for years with fewer than 10 people, most of whom loved the work and the team so much that we just stayed on it for years. also, despite the understaffing, we had managers who were extremely good about maintaining work/life balance and the "marathon, not sprint" approach to work. as i said in another comment, it's the best job i've ever had, and i'll miss it deeply.

fluffyfe 3y ago
That is a lot of great work. Sorry to hear that it is gone. Take some time to rest and hope find somewhere with better vision!
deleted 3y ago
[deleted]
iandanforth 3y ago
When was pytype invoked? Runtime? CI/CD? Did it output types to be added to files or did it infer and then enable downstream tasks based on those inferences?
- zem 3y ago
  it was invoked manually at build time, and then at CI/CD time (there was no runtime type checking). we did output types to pyi files (the python equivalent of C header files), but mainly we used the inferred types to check the code for type correctness - one of the unique selling points of pytype, and why google was developing its own typechecker in the first place, was that the other available python type checkers would check annotated code for consistency, but would not do anything about type errors in unannotated code. if pytype did detect type errors, the code failed CI and had to be fixed before it was checked in. we did also have a separate tool that could merge the inferred types back into the source code as annotations, but it never got much uptake.
zem 3y ago
> the rule at google is if you check any code in, you are responsible for every single breakage it causes i can no longer edit my post to clarify this, but by "responsible for breakages" i meant that if your new check-in caused any CI tests anywhere within the codebase to fail, even if due to bugs in the other code, you had to stop and fix it, or get the owners to fix it, or find some principled way to temporarily disable those tests, before you could check your code in. this was a very real issue for things like the python runtime or widely used code checkers like pytype or pylint, because if e.g. the new version of python, or some improved check in pytype, started raising failures in a code pattern that was technically wrong but which the existing toolchain did not complain about, you could not release the new version until you had fixed all those new breakages. contrast this with non-monorepo codebases, where the other code would have had to deal with the fact that "oh, our code works under 3.10 but 3.11 broke it, guess we have to pin our own repo to 3.10 until we fix it", but as the python team we could just say "we support 3.11 now, you need to catch up"
- Arainach 3y ago
  > i can no longer edit my post to clarify this, but by "responsible for breakages" i meant that if your new check-in caused any CI tests anywhere within the codebase to fail, even if due to bugs in the other code, you had to stop and fix it, or get the owners to fix it, or find some principled way to temporarily disable those tests, before you could check your code Google engineers hate this one weird hack: just send a mail saying "we're deleting/deprecating [everything your product depends on] in [14-180] days, good luck!"
  - lbruno 3y ago
    this reminds me of kenton v's complaint about working on infra being so hard
  - pjc50 3y ago
    I'm guessing these are related. If every team is potentially responsible for breakages in every product, there's no such thing as a "just chugging along" product and there will be a constant demand to delete products that are not popular within Google.
stevage 3y ago
Can you say more about the automated refactoring? How did it work? How was it triggered? How much human involvement was there? Are any parts of it public?
- zem 3y ago
  can't say too much about it because it mostly used a lot of internal tools, but it was a combination of writing per-file utilities to do specific code transforms (think something like libcst), and then a framework to scan the google codebase, find affected files, and apply the transforms to them.
  - banish-m4 3y ago
    Code mods are fun. Were they gated by human PR reviews, perhaps in batches, or applied without review?
    - zem 3y ago
      yes, gated by human reviews based on who owned the code, unless they were completely trivial and then were gated by a single global approver. there were always humans in the loop though.
      - banish-m4 3y ago
        Ah makes sense. At another MAANG, there was a code mod template job with a half dozen subvariants. I recall using a variant that supported ripgrep piping to sed -i for a simple replacement mod. There were jobs properties controlling how many mods per diff and whether it needed human approval or not. I think I did one batched by 1000 running once every 2 weeks to keep noise down.
iamcreasy 2y ago
> needed to do extraordinary things to keep it performant and maintainable. This is very interesting. Can you share more about this?
- zem 2y ago
  i can't, unfortunately, it was youtube internal code. but by the same token a lot of the performance stuff was specific optimisations for that code and would not really generalise. one of the current "extremely large python codebase" projects is instagram, and they do have some public repos, notably monkeytype (https://github.com/Instagram/MonkeyType https://github.com/Instagram/MonkeyType) which youtube did have its own analogue of, and cinder (https://engineering.fb.com/2022/05/02/open-source/cinder-jits-instagram/ https://engineering.fb.com/2022/05/02/open-source/cinder-jit...). in general facebook's engineering blog is a great place to read about this sort of thing.
dotancohen 3y ago
> very large projects with a one-file-at-a-time architecture (this was my primary job at google, ama) Is there any place where we could see such tools? A lot of the scripts that I write are contained in a single file, and I'm sure that there would be much to learn from seeing such project scripts. Thank you.
- zem 3y ago
  it's open source! check out https://github.com/google/pytype https://github.com/google/pytype and https://github.com/google/pytype/blob/main/docs/developers/tools.md#analyze_project https://github.com/google/pytype/blob/main/docs/developers/t... for more on the multi-file runner
  - dotancohen 3y ago
    Terrific! Thank you.
jpdaigle 3y ago
I’m so sorry.
banish-m4 3y ago
When Wall St. concerns lead to imposing cannibalistic capitalism plans that are pennywise but pound foolish. Just wait a few years. When they come calling for consulting services, charge them triple.
elbear 3y ago
Frankly, it sounds like a dream job for me too, to be responsible for a particular technology and make sure that it works for all your internal (and external) users.
- zem 3y ago
  agreed! it takes a certain kind of person to find that fun, but the people who enjoy it really enjoy it.
offbyone 3y ago
I was your opposite number at AWS for many years. Like you say, it was one of the best jobs I've ever had, but over there it was a career dead end. I wasn't laid off, but I had to leave it to have any hope of growing myself. I feel for ya, zem; if you ever turn up at a PyCon in person, lemme buy you a drink.
- zem 3y ago
  I guess it was in some ways a career dead end at google too, I joined as a senior engineer and never bothered applying for promotion to staff, and most of my teammates were similar. On the other hand, I never felt I wasn't growing myself because I learnt a lot, took on some very ambitious projects, and was trusted with a lot of ownership and input into "what do you feel the most effective thing you could be working on right now is?"
gpshead 3y ago
Adding to the wonderful writeup by my now-ex teammate (thanks!): Several of us were/are/TBD also involved in both long term strategic leadership and maintenance of the open source CPython project itself. That direct feedback line from a major diverse needs user into the project and ecosystem was valuable for the world. The reason I stayed on this team for 12+ years is as zem said. It was an ideal impactful alignment of people, abilities, priorities, and work life balance. My prior teams at Google... were often not. For the first half of our Python teams existence, there were only ~5 of us. Many early years were spent paying down internal tech debt accumulated from prior years of neglecting to have a strong Python strategy and letting too many do their own thing. Python was one of the very first languages used widely at Google. It was the last major backend language to get a language team. Signed, -- the now-ex runtimes TL
- zem 3y ago
  yes, i definitely should have said more about the cpython leadership work! i was responding to a comment that had already mentioned upstream contributions so i didn't think to highlight it more, but it was a huge contribution to both google and to cpython at large.
- gooob 3y ago
  well, why don't you guys stay together and continue work? start your own thing! google fired you, yes, but you're still alive and able to communicate. maybe you won't get paid as much as you did at google. but prove to them that you have something of value to provide to society.
CydeWeys 3y ago
Any guess how many SWE-yrs are now going to be spent migrating existing Python projects away from Python, now that Python seems to be unsupported at Google going forward?
mezyt 2y ago
[dead]
arcamox 3y ago
I feel your pain. All the best!
emmy1989 3y ago
[dead]
- samstave 3y ago
  [dead]
emmy1989 3y ago
[dead]
nocte 3y ago
How can someone hire the whole team? :)
Mxmilan 3y ago
sorry to hear about the situation at google, would love to help out. we connect laid-off engineers to startups. if anyone from the team is interested here is our discord server https://discord.com/invite/Bnp5zyc8nM https://discord.com/invite/Bnp5zyc8nM
e40 3y ago
My guess: the people in your group have very high salaries and they want to dump them. I would not be surprised if they hire people to make a new team at some point in the future. I run HR at a small software company reports and my HR person has a lot of friends in HR at FAANGs. For 10+ years we've been talking about hiring and salaries at Google (and others). It's just crazy to me how high salaries have gotten. I know someone that switched from Google to FB when they made $750k at Google and got $1M at FB. I said all this to say: I've talked to a bunch of people over the last 6 months that know someone laid off and: 1) they had a very high salary, and 2) the group they were in was hiring.
Canders42 3y ago
We are hiring here at Cruise and I believe your team could be a good fit for one of the roles that we are hiring for. Here is the job description: https://www.getcruise.com/careers/jobs/2831153/ https://www.getcruise.com/careers/jobs/2831153/ If any of you are interested, feel free to directly reach out to the recruiter Connor Anderson (Connor.Anderson@getcruise.com) or feel free to apply directly.
- webprofusion 3y ago
  Only issue I can immediately see is that the salary is about a quarter of the FANG rate.
  - nmfisher 3y ago
    I mean, that probably at least partly explains why they were laid off..
  - Canders42 3y ago
    Please keep in mind that the listed salary range on the job description is only the median base salary range. Not the full total compensation range that includes bonus and other factors to it. We can pay much more than what is listed there from a total compensation standpoint.
cheviana37 3y ago
We're hiring! https://boards.greenhouse.io/blackcanyonconsulting/jobs/5754387003 https://boards.greenhouse.io/blackcanyonconsulting/jobs/5754... Good life work balance
deleted 3y ago
[deleted]
fullmetal09 3y ago
> maintained tools to keep thousands of third party packages constantly updated from their open source versions. Hi Zem could you say more about how you maintained those thousands of third party packages? We are facing the same issue and would love to hear your insight!
- zem 3y ago
  sure! it used a collection of tools, the main ones were * copybara [https://github.com/google/copybara https://github.com/google/copybara] to mirror between github and our internal repo and transform the source tree layout to our internal conventions * mercurial patch queues [https://hgbook.red-bean.com/read/managing-change-with-mercurial-queues.html https://hgbook.red-bean.com/read/managing-change-with-mercur...] to apply internal changes there are internal tools to automate the copybara configuration for python packages specifically, and to map between mercurial and google's version control system, but that's the main idea, and a specific review process for adding new packages. the process is actually documented here though not all the referenced tools are available (or even applicable) externally: https://opensource.google/documentation/reference/thirdparty/python https://opensource.google/documentation/reference/thirdparty...
DavidAmelia 3y ago
[dead]

by zem 3y ago

fluffyfe 3y ago

That is a lot of great work. Sorry to hear that it is gone. Take some time to rest and hope find somewhere with better vision!

deleted 3y ago

[deleted]

iandanforth 3y ago

When was pytype invoked? Runtime? CI/CD? Did it output types to be added to files or did it infer and then enable downstream tasks based on those inferences?

zem 3y ago
it was invoked manually at build time, and then at CI/CD time (there was no runtime type checking). we did output types to pyi files (the python equivalent of C header files), but mainly we used the inferred types to check the code for type correctness - one of the unique selling points of pytype, and why google was developing its own typechecker in the first place, was that the other available python type checkers would check annotated code for consistency, but would not do anything about type errors in unannotated code. if pytype did detect type errors, the code failed CI and had to be fixed before it was checked in. we did also have a separate tool that could merge the inferred types back into the source code as annotations, but it never got much uptake.

zem 3y ago

> the rule at google is if you check any code in, you are responsible for every single breakage it causes i can no longer edit my post to clarify this, but by "responsible for breakages" i meant that if your new check-in caused any CI tests anywhere within the codebase to fail, even if due to bugs in the other code, you had to stop and fix it, or get the owners to fix it, or find some principled way to temporarily disable those tests, before you could check your code in. this was a very real issue for things like the python runtime or widely used code checkers like pytype or pylint, because if e.g. the new version of python, or some improved check in pytype, started raising failures in a code pattern that was technically wrong but which the existing toolchain did not complain about, you could not release the new version until you had fixed all those new breakages. contrast this with non-monorepo codebases, where the other code would have had to deal with the fact that "oh, our code works under 3.10 but 3.11 broke it, guess we have to pin our own repo to 3.10 until we fix it", but as the python team we could just say "we support 3.11 now, you need to catch up"

Arainach 3y ago
> i can no longer edit my post to clarify this, but by "responsible for breakages" i meant that if your new check-in caused any CI tests anywhere within the codebase to fail, even if due to bugs in the other code, you had to stop and fix it, or get the owners to fix it, or find some principled way to temporarily disable those tests, before you could check your code Google engineers hate this one weird hack: just send a mail saying "we're deleting/deprecating [everything your product depends on] in [14-180] days, good luck!"
- lbruno 3y ago
  this reminds me of kenton v's complaint about working on infra being so hard
- pjc50 3y ago
  I'm guessing these are related. If every team is potentially responsible for breakages in every product, there's no such thing as a "just chugging along" product and there will be a constant demand to delete products that are not popular within Google.

stevage 3y ago

Can you say more about the automated refactoring? How did it work? How was it triggered? How much human involvement was there? Are any parts of it public?

zem 3y ago
can't say too much about it because it mostly used a lot of internal tools, but it was a combination of writing per-file utilities to do specific code transforms (think something like libcst), and then a framework to scan the google codebase, find affected files, and apply the transforms to them.
- banish-m4 3y ago
  Code mods are fun. Were they gated by human PR reviews, perhaps in batches, or applied without review?
  - zem 3y ago
    yes, gated by human reviews based on who owned the code, unless they were completely trivial and then were gated by a single global approver. there were always humans in the loop though.
    - banish-m4 3y ago
      Ah makes sense. At another MAANG, there was a code mod template job with a half dozen subvariants. I recall using a variant that supported ripgrep piping to sed -i for a simple replacement mod. There were jobs properties controlling how many mods per diff and whether it needed human approval or not. I think I did one batched by 1000 running once every 2 weeks to keep noise down.

iamcreasy 2y ago

> needed to do extraordinary things to keep it performant and maintainable. This is very interesting. Can you share more about this?

zem 2y ago
i can't, unfortunately, it was youtube internal code. but by the same token a lot of the performance stuff was specific optimisations for that code and would not really generalise. one of the current "extremely large python codebase" projects is instagram, and they do have some public repos, notably monkeytype (https://github.com/Instagram/MonkeyType https://github.com/Instagram/MonkeyType) which youtube did have its own analogue of, and cinder (https://engineering.fb.com/2022/05/02/open-source/cinder-jits-instagram/ https://engineering.fb.com/2022/05/02/open-source/cinder-jit...). in general facebook's engineering blog is a great place to read about this sort of thing.

dotancohen 3y ago

> very large projects with a one-file-at-a-time architecture (this was my primary job at google, ama) Is there any place where we could see such tools? A lot of the scripts that I write are contained in a single file, and I'm sure that there would be much to learn from seeing such project scripts. Thank you.

zem 3y ago
it's open source! check out https://github.com/google/pytype https://github.com/google/pytype and https://github.com/google/pytype/blob/main/docs/developers/tools.md#analyze_project https://github.com/google/pytype/blob/main/docs/developers/t... for more on the multi-file runner
- dotancohen 3y ago
  Terrific! Thank you.

jpdaigle 3y ago

I’m so sorry.

banish-m4 3y ago

When Wall St. concerns lead to imposing cannibalistic capitalism plans that are pennywise but pound foolish. Just wait a few years. When they come calling for consulting services, charge them triple.

elbear 3y ago

Frankly, it sounds like a dream job for me too, to be responsible for a particular technology and make sure that it works for all your internal (and external) users.

zem 3y ago
agreed! it takes a certain kind of person to find that fun, but the people who enjoy it really enjoy it.

offbyone 3y ago

I was your opposite number at AWS for many years. Like you say, it was one of the best jobs I've ever had, but over there it was a career dead end. I wasn't laid off, but I had to leave it to have any hope of growing myself. I feel for ya, zem; if you ever turn up at a PyCon in person, lemme buy you a drink.

zem 3y ago
I guess it was in some ways a career dead end at google too, I joined as a senior engineer and never bothered applying for promotion to staff, and most of my teammates were similar. On the other hand, I never felt I wasn't growing myself because I learnt a lot, took on some very ambitious projects, and was trusted with a lot of ownership and input into "what do you feel the most effective thing you could be working on right now is?"

gpshead 3y ago

Adding to the wonderful writeup by my now-ex teammate (thanks!): Several of us were/are/TBD also involved in both long term strategic leadership and maintenance of the open source CPython project itself. That direct feedback line from a major diverse needs user into the project and ecosystem was valuable for the world. The reason I stayed on this team for 12+ years is as zem said. It was an ideal impactful alignment of people, abilities, priorities, and work life balance. My prior teams at Google... were often not. For the first half of our Python teams existence, there were only ~5 of us. Many early years were spent paying down internal tech debt accumulated from prior years of neglecting to have a strong Python strategy and letting too many do their own thing. Python was one of the very first languages used widely at Google. It was the last major backend language to get a language team. Signed, -- the now-ex runtimes TL

zem 3y ago
yes, i definitely should have said more about the cpython leadership work! i was responding to a comment that had already mentioned upstream contributions so i didn't think to highlight it more, but it was a huge contribution to both google and to cpython at large.
gooob 3y ago
well, why don't you guys stay together and continue work? start your own thing! google fired you, yes, but you're still alive and able to communicate. maybe you won't get paid as much as you did at google. but prove to them that you have something of value to provide to society.

CydeWeys 3y ago

Any guess how many SWE-yrs are now going to be spent migrating existing Python projects away from Python, now that Python seems to be unsupported at Google going forward?

mezyt 2y ago

[dead]

arcamox 3y ago

I feel your pain. All the best!

emmy1989 3y ago

[dead]

samstave 3y ago
[dead]

emmy1989 3y ago

[dead]

nocte 3y ago

How can someone hire the whole team? :)

Mxmilan 3y ago

sorry to hear about the situation at google, would love to help out. we connect laid-off engineers to startups. if anyone from the team is interested here is our discord server https://discord.com/invite/Bnp5zyc8nM https://discord.com/invite/Bnp5zyc8nM

e40 3y ago

My guess: the people in your group have very high salaries and they want to dump them. I would not be surprised if they hire people to make a new team at some point in the future. I run HR at a small software company reports and my HR person has a lot of friends in HR at FAANGs. For 10+ years we've been talking about hiring and salaries at Google (and others). It's just crazy to me how high salaries have gotten. I know someone that switched from Google to FB when they made $750k at Google and got $1M at FB. I said all this to say: I've talked to a bunch of people over the last 6 months that know someone laid off and: 1) they had a very high salary, and 2) the group they were in was hiring.

Canders42 3y ago

We are hiring here at Cruise and I believe your team could be a good fit for one of the roles that we are hiring for. Here is the job description: https://www.getcruise.com/careers/jobs/2831153/ https://www.getcruise.com/careers/jobs/2831153/ If any of you are interested, feel free to directly reach out to the recruiter Connor Anderson (Connor.Anderson@getcruise.com) or feel free to apply directly.

webprofusion 3y ago
Only issue I can immediately see is that the salary is about a quarter of the FANG rate.
- nmfisher 3y ago
  I mean, that probably at least partly explains why they were laid off..
- Canders42 3y ago
  Please keep in mind that the listed salary range on the job description is only the median base salary range. Not the full total compensation range that includes bonus and other factors to it. We can pay much more than what is listed there from a total compensation standpoint.

cheviana37 3y ago

We're hiring! https://boards.greenhouse.io/blackcanyonconsulting/jobs/5754387003 https://boards.greenhouse.io/blackcanyonconsulting/jobs/5754... Good life work balance

deleted 3y ago

[deleted]

fullmetal09 3y ago

> maintained tools to keep thousands of third party packages constantly updated from their open source versions. Hi Zem could you say more about how you maintained those thousands of third party packages? We are facing the same issue and would love to hear your insight!

zem 3y ago
sure! it used a collection of tools, the main ones were * copybara [https://github.com/google/copybara https://github.com/google/copybara] to mirror between github and our internal repo and transform the source tree layout to our internal conventions * mercurial patch queues [https://hgbook.red-bean.com/read/managing-change-with-mercurial-queues.html https://hgbook.red-bean.com/read/managing-change-with-mercur...] to apply internal changes there are internal tools to automate the copybara configuration for python packages specifically, and to map between mercurial and google's version control system, but that's the main idea, and a specific review process for adding new packages. the process is actually documented here though not all the referenced tools are available (or even applicable) externally: https://opensource.google/documentation/reference/thirdparty/python https://opensource.google/documentation/reference/thirdparty...

DavidAmelia 3y ago

[dead]