4 ms·

For the past decade Google has seemed like a deer in headlights, unable to move in any direction, failing at introducing new products, now getting crushed on AI

by pavlov 4y ago

For the past decade Google has seemed like a deer in headlights, unable to move in any direction, failing at introducing new products, now getting crushed on AI. Maybe the problem has something to do with the way they hire. The article mentions that this string-processing problem had to be approved by a committee: ”Twice I asked the hiring committee if I should stop using this and they had no problem with it.” Not even the Soviet Union had a process where job interview questions were regularly approved by a committee. The level of bureaucracy in administering these brain puzzles at mass scale is absurd. The article mentions that the candidate who did best on this question was “the type who did high-school competitions.” I wonder if anyone stopped at any point to question whether that’s the ideal type of hire.

alex_sf 4y ago
> The level of bureaucracy in administering these brain puzzles at mass scale is absurd. It's unavoidable when you allow too many different interests to have a say in the process.
- atoav 4y ago
  The question is why you would have to many interests in one hire? That hints to me that there are problems of an organisational nature.
xmprt 4y ago
> The article mentions that the candidate who did best on this question was “the type who did high-school competitions.” I wonder if anyone stopped at any point to question whether that’s the ideal type of hire. I think this is the most telling bit. Time after time, I've seen excellent competitive programmers end up being mediocre engineers. Almost none of them are bad and it's also a very objective (at least on paper) process which is probably why Google is happy to stick with the process but most of them also aren't amazing innovators. I'm sure that university student who aced the interview turned out to be very smart - they went to a great university and can solve challenging puzzles. But I don't think they'll end up being the engineer that builds Google's competition to ChatGPT. I wonder whether a course correction where Google starts looking for people who create really cool projects even at the risk of not being able to solve any coding challenges would actually help.
- fifilura 4y ago
  > Almost none of them are bad and it's also a very objective (at least on paper) process which is probably why Google is happy to stick with the process but most of them also aren't amazing innovators. Oh, wow, if I could get that kind of result ("almost none of them bad", my addition: "possibly great") from an interview, I would stick with that process. I have also conducted interviews in a company with heavy focus on code tests, which obviously favors competitive programmers, and my anecdata was that I got very good results compared to the alternative (i.e. not doing that).
  - fifilura 4y ago
    That said - that way of conducting interviews pretty often makes you feel like a jerk. And that is uncomfortable for both sides. And I believe this is why this discussion always keeps popping up. And it makes you discard a lot of "possibly great" candidates at the cost of "almost none of them bad". It is the eternal precision vs recall tradeoff.
- gfd 4y ago
  >But I don't think they'll end up being the engineer that builds Google's competition to ChatGPT OpenAI has a lot of strong competition programmers. Search IOI or ICPC in their team update blogs: https://openai.com/blog/team-update https://openai.com/blog/team-update https://openai.com/blog/team-update-august https://openai.com/blog/team-update-august https://openai.com/blog/team-update-january https://openai.com/blog/team-update-january
- candleknight 4y ago
  > I've seen excellent competitive programmers end up being mediocre engineers I've seen the opposite. I know many extremely talented competitive programmers (IOI, CCO, USACO platinum, Codeforces grandmasters, ICPC, and the list goes on) and they are all excellent engineers. It's not about having a wealth of knowledge about DSA (although this occasionally helps, depending on the type of work) or competitive programming itself, but rather the skills it teaches. They all know how to learn new things very quickly and thoroughly, and it goes without saying that their general problem-solving skills are incredible. They've interned at companies like Snowflake, Google, Jump, Citadel, Jane Street, Waabi, Uber R&D, DataDog, SingleStore, etc. (all before or during their 3rd year, which is practically unheard of for "normal" students, especially during the tech recession) and have consistently received the highest performance grade (our co-op program requires employees to give them a rating on a scale from 1 to 7). There are also a few I don't know directly who are in fact working on things like Bard, or have founded their own companies (TabNine was one of them IIRC).
atoav 4y ago
I suck at those questions, and I hated those competitions. Yet I was the only one who would create things (hard- and software) in their spare time. I think some of these questions are insulting. Do you wanna get someone who solves actual realworld issues and interacts with real world challenges or what? That being said I am easily the best/fastest programmer in my organization despite there being people who have 20 years more experience.
- cultureswitch 4y ago
  A mix of hubris and disdain of theoretical pursuits.
- Aeolos 4y ago
  > Yet I was the only one who would create things (hard- and software) in their spare time. I have found this to be a very good signal in the hiring process. People who have technical side-projects (machining, 3d printing, electronics, game programming, open-source) tend to be really good at conceptualization, problem-solving, cost-benefit analysis, problem-solving, staying-power, which are critical elements to the success of a software project. Trivia CS knowledge questions, such as the one presented here, have their place too: they test whether someone can think on their feet, which is not a bad skill to have. However, all things consider, who would you rather hire? - someone who can do an O-time analysis, but has not finished any real project? - someone who failed the O-time analysis, but has built something to the end? I'd obviously prefer someone who can do both, but the person who has released an open-source project for their ultra-niche hobby (say, a sock knitting pattern designer, used by 5 very passionate individuals; or a climbing wall with LED lights showing you the path) gets my vote otherwise. They may not have come up with the trie solution, but what they've done is conceptualize, build and deliver something concrete to users. Which means they have proven and refined the precise skills every software project needs.
  - the_af 4y ago
    > Trivia CS knowledge questions, such as the one presented here, have their place too: they test whether someone can think on their feet, which is not a bad skill to have. Well, the author does mention a surprisingly high number of allegedly senior programmers fail at coming up with the simple brute-force solution. At this early stage in the challenge, this isn't "a trivia CS knowledge question", it's simply knowing how to code the simplest iteration over a string and knowing there is something called "a dictionary" which allows lookups! If you don't know this, what do you know?
dilyevsky 4y ago
HC is just a team of very senior engineers who make a final determination based on interview feedback. It has existed since pretty early and was a good way to massively scale the hiring pipeline without compromising on quality which ime is not true for many other big shops which did not have this
martopix 4y ago
The article does not state that "interview questions were regularly approved by a committee". It says that this question was banned because it was leaked to the public. And that he, very reasonably, asked the hiring committee if he could use it anyway.
- shaftway 4y ago
  This. Hiring committee's job is to take the interview feedback and provide a "Hire" or "No Hire" decision. 99% of the time this is done entirely based on the interviewers' feedback without any further interaction, but occasionally the hiring committee will reach out to ask clarifying questions or provide feedback on the report. Sometimes that feedback is along the lines of "this question is more complex than the candidate's target level" and sometimes it's "hey, that question has been widely posted as a 'Google interview question', so people are studying it specifically; you need to stop asking it." Good interviewers will proactively ask for feedback from hiring committees so that they can provide a better signal. Also, it's worth pointing out that there is no "one" hiring committee. Many people are in the pool for that task (it counts as a "citizenship contribution" during perf) and you meet for a few hours once every few weeks and grind through a stack of feedback. That's why he asked multiple times; he effectively asked committees made up of different sets of people.
bolanyo 4y ago
> Not even the Soviet Union had a process where job interview questions were regularly approved by a committee. Not defending Google's practices here, but this claim is nonsense. Many public sector/third sector jobs in Western society will only use interview questions which have been pre-approved by explicit policy.
- wickedsickeune 4y ago
  Even in the public sector, you run the committee once, finalise the questions and that's it. The OP means that in Google they repeat this process many times (I don't know if you can infer as much from the article)
barrkel 4y ago
This problem is not great. It is a classic of the programming puzzle kind, and does not measure metrics which are important for most engineering. For some roles, like working deep in the weeds of a compiler or database optimizer, I think it's not that bad. That kind of programming has a puzzle feel: searches for near optimality in combinatorial (or worse) solution spaces. But it's not very applicable to most engineering. On the point of committees, though. Hiring committees make a hire/no-hire decision. They don't approve questions, but they do make decisions based on the results of asking a question. So they rely on the signal that the question generates, and if a question is not giving good signal, they will have an opinion. There's a different reason for wanting to standardize on interview questions. It's very easy to embed cultural bias. If the topic of a programming problem uses the rules of a game, for example, that will probably mean that people who are familiar with the game will understand the gist of the problem much faster than people who've never played the game.
deleted 4y ago
[deleted]
boshalfoshal 4y ago
No, its not the interviews. Its the proliferation of the idea that Google (and co) as the "final boss" interviews/companies and once you get in its just resting and vesting. Back in the day these companies were spots for hackers and innovators, now they are mostly just the final destination for clout or TC chasers. That is what stifles innovation, not a few innocious coding questions. IMO any interview type would be gameable. Also at Googles scale this is the most realistic pipeline to hire people, and having it go through a committee ensures that candidates are deterministically and fairly assessed. Also I recommend you try these "high school problems" you seem to be talking down on. I for one greatly respect people who can solve those types of problems, regardless of their age. These are not trivial problems, mind you. They require a lot of mathematical and logical intuition that 99% of people (and I'd wager the majority of SWEs) don't have.

by pavlov 4y ago

alex_sf 4y ago

> The level of bureaucracy in administering these brain puzzles at mass scale is absurd. It's unavoidable when you allow too many different interests to have a say in the process.

atoav 4y ago
The question is why you would have to many interests in one hire? That hints to me that there are problems of an organisational nature.

xmprt 4y ago

> The article mentions that the candidate who did best on this question was “the type who did high-school competitions.” I wonder if anyone stopped at any point to question whether that’s the ideal type of hire. I think this is the most telling bit. Time after time, I've seen excellent competitive programmers end up being mediocre engineers. Almost none of them are bad and it's also a very objective (at least on paper) process which is probably why Google is happy to stick with the process but most of them also aren't amazing innovators. I'm sure that university student who aced the interview turned out to be very smart - they went to a great university and can solve challenging puzzles. But I don't think they'll end up being the engineer that builds Google's competition to ChatGPT. I wonder whether a course correction where Google starts looking for people who create really cool projects even at the risk of not being able to solve any coding challenges would actually help.

fifilura 4y ago
> Almost none of them are bad and it's also a very objective (at least on paper) process which is probably why Google is happy to stick with the process but most of them also aren't amazing innovators. Oh, wow, if I could get that kind of result ("almost none of them bad", my addition: "possibly great") from an interview, I would stick with that process. I have also conducted interviews in a company with heavy focus on code tests, which obviously favors competitive programmers, and my anecdata was that I got very good results compared to the alternative (i.e. not doing that).
- fifilura 4y ago
  That said - that way of conducting interviews pretty often makes you feel like a jerk. And that is uncomfortable for both sides. And I believe this is why this discussion always keeps popping up. And it makes you discard a lot of "possibly great" candidates at the cost of "almost none of them bad". It is the eternal precision vs recall tradeoff.
gfd 4y ago
>But I don't think they'll end up being the engineer that builds Google's competition to ChatGPT OpenAI has a lot of strong competition programmers. Search IOI or ICPC in their team update blogs: https://openai.com/blog/team-update https://openai.com/blog/team-update https://openai.com/blog/team-update-august https://openai.com/blog/team-update-august https://openai.com/blog/team-update-january https://openai.com/blog/team-update-january
candleknight 4y ago
> I've seen excellent competitive programmers end up being mediocre engineers I've seen the opposite. I know many extremely talented competitive programmers (IOI, CCO, USACO platinum, Codeforces grandmasters, ICPC, and the list goes on) and they are all excellent engineers. It's not about having a wealth of knowledge about DSA (although this occasionally helps, depending on the type of work) or competitive programming itself, but rather the skills it teaches. They all know how to learn new things very quickly and thoroughly, and it goes without saying that their general problem-solving skills are incredible. They've interned at companies like Snowflake, Google, Jump, Citadel, Jane Street, Waabi, Uber R&D, DataDog, SingleStore, etc. (all before or during their 3rd year, which is practically unheard of for "normal" students, especially during the tech recession) and have consistently received the highest performance grade (our co-op program requires employees to give them a rating on a scale from 1 to 7). There are also a few I don't know directly who are in fact working on things like Bard, or have founded their own companies (TabNine was one of them IIRC).

atoav 4y ago

I suck at those questions, and I hated those competitions. Yet I was the only one who would create things (hard- and software) in their spare time. I think some of these questions are insulting. Do you wanna get someone who solves actual realworld issues and interacts with real world challenges or what? That being said I am easily the best/fastest programmer in my organization despite there being people who have 20 years more experience.

cultureswitch 4y ago
A mix of hubris and disdain of theoretical pursuits.
Aeolos 4y ago
> Yet I was the only one who would create things (hard- and software) in their spare time. I have found this to be a very good signal in the hiring process. People who have technical side-projects (machining, 3d printing, electronics, game programming, open-source) tend to be really good at conceptualization, problem-solving, cost-benefit analysis, problem-solving, staying-power, which are critical elements to the success of a software project. Trivia CS knowledge questions, such as the one presented here, have their place too: they test whether someone can think on their feet, which is not a bad skill to have. However, all things consider, who would you rather hire? - someone who can do an O-time analysis, but has not finished any real project? - someone who failed the O-time analysis, but has built something to the end? I'd obviously prefer someone who can do both, but the person who has released an open-source project for their ultra-niche hobby (say, a sock knitting pattern designer, used by 5 very passionate individuals; or a climbing wall with LED lights showing you the path) gets my vote otherwise. They may not have come up with the trie solution, but what they've done is conceptualize, build and deliver something concrete to users. Which means they have proven and refined the precise skills every software project needs.
- the_af 4y ago
  > Trivia CS knowledge questions, such as the one presented here, have their place too: they test whether someone can think on their feet, which is not a bad skill to have. Well, the author does mention a surprisingly high number of allegedly senior programmers fail at coming up with the simple brute-force solution. At this early stage in the challenge, this isn't "a trivia CS knowledge question", it's simply knowing how to code the simplest iteration over a string and knowing there is something called "a dictionary" which allows lookups! If you don't know this, what do you know?

dilyevsky 4y ago

HC is just a team of very senior engineers who make a final determination based on interview feedback. It has existed since pretty early and was a good way to massively scale the hiring pipeline without compromising on quality which ime is not true for many other big shops which did not have this

martopix 4y ago

The article does not state that "interview questions were regularly approved by a committee". It says that this question was banned because it was leaked to the public. And that he, very reasonably, asked the hiring committee if he could use it anyway.

shaftway 4y ago
This. Hiring committee's job is to take the interview feedback and provide a "Hire" or "No Hire" decision. 99% of the time this is done entirely based on the interviewers' feedback without any further interaction, but occasionally the hiring committee will reach out to ask clarifying questions or provide feedback on the report. Sometimes that feedback is along the lines of "this question is more complex than the candidate's target level" and sometimes it's "hey, that question has been widely posted as a 'Google interview question', so people are studying it specifically; you need to stop asking it." Good interviewers will proactively ask for feedback from hiring committees so that they can provide a better signal. Also, it's worth pointing out that there is no "one" hiring committee. Many people are in the pool for that task (it counts as a "citizenship contribution" during perf) and you meet for a few hours once every few weeks and grind through a stack of feedback. That's why he asked multiple times; he effectively asked committees made up of different sets of people.

bolanyo 4y ago

> Not even the Soviet Union had a process where job interview questions were regularly approved by a committee. Not defending Google's practices here, but this claim is nonsense. Many public sector/third sector jobs in Western society will only use interview questions which have been pre-approved by explicit policy.

wickedsickeune 4y ago
Even in the public sector, you run the committee once, finalise the questions and that's it. The OP means that in Google they repeat this process many times (I don't know if you can infer as much from the article)

barrkel 4y ago

This problem is not great. It is a classic of the programming puzzle kind, and does not measure metrics which are important for most engineering. For some roles, like working deep in the weeds of a compiler or database optimizer, I think it's not that bad. That kind of programming has a puzzle feel: searches for near optimality in combinatorial (or worse) solution spaces. But it's not very applicable to most engineering. On the point of committees, though. Hiring committees make a hire/no-hire decision. They don't approve questions, but they do make decisions based on the results of asking a question. So they rely on the signal that the question generates, and if a question is not giving good signal, they will have an opinion. There's a different reason for wanting to standardize on interview questions. It's very easy to embed cultural bias. If the topic of a programming problem uses the rules of a game, for example, that will probably mean that people who are familiar with the game will understand the gist of the problem much faster than people who've never played the game.

deleted 4y ago

[deleted]

boshalfoshal 4y ago

No, its not the interviews. Its the proliferation of the idea that Google (and co) as the "final boss" interviews/companies and once you get in its just resting and vesting. Back in the day these companies were spots for hackers and innovators, now they are mostly just the final destination for clout or TC chasers. That is what stifles innovation, not a few innocious coding questions. IMO any interview type would be gameable. Also at Googles scale this is the most realistic pipeline to hire people, and having it go through a committee ensures that candidates are deterministically and fairly assessed. Also I recommend you try these "high school problems" you seem to be talking down on. I for one greatly respect people who can solve those types of problems, regardless of their age. These are not trivial problems, mind you. They require a lot of mathematical and logical intuition that 99% of people (and I'd wager the majority of SWEs) don't have.