8 ms·

"As far as this outsider can tell, the systems and processes for writing code at Google must be among the best in the world, given both the scale of the company

"As far as this outsider can tell, the systems and processes for writing code at Google must be among the best in the world, given both the scale of the company and how often people sing their praises." Is this really the case? In high school I sure thought Google was a magical software heaven us mortals could but dream of working for, but now (and increasingly as of late) I'm strongly under the impression that 20 years ago they discovered unobtanium in the form of a very under-served search and ads market, they plopped down camp on top of it and turned it into a giant money firehose, and now to this day are simply operating that ridiculously high-margin business while trying desperately to find another vein of unobtanium because when you already have one, that's the only thing that even registers on your radar. "Revenue from Google Network ads hit $7.5 billion (10.7%), and YouTube ad revenues registered at $6.7 billion (9.6%). In other words, at $54.5 billion, the total ad revenue from Google constituted 78.2% of the company's total revenue in the quarter." https://www.oberlo.com/statistics/how-does-google-make-money https://www.oberlo.com/statistics/how-does-google-make-money 78% between google and youtube. And given youtube isn't nearly as high-margin as the search product (can you even imagine the cost related to essentially storing the video history of the world?), I'm hesitant to say youtube is their 2nd vein. So yeah. Turns out when you have essentially a monopoly in a high-margin business, you can do basically anything and build an entire private software ecosystem without going bankrupt. I interpret "how google does it" to simply mean "look at all this cool stuff you can do when your product is this high-margin".

nosefrog 3y ago
I used to work there. The tooling is pretty good.
- fnordpiglet 3y ago
  I’ve not worked there but I’ve worked with many exgooglers. They tend to think Google * is the best in the world and they created it before anyone else and everyone else with similar capabilities copied them, and that Googlers are smarter. However every company of their size and technical complexity has built almost exactly the same tooling and processes, but don’t carry the same attitude around. At a certain point I’ve learned to smile and nod as they explain and take credit for things I’ve developed myself at other FAANG and adjacent highly technical megacorps. The final thing of note is they seem to be unable to replicate their magic outside of Google, and blame the org they’re now in for being deficient in some way for that. “At Google we had X and didn’t have to worry about Y, without Y we can’t do X,” where Y is some nebulous qualitative feature of google culture. After I finish smiling and nodding I just go build X. N.b., I’m not saying nosefrog is this person, or even everyone at google or ex-google is like this. This was just my experience working with senior principal or other very senior engineering staff.
  - jollyllama 3y ago
    Great comment. Google and others blazed a trail, but by 2010 you could find similar, often specialized tooling and practices emerging at most Fortune 500 tech companies.
    - teaearlgraycold 3y ago
      Today you can almost universally get a better off-the-shelf solution with a SaaS product than what Googlers are able to access internally.
mabbo 3y ago
> I'm strongly under the impression that 20 years ago they discovered unobtanium in the form of a very under-served search and ads market, they plopped down camp on top of it and turned it into a giant money firehose In 2010 when I interned there, one of my mentors said something to me that stuck with me: "Google found a hose that money pours out of, and it's name is 'online advertising'. All we do is optimize that hose, and search for another one."
- paul7986 3y ago
  Yes and just like all tech companies.. Google steals from smaller companies to the dreamers they inspired to invent and create. Sonos: https://www.theverge.com/2023/5/26/23739273/google-sonos-smart-speaker-patent-lawsuit-ruling https://www.theverge.com/2023/5/26/23739273/google-sonos-sma... A MIT student: https://news.ycombinator.com/item?id=18566929 https://news.ycombinator.com/item?id=18566929 I met them (they were horribly unprofessional) in 2013 around time Sonos did to discuss my audio syncing tech. Best to steal tech then bother investing into R&D. Quickest way to make more money .. steal .. spend less and make more.
  - paul7986 3y ago
    I've been told I shouldn't talk about this anymore on hacker news by Dang. I haven't mentioned my experience in years cause it's old news but Sonos court victory is new relevant news to my experience meeting Google R&D that same year. Though I can say for sure and claim is that they were extremely and laughably unprofessional to me and I met with other tech companies then who were very professional and polite.
ChrisMarshallNY 3y ago
Apple has undergone a similar ossification. I think it's just something that happens to companies that get a certain size. The Process becomes more important than the Product. When I was younger, I wanted so badly, to work at Apple. A few years ago, they actually approached me, and I found out that the culture seems to have drastically changed. They ended up not wanting me anyway, so maybe it's just sour grapes, on my part.
- lapcat 3y ago
  > I found out that the culture seems to have drastically changed. Can you explain in detail?
  - NetOpWibby 3y ago
    Steve isn't there, for one.
    - lapcat 3y ago
      Everyone knows that Steve Jobs died. That's not an answer to my question.
  - ChrisMarshallNY 3y ago
    I'm not going to go into detail, as I don't think that it's helpful. But I don't think that someone like me would be very welcome there. That is both good and bad. I am certainly not God's Gift to Programming, so they may well be better off without folks like me. They certainly seem to be making a lot of green.
    - lapcat 3y ago
      > I'm not going to go into detail, as I don't think that it's helpful. Wouldn't it be helpful to other people who are interested in working there?
      - 3y ago
teaearlgraycold 3y ago
I used to work there and feel about the same. Google’s internal tech is really good where it needs to be, and barely good enough where it doesn’t. Borg, Blaze, d, google3, codesearch, etc. are all very good. Almost everything else falls by the wayside. Generally programmers at Google are better than programmers I’ve worked with outside of the company. But not by leaps and bounds. And in my opinion it doesn’t matter as your pace of development is so much slower it matters little how good of a programmer you are.
afavour 3y ago
I’d be interested in the perspective of a high schooler/fresh college grad on this too. I was in college when Google appeared on the scene and my impression was very much that it was an engineering paradise (20% time had a lot to do with that!), in the many years since most of the interactions I’ve had with Google have changed that perception a ton. But I’m not sure what it looks like to someone just starting out.
yowlingcat 3y ago
I think this is broadly correct but I do think their cloud services business holds some promise. Although a distant third at best behind AWS/Azure (and maybe fourth behind OCI), I have seen some pretty large shops with a ton of volume of all of their compute to GCP. It does seem like the immense infrastructure Google had to build to power this firehose of money has the skeleton of a great product to power the cloud workloads of many other businesses, in ways that can compete with AWS on their home turf. But this is conjecture from my end, I haven't used GCP in the heat of production.
yafbum 3y ago
It's possible that such techniques are unreasonable in low margin companies, and it could also be said that perhaps these engineering techniques have been a factor in allowing Google to keep its margins high as it scaled up by many orders of magnitude. Google in 2003 was making less than $1bn/year in revenue... 200x growth in 20 years
FartyMcFarter 3y ago
I don't understand your argument - why is the one trick pony revenue generation incompatible with high quality systems and processes for writing code? Both can be true at the same time, and my feeling is that they are, having worked there. The business is not very diversified, and the tools are great.
- BurningFrog 3y ago
  If Google's code writing process really was superior, you'd expect them to consistently produce killer products in other fields. But I can't think of anything like that in the last 15 years.
  - fsociety 3y ago
    Unfortunately good code does not mean a good product, and Google is anything but a good product company.
    - shadowgovt 3y ago
      If anything, these are orthogonal. I had a front-row seat to some really revolutionary ideas in Google getting to the prototype stage before being squashed in the gears of "We're chasing a market and that's not how users see this product working." Stuff where, if it caught on, it'd be a paradigm shift... But it turns out users don't want every paradigm shift that comes down the lane. Because Google has (traditionally; this has changed in recent years) a real push-pull in authority between management and engineering leadership, the company can't commit fully to building a quality implementation of a status quo. Nor can it commit fully to chasing entirely new ways of doing things that could shake up an established market. In general, this... Actually kind of works out fine for them, more fine than critics often realize, because neither of those answers are always correct. Sometimes you get Gmail. Sometimes you get Google Drive. And sometimes you get iGoogle or Wave. And sometimes you get the stuff in between, like Reader or App Engine (really popular among the users, but the users don't have the money to make it profitable to commit to it).
  - danans 3y ago
londons_explore 3y ago
Google search was 1998. Google Maps was 2005. Gmail was 2004. Android was 2008. Youtube was 2005. Chrome was 2008. Docs was 2006. Translate was 2006. Yet in the last decade, they really haven't had many successes (perhaps with the exception of Google Photos - 2015) One would imagine that with nearly 200,000 employees at least one of them would have a good enough idea for a new product people like. But management and culture, with 'more wood behind fewer arrows', is no longer conducive to the good ideas getting launched.
- danpalmer 3y ago
  This is a common meme, but it depends how you define "product". There are companies with millions in VC backing that would just be a feature on a Google (or other big tech company) product as most people define them. I would argue that taking a product from 100m to 1bn users is a whole new level of success, and that has been done multiple times in the last decade.
  - water-your-self 3y ago
    Why should scaling matter? I thought google tried to solve problems regardless of scale? Or maybe youre commending google for its ability to more effectively commodify datacenter labour in the past decade?
    - danpalmer 3y ago
      I don't think we can expect businesses to solve problems without any business upside. I don't hold that against any company. That means needing some amount of scale in order to justify the work and opportunity cost.
    - londons_explore 3y ago
      'free' ad supported products usually have engineering costs greater than compute costs, even up to billions of users. Compute scales with the userbase. Engineering scales with the product featureset. That means a small userbase in general cannot support much engineering and therefore cannot have much complexity/features if it wants to be profitable. That in turn puts big companies at a massive advantage. And in fact we see that - the vast majority of my time using the web is spent on big products (google, youtube, reddit, twitter), and only a small fraction on small sites (perhaps HN being the one exception here). Those big products won the battle for my attention.
flohofwoe 3y ago
I can't think of a single C++ library coming out of Google I had to deal with and where I didn't have massive problems integrating them into my own C++ projects because of all the Google specialties baked into those projects (their own weird build tools which don't seem to have changed much since the early 2000's and the use of other Google dependencies like abseil). Maybe it works well inside Google, but definitely not in the real world.
- DoingIsLearning 3y ago
  Off the top of my head I used FlatBuffers in quite a few projects many moons ago and that was pretty seamless. Arguably it was a bit thin on documentation but it just worked.
danpalmer 3y ago
The tooling is great in some ways, but possibly not in the way you'd expect. There is a tool that will do almost anything, but the tools are often just a bit janky. Rather than thinking of the toolchain as some polished, tightly integrated, perfect system, think of it more like an internal open-source ecosystem of things that work together but which are made by individuals, with limited resources, that still feature bugs. This is not to downplay what we have, it's amazing in many ways, but most tools are exactly the same sort of thing that you'd create in any other place, we just have more of them that cover almost every aspect of development. So much of the time I run into neat little systems and then discover they're a few Python scripts running under someone's personal quota. Sometimes these are 20% time projects, sometimes they are bits of tooling teams build for their own needs, sometimes they are more concerted efforts. There is a core, tightly integrated set of systems that work well and are more "productized", but most of those aren't that much more special than what you can buy at any other company. If you've used Datadog then you'd feel at home in much of the modern infra tooling. Spanner is great, but anyone can pay to use it. Our CDN is very good, but there are other CDNs and from a developer perspective there isn't a whole lot of difference.
- 88913527 3y ago
  I have yet to be impressed by internal software tools developed by one team and consumed by another. You're likely being friendly and understating the degree of jank which is present.
  - danpalmer 3y ago
    Is this at Google, or elsewhere? If it's at Google I'd be happy to try to persuade you via chat :) If elsewhere, I think there are 2 kinds of tool: those developed by individuals/small groups in informal ways, some of these are bad, but much like with open-source tools, the ones that are good survive and become popular. The other type is those with more formal backing, dedicated dev/design/UX/PM resources, and those are good at surveying users to figure out what the right problem to solve is, so they're normally pretty good, and when they're not, they still accept fixes (I regularly contribute fixes to other teams codebases when I find issues).
mliker 3y ago
Google software engineering may not be perfect, but that's because we all have high standards probably. I've worked at both FAANG and non-FAANG companies, and while the sample size is small and not representative of ALL non-FAANG companies, I will say that my experience at FAANG companies is the engineers there do to get dream big and come up with ingenious engineering solutions that make engineering work at non-FAANG look like college senior undergrad projects. I think we should celebrate what Google has done for software engineering instead of nitpicking every little detail that they got wrong.
- fatnoah 3y ago
  > I will say that my experience at FAANG companies is the engineers there do to get dream big and come up with ingenious engineering solutions that make engineering work at non-FAANG look like college senior undergrad projects My own experience at FAANG and non-FAANG is that __some engineers__ at FAANG get to dream big and demonstrate ingenuity, whereas at non-FAANG (especially startups I've been at) everyone can do this, though not all take the opportunity. At my last FAANG where I was a manager, my opinion is that performance management and the focus on bullet points, metrics, and peer feedback for review time is a huge limiter for everyone else. I spent a lot of time creating cover for people on my team to actually try something new and innovative. More senior management constantly wanted to know why person X on my team was working on a new thing instead of trying to move some incremental metric somewhere else. Based on my experience during performance reviews for senior ICs, risk taking wasn't particularly encouraged at L6 (or even L7) and below as the cost of failure was substantial.
shadowgovt 3y ago
You're not wrong. Google gets to sidestep a lot of problems other lower-margin companies have because they've spent their whole history being near market-leader on search and ads. Things like "deadlines" were foreign to the company for a long time; when you're already the pack leader, you release new features when you feel like it, not to play catch-up with competitors. You can see them struggle in spaces where that's not true (Cloud, for example). There is one underlying technology special sauce they refined to an art form, which is building a reliable system on incredibly unreliable components. This comes from their inception when the two guys building out the company knew how to write code but not build custom machines; their initial attempts at racks (hard drives and motherboards mounted to flexible plywood) are hilarious-looking by modern standards and broke down all the time, so they had to learn how to write software that was fault-tolerant at every link in the chain. That resulted in building an infrastructure that was extremely supportive of experimentation (if the system can survive half of it disappearing, it can survive a software bug crashing half the machines), and the rest of the shape of the company kind of flows from that. It is worth repeating a lot that most companies do not operate at the scale or constraints Google does, and their approach to engineering is emphatically not one-size-fits-all.
- lokar 3y ago
  I think people forget that the standard back then in the industry was very high margin servers with redundant everything (psu, network/nic, disks, etc). Going to (eventually) mostly dependable simple servers with no redundancy was a big leap.
  - shadowgovt 3y ago
    There were a couple places in Google's internal infrastructure where they had picked up traditional "big iron" architectures (mostly by acquisitions), and it was funny to observe the sheer plane between their regular best practices and those systems (when they hadn't yet been replaced). One high-ticket item relied on a huge Oracle database. Every other aspect of that product was patch-on-the-fly, silent-release new versions, forward-and-backwards compatibility of software... But the whole product had regular scheduled outages on the weekends because the Oracle part could not be patched on the fly, had a rigid schema baked into its relational architecture, and had to be brought offline for updates and migrations. It was like everyone was zipping around in racecars and then they all had to pause for Grandpa to cross the street.
ownagefool 3y ago
Probably all of the above is true. From an outsiders, it looks like all of FAANG has scaled to the point where delivery, perf and managing the business has become quite cumbersome. This, I suspect is why they're all used the economic climate to cut back, as they largely are unable to see the benefit of the additional hiring. I have no doubt most of the talent is top tier, but being a strong engineer doesn't deliver profits nor cost savings on it's own.
- water-your-self 3y ago
  Hiring is always profitable as it lowers the average cost of your engineers. Layoffs are a way to accelerate that cost averaging.
deleted 3y ago
[deleted]
- deleted 3y ago
  [deleted]

nosefrog 3y ago

I used to work there. The tooling is pretty good.

fnordpiglet 3y ago
I’ve not worked there but I’ve worked with many exgooglers. They tend to think Google * is the best in the world and they created it before anyone else and everyone else with similar capabilities copied them, and that Googlers are smarter. However every company of their size and technical complexity has built almost exactly the same tooling and processes, but don’t carry the same attitude around. At a certain point I’ve learned to smile and nod as they explain and take credit for things I’ve developed myself at other FAANG and adjacent highly technical megacorps. The final thing of note is they seem to be unable to replicate their magic outside of Google, and blame the org they’re now in for being deficient in some way for that. “At Google we had X and didn’t have to worry about Y, without Y we can’t do X,” where Y is some nebulous qualitative feature of google culture. After I finish smiling and nodding I just go build X. N.b., I’m not saying nosefrog is this person, or even everyone at google or ex-google is like this. This was just my experience working with senior principal or other very senior engineering staff.
- jollyllama 3y ago
  Great comment. Google and others blazed a trail, but by 2010 you could find similar, often specialized tooling and practices emerging at most Fortune 500 tech companies.
  - teaearlgraycold 3y ago
    Today you can almost universally get a better off-the-shelf solution with a SaaS product than what Googlers are able to access internally.

mabbo 3y ago

> I'm strongly under the impression that 20 years ago they discovered unobtanium in the form of a very under-served search and ads market, they plopped down camp on top of it and turned it into a giant money firehose In 2010 when I interned there, one of my mentors said something to me that stuck with me: "Google found a hose that money pours out of, and it's name is 'online advertising'. All we do is optimize that hose, and search for another one."

paul7986 3y ago
Yes and just like all tech companies.. Google steals from smaller companies to the dreamers they inspired to invent and create. Sonos: https://www.theverge.com/2023/5/26/23739273/google-sonos-smart-speaker-patent-lawsuit-ruling https://www.theverge.com/2023/5/26/23739273/google-sonos-sma... A MIT student: https://news.ycombinator.com/item?id=18566929 https://news.ycombinator.com/item?id=18566929 I met them (they were horribly unprofessional) in 2013 around time Sonos did to discuss my audio syncing tech. Best to steal tech then bother investing into R&D. Quickest way to make more money .. steal .. spend less and make more.
- paul7986 3y ago
  I've been told I shouldn't talk about this anymore on hacker news by Dang. I haven't mentioned my experience in years cause it's old news but Sonos court victory is new relevant news to my experience meeting Google R&D that same year. Though I can say for sure and claim is that they were extremely and laughably unprofessional to me and I met with other tech companies then who were very professional and polite.

ChrisMarshallNY 3y ago

Apple has undergone a similar ossification. I think it's just something that happens to companies that get a certain size. The Process becomes more important than the Product. When I was younger, I wanted so badly, to work at Apple. A few years ago, they actually approached me, and I found out that the culture seems to have drastically changed. They ended up not wanting me anyway, so maybe it's just sour grapes, on my part.

lapcat 3y ago
> I found out that the culture seems to have drastically changed. Can you explain in detail?
- NetOpWibby 3y ago
  Steve isn't there, for one.
  - lapcat 3y ago
    Everyone knows that Steve Jobs died. That's not an answer to my question.
- ChrisMarshallNY 3y ago
  I'm not going to go into detail, as I don't think that it's helpful. But I don't think that someone like me would be very welcome there. That is both good and bad. I am certainly not God's Gift to Programming, so they may well be better off without folks like me. They certainly seem to be making a lot of green.
  - lapcat 3y ago
    > I'm not going to go into detail, as I don't think that it's helpful. Wouldn't it be helpful to other people who are interested in working there?
    - 3y ago

teaearlgraycold 3y ago

I used to work there and feel about the same. Google’s internal tech is really good where it needs to be, and barely good enough where it doesn’t. Borg, Blaze, d, google3, codesearch, etc. are all very good. Almost everything else falls by the wayside. Generally programmers at Google are better than programmers I’ve worked with outside of the company. But not by leaps and bounds. And in my opinion it doesn’t matter as your pace of development is so much slower it matters little how good of a programmer you are.

afavour 3y ago

I’d be interested in the perspective of a high schooler/fresh college grad on this too. I was in college when Google appeared on the scene and my impression was very much that it was an engineering paradise (20% time had a lot to do with that!), in the many years since most of the interactions I’ve had with Google have changed that perception a ton. But I’m not sure what it looks like to someone just starting out.

yowlingcat 3y ago

I think this is broadly correct but I do think their cloud services business holds some promise. Although a distant third at best behind AWS/Azure (and maybe fourth behind OCI), I have seen some pretty large shops with a ton of volume of all of their compute to GCP. It does seem like the immense infrastructure Google had to build to power this firehose of money has the skeleton of a great product to power the cloud workloads of many other businesses, in ways that can compete with AWS on their home turf. But this is conjecture from my end, I haven't used GCP in the heat of production.

yafbum 3y ago

It's possible that such techniques are unreasonable in low margin companies, and it could also be said that perhaps these engineering techniques have been a factor in allowing Google to keep its margins high as it scaled up by many orders of magnitude. Google in 2003 was making less than $1bn/year in revenue... 200x growth in 20 years

FartyMcFarter 3y ago

I don't understand your argument - why is the one trick pony revenue generation incompatible with high quality systems and processes for writing code? Both can be true at the same time, and my feeling is that they are, having worked there. The business is not very diversified, and the tools are great.

BurningFrog 3y ago
If Google's code writing process really was superior, you'd expect them to consistently produce killer products in other fields. But I can't think of anything like that in the last 15 years.
- fsociety 3y ago
  Unfortunately good code does not mean a good product, and Google is anything but a good product company.
  - shadowgovt 3y ago
    If anything, these are orthogonal. I had a front-row seat to some really revolutionary ideas in Google getting to the prototype stage before being squashed in the gears of "We're chasing a market and that's not how users see this product working." Stuff where, if it caught on, it'd be a paradigm shift... But it turns out users don't want every paradigm shift that comes down the lane. Because Google has (traditionally; this has changed in recent years) a real push-pull in authority between management and engineering leadership, the company can't commit fully to building a quality implementation of a status quo. Nor can it commit fully to chasing entirely new ways of doing things that could shake up an established market. In general, this... Actually kind of works out fine for them, more fine than critics often realize, because neither of those answers are always correct. Sometimes you get Gmail. Sometimes you get Google Drive. And sometimes you get iGoogle or Wave. And sometimes you get the stuff in between, like Reader or App Engine (really popular among the users, but the users don't have the money to make it profitable to commit to it).
- danans 3y ago

londons_explore 3y ago

Google search was 1998. Google Maps was 2005. Gmail was 2004. Android was 2008. Youtube was 2005. Chrome was 2008. Docs was 2006. Translate was 2006. Yet in the last decade, they really haven't had many successes (perhaps with the exception of Google Photos - 2015) One would imagine that with nearly 200,000 employees at least one of them would have a good enough idea for a new product people like. But management and culture, with 'more wood behind fewer arrows', is no longer conducive to the good ideas getting launched.

danpalmer 3y ago
This is a common meme, but it depends how you define "product". There are companies with millions in VC backing that would just be a feature on a Google (or other big tech company) product as most people define them. I would argue that taking a product from 100m to 1bn users is a whole new level of success, and that has been done multiple times in the last decade.
- water-your-self 3y ago
  Why should scaling matter? I thought google tried to solve problems regardless of scale? Or maybe youre commending google for its ability to more effectively commodify datacenter labour in the past decade?
  - danpalmer 3y ago
    I don't think we can expect businesses to solve problems without any business upside. I don't hold that against any company. That means needing some amount of scale in order to justify the work and opportunity cost.
  - londons_explore 3y ago
    'free' ad supported products usually have engineering costs greater than compute costs, even up to billions of users. Compute scales with the userbase. Engineering scales with the product featureset. That means a small userbase in general cannot support much engineering and therefore cannot have much complexity/features if it wants to be profitable. That in turn puts big companies at a massive advantage. And in fact we see that - the vast majority of my time using the web is spent on big products (google, youtube, reddit, twitter), and only a small fraction on small sites (perhaps HN being the one exception here). Those big products won the battle for my attention.

flohofwoe 3y ago

I can't think of a single C++ library coming out of Google I had to deal with and where I didn't have massive problems integrating them into my own C++ projects because of all the Google specialties baked into those projects (their own weird build tools which don't seem to have changed much since the early 2000's and the use of other Google dependencies like abseil). Maybe it works well inside Google, but definitely not in the real world.

DoingIsLearning 3y ago
Off the top of my head I used FlatBuffers in quite a few projects many moons ago and that was pretty seamless. Arguably it was a bit thin on documentation but it just worked.

danpalmer 3y ago

The tooling is great in some ways, but possibly not in the way you'd expect. There is a tool that will do almost anything, but the tools are often just a bit janky. Rather than thinking of the toolchain as some polished, tightly integrated, perfect system, think of it more like an internal open-source ecosystem of things that work together but which are made by individuals, with limited resources, that still feature bugs. This is not to downplay what we have, it's amazing in many ways, but most tools are exactly the same sort of thing that you'd create in any other place, we just have more of them that cover almost every aspect of development. So much of the time I run into neat little systems and then discover they're a few Python scripts running under someone's personal quota. Sometimes these are 20% time projects, sometimes they are bits of tooling teams build for their own needs, sometimes they are more concerted efforts. There is a core, tightly integrated set of systems that work well and are more "productized", but most of those aren't that much more special than what you can buy at any other company. If you've used Datadog then you'd feel at home in much of the modern infra tooling. Spanner is great, but anyone can pay to use it. Our CDN is very good, but there are other CDNs and from a developer perspective there isn't a whole lot of difference.

88913527 3y ago
I have yet to be impressed by internal software tools developed by one team and consumed by another. You're likely being friendly and understating the degree of jank which is present.
- danpalmer 3y ago
  Is this at Google, or elsewhere? If it's at Google I'd be happy to try to persuade you via chat :) If elsewhere, I think there are 2 kinds of tool: those developed by individuals/small groups in informal ways, some of these are bad, but much like with open-source tools, the ones that are good survive and become popular. The other type is those with more formal backing, dedicated dev/design/UX/PM resources, and those are good at surveying users to figure out what the right problem to solve is, so they're normally pretty good, and when they're not, they still accept fixes (I regularly contribute fixes to other teams codebases when I find issues).

mliker 3y ago

Google software engineering may not be perfect, but that's because we all have high standards probably. I've worked at both FAANG and non-FAANG companies, and while the sample size is small and not representative of ALL non-FAANG companies, I will say that my experience at FAANG companies is the engineers there do to get dream big and come up with ingenious engineering solutions that make engineering work at non-FAANG look like college senior undergrad projects. I think we should celebrate what Google has done for software engineering instead of nitpicking every little detail that they got wrong.

fatnoah 3y ago
> I will say that my experience at FAANG companies is the engineers there do to get dream big and come up with ingenious engineering solutions that make engineering work at non-FAANG look like college senior undergrad projects My own experience at FAANG and non-FAANG is that __some engineers__ at FAANG get to dream big and demonstrate ingenuity, whereas at non-FAANG (especially startups I've been at) everyone can do this, though not all take the opportunity. At my last FAANG where I was a manager, my opinion is that performance management and the focus on bullet points, metrics, and peer feedback for review time is a huge limiter for everyone else. I spent a lot of time creating cover for people on my team to actually try something new and innovative. More senior management constantly wanted to know why person X on my team was working on a new thing instead of trying to move some incremental metric somewhere else. Based on my experience during performance reviews for senior ICs, risk taking wasn't particularly encouraged at L6 (or even L7) and below as the cost of failure was substantial.

shadowgovt 3y ago

You're not wrong. Google gets to sidestep a lot of problems other lower-margin companies have because they've spent their whole history being near market-leader on search and ads. Things like "deadlines" were foreign to the company for a long time; when you're already the pack leader, you release new features when you feel like it, not to play catch-up with competitors. You can see them struggle in spaces where that's not true (Cloud, for example). There is one underlying technology special sauce they refined to an art form, which is building a reliable system on incredibly unreliable components. This comes from their inception when the two guys building out the company knew how to write code but not build custom machines; their initial attempts at racks (hard drives and motherboards mounted to flexible plywood) are hilarious-looking by modern standards and broke down all the time, so they had to learn how to write software that was fault-tolerant at every link in the chain. That resulted in building an infrastructure that was extremely supportive of experimentation (if the system can survive half of it disappearing, it can survive a software bug crashing half the machines), and the rest of the shape of the company kind of flows from that. It is worth repeating a lot that most companies do not operate at the scale or constraints Google does, and their approach to engineering is emphatically not one-size-fits-all.

lokar 3y ago
I think people forget that the standard back then in the industry was very high margin servers with redundant everything (psu, network/nic, disks, etc). Going to (eventually) mostly dependable simple servers with no redundancy was a big leap.
- shadowgovt 3y ago
  There were a couple places in Google's internal infrastructure where they had picked up traditional "big iron" architectures (mostly by acquisitions), and it was funny to observe the sheer plane between their regular best practices and those systems (when they hadn't yet been replaced). One high-ticket item relied on a huge Oracle database. Every other aspect of that product was patch-on-the-fly, silent-release new versions, forward-and-backwards compatibility of software... But the whole product had regular scheduled outages on the weekends because the Oracle part could not be patched on the fly, had a rigid schema baked into its relational architecture, and had to be brought offline for updates and migrations. It was like everyone was zipping around in racecars and then they all had to pause for Grandpa to cross the street.

ownagefool 3y ago

Probably all of the above is true. From an outsiders, it looks like all of FAANG has scaled to the point where delivery, perf and managing the business has become quite cumbersome. This, I suspect is why they're all used the economic climate to cut back, as they largely are unable to see the benefit of the additional hiring. I have no doubt most of the talent is top tier, but being a strong engineer doesn't deliver profits nor cost savings on it's own.

water-your-self 3y ago
Hiring is always profitable as it lowers the average cost of your engineers. Layoffs are a way to accelerate that cost averaging.

deleted 3y ago

[deleted]

deleted 3y ago
[deleted]