4 ms·

There are a bunch of comments saying that Google is just like any other big tech company and that the exciting engineering bit has gone. My experience is only f

There are a bunch of comments saying that Google is just like any other big tech company and that the exciting engineering bit has gone. My experience is only from the last 2.5 years, but I've got a slightly different take. Engineering from >10 years ago seems like it was a wild west. Some truly stunning pieces of technology, strung together with duct tape. Everything had its own configuration language, workflow engine, monitoring solution, etc. Deployments were infrequent, unreliable, and sometimes even done from a dev's machine. I don't want to disparage engineers who worked there during that time, the systems were amazing, but everything around the edge seemed pretty disparate, and I suspect gave rise to the "promo project" style development meme. Nowadays we've got Boq/Pod, the P2020 suite, Rollouts, the automated job sizing technologies, even BCID. None of these are perfect by any means, but the convergence is a really good thing. I switched team, PA, and discipline 6 months ago, and it was dead easy to get up and running because I already knew the majority of the tech, and most of that tech is pretty mature and capable. Maybe Google has become more like other tech companies (although I doubt they have this level of convergence), but I think people glorify the old days at Google and miss that a lot of bad engineering was done. Just one example, but I suspect Google has some of the best internal security of any software company on the planet, and that's a very good thing, but it most certainly didn't have that back in the day.

whiterknight 2y ago
"Bad engineering" in adhoc ways tends to mean new ideas are being explored. Sophisticated deployments, logistics, procedures, tends to mean you're optimizing or extending existing system. That's not to disparage the latter. making things work at scale is hard engineering. But when people praise the glory days, it may be a preference for working on new ideas in small projects.
- danpalmer 2y ago
  There are most definitely trade-offs. Getting a small project off the ground now is hard because the size of the organisation puts so many requirements on you. The way I like to look at it is that small things take a while, but big things happen remarkably quickly. For example, rolling out a "hello world" service might take a few days, but having that service serve 1M QPS is pretty much free (in terms of effort). At my previous place a new service might have taken an hour to set up a deployment for, but having it serve 1M QPS would have required overhauling several aspects of our infrastructure over months.
skybrian 2y ago
Although there were some rough areas, Google has been considered to have pretty advanced software engineering for 20 years or more. There was plenty for a new engineer to boggle at even then. On the other hand, someone who started in 2015 missed out on the years when Google was mostly considered to be the good guys and given the benefit of the doubt. That’s about when the culture started turning against “big tech” in general (rather than specific companies like Microsoft).
- ethbr1 2y ago
  > the years when Google was mostly considered to be the good guys and given the benefit of the doubt It's been interesting to see the tech and public zeitgeist shift on this. Lesson: if you're going to be popular for killing a king, be careful how similarly the thing you place on your head starts to look to a crown.
  - zrobotics 2y ago
    Serious question: what king did Google kill? They really didn't take down a big player in an existing industry, they started by being very good at search and displaced other search companies at a time when internet search wasn't a big business. I don't have a great answer here, but one of the things that I think caused public opinion to shift is when people started realizing how truly massive they had gotten. It's possible to avoid Facebook, Apple, Amazon, and Netflix, but by 2015 it started to become nearly impossible to avoid Google or their products. Not sure if that's the reason public opinion shifted, I'd started to get worried about the same time I also dropped my Facebook account, so ~2011. I have successfully managed to avoid Facebook products and tracking, but I had to give up on avoiding Google, it's simply not possible unless one is willing to make extreme sacrifices. So i kinda gave up, it was starting to become digital masochism. There's a very bitter irony in me typing this on a pixel 8,but I've just accepted that there isn't any avoiding Google tracking my online life so I've just stopped caring as much as I used to.
    - justinclift 2y ago
      > I think caused public opinion to shift is when people started realizing how truly massive they had gotten. I'm personally more of the opinion that Google has caused enough serious issues for enough people - with famously no way to get the issues resolved - that they themselves seeded or caused the negative public opinion. Combine that with Google behaviour of clearly doing things in their own interest even when not to their user's benefit (manifest v3 proposal is a good example), and many people are like "screw Google". ;)
throwaway20222 2y ago
From my experience crossing eras; the tools have improved, and there are still some amazingly brilliant people, but the amount of overhead and the ratio of top level, passionate builders has dropped meaningfully. I have had a priority project stuck in legal hell for six months because it’s just a morass of logistics. The project is benign but it doesn’t matter. We are ossified. I look at it as somewhat inevitable considering the path the company has taken, but it is certainly different. We are cranking out money and that’s fine, but it is a change.
- xnx 2y ago
  Success is its own trap. At a certain size, there's more to lose than to gain with any change.
hiddencost 2y ago
Oof. I knew it was going to happen, but people thinking boq/pod are good rather than actually killing the company makes me pretty sad.
- alienchow 2y ago
  Boq/PoD merely standardizes production deployments. You can choose to not do it, and end up having to redo everything yourself. The vast majority of all services uses it perfectly fine. Unified rollouts, unified versioning, universal dashboards, security compliance, standardized quota management, standard alert configs. It's opinionated, but I can drop into any team and hit the ground running. I don't want to learn your custom dashboards doing the exact same thing with different names. The issue with PoD is that it's a great concept and implementation that's tight on resources, and doesn't have much of a plan to expand beyond its current paradigm. The P2020 team deserves way more recognition for all the work they have done.
- elevatedastalt 2y ago
  Please tell me how you think Boq is killing the company.
miki123211 2y ago
> Nowadays we've got Boq/Pod, the P2020 suite, Rollouts, the automated job sizing technologies, even BCID Could you elaborate on what these are for a non-googler? Ironically enough, Google isn't very helpful.
- aoeusnth1 2y ago
  Boq/Pod: canonical service frameworks and configuration automation all-in-one system. Boq and Pod give you blueprints for server configs, deployment, server discovery, environments + release pipelines, monitoring, alerting, canary analysis, functional testing, integration testing, unit testing and presubmit, server throttling, etc etc all for free with automated setup. P2020 + Rollouts: This is for intent-driven deployment, where deployment configuration for jobs, networks, pubsub topics, and other resources are declaratively checked into source, and the Annealing system automatically finds diffs against production state and resolves those diffs (aka a rollout). Automated job sizing: load-trend driven automated capacity planning. Separate from autopilot, which is a more time-sensitive autoscaler. This will actually make configuration changes based on trends in traffic, and request quota for you automatically (with your approval). BCID: this is for verifiable binaries and configs in prod, where it requires two parties to make source changes, two parties to make config changes, two parties to approve non-automated production changes, and only verified check-in binaries and configs can run in prod, not stuff you build on your desktop machine.
  - munksbeer 2y ago
    > Boq/Pod: canonical service frameworks and configuration automation all-in-one system. Boq and Pod give you blueprints for server configs, deployment, server discovery, environments + release pipelines, monitoring, alerting, canary analysis, functional testing, integration testing, unit testing and presubmit, server throttling, etc etc all for free with automated setup. That sounds amazing.
- nazcan 2y ago
  These are well-lit paths, that try to make it easy to take a server you want to run in production, and give you reasonable releases/monitoring/canarying for free (e.g. no one should have to configure something that stops pushes that are causing crashes of the updates tasks).
aorloff 2y ago
> 10 years ago if you were operating at scale you still racked your own servers
Rebelgecko 2y ago
On the flip side, clueless execs are killing tools like Code Search because they don't understand the value. They're willing to layoff off a team to save money even if it reduces the productivity of 20,000 employees by 1%
mike_hearn 2y ago
I left over ten years ago and it's hard to understand that perspective. Back when I was an SRE (~2006 to 2009) there were only one or two monitoring systems (which didn't overlap, so you could argue there was one) and a handful of config languages. Compared to anywhere else Google had military levels of discipline and order. > Deployments were infrequent, unreliable, and sometimes even done from a dev's machine. Deployments were weekly and done from a dev machine because that way someone was watching it and could intervene in case of unexpected problems. Some teams didn't do that and tried to automate rollouts completely. I could always tell which products weren't doing enough manual work because I'd encounter obviously broken features live in production, do a bit of digging and discover end user complaints had been piling up in the support forums for months. But nobody was reading them, and the metrics didn't show any problem, and changes flowed into prod so the team just ... didn't realize their product wasn't working. There's no substitute for trying stuff out for yourself. I encounter clearly broken software that never seems to get fixed way too often these days and I'm sure it's partly because the teams in question don't use their own product much and don't even realize anything is wrong.
- danpalmer 2y ago
  I think the state of the art has moved on quite a way from this. I understand the point of view that someone should be watching a release, but the alternative is not "no one watching a release", but more that binary releases should be no-ops. With feature flagging the binary release should do nothing different so that no one watching it is not a problem. Additionally, rolling out from a dev machine brings so many risks – security, reproducibility, human error, and so on. I'm glad this is not the way things work anymore, and for the most part things are more reliable as a result.
  - mike_hearn 2y ago
    Well, to be clear "rollout from a dev machine" meant just that the rollout controller ran locally, the actual software being released was built by a release pipeline, placed into signed packages and so on. So it was all auditable. The people doing the rollouts were those who had production administrator access anyway for on-call troubleshooting and debugging and permissions were enforced, so there was no security impact. And the same process was used for flag flips so just putting everything behind flags didn't make much difference. It doesn't sound like what's done now is a whole lot different tbh.
contrarian1234 2y ago
So sure they can very reliably and safely deploy... It just doesn't feel like they've actually deployed anything interesting in 5+ years (except maybe some AI stuff). From the outside it looks like the company is perpetually refactoring history's biggest yakshave...? not surprising for a programmer-run company I'm sure it still matters to bring down the power and server bills. But one can't help feel like they could be doing much more
tarsinge 2y ago
What’s the point if the capacity to deploy good products has been lost in the process? What if what you see as unreliable was at the hearth of what enabled great things to happen. This is a typical caveat of a programmer centric organization. Things get more reliable but more frozen and innovation die.

whiterknight 2y ago

"Bad engineering" in adhoc ways tends to mean new ideas are being explored. Sophisticated deployments, logistics, procedures, tends to mean you're optimizing or extending existing system. That's not to disparage the latter. making things work at scale is hard engineering. But when people praise the glory days, it may be a preference for working on new ideas in small projects.

danpalmer 2y ago
There are most definitely trade-offs. Getting a small project off the ground now is hard because the size of the organisation puts so many requirements on you. The way I like to look at it is that small things take a while, but big things happen remarkably quickly. For example, rolling out a "hello world" service might take a few days, but having that service serve 1M QPS is pretty much free (in terms of effort). At my previous place a new service might have taken an hour to set up a deployment for, but having it serve 1M QPS would have required overhauling several aspects of our infrastructure over months.

skybrian 2y ago

Although there were some rough areas, Google has been considered to have pretty advanced software engineering for 20 years or more. There was plenty for a new engineer to boggle at even then. On the other hand, someone who started in 2015 missed out on the years when Google was mostly considered to be the good guys and given the benefit of the doubt. That’s about when the culture started turning against “big tech” in general (rather than specific companies like Microsoft).

ethbr1 2y ago
> the years when Google was mostly considered to be the good guys and given the benefit of the doubt It's been interesting to see the tech and public zeitgeist shift on this. Lesson: if you're going to be popular for killing a king, be careful how similarly the thing you place on your head starts to look to a crown.
- zrobotics 2y ago
  Serious question: what king did Google kill? They really didn't take down a big player in an existing industry, they started by being very good at search and displaced other search companies at a time when internet search wasn't a big business. I don't have a great answer here, but one of the things that I think caused public opinion to shift is when people started realizing how truly massive they had gotten. It's possible to avoid Facebook, Apple, Amazon, and Netflix, but by 2015 it started to become nearly impossible to avoid Google or their products. Not sure if that's the reason public opinion shifted, I'd started to get worried about the same time I also dropped my Facebook account, so ~2011. I have successfully managed to avoid Facebook products and tracking, but I had to give up on avoiding Google, it's simply not possible unless one is willing to make extreme sacrifices. So i kinda gave up, it was starting to become digital masochism. There's a very bitter irony in me typing this on a pixel 8,but I've just accepted that there isn't any avoiding Google tracking my online life so I've just stopped caring as much as I used to.
  - justinclift 2y ago
    > I think caused public opinion to shift is when people started realizing how truly massive they had gotten. I'm personally more of the opinion that Google has caused enough serious issues for enough people - with famously no way to get the issues resolved - that they themselves seeded or caused the negative public opinion. Combine that with Google behaviour of clearly doing things in their own interest even when not to their user's benefit (manifest v3 proposal is a good example), and many people are like "screw Google". ;)

throwaway20222 2y ago

From my experience crossing eras; the tools have improved, and there are still some amazingly brilliant people, but the amount of overhead and the ratio of top level, passionate builders has dropped meaningfully. I have had a priority project stuck in legal hell for six months because it’s just a morass of logistics. The project is benign but it doesn’t matter. We are ossified. I look at it as somewhat inevitable considering the path the company has taken, but it is certainly different. We are cranking out money and that’s fine, but it is a change.

xnx 2y ago
Success is its own trap. At a certain size, there's more to lose than to gain with any change.

hiddencost 2y ago

Oof. I knew it was going to happen, but people thinking boq/pod are good rather than actually killing the company makes me pretty sad.

alienchow 2y ago
Boq/PoD merely standardizes production deployments. You can choose to not do it, and end up having to redo everything yourself. The vast majority of all services uses it perfectly fine. Unified rollouts, unified versioning, universal dashboards, security compliance, standardized quota management, standard alert configs. It's opinionated, but I can drop into any team and hit the ground running. I don't want to learn your custom dashboards doing the exact same thing with different names. The issue with PoD is that it's a great concept and implementation that's tight on resources, and doesn't have much of a plan to expand beyond its current paradigm. The P2020 team deserves way more recognition for all the work they have done.
elevatedastalt 2y ago
Please tell me how you think Boq is killing the company.

miki123211 2y ago

> Nowadays we've got Boq/Pod, the P2020 suite, Rollouts, the automated job sizing technologies, even BCID Could you elaborate on what these are for a non-googler? Ironically enough, Google isn't very helpful.

aoeusnth1 2y ago
Boq/Pod: canonical service frameworks and configuration automation all-in-one system. Boq and Pod give you blueprints for server configs, deployment, server discovery, environments + release pipelines, monitoring, alerting, canary analysis, functional testing, integration testing, unit testing and presubmit, server throttling, etc etc all for free with automated setup. P2020 + Rollouts: This is for intent-driven deployment, where deployment configuration for jobs, networks, pubsub topics, and other resources are declaratively checked into source, and the Annealing system automatically finds diffs against production state and resolves those diffs (aka a rollout). Automated job sizing: load-trend driven automated capacity planning. Separate from autopilot, which is a more time-sensitive autoscaler. This will actually make configuration changes based on trends in traffic, and request quota for you automatically (with your approval). BCID: this is for verifiable binaries and configs in prod, where it requires two parties to make source changes, two parties to make config changes, two parties to approve non-automated production changes, and only verified check-in binaries and configs can run in prod, not stuff you build on your desktop machine.
- munksbeer 2y ago
  > Boq/Pod: canonical service frameworks and configuration automation all-in-one system. Boq and Pod give you blueprints for server configs, deployment, server discovery, environments + release pipelines, monitoring, alerting, canary analysis, functional testing, integration testing, unit testing and presubmit, server throttling, etc etc all for free with automated setup. That sounds amazing.
nazcan 2y ago
These are well-lit paths, that try to make it easy to take a server you want to run in production, and give you reasonable releases/monitoring/canarying for free (e.g. no one should have to configure something that stops pushes that are causing crashes of the updates tasks).

aorloff 2y ago

> 10 years ago if you were operating at scale you still racked your own servers

Rebelgecko 2y ago

On the flip side, clueless execs are killing tools like Code Search because they don't understand the value. They're willing to layoff off a team to save money even if it reduces the productivity of 20,000 employees by 1%

mike_hearn 2y ago

I left over ten years ago and it's hard to understand that perspective. Back when I was an SRE (~2006 to 2009) there were only one or two monitoring systems (which didn't overlap, so you could argue there was one) and a handful of config languages. Compared to anywhere else Google had military levels of discipline and order. > Deployments were infrequent, unreliable, and sometimes even done from a dev's machine. Deployments were weekly and done from a dev machine because that way someone was watching it and could intervene in case of unexpected problems. Some teams didn't do that and tried to automate rollouts completely. I could always tell which products weren't doing enough manual work because I'd encounter obviously broken features live in production, do a bit of digging and discover end user complaints had been piling up in the support forums for months. But nobody was reading them, and the metrics didn't show any problem, and changes flowed into prod so the team just ... didn't realize their product wasn't working. There's no substitute for trying stuff out for yourself. I encounter clearly broken software that never seems to get fixed way too often these days and I'm sure it's partly because the teams in question don't use their own product much and don't even realize anything is wrong.

danpalmer 2y ago
I think the state of the art has moved on quite a way from this. I understand the point of view that someone should be watching a release, but the alternative is not "no one watching a release", but more that binary releases should be no-ops. With feature flagging the binary release should do nothing different so that no one watching it is not a problem. Additionally, rolling out from a dev machine brings so many risks – security, reproducibility, human error, and so on. I'm glad this is not the way things work anymore, and for the most part things are more reliable as a result.
- mike_hearn 2y ago
  Well, to be clear "rollout from a dev machine" meant just that the rollout controller ran locally, the actual software being released was built by a release pipeline, placed into signed packages and so on. So it was all auditable. The people doing the rollouts were those who had production administrator access anyway for on-call troubleshooting and debugging and permissions were enforced, so there was no security impact. And the same process was used for flag flips so just putting everything behind flags didn't make much difference. It doesn't sound like what's done now is a whole lot different tbh.

contrarian1234 2y ago

So sure they can very reliably and safely deploy... It just doesn't feel like they've actually deployed anything interesting in 5+ years (except maybe some AI stuff). From the outside it looks like the company is perpetually refactoring history's biggest yakshave...? not surprising for a programmer-run company I'm sure it still matters to bring down the power and server bills. But one can't help feel like they could be doing much more

tarsinge 2y ago

What’s the point if the capacity to deploy good products has been lost in the process? What if what you see as unreliable was at the hearth of what enabled great things to happen. This is a typical caveat of a programmer centric organization. Things get more reliable but more frozen and innovation die.