6 ms·

Google's internal developer hazing and poor DX surely have something to do with this. I've talked with multiple ex coworker developers at Google about the 3 to

by joshe 4y ago

Google's internal developer hazing and poor DX surely have something to do with this. I've talked with multiple ex coworker developers at Google about the 3 to 6 months it takes to actually figure out how to deploy something. Also the constant overhead of approvals and oddball systems (gerrit!) thereafter. Eventually good DX seems unimportant and even associated with failure. Real programmers grovel to weird dependancies and find out stuff the hard way. There's a better alternate universe where Firebase and 2010 era Google App Engine are the template for GCP.

kotlin2 4y ago
I still have no idea how to navigate a Gerrit PR. I sometimes land on one trying to investigate a bug in Chromium or something like that. I’m sure Gerrit’s approach has some benefits, but it definitely fails at showing a simple before and after view, i.e. “here’s what changed.”
- hinkley 4y ago
  When change is hard people start over. Anything in your system that adds friction to modifying existing code (be it source structure, lack of static analysis tools, build shenanigans, diff tool) encourages people to copy instead, or reinvent.
SkittlesNTwix 4y ago
> internal developer hazing Could you expand upon what you mean by this part?
- piotrkaminski 4y ago
  Not OP, but I'm guessing they're referring to the "readability" approval process, which can sometimes be pretty brutal. (Or at least that was the case a decade ago, don't know how it's evolved since.)
hinkley 4y ago
People with a high pain threshold create systems that require a high pain threshold. That filters out a lot of people with a low tolerance for bullshit. Then the “indispensability” of people who built job security into the system filters again.
deleted 4y ago
[deleted]
asciimike 4y ago
> I've talked with multiple ex coworker developers at Google about the 3 to 6 months it takes to actually figure out how to deploy something. In typical developer environments, you end up with a bell curve of "easy things are relatively easy (host a static site, 30 seconds), but some things end up being totally impossible (build a global CDN with 99.999... availability)." Developing at Google is not at all like this: building anything at Google is medium-hard. It takes longer to do simple things, but at the same time, basically nothing is technically impossible. In many cases, doing the hard things is much easier than elsewhere because, yeah, sure, you have to set up 20 different config files, but those config files abstract damn near anything a production system needs to operate. Example: I wanted to add improved image serving functionality (e.g. imgix style URL params), and I was able to integrate the existing infra Photos uses in about 15 lines of code and a few hours. I don't think there are too many places in the world where it's possible to provide that functionality in an afternoon. Why it never shipped is a separate story involving non-technical reasons (cross-PA politics). > Also the constant overhead of approvals and oddball systems (gerrit!) thereafter. IIRC only the Android codebase used Gerrit, the rest of Firebase and GCP were on Google3. The approvals and other stuff... true. > Eventually good DX seems unimportant and even associated with failure. I think this comes down to a definition of "good DX". I think the HN definition (indeed by default one) is "how quickly can I solve for an issue using the happy path." Indeed, Firebase solves that for a good number of problems (why people use it, why there are people who love it). At some point though, you can't solve for _all_ use cases along the happy path (see e.g. SAML, ABAC, any $ENTERPRISE_FEATURE). If you want to expand the business, you need to solve for some things that don't have a perfect happy path, and that's where things start getting messy. IMO, the goal of Firebase's acquisition was to keep Firebase solving the 80-90% of things that had a happy path, and GCP was the remaining 10-20% where folks had to break out of to do arbitrary things. It seems like a lot of folks in other threads here are complaining about that because the abstractions leak out--I think it's a fair criticism, but I also am not entirely sure how to solve for that.
- joshe 4y ago
  Great nuance, thanks for commenting. DX to me is reduced cognitive load. We are all operating at the limits of abilities, better DX expands the scope of what we can accomplish in our weeks and months. Often things sold as DX are absolute garbage dead end schlock, like a Flash to iphone app builder or something. So I get some of the indifference (google's not yours). But I don't think we should think of it that way. Just on your last paragraph, I think it started that way. That Firebase would be the competent general problem solver that you might need to break out of. Firebase and App Engine have tended to weaken over time, so that you needed to move to GCP more often. Ideally Firebase would get more powerful, cover more cases, and have higher quota limits. Instead it feels like they are being degraded or abandoned rather than embraced. I've never gotten the impression that at google DX was lionized. Examples of doing this well culturally are pre-Salesforce Heroku, Github, vscode, and Stripe. A github/heroku style culture doing GCP would crush all competitors. Instead if anything GCP is a little worse than AWS.
jeffbee 4y ago
Googlers deploy software on production systems on Day 2 of their on-boarding training. I've never worked anywhere that had as straightforward and usable developer tools.
- asciimike 4y ago
  Running `borgcfg up` is very different than spending weeks writing monarch configs :P I totally agree that Google's tooling is _legendary_ when you understand it all, but I also agree that having to learn 15-20 different config languages to get a service running in production was pretty damn painful when we only had three months to do it. There's also been a good amount of work in the world outside of google to get tooling to a similar place (and in many cases, without 20 years of baggage).
  - jeffbee 4y ago
    I guess the trick is to understand that it's all protobufs anyway, and the config langs are just ways of generating protobufs. Outside of Google, it is useful to understand that, for example, Helm is not part of Kubernetes, in the same way that borgcfg is not part of borg and many people use borg without borgcfg.
    - asciimike 4y ago
      Truly, all we did was move data from one proto to another...

by joshe 4y ago

kotlin2 4y ago

I still have no idea how to navigate a Gerrit PR. I sometimes land on one trying to investigate a bug in Chromium or something like that. I’m sure Gerrit’s approach has some benefits, but it definitely fails at showing a simple before and after view, i.e. “here’s what changed.”

hinkley 4y ago
When change is hard people start over. Anything in your system that adds friction to modifying existing code (be it source structure, lack of static analysis tools, build shenanigans, diff tool) encourages people to copy instead, or reinvent.

SkittlesNTwix 4y ago

> internal developer hazing Could you expand upon what you mean by this part?

piotrkaminski 4y ago
Not OP, but I'm guessing they're referring to the "readability" approval process, which can sometimes be pretty brutal. (Or at least that was the case a decade ago, don't know how it's evolved since.)

hinkley 4y ago

People with a high pain threshold create systems that require a high pain threshold. That filters out a lot of people with a low tolerance for bullshit. Then the “indispensability” of people who built job security into the system filters again.

deleted 4y ago

[deleted]

asciimike 4y ago

> I've talked with multiple ex coworker developers at Google about the 3 to 6 months it takes to actually figure out how to deploy something. In typical developer environments, you end up with a bell curve of "easy things are relatively easy (host a static site, 30 seconds), but some things end up being totally impossible (build a global CDN with 99.999... availability)." Developing at Google is not at all like this: building anything at Google is medium-hard. It takes longer to do simple things, but at the same time, basically nothing is technically impossible. In many cases, doing the hard things is much easier than elsewhere because, yeah, sure, you have to set up 20 different config files, but those config files abstract damn near anything a production system needs to operate. Example: I wanted to add improved image serving functionality (e.g. imgix style URL params), and I was able to integrate the existing infra Photos uses in about 15 lines of code and a few hours. I don't think there are too many places in the world where it's possible to provide that functionality in an afternoon. Why it never shipped is a separate story involving non-technical reasons (cross-PA politics). > Also the constant overhead of approvals and oddball systems (gerrit!) thereafter. IIRC only the Android codebase used Gerrit, the rest of Firebase and GCP were on Google3. The approvals and other stuff... true. > Eventually good DX seems unimportant and even associated with failure. I think this comes down to a definition of "good DX". I think the HN definition (indeed by default one) is "how quickly can I solve for an issue using the happy path." Indeed, Firebase solves that for a good number of problems (why people use it, why there are people who love it). At some point though, you can't solve for _all_ use cases along the happy path (see e.g. SAML, ABAC, any $ENTERPRISE_FEATURE). If you want to expand the business, you need to solve for some things that don't have a perfect happy path, and that's where things start getting messy. IMO, the goal of Firebase's acquisition was to keep Firebase solving the 80-90% of things that had a happy path, and GCP was the remaining 10-20% where folks had to break out of to do arbitrary things. It seems like a lot of folks in other threads here are complaining about that because the abstractions leak out--I think it's a fair criticism, but I also am not entirely sure how to solve for that.

joshe 4y ago
Great nuance, thanks for commenting. DX to me is reduced cognitive load. We are all operating at the limits of abilities, better DX expands the scope of what we can accomplish in our weeks and months. Often things sold as DX are absolute garbage dead end schlock, like a Flash to iphone app builder or something. So I get some of the indifference (google's not yours). But I don't think we should think of it that way. Just on your last paragraph, I think it started that way. That Firebase would be the competent general problem solver that you might need to break out of. Firebase and App Engine have tended to weaken over time, so that you needed to move to GCP more often. Ideally Firebase would get more powerful, cover more cases, and have higher quota limits. Instead it feels like they are being degraded or abandoned rather than embraced. I've never gotten the impression that at google DX was lionized. Examples of doing this well culturally are pre-Salesforce Heroku, Github, vscode, and Stripe. A github/heroku style culture doing GCP would crush all competitors. Instead if anything GCP is a little worse than AWS.

jeffbee 4y ago

Googlers deploy software on production systems on Day 2 of their on-boarding training. I've never worked anywhere that had as straightforward and usable developer tools.

asciimike 4y ago
Running `borgcfg up` is very different than spending weeks writing monarch configs :P I totally agree that Google's tooling is _legendary_ when you understand it all, but I also agree that having to learn 15-20 different config languages to get a service running in production was pretty damn painful when we only had three months to do it. There's also been a good amount of work in the world outside of google to get tooling to a similar place (and in many cases, without 20 years of baggage).
- jeffbee 4y ago
  I guess the trick is to understand that it's all protobufs anyway, and the config langs are just ways of generating protobufs. Outside of Google, it is useful to understand that, for example, Helm is not part of Kubernetes, in the same way that borgcfg is not part of borg and many people use borg without borgcfg.
  - asciimike 4y ago
    Truly, all we did was move data from one proto to another...