4 ms·

These kind of organizational problems happen everywhere, that doesn't bug me. What bugs me is when leadership knows about it and doesn't care. After low-level e

by throwaway20371 5y ago

These kind of organizational problems happen everywhere, that doesn't bug me. What bugs me is when leadership knows about it and doesn't care. After low-level engineers stick their professional neck out to complain in internal town halls and through feedback forms, and leadership gives some bullshit answer that doesn't address or even acknowledge the problem. It would be less infuriating if they just said "I don't give a shit." It's the weasel words and pretending the problem doesn't exist that infuriates me. A lot of the time it doesn't even take much work at all to begin addressing the issue, like a working group for continuous improvement of highly-painful high-value processes. You don't even have to solve it. Just attempt to address it.

calmlynarczyk 5y ago
I work at a global corporation with 50,000 employees. Even though I've never been at Google I felt every pain point this video was getting at because our company is trying to implement all of this stuff right now. "Oh you want to go to production? Here's a list from A-XX stating what you need to accomplish that." Thing is I thought they actually handled this gracefully when I started because lots of requirements were tiered with various criteria you had to meet to move up (mostly for brownie points). But then one day the Tech Execs lose their minds and decide "everything needs to meet all criteria for every single process." You want to create an S3 bucket to store data? That will be a week of submitting paperwork and another month of meetings and approvals from various teams you've never heard of. Plus you have to register your schema, implement data quality checks, unit tests, regression tests, get a PR and CO approved for your central config change, remediate any CVEs in the tooling that you used, and build all of this using our in-house CI/CD platform we created because we're just soooo special. Now you're allowed to launch. Oh wait, NO because we've put the entire corporation on hold from launching new systems for the last calendar year because we're still trying to agree on the final process everyone needs to follow to go to production. It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems.
- acdha 5y ago
  > It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems. Followed by the inevitable ranting about “shadow IT”, AKA the requirements gathering they really should have done.
- unethical_ban 5y ago
  In my previous role, the secdevops groups (matrixed teams) were building custom terraform modules for our devs to use in order to easily deploy compliant AWS infrastructure - and devs could only deploy via terraform/CI-CD. While TF specifically states that custom modules are not meant to be used as wrappers, I thought it was a clever way to try getting security "out of the way" while still enforcing best practices.
  - darkwater 5y ago
    > While TF specifically states that custom modules are not meant to be used as wrappers What do you mean with this?
    - easton 5y ago
      "We do not recommend writing modules that are just thin wrappers around single other resource types. If you have trouble finding a name for your module that isn't the same as the main resource type inside it, that may be a sign that your module is not creating any new abstraction and so the module is adding unnecessary complexity. Just use the resource type directly in the calling module instead." from https://www.terraform.io/docs/language/modules/develop/index.html https://www.terraform.io/docs/language/modules/develop/index... If you write different versions of the terraform modules that do some corporate specific magic, I think that would be okay under this rule. It's when you're writing a module that doesn't do any useful magic that they want you to stop and think.
- oblio 5y ago
  > It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems. It's hard to find the right balance. You want a bit of process, but not too much. But it's one of the hardest problems in the existence of humanity and whoever solves it should probably get all the Nobel prizes available (including peace and chemistry!).
ts4z 5y ago
To be fair, they did, and many things have improved. And this video was used as an uncomfortable reminder to make some of those changes.
TideAd 5y ago
My team has issues deploying builds to test machines. It's like 15 steps and takes an hour. The tooling is atrocious and recently got even worse. We eventually found the team responsible for this (the org structure is hard to penetrate because no one answers emails). They said they had no idea anyone was dissatisfied. Then they said that it was a low priority so they didn't care and nothing would be done. In my experience, you can usually convince an engineer that their stuff has a problem and they need to fix it. But it's often impossible to convince management if they aren't on the hook for user satisfaction.
m0zg 5y ago
At Google back then "leadership" might as well not even show up. It was super bottom-up, and _you_, not "leadership" were supposed to identify and fix issues. No "leadership" would stop you, either, at least in most cases. I don't believe that in all my years there anyone ever told me what to do. It was very easy to start projects, shut down projects, get headcount, get resources (if your business case is sufficiently persuasive to others). Not a complete free for all, but certainly _a lot_ more freedom than you'd normally see in companies of that size. And (IMO) people used that freedom and autonomy pretty well. That kinda deteriorated over time, culminating with Sundar "McKinsey" Pichai, and then went rapidly downhill from there, and now I flat out reject their recruiters, based on the feedback from friends still employed there.

calmlynarczyk 5y ago

I work at a global corporation with 50,000 employees. Even though I've never been at Google I felt every pain point this video was getting at because our company is trying to implement all of this stuff right now. "Oh you want to go to production? Here's a list from A-XX stating what you need to accomplish that." Thing is I thought they actually handled this gracefully when I started because lots of requirements were tiered with various criteria you had to meet to move up (mostly for brownie points). But then one day the Tech Execs lose their minds and decide "everything needs to meet all criteria for every single process." You want to create an S3 bucket to store data? That will be a week of submitting paperwork and another month of meetings and approvals from various teams you've never heard of. Plus you have to register your schema, implement data quality checks, unit tests, regression tests, get a PR and CO approved for your central config change, remediate any CVEs in the tooling that you used, and build all of this using our in-house CI/CD platform we created because we're just soooo special. Now you're allowed to launch. Oh wait, NO because we've put the entire corporation on hold from launching new systems for the last calendar year because we're still trying to agree on the final process everyone needs to follow to go to production. It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems.

acdha 5y ago
> It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems. Followed by the inevitable ranting about “shadow IT”, AKA the requirements gathering they really should have done.
unethical_ban 5y ago
In my previous role, the secdevops groups (matrixed teams) were building custom terraform modules for our devs to use in order to easily deploy compliant AWS infrastructure - and devs could only deploy via terraform/CI-CD. While TF specifically states that custom modules are not meant to be used as wrappers, I thought it was a clever way to try getting security "out of the way" while still enforcing best practices.
- darkwater 5y ago
  > While TF specifically states that custom modules are not meant to be used as wrappers What do you mean with this?
  - easton 5y ago
    "We do not recommend writing modules that are just thin wrappers around single other resource types. If you have trouble finding a name for your module that isn't the same as the main resource type inside it, that may be a sign that your module is not creating any new abstraction and so the module is adding unnecessary complexity. Just use the resource type directly in the calling module instead." from https://www.terraform.io/docs/language/modules/develop/index.html https://www.terraform.io/docs/language/modules/develop/index... If you write different versions of the terraform modules that do some corporate specific magic, I think that would be okay under this rule. It's when you're writing a module that doesn't do any useful magic that they want you to stop and think.
oblio 5y ago
> It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems. It's hard to find the right balance. You want a bit of process, but not too much. But it's one of the hardest problems in the existence of humanity and whoever solves it should probably get all the Nobel prizes available (including peace and chemistry!).

ts4z 5y ago

To be fair, they did, and many things have improved. And this video was used as an uncomfortable reminder to make some of those changes.

TideAd 5y ago

My team has issues deploying builds to test machines. It's like 15 steps and takes an hour. The tooling is atrocious and recently got even worse. We eventually found the team responsible for this (the org structure is hard to penetrate because no one answers emails). They said they had no idea anyone was dissatisfied. Then they said that it was a low priority so they didn't care and nothing would be done. In my experience, you can usually convince an engineer that their stuff has a problem and they need to fix it. But it's often impossible to convince management if they aren't on the hook for user satisfaction.

m0zg 5y ago

At Google back then "leadership" might as well not even show up. It was super bottom-up, and _you_, not "leadership" were supposed to identify and fix issues. No "leadership" would stop you, either, at least in most cases. I don't believe that in all my years there anyone ever told me what to do. It was very easy to start projects, shut down projects, get headcount, get resources (if your business case is sufficiently persuasive to others). Not a complete free for all, but certainly _a lot_ more freedom than you'd normally see in companies of that size. And (IMO) people used that freedom and autonomy pretty well. That kinda deteriorated over time, culminating with Sundar "McKinsey" Pichai, and then went rapidly downhill from there, and now I flat out reject their recruiters, based on the feedback from friends still employed there.