5 ms·

Partly yes, but not fully. The idea is not to go back to the Software Engineer asking the Ops team "Hey, can you provision a Postgres database for me please?"

Partly yes, but not fully. The idea is not to go back to the Software Engineer asking the Ops team "Hey, can you provision a Postgres database for me please?" and then waiting a week for it. It's that the Software Engineer takes a module that was prepared by the Platform team - i.e. "terraform-postgres-mycompany" - which already includes all the requirements the company has for handling databases (think backups, monitoring, encryption, etc.). They can then proceed to use it in the small service-specific Terraform configuration, which really is just putting such ready-made modules together. The important bit being - the Platform team isn't a bottleneck here.

zwischenzug 4y ago
Agreed.
steveBK123 4y ago
Sure, but early in this transition it is slow&painful. At my last org.. The old process of "Hey, can you provision a Postgres database for me please?" was managed in a web ticketing system, change managed, and had 24 hours turnaround! As was most other requests - VMs, NFS shares, FTP servers, network/FW changes, etc. The new process was "Hey is there a terraform module for Postgres?" followed by weeks/months of prioritization and specification battles with the platform team. Somehow despite the platform existing 18 months, we were the first team to need a.. database? Then they didn't want to support Postgres and were forcing everyone onto Aurora. Then it look N months for it to come up in their queue. Rinse&repeat with every fundamental building block of a cloud offering, ad infinitum. This is why I strongly prefer the embed model until the platform has proven itself to at least be at the "80% solved" end of the spectrum.
- swozey 4y ago
  As an Ops, this sounds like a very overworked ops team. I can write a production worthy postgres module in less than a day, maybe a few hours. Or... just a subpar team, as much as I hate to say that.
  - steveBK123 4y ago
    Mostly overworked but also not a team of veterans who knew what they were doing already. Platform Ops:Product App Devs ratio was like 1:30, and they had very little implemented yet so just completely overscheduled and understaffed.
    - swozey 4y ago
      I've been ops for about 15 years and understaffing us is such a huge problem. I just joined a company with 2 other Ops for 80 developers over 8-10 teams, my previous had 7 for 70 and was one of the few places I felt like I could take a breath and relax, easily take days off, etc. I felt more like a SWE than ops with regard to work-life balance/on-call rotations/ability to do creative & research work, etc. I'm glad those 2 ops at my current spot finally have help but they were alone for years while the company was generating a ton of revenue which bothers me but seemingly was their own decision; nobody told them they were overworked until us more SR ops people arrived and went out of our way to pull work away from them. Having all sorts of different teams (style, culture, language) relying on you to not gate-keep is.. stressful at times. I try to context switch as little as possible but sometimes it just can't be avoided.
  - yarosh 4y ago
    As an Ops, who literally replaced Patroni with a Terraform CTS module, in about a week... can say that it would be nearly impossible to do in a non Pizza size team due to communication and confirmation biases, alongside the respective anti-patterns.
timcavel 4y ago
chii 4y ago
> They can then proceed to use it in the small service-specific Terraform configuration, which really is just putting such ready-made modules together. now, a new requirement comes in for the product team(s) - a new service which bulk reads/processes another team's data (e.g., an export service). Turns out this is too slow if the bulk reads use the usual web api route. So direct access to another team's provisioned database is required. This now falls onto the platform team to produce a method for doing so. Imagine this, but multiplied by N, where N is the number of different features being worked on at the same time, and all of them needs time from the platform team to produce something new for them!

zwischenzug 4y ago

Agreed.

steveBK123 4y ago

Sure, but early in this transition it is slow&painful. At my last org.. The old process of "Hey, can you provision a Postgres database for me please?" was managed in a web ticketing system, change managed, and had 24 hours turnaround! As was most other requests - VMs, NFS shares, FTP servers, network/FW changes, etc. The new process was "Hey is there a terraform module for Postgres?" followed by weeks/months of prioritization and specification battles with the platform team. Somehow despite the platform existing 18 months, we were the first team to need a.. database? Then they didn't want to support Postgres and were forcing everyone onto Aurora. Then it look N months for it to come up in their queue. Rinse&repeat with every fundamental building block of a cloud offering, ad infinitum. This is why I strongly prefer the embed model until the platform has proven itself to at least be at the "80% solved" end of the spectrum.

swozey 4y ago
As an Ops, this sounds like a very overworked ops team. I can write a production worthy postgres module in less than a day, maybe a few hours. Or... just a subpar team, as much as I hate to say that.
- steveBK123 4y ago
  Mostly overworked but also not a team of veterans who knew what they were doing already. Platform Ops:Product App Devs ratio was like 1:30, and they had very little implemented yet so just completely overscheduled and understaffed.
  - swozey 4y ago
    I've been ops for about 15 years and understaffing us is such a huge problem. I just joined a company with 2 other Ops for 80 developers over 8-10 teams, my previous had 7 for 70 and was one of the few places I felt like I could take a breath and relax, easily take days off, etc. I felt more like a SWE than ops with regard to work-life balance/on-call rotations/ability to do creative & research work, etc. I'm glad those 2 ops at my current spot finally have help but they were alone for years while the company was generating a ton of revenue which bothers me but seemingly was their own decision; nobody told them they were overworked until us more SR ops people arrived and went out of our way to pull work away from them. Having all sorts of different teams (style, culture, language) relying on you to not gate-keep is.. stressful at times. I try to context switch as little as possible but sometimes it just can't be avoided.
- yarosh 4y ago
  As an Ops, who literally replaced Patroni with a Terraform CTS module, in about a week... can say that it would be nearly impossible to do in a non Pizza size team due to communication and confirmation biases, alongside the respective anti-patterns.

timcavel 4y ago

chii 4y ago

> They can then proceed to use it in the small service-specific Terraform configuration, which really is just putting such ready-made modules together. now, a new requirement comes in for the product team(s) - a new service which bulk reads/processes another team's data (e.g., an export service). Turns out this is too slow if the bulk reads use the usual web api route. So direct access to another team's provisioned database is required. This now falls onto the platform team to produce a method for doing so. Imagine this, but multiplied by N, where N is the number of different features being worked on at the same time, and all of them needs time from the platform team to produce something new for them!