3 ms·

> My leadership is currently promoting "better to ask forgiveness", or put another way: "a bias towards action" lol, that works well until a big issue occurs i

by JTbane 1y ago

> My leadership is currently promoting "better to ask forgiveness", or put another way: "a bias towards action" lol, that works well until a big issue occurs in production

hkt 1y ago
Many companies will roll out to slices of production and monitor error rates. It is part of SRE and I would eat my hat if that wasn't the case here.
- dekhn 1y ago
  Yes, I was SRE at Google (Ads) for several years and that influences my work today. SRE was the first time I was on an ops team that actually was completely empowered to push back against intrusive external changes.
- crabbone 1y ago
  The big events that shatter everything to smithereens aren't that common or really dangerous: most of the time you can lose something, revert and move on from such an event. The real unmitigated danger of unchecked push to production is the velocity with which this generates technical debt. Shipping something implicitly promises the user that that feature will live on for some time, and that removal will be gradual and may require substitute or compensation. So, if you keep shipping half-baked product over and over, you'll be drowning in features that you wish you never shipped, and your support team will be overloaded, and, eventually, the product will become such a mess that developing it further will become too expensive or just too difficult, and then you'll have to spend a lot of money and time doing it all over... and it's also possible you won't have that much money and time.
Aperocky 1y ago
That assume big issue don't occur in production otherwise, with everything having gone through 5 layer of approvals.
- treis 1y ago
  In that case at least 6 people are responsible so nobody is.
mgiampapa 1y ago
Have we learned nothing from Cambridge Analytica?
- munk-a 1y ago
  We learned not to publish as much information about contracts and to have huge networks of third party data sharing so that any actually concerning ones get buried in noise.
itronitron 1y ago
I suppose that's a consequence of having to A/B test everything in order to develop a product
ponector 1y ago
But then it also works. Managers can scapegoat engineer who is asking for forgiveness. It's a total win for the management: they take credits if initiative is successful but blame someone else for failure.
- idrios 1y ago
  Which brings it full circle to engineers saying no to product releases after being burned too harshly by being scapegoated

by JTbane 1y ago

> My leadership is currently promoting "better to ask forgiveness", or put another way: "a bias towards action" lol, that works well until a big issue occurs in production

hkt 1y ago

Many companies will roll out to slices of production and monitor error rates. It is part of SRE and I would eat my hat if that wasn't the case here.

dekhn 1y ago
Yes, I was SRE at Google (Ads) for several years and that influences my work today. SRE was the first time I was on an ops team that actually was completely empowered to push back against intrusive external changes.
crabbone 1y ago
The big events that shatter everything to smithereens aren't that common or really dangerous: most of the time you can lose something, revert and move on from such an event. The real unmitigated danger of unchecked push to production is the velocity with which this generates technical debt. Shipping something implicitly promises the user that that feature will live on for some time, and that removal will be gradual and may require substitute or compensation. So, if you keep shipping half-baked product over and over, you'll be drowning in features that you wish you never shipped, and your support team will be overloaded, and, eventually, the product will become such a mess that developing it further will become too expensive or just too difficult, and then you'll have to spend a lot of money and time doing it all over... and it's also possible you won't have that much money and time.

Aperocky 1y ago

That assume big issue don't occur in production otherwise, with everything having gone through 5 layer of approvals.

treis 1y ago
In that case at least 6 people are responsible so nobody is.

mgiampapa 1y ago

Have we learned nothing from Cambridge Analytica?

munk-a 1y ago
We learned not to publish as much information about contracts and to have huge networks of third party data sharing so that any actually concerning ones get buried in noise.

itronitron 1y ago

I suppose that's a consequence of having to A/B test everything in order to develop a product

ponector 1y ago

But then it also works. Managers can scapegoat engineer who is asking for forgiveness. It's a total win for the management: they take credits if initiative is successful but blame someone else for failure.

idrios 1y ago
Which brings it full circle to engineers saying no to product releases after being burned too harshly by being scapegoated