4 ms·

Integrating with other services makes your software more brittle as there are more points of failure from hardware and networking on your end, to API gateways,

Integrating with other services makes your software more brittle as there are more points of failure from hardware and networking on your end, to API gateways, networking, and hardware on their end. Periodically those things tend to go sideways which means whatever it was your system was trying to do doesn't happen, or worse it only partially happens. Usually some poor engineer is then tasked with figuring out if this is an on-going problem or a one-off problem? Did it affect a single customer, or is it affecting all customers? Is it affecting a single instance, or is the problem happening across all instances? Is it a problem on our end, or is the 3rd party API misbehaving (again) today? Generally you need solid APM and logging to support the operations of this. Then there's queueing tasks for later and making sure you never lose messages between systems. All sorts of fun happens when you hit a capacity limit in your queueing system due to unintended/unnoticed design flaw and all of a sudden system performance starts to tank as things aren't getting processed due to too much stuff in the queue that isn't draining at a sufficient rate. You need to figure out is the queue actually draining or is it still increasing in size? Are all the worker nodes still running or did one (or several) of them hang and didn't die but simply isn't processing any jobs out of the queue but still consuming a spot in your auto-scaling group blocking a new healthy instance from coming online? Is it a problem in your software or is it actually a blip in operations from your cloud-provider? It's way, way, way more complex than simply "chuck some API keys in a file and she'll be right!". That's hobby project level stuff, not realistic ops of a production system.

wruza 4y ago
I’m literally working at automated finance and we are offloading these cases to client support, and late/next day status requests for b2b peers. The only devops-level technicality there is an in-process keep-alive system which prevents overwhelming the support repeatedly. What you’re describing tries to deal with these issues at the level where it is hard, and it is, but that’s exactly the road to complexity and infra costs (and infra issues as well). As a result, it trades simple client issues for hard technical ones, and you still need support. It’s cool to have an ideal incident-free system, but only when you can cover its costs - and more importantly its evolution barriers - by exactly that quality. Iow, don’t replace humans with machines either until machines learn to wipe their own ass, or until it’s just too much ass to manage efficiently anyway. I’d fight at your side few years ago, but since starting to work in this field and asking around I realized nobody cares about that ideal way, cause it’s so expensive and hard to maintain, and the cost of change is prohibitive. It even left few scars on my mental health, exactly because I was too anxious to go this way after rotating in environments where devops is praised as something inevitable to survive. If you squint at us at the right angle you still can see devops, but I think we are just expressing two opposite sets of ideas, each applicable in its own type of a business env. You may still call that a hobby-level, and I’d even agree (cause it essentially is) if that hobby didn’t bring as much as to sustain and profit the company at the levels many companies would wish they were at. If it works and brings revenue who cares how it’s categorized.

wruza 4y ago

I’m literally working at automated finance and we are offloading these cases to client support, and late/next day status requests for b2b peers. The only devops-level technicality there is an in-process keep-alive system which prevents overwhelming the support repeatedly. What you’re describing tries to deal with these issues at the level where it is hard, and it is, but that’s exactly the road to complexity and infra costs (and infra issues as well). As a result, it trades simple client issues for hard technical ones, and you still need support. It’s cool to have an ideal incident-free system, but only when you can cover its costs - and more importantly its evolution barriers - by exactly that quality. Iow, don’t replace humans with machines either until machines learn to wipe their own ass, or until it’s just too much ass to manage efficiently anyway. I’d fight at your side few years ago, but since starting to work in this field and asking around I realized nobody cares about that ideal way, cause it’s so expensive and hard to maintain, and the cost of change is prohibitive. It even left few scars on my mental health, exactly because I was too anxious to go this way after rotating in environments where devops is praised as something inevitable to survive. If you squint at us at the right angle you still can see devops, but I think we are just expressing two opposite sets of ideas, each applicable in its own type of a business env. You may still call that a hobby-level, and I’d even agree (cause it essentially is) if that hobby didn’t bring as much as to sustain and profit the company at the levels many companies would wish they were at. If it works and brings revenue who cares how it’s categorized.