3 ms·

If you're a developer working on a service your primary objective is service delivery. It's really, really difficult to do that without understanding (intimatel

by bradhe 11y ago

If you're a developer working on a service your primary objective is service delivery. It's really, really difficult to do that without understanding (intimately) all the components in play from top to bottom.

atarian 11y ago
Understanding the entire lifecycle of a service (from development to production) is a must for a developer. But it should not be the primary objective of the developer to deliver the service; developers should be working on development. When developers are in charge of delivering a service, they will end up spending most, if not all, of their time debugging operational issues instead of improving the service. This problem gets worse over time as the service will need to be deployed to more places to address scale. An organization with good technical leadership should know this.
onion2k 11y ago
It's really, really difficult to do that without understanding (intimately) all the components in play from top to bottom. I disagree. If a component is essentially a black box, where I know the inputs and I can test to check I get the expected output in known conditions, then that's enough. It has to be, otherwise there wouldn't be any way to use a middleware, or a third-party API, or anything too complex for one person to understand. There are lots of software businesses where no one fully understands the internals of the product. That's fine. So long as there are people who expertly understand the part they're responsible for, and the APIs are well documented, everything will work great. The problem with devops is that it gives people the impression that everyone can be responsible for everything, with everyone on the team having a hand in the design, the code, the testing, the deployment, etc. For a sufficiently complex product that is just not possible. At some point it gets too big, and people have to start accepting that they can't do it all, and that they'll have to trust their colleagues to get things right.
- MichaelGG 11y ago
  This works great when it works great. Black boxes everywhere. But then you run into some issue, like, I dunno, your monitoring service is telling clients the site is going down, even though you can't find any evidence of this. If you know how to go deeper then you run a packet capture and start seeing that, whatdya know, your top-3 IaaS provider's network is shit and is often dropping packets, so those timeouts are actually occurring, though not on anything you control. Now you're able to try to setup a repro, then open a ticket with enough details to get something done. Versus opening a ticket along the lines of "Hi Top-3 IaaS provider. Some monitoring service says our site doesn't always respond, but I'm sure it's not me. Please fix your network, thanks." Not knowing how things work at a deeper level leads to experienced teams working on high-profile sites to come up with stuff like: "We have 2 domains in use, but need to go SSL only. We only have a certificate for one domain. So please implement a DNS-level redirect from domain A to domain B to avoid browser SSL errors." Or an experienced dev insisting that a major vendor is shipping invalid binaries because the runtime says so. And insists they aren't metadata-only libs, but isn't comfortable opening up the binaries to verify and thus spends a day trying to figure things out another way. Being able to get to a baseline of what's working is a fundamental step in troubleshooting. Just like you would always ask a home user to make sure stuff is plugged in, verifying lower-levels of the stack are correctly operating should be a common step - it's unfortunate how common it is that something else is broken under the level you're working at.
  - onion2k 11y ago
    You're confusing 'needs a deep knowledge of how components work' with 'needs a deep of knowledge debugging'. A suitably senior developer should definitely be able to deduce where a problem is most likely to be happening, and figure out a way to test it. In the case you're alluding to, it'd be much more useful to know general information about why any monitoring service might think any IaaS provider is not responding, and have the skills to test that. Knowing details about the components you're using (eg AWS and Pingdom) is much less useful.

by bradhe 11y ago

atarian 11y ago

Understanding the entire lifecycle of a service (from development to production) is a must for a developer. But it should not be the primary objective of the developer to deliver the service; developers should be working on development. When developers are in charge of delivering a service, they will end up spending most, if not all, of their time debugging operational issues instead of improving the service. This problem gets worse over time as the service will need to be deployed to more places to address scale. An organization with good technical leadership should know this.

onion2k 11y ago

It's really, really difficult to do that without understanding (intimately) all the components in play from top to bottom. I disagree. If a component is essentially a black box, where I know the inputs and I can test to check I get the expected output in known conditions, then that's enough. It has to be, otherwise there wouldn't be any way to use a middleware, or a third-party API, or anything too complex for one person to understand. There are lots of software businesses where no one fully understands the internals of the product. That's fine. So long as there are people who expertly understand the part they're responsible for, and the APIs are well documented, everything will work great. The problem with devops is that it gives people the impression that everyone can be responsible for everything, with everyone on the team having a hand in the design, the code, the testing, the deployment, etc. For a sufficiently complex product that is just not possible. At some point it gets too big, and people have to start accepting that they can't do it all, and that they'll have to trust their colleagues to get things right.

MichaelGG 11y ago
This works great when it works great. Black boxes everywhere. But then you run into some issue, like, I dunno, your monitoring service is telling clients the site is going down, even though you can't find any evidence of this. If you know how to go deeper then you run a packet capture and start seeing that, whatdya know, your top-3 IaaS provider's network is shit and is often dropping packets, so those timeouts are actually occurring, though not on anything you control. Now you're able to try to setup a repro, then open a ticket with enough details to get something done. Versus opening a ticket along the lines of "Hi Top-3 IaaS provider. Some monitoring service says our site doesn't always respond, but I'm sure it's not me. Please fix your network, thanks." Not knowing how things work at a deeper level leads to experienced teams working on high-profile sites to come up with stuff like: "We have 2 domains in use, but need to go SSL only. We only have a certificate for one domain. So please implement a DNS-level redirect from domain A to domain B to avoid browser SSL errors." Or an experienced dev insisting that a major vendor is shipping invalid binaries because the runtime says so. And insists they aren't metadata-only libs, but isn't comfortable opening up the binaries to verify and thus spends a day trying to figure things out another way. Being able to get to a baseline of what's working is a fundamental step in troubleshooting. Just like you would always ask a home user to make sure stuff is plugged in, verifying lower-levels of the stack are correctly operating should be a common step - it's unfortunate how common it is that something else is broken under the level you're working at.
- onion2k 11y ago
  You're confusing 'needs a deep knowledge of how components work' with 'needs a deep of knowledge debugging'. A suitably senior developer should definitely be able to deduce where a problem is most likely to be happening, and figure out a way to test it. In the case you're alluding to, it'd be much more useful to know general information about why any monitoring service might think any IaaS provider is not responding, and have the skills to test that. Knowing details about the components you're using (eg AWS and Pingdom) is much less useful.