5 ms·

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. Howe

I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments. However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs. It’s incredibly infuriating and time consuming, and I absolutely hate doing it this way. 90% of the time you also get incredibly vague bug reports with irrelevant logs, and a description of what they think the problem is. Most of the time you need to spend another day finding correct logs and somehow debugging it. Most teams log every single request with all parameters and payloads because they can just replicate the problem locally instead of relying on production support. We’ve long advocated for either having dedicated support or have engineers on some sort of schedule that can do support.

pawelmi 6y ago
I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software. They can see all the data and in theory modify system behavior, eg modify/install any binary. Why are thrall developers less trusted, is it some kind of logic or regulation or just "the way it has always been done" in finance?
- goatinaboat 6y ago
  I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software They can, they just can’t have direct access to live systems due to separation of duties. But there are methods for dealing with this, like centralised logging so a developer never needs to see the original log file on the problematic box.
  - woutr_be 6y ago
    > But there are methods for dealing with this, like centralised logging so a developer never needs to see the original log file on the problematic box. Indeed, most of our newer services have been doing that, so developers have direct access to logs, which makes our lives a bit easier. But sadly a lot of systems our outdated, and nobody wants to invest time and money into implementing things there.
- woutr_be 6y ago
  Part of it probably stems from that it's always been done this way, and the processes haven't evolved over time. Another part is also to protect the bank against rogue employees, it wouldn't be the first time a developer made changes against a production database. You're right that production support has access to those systems, and could potentially make changes and install different binaries, but the amount of people that can do that is extremely limited. Every change also requires a change request that needs several approvals, to request data you need another data request.
  - HenryBemis 6y ago
    Adding to that (because only when I use this example people understand clearly): CompanyA is using ITS OWN assets, funds, IP, etc. you own it, you can burn to the ground. BankB is holding other people's money. You can't go make a mistake, a bank losing 100m of OUR money and say "oops my dev made a mistake". Edit: similar expectations are in publicly traded companies (aka companies where they use OUR money - we give them our cash and they give us stocks). This is why external auditors (e.g. Big4) do not like when they see "poor change management processes", such as inconsistent SoD.
    - woutr_be 6y ago
      > BankB is holding other people's money. You can't go make a mistake, a bank losing 100m of OUR money and say "oops my dev made a mistake". Not only that, but once that happens, regulators will come in, and everybody involved can be held liable. Not only will the bank be fined, but depending on how bad your fuck up was, you'll probably end up losing your job and might face further penalties. So in the interest of everyone, it's best to just avoid it all together.
      - pawelmi 6y ago
        Thanks for answers in this thread. I didn't really mean to have YOLO-type random access to production. I was hoping there are ways in between to bridge that gap between dev and ops in those systems, similarly how it has been done eg with SRE in more relaxed security applications. I was hoping for some solutions on the spectrum are adopted more, like mentioned cetralized logs stripped of private data or granting temporary audited access. But it seems with legacy systems this is much harder to implements. I believe there is an optimum balance where actually fewer mistakes could be made if both people developing and operating te system had more visibility into each other field. As for willful fraud attempts, well you can't rule out devs would do it, so of course there should be various barriers preventing that and proper change management, but, my sampling bias aside, when I look at some recent scandals in finance, take eg Wirecard as the last one, there is more often higher management involved than devs.
- HenryBemis 6y ago
  Seems like we were writing in parallel. "We don't trust" a dev. The change management processes demand the existence of 1) Dev, 2) Librarian (we used to call them that)(that would review and transfer the code, or review and compile the code), 3) the prod sys admin. Some orgs may have a slightly different setup, but in some form or another, but (these general) rules apply. Today with tools like CyberArk it is easier to grant temporarily privileged access to a dev for production support, we also got the tools to trace/monitor/record access, so it makes the process auditor-friendly.
- x87678r 6y ago
  > just "the way it has always been done" in finance Twenty years ago for many systems devs could do whatever they wanted in production. There were insider trading scandals and combined with SOX, regulators cracked down on it so now devs have lost at least write access. If you have an old system that relied on knowledgeable devs to fix stuff its a terrible situation where people just quit and no one can support it.
HenryBemis 6y ago
Adding to the above 100/% spot on comment; "we" in finance/banking do this because segregation of duties is mandatory, none of that DevOps nonsense ;) In the off chance that a dev has the unique knowledge to solve a problem, they may get the firefighter/temporary elevated access needed, but will have to document the reason and the dev's actions very very well, because both internal and external auditors will zero in on that.
user5994461 6y ago
Also in finance with dedicated support teams. Our support was great, some of our guys in Asia were outright fantastic at debugging. To be fair, being great wasn't enough, their job was only possible because the company had unified tooling. A single deployment solution that was deploying near 1M tasks a day in the company, allowing all employees to lookup what is running where and see logs. This made me appreciate just how useful it is to have both dedicated support AND unified tooling. The average company couldn't benefit from having folks on rota because it's impossible to figure out where anything is running.
- woutr_be 6y ago
  We still don't have a single deployment solution, we don't even have single hosting solution. We can choose between 3 different cloud providers, or dedicated servers. (They're pushing for cloud now) The thing is, this all is pushed down from management. In my previous project, we tried to automate as much as possible, but at the end of the day, our production support still wanted to deploy manually. Our business still wanted to see manual end-to-end tests with screenshots. Then there's also different regulations in certain countries where you need to host your application and database in the country itself, so that's another solution. Working in finance can be a real eye opener sometimes.
arminiusreturns 6y ago
>However, production support teams don’t have a real understanding of our application and how it’s build. This reeks of bad documentation to me (which finance is notorious for). If a dev has to be on to support normal prod ops thats largely due to errors in both documentation and often in poor tooling. Sometimes those errors aren't as much the devs fault because of management decisions, usually related to understaffing, but I hate how prod support gets shit on so often for failing to fix an issue when it's not really their fault.
- woutr_be 6y ago
  You're not wrong, the entire thing is because of poor management decisions and poor processes. I don't really shit on specific people, more on the entire process. > This reeks of bad documentation to me Not necessarily, you can document your entire application, but production support only looks at the logs, and does a data extract based on what they see. It would be far more beneficial if you had someone who has a clear understanding of the application so that they can help with debugging and actually solving the problem. At the end of the day, production support are teams who help with 10-20 applications, it's impossible for them to truly understand specific applications. They receive a bug report from the business, investigate and extract logs, then pass it to the relevant development teams. If you need extra info, well though luck, you can reply to the ticket and wait for it to be picked up again. It's no surprise companies like this move so slow.

pawelmi 6y ago

I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software. They can see all the data and in theory modify system behavior, eg modify/install any binary. Why are thrall developers less trusted, is it some kind of logic or regulation or just "the way it has always been done" in finance?

goatinaboat 6y ago
I'm curious, why can't people that created the system take part in production support. You've mentioned finance, that I presume require high level of security, but at the same time there are also people on the other side, just not knowing the system first hand and perhaps having skills different from knowing how to debug software They can, they just can’t have direct access to live systems due to separation of duties. But there are methods for dealing with this, like centralised logging so a developer never needs to see the original log file on the problematic box.
- woutr_be 6y ago
  > But there are methods for dealing with this, like centralised logging so a developer never needs to see the original log file on the problematic box. Indeed, most of our newer services have been doing that, so developers have direct access to logs, which makes our lives a bit easier. But sadly a lot of systems our outdated, and nobody wants to invest time and money into implementing things there.
woutr_be 6y ago
Part of it probably stems from that it's always been done this way, and the processes haven't evolved over time. Another part is also to protect the bank against rogue employees, it wouldn't be the first time a developer made changes against a production database. You're right that production support has access to those systems, and could potentially make changes and install different binaries, but the amount of people that can do that is extremely limited. Every change also requires a change request that needs several approvals, to request data you need another data request.
- HenryBemis 6y ago
  Adding to that (because only when I use this example people understand clearly): CompanyA is using ITS OWN assets, funds, IP, etc. you own it, you can burn to the ground. BankB is holding other people's money. You can't go make a mistake, a bank losing 100m of OUR money and say "oops my dev made a mistake". Edit: similar expectations are in publicly traded companies (aka companies where they use OUR money - we give them our cash and they give us stocks). This is why external auditors (e.g. Big4) do not like when they see "poor change management processes", such as inconsistent SoD.
  - woutr_be 6y ago
    > BankB is holding other people's money. You can't go make a mistake, a bank losing 100m of OUR money and say "oops my dev made a mistake". Not only that, but once that happens, regulators will come in, and everybody involved can be held liable. Not only will the bank be fined, but depending on how bad your fuck up was, you'll probably end up losing your job and might face further penalties. So in the interest of everyone, it's best to just avoid it all together.
    - pawelmi 6y ago
      Thanks for answers in this thread. I didn't really mean to have YOLO-type random access to production. I was hoping there are ways in between to bridge that gap between dev and ops in those systems, similarly how it has been done eg with SRE in more relaxed security applications. I was hoping for some solutions on the spectrum are adopted more, like mentioned cetralized logs stripped of private data or granting temporary audited access. But it seems with legacy systems this is much harder to implements. I believe there is an optimum balance where actually fewer mistakes could be made if both people developing and operating te system had more visibility into each other field. As for willful fraud attempts, well you can't rule out devs would do it, so of course there should be various barriers preventing that and proper change management, but, my sampling bias aside, when I look at some recent scandals in finance, take eg Wirecard as the last one, there is more often higher management involved than devs.
HenryBemis 6y ago
Seems like we were writing in parallel. "We don't trust" a dev. The change management processes demand the existence of 1) Dev, 2) Librarian (we used to call them that)(that would review and transfer the code, or review and compile the code), 3) the prod sys admin. Some orgs may have a slightly different setup, but in some form or another, but (these general) rules apply. Today with tools like CyberArk it is easier to grant temporarily privileged access to a dev for production support, we also got the tools to trace/monitor/record access, so it makes the process auditor-friendly.
x87678r 6y ago
> just "the way it has always been done" in finance Twenty years ago for many systems devs could do whatever they wanted in production. There were insider trading scandals and combined with SOX, regulators cracked down on it so now devs have lost at least write access. If you have an old system that relied on knowledgeable devs to fix stuff its a terrible situation where people just quit and no one can support it.

HenryBemis 6y ago

Adding to the above 100/% spot on comment; "we" in finance/banking do this because segregation of duties is mandatory, none of that DevOps nonsense ;) In the off chance that a dev has the unique knowledge to solve a problem, they may get the firefighter/temporary elevated access needed, but will have to document the reason and the dev's actions very very well, because both internal and external auditors will zero in on that.

user5994461 6y ago

Also in finance with dedicated support teams. Our support was great, some of our guys in Asia were outright fantastic at debugging. To be fair, being great wasn't enough, their job was only possible because the company had unified tooling. A single deployment solution that was deploying near 1M tasks a day in the company, allowing all employees to lookup what is running where and see logs. This made me appreciate just how useful it is to have both dedicated support AND unified tooling. The average company couldn't benefit from having folks on rota because it's impossible to figure out where anything is running.

woutr_be 6y ago
We still don't have a single deployment solution, we don't even have single hosting solution. We can choose between 3 different cloud providers, or dedicated servers. (They're pushing for cloud now) The thing is, this all is pushed down from management. In my previous project, we tried to automate as much as possible, but at the end of the day, our production support still wanted to deploy manually. Our business still wanted to see manual end-to-end tests with screenshots. Then there's also different regulations in certain countries where you need to host your application and database in the country itself, so that's another solution. Working in finance can be a real eye opener sometimes.

arminiusreturns 6y ago

>However, production support teams don’t have a real understanding of our application and how it’s build. This reeks of bad documentation to me (which finance is notorious for). If a dev has to be on to support normal prod ops thats largely due to errors in both documentation and often in poor tooling. Sometimes those errors aren't as much the devs fault because of management decisions, usually related to understaffing, but I hate how prod support gets shit on so often for failing to fix an issue when it's not really their fault.

woutr_be 6y ago
You're not wrong, the entire thing is because of poor management decisions and poor processes. I don't really shit on specific people, more on the entire process. > This reeks of bad documentation to me Not necessarily, you can document your entire application, but production support only looks at the logs, and does a data extract based on what they see. It would be far more beneficial if you had someone who has a clear understanding of the application so that they can help with debugging and actually solving the problem. At the end of the day, production support are teams who help with 10-20 applications, it's impossible for them to truly understand specific applications. They receive a bug report from the business, investigate and extract logs, then pass it to the relevant development teams. If you need extra info, well though luck, you can reply to the ticket and wait for it to be picked up again. It's no surprise companies like this move so slow.