5 ms·

Awful hard to attach a debugger to a production system?

gkhartman 5y ago
Agreed. In a perfect world all production bugs can be reproduced on a test instance, but that's often not the case in the real world. Also, I've seen plenty of timing sensitive bugs that get covered up by the delay introduced by the attached debugger. The much feared "Heisenbug".
rad_gruchalski 5y ago
What if you don’t have physical access to a production system?
- 908B64B197 5y ago
  That's an organizational problem, not a technical one.
  - ahonhn 5y ago
    yeah? well its a pretty standard organisational problem too
  - sorokod 5y ago
    That developers don't have live access is dictated by industry rules in some cases. Not a problem but rather a fact of life.
    - xwolfi 5y ago
      So who has access, if at all ? If nobody has access, how is deployment done or setup ? Can you imagine the "deployer" person ... learning or being replaced by a developer ? This is an organisational weakness not to have figured out a way to make the people responsible for production, also responsible for reproduction and bug fixing. I've seen companies with a full team of developers who never add new features but only reproduce and fix the errors of the other feature teams, companies with read-only access to obfuscated informations good enough to reproduce without any client reference, companies who localize enough people in each jurisdiction the production is regulated in rather than have everyone out and unable to reach it. It's an organisational problem, fix it instead of saying it's impossible to be creative with it. And if my experience if anything to go by, regulators and auditors are either misunderstood in wayyy too conservative ways or are themselves misunderstanding in conservative ways their own rules: there always are ways to explain that it's more risky not to maintain production rather than let if rot slowly because "I can't ssh to the box". While it's always easier to be conservative with vague rules, it's also always possible to be clear faced with absurdity and it's rare people embrace the absurdity: usually they tell you: "I'll lose my job if I don't follow the rule X put in place 20 years ago" and not "I want the production to be offline the entire day during heavy traffic because I think clients prefer when they can't use our software". So, easy: "20 years ago, X. made a mistake - let's show him", and the answer from X is almost always "Why the fuck are you even asking me, ofc exceptions are possible, fix now we do the process after". And "doing the process" is often to can the old rule, or make it change so much it loses all fear factor and start being actually useful. Give up and rot, give feedback and soar :)
      - theshrike79 5y ago
        Lets's say you run a company and buy a piece of software. The business you're in is regulated and deals with people's personal information. Would you let a random coder come in an attach their laptop to your network and start digging around for a problem in their software? Especially if it's your ass on the line if any regulated data is seen by the wrong pair of eyes? Or would you just ship them the logs from their own software and tell them to figure it out.
        908B64B197 5y ago
        > Would you let a random coder come in No. But I would a proper Engineer. The same way I don’t go seek treatment in a mall, but rather in a legitimate medical clinic by real MDs.
      - sorokod 5y ago
        Didn't say that "nobody has access" - only explicitly authorised people do. In addition log data is often stored and indexed in a central location. Access to that data also requires authorization. Does this clarifies things?
  - ghotli 5y ago
    Just a bit of color to other replies here. Financial industries of course have audits and those audits require controls to be written. Controls can have human / organizational solutions as well as technical solutions and they'll ask for proof that these controls are followed. Figure out how to write a control and prove it to the auditors that it's followed and adheres to both of these, then I guess you're golden. Probably easier said than done. FFIEC audit handbook excerpts for "Segregation of Duties" [1] and "Principle of least privilege" [2] [1] https://ithandbook.ffiec.gov/it-booklets/information-security/ii-information-security-program-management/iic-risk-mitigation/iic7-user-security-controls/iic7(c)-segregation-of-duties.aspx https://ithandbook.ffiec.gov/it-booklets/information-securit... [2] https://ithandbook.ffiec.gov/it-booklets/information-security/ii-information-security-program-management/iic-risk-mitigation/iic7-user-security-controls/iic7(b)-user-access-program.aspx https://ithandbook.ffiec.gov/it-booklets/information-securit...
  - rad_gruchalski 5y ago
    What if it's a compliance problem?
_ZeD_ 5y ago
I hope so
theshrike79 5y ago
Usually impossible. It's easier to tell the customers to send the logs from the relevant timeframe and narrow down the problem. Maybe instruct them to up the log level a bit and see if the problem surfaces again. Always log enough to see where the bug is just by looking at the logs. Getting permission to deploy a new version with additional logging might be a bit of a pain.

Agreed. In a perfect world all production bugs can be reproduced on a test instance, but that's often not the case in the real world. Also, I've seen plenty of timing sensitive bugs that get covered up by the delay introduced by the attached debugger. The much feared "Heisenbug".

rad_gruchalski 5y ago

What if you don’t have physical access to a production system?

908B64B197 5y ago
That's an organizational problem, not a technical one.
- ahonhn 5y ago
  yeah? well its a pretty standard organisational problem too
- sorokod 5y ago
  That developers don't have live access is dictated by industry rules in some cases. Not a problem but rather a fact of life.
  - xwolfi 5y ago
    So who has access, if at all ? If nobody has access, how is deployment done or setup ? Can you imagine the "deployer" person ... learning or being replaced by a developer ? This is an organisational weakness not to have figured out a way to make the people responsible for production, also responsible for reproduction and bug fixing. I've seen companies with a full team of developers who never add new features but only reproduce and fix the errors of the other feature teams, companies with read-only access to obfuscated informations good enough to reproduce without any client reference, companies who localize enough people in each jurisdiction the production is regulated in rather than have everyone out and unable to reach it. It's an organisational problem, fix it instead of saying it's impossible to be creative with it. And if my experience if anything to go by, regulators and auditors are either misunderstood in wayyy too conservative ways or are themselves misunderstanding in conservative ways their own rules: there always are ways to explain that it's more risky not to maintain production rather than let if rot slowly because "I can't ssh to the box". While it's always easier to be conservative with vague rules, it's also always possible to be clear faced with absurdity and it's rare people embrace the absurdity: usually they tell you: "I'll lose my job if I don't follow the rule X put in place 20 years ago" and not "I want the production to be offline the entire day during heavy traffic because I think clients prefer when they can't use our software". So, easy: "20 years ago, X. made a mistake - let's show him", and the answer from X is almost always "Why the fuck are you even asking me, ofc exceptions are possible, fix now we do the process after". And "doing the process" is often to can the old rule, or make it change so much it loses all fear factor and start being actually useful. Give up and rot, give feedback and soar :)
    - theshrike79 5y ago
      Lets's say you run a company and buy a piece of software. The business you're in is regulated and deals with people's personal information. Would you let a random coder come in an attach their laptop to your network and start digging around for a problem in their software? Especially if it's your ass on the line if any regulated data is seen by the wrong pair of eyes? Or would you just ship them the logs from their own software and tell them to figure it out.
      - 908B64B197 5y ago
        > Would you let a random coder come in No. But I would a proper Engineer. The same way I don’t go seek treatment in a mall, but rather in a legitimate medical clinic by real MDs.
    - sorokod 5y ago
      Didn't say that "nobody has access" - only explicitly authorised people do. In addition log data is often stored and indexed in a central location. Access to that data also requires authorization. Does this clarifies things?
- ghotli 5y ago
  Just a bit of color to other replies here. Financial industries of course have audits and those audits require controls to be written. Controls can have human / organizational solutions as well as technical solutions and they'll ask for proof that these controls are followed. Figure out how to write a control and prove it to the auditors that it's followed and adheres to both of these, then I guess you're golden. Probably easier said than done. FFIEC audit handbook excerpts for "Segregation of Duties" [1] and "Principle of least privilege" [2] [1] https://ithandbook.ffiec.gov/it-booklets/information-security/ii-information-security-program-management/iic-risk-mitigation/iic7-user-security-controls/iic7(c)-segregation-of-duties.aspx https://ithandbook.ffiec.gov/it-booklets/information-securit... [2] https://ithandbook.ffiec.gov/it-booklets/information-security/ii-information-security-program-management/iic-risk-mitigation/iic7-user-security-controls/iic7(b)-user-access-program.aspx https://ithandbook.ffiec.gov/it-booklets/information-securit...
- rad_gruchalski 5y ago
  What if it's a compliance problem?

_ZeD_ 5y ago

I hope so

theshrike79 5y ago

Usually impossible. It's easier to tell the customers to send the logs from the relevant timeframe and narrow down the problem. Maybe instruct them to up the log level a bit and see if the problem surfaces again. Always log enough to see where the bug is just by looking at the logs. Getting permission to deploy a new version with additional logging might be a bit of a pain.