3 ms·

Have you tried to debug the same two problems when you do not have access to the infrastructure, versus when you do have infrastructure access? It's a radically

by sargun 9y ago

Have you tried to debug the same two problems when you do not have access to the infrastructure, versus when you do have infrastructure access? It's a radically different experience. In addition, even if no one needs access, a bad actor who writes the code can always put a backdoor in place and read whatever they want.

typicalbender 9y ago
The best approach is to make it the exception not the rule. This ultimately protects everyone and provides a nice paper trail in the event something does happen and needs to be audited. You can partially mitigate the bad actor problem by putting safe guards in place to make sure all changes have to be peer reviewed before shipping.
djsumdog 9y ago
I agree with you. I worked at a BlueCross and when debugging front-end apps, and I'd often dig through the database to check to see what is in each table and what could be throwing an error. I'd often bright up a debugging tool called TeaLeaf that let me watch the entire web session (this was back in 2009; so the full DOM tracking we complain about now has been available for quite some time). Technically, we couldn't see any Group44 information (BlueCross employees) without special permissions. But we had to build that logic into all the debugging tools, and if I wanted to, I could query the database directly. We were all covered by HIPPA, so we could face serious fines over leaked personal medical information. And honestly, the security and audit team were actually pretty good; often identifying Group44 issues and they'd have SSH and su log scanners to constantly check for unauthorized access on servers. It's all about trade-offs. Lyft is still a growing company and they probably just need to implement some more automated auditing. And to be fair, these aren't medical records. It's going to become an issue if a customer service rep gives anything to the police without a warrant though.
- mattmanser 9y ago
  That you casually talk about breaking the law with incredibly sensitive data is shocking. And worse still, you built in a system that circumvented the official policy, simply because you weren't capable of doing your job. You basically built in an illegal back door. If you were my employee I would fire you on the spot for gross negligence when I found that. Or maybe if you were a junior you'd be given a massive dressing down, put on notice, re-sent on all the mandatory company tutorials about sensitive security of data and an urgent fix put in to remove the backdoor. This is just bad engineering, if you can't recreate bugs without having to constantly resort to live data, get someone in who can. It's not particularly hard, it's a learn-able skill of our profession. And if you're really having a problem, and it does happen once in a blue moon, you should have the ability to create an anonymized version of the live db that strips out all the personally identifiable info. That you can still only get with a signed, approved request and a full justification of why you needed it. For the record I have worked somewhere that had highly sensitive data. And they took it seriously, unlike you two. None of the engineering team had access to the live db, the live site or the passwords, apart from some senior team members.
  - djsumdog 9y ago
    > You basically built in an illegal back door. Wat? No, we build the system to block group44 access to all our own diagnostic tools for downstream users. I'm just saying we could also get around it because we build it and had direct db access. We didn't though, or at least I didn't. I did not violate the law at all. We might have had to look at peoples' health claims information to debug issues, and that's fine, so long as we never disclose that data. That was a decade ago, so they might have tightened things up. Still, at some point, some people are going to have access to all the data (A DBA is not going to be very effective at his/her job if they don't have admin access on production).
  - organsnyder 9y ago
    > This is just bad engineering, if you can't recreate bugs without having to constantly resort to live data, get someone in who can. It's not particularly hard, it's a learn-able skill of our profession. In any large enterprise, you're probably working with data spanning multiple in-house databases, vendor applications, message buses, responsible teams... I can develop a greenfield system that makes identifying bugs trivial without accessing production data. Unfortunately, the real world is rarely that simple—especially for any enterprise that has grown organically over time (i.e., all of them). My employer recently retired some systems that were older than me (I'm in my mid 30's), and the organization is simply too large to be understood by any one individual.

by sargun 9y ago

typicalbender 9y ago

The best approach is to make it the exception not the rule. This ultimately protects everyone and provides a nice paper trail in the event something does happen and needs to be audited. You can partially mitigate the bad actor problem by putting safe guards in place to make sure all changes have to be peer reviewed before shipping.

djsumdog 9y ago

I agree with you. I worked at a BlueCross and when debugging front-end apps, and I'd often dig through the database to check to see what is in each table and what could be throwing an error. I'd often bright up a debugging tool called TeaLeaf that let me watch the entire web session (this was back in 2009; so the full DOM tracking we complain about now has been available for quite some time). Technically, we couldn't see any Group44 information (BlueCross employees) without special permissions. But we had to build that logic into all the debugging tools, and if I wanted to, I could query the database directly. We were all covered by HIPPA, so we could face serious fines over leaked personal medical information. And honestly, the security and audit team were actually pretty good; often identifying Group44 issues and they'd have SSH and su log scanners to constantly check for unauthorized access on servers. It's all about trade-offs. Lyft is still a growing company and they probably just need to implement some more automated auditing. And to be fair, these aren't medical records. It's going to become an issue if a customer service rep gives anything to the police without a warrant though.

mattmanser 9y ago
That you casually talk about breaking the law with incredibly sensitive data is shocking. And worse still, you built in a system that circumvented the official policy, simply because you weren't capable of doing your job. You basically built in an illegal back door. If you were my employee I would fire you on the spot for gross negligence when I found that. Or maybe if you were a junior you'd be given a massive dressing down, put on notice, re-sent on all the mandatory company tutorials about sensitive security of data and an urgent fix put in to remove the backdoor. This is just bad engineering, if you can't recreate bugs without having to constantly resort to live data, get someone in who can. It's not particularly hard, it's a learn-able skill of our profession. And if you're really having a problem, and it does happen once in a blue moon, you should have the ability to create an anonymized version of the live db that strips out all the personally identifiable info. That you can still only get with a signed, approved request and a full justification of why you needed it. For the record I have worked somewhere that had highly sensitive data. And they took it seriously, unlike you two. None of the engineering team had access to the live db, the live site or the passwords, apart from some senior team members.
- djsumdog 9y ago
  > You basically built in an illegal back door. Wat? No, we build the system to block group44 access to all our own diagnostic tools for downstream users. I'm just saying we could also get around it because we build it and had direct db access. We didn't though, or at least I didn't. I did not violate the law at all. We might have had to look at peoples' health claims information to debug issues, and that's fine, so long as we never disclose that data. That was a decade ago, so they might have tightened things up. Still, at some point, some people are going to have access to all the data (A DBA is not going to be very effective at his/her job if they don't have admin access on production).
- organsnyder 9y ago
  > This is just bad engineering, if you can't recreate bugs without having to constantly resort to live data, get someone in who can. It's not particularly hard, it's a learn-able skill of our profession. In any large enterprise, you're probably working with data spanning multiple in-house databases, vendor applications, message buses, responsible teams... I can develop a greenfield system that makes identifying bugs trivial without accessing production data. Unfortunately, the real world is rarely that simple—especially for any enterprise that has grown organically over time (i.e., all of them). My employer recently retired some systems that were older than me (I'm in my mid 30's), and the organization is simply too large to be understood by any one individual.