4 ms·

I think you misunderstand. Even with code review policies, there is still a short list of people who can push to production without going through code review. N

I think you misunderstand. Even with code review policies, there is still a short list of people who can push to production without going through code review. Not from a policy standpoint but from a security and access perspective.

falcolas 8y ago
Which indicates a different problem if the sabatuer was on that short list. And in this day and age of cryptographically signed commits, the number of people who could do this should go down even further. Given this, I think it’s much more likely there were few or bad controls, than a person in an incredibly privileged position working in the margins.
- SolarNet 8y ago
  Someone builds the code, it doesn't have to be committed.
dingaling 8y ago
The chain we had in ${BIGCORP}: Programmers: read-write to repository Staging team: read-only on repository, read-write to test servers and staging zone Deployment team: read-only on repository and staging zone, read-write to production It wouldn't prevent malicious code going out but at least would require a chain of cooperation between employees, which would be harder to achieve.
- kbenson 8y ago
  > It wouldn't prevent malicious code going out but at least would require a chain of cooperation between employees, which would be harder to achieve. No, it just requires a chain of cooperation between authorized accounts. That's a very important distinction, especially here, where the email in question alleged the following: This included making direct code changes to the Tesla Manufacturing Operating System under false usernames
- cbzoiav 8y ago
  Did your staging or deployment team review code? And what stops a single member of the staging or deployment team patching the build scripts, binaries or just installing their own software to a server?
- tomp 8y ago
  So the deployment team could write and push malicious code to production?
  - XorNot 8y ago
    They pretty much always can. Sure sure you can imagine some perfect system which would mitigate it but no one - definitely not your bank - is doing that. It very much sounds like thats the case here - production code was edited, and subsequent auditing has found what should've been deployed and what is deployed differs.
    - sanderjd 8y ago
      Lots of people do work really hard on mitigating this problem. It's a tough and constant battle, but that doesn't mean you have to throw your hands up and not bother working on it. I'm sure you're right that my bank isn't working to mitigate insider threat to the extent I'd like, but Tesla's code is more safety critical than my bank and I think it would be worth their while to work very hard on keeping this from happening with their computer-on-wheels.
- qaq 8y ago
  It would not require any chain of cooperation I highly doubt staging team would catch some coefficient used for particular industrial robot being off by 0.1% in some PR.
- justinclift 8y ago
  As a data point, in most of the ${BIGCORP}'s I've worked there are also infrastructure roles most people don't see or think about, which have access across wider environments. * Storage engineers: Generally have access to most storage (all of dev/test/prod) in their group. Sometimes their access is silo'd, sometimes not. * Backup engineers: Generally have read/write access to _everything_, and all historical versions of it, as backup systems need to be able to do both read/write. Fairly often there are ways for this access to be "unlogged" too, so the actions aren't captured into any system auditing logs (otherwise it can screw things up). I've not (yet) seen access for backup engineers ever be silo-d, but some places might be doing it and I've just never seen it. :)
- closeparen 8y ago
  You generally can't stop someone with administrative access to production from running something that didn't come from your normal process/VCS.
masklinn 8y ago
> Even with code review policies, there is still a short list of people who can push to production without going through code review. That's completely unnecessary and should not be the case. If you need something pushed quickly, you can get a colleague with review bit and get them to ack for "urgency" reasons after a quick lookover.
- greglindahl 8y ago
  I'm glad to see evidence that security theater fantasies are alive and well!
  - falsedan 8y ago
    To be fair, this is more likely to be compliance rather than security.
- vidarh 8y ago
  I've seen lots of places that think they have a process in place that prevents this. I've very rarely seen places that actually had sufficient security in place to prevent someone with malicious intent from actually finding ways of bypassing it if they were prepared to break company policies and/or the law. I'm sure they exist, and more places ought to take this seriously, but part of the problem is a lot of places think they have processes in place that ensures they're not vulnerable. Often it boils down to not taking sufficient measures against social engineering. In this case the claim is they used fake usernames - most places I've worked, successfully getting a fake account if you already work there would tend to "only" require a willingness to lie on a form or two ("fake" a contractor) and then request elevated privileges. Very few places I've done work requires sufficient checks or counter-signatures to require additional accomplices or make it harder than that. They do exist, but they're rare. The state of security most places is quite depressing at times. Then again, most of the time it's enough.
- sokoloff 8y ago
  I wrote the policy for our company (and got it through the audit and compliance processes, including SOX404 and PCI-DSS) that specifically and intentionally allows a specific group to take whatever action they determine is appropriate in the face of a production emergency, provided they declared the emergency, their intent, and documented/published what they did afterwards. I believe this policy, used only a few times per year, has saved us 8 figures in outage costs over a decade. (More than half of the benefit is from a clear statement and instilled sense of ownership, and only secondarily the defusing/unraveling of people would otherwise wait or insert tangles of “best practice” red tape while the website or a factory was hard down.) I based it on 14 CFR 91.3 (in intent) which says, in part: 91.3 Responsibility and authority of the pilot in command. (a) The pilot in command of an aircraft is directly responsible for, and is the final authority as to, the operation of that aircraft. (b) In an in-flight emergency requiring immediate action, the pilot in command may deviate from any rule of this part to the extent required to meet that emergency. (I made part c of the law, the reporting requirement, mandatory where it’s only on-demand in the aviation law.) When we explain the policy to new employees, we often cite the aviation law directly, to help clearly communicate our intent. [0] https://www.law.cornell.edu/cfr/text/14/91.3 https://www.law.cornell.edu/cfr/text/14/91.3
sanderjd 8y ago
No I understood your point. I just don't think it must be true, in the strong formulation you are using, that there must be a short list of people who can independently cause new code to run on a vehicle. I believe security and access can be set up such that no one person can accomplish that. It may be very difficult to set that up, but it seems worthwhile in this case.

falcolas 8y ago

Which indicates a different problem if the sabatuer was on that short list. And in this day and age of cryptographically signed commits, the number of people who could do this should go down even further. Given this, I think it’s much more likely there were few or bad controls, than a person in an incredibly privileged position working in the margins.

SolarNet 8y ago
Someone builds the code, it doesn't have to be committed.

dingaling 8y ago

The chain we had in ${BIGCORP}: Programmers: read-write to repository Staging team: read-only on repository, read-write to test servers and staging zone Deployment team: read-only on repository and staging zone, read-write to production It wouldn't prevent malicious code going out but at least would require a chain of cooperation between employees, which would be harder to achieve.

kbenson 8y ago
> It wouldn't prevent malicious code going out but at least would require a chain of cooperation between employees, which would be harder to achieve. No, it just requires a chain of cooperation between authorized accounts. That's a very important distinction, especially here, where the email in question alleged the following: This included making direct code changes to the Tesla Manufacturing Operating System under false usernames
cbzoiav 8y ago
Did your staging or deployment team review code? And what stops a single member of the staging or deployment team patching the build scripts, binaries or just installing their own software to a server?
tomp 8y ago
So the deployment team could write and push malicious code to production?
- XorNot 8y ago
  They pretty much always can. Sure sure you can imagine some perfect system which would mitigate it but no one - definitely not your bank - is doing that. It very much sounds like thats the case here - production code was edited, and subsequent auditing has found what should've been deployed and what is deployed differs.
  - sanderjd 8y ago
    Lots of people do work really hard on mitigating this problem. It's a tough and constant battle, but that doesn't mean you have to throw your hands up and not bother working on it. I'm sure you're right that my bank isn't working to mitigate insider threat to the extent I'd like, but Tesla's code is more safety critical than my bank and I think it would be worth their while to work very hard on keeping this from happening with their computer-on-wheels.
qaq 8y ago
It would not require any chain of cooperation I highly doubt staging team would catch some coefficient used for particular industrial robot being off by 0.1% in some PR.
justinclift 8y ago
As a data point, in most of the ${BIGCORP}'s I've worked there are also infrastructure roles most people don't see or think about, which have access across wider environments. * Storage engineers: Generally have access to most storage (all of dev/test/prod) in their group. Sometimes their access is silo'd, sometimes not. * Backup engineers: Generally have read/write access to _everything_, and all historical versions of it, as backup systems need to be able to do both read/write. Fairly often there are ways for this access to be "unlogged" too, so the actions aren't captured into any system auditing logs (otherwise it can screw things up). I've not (yet) seen access for backup engineers ever be silo-d, but some places might be doing it and I've just never seen it. :)
closeparen 8y ago
You generally can't stop someone with administrative access to production from running something that didn't come from your normal process/VCS.

masklinn 8y ago

> Even with code review policies, there is still a short list of people who can push to production without going through code review. That's completely unnecessary and should not be the case. If you need something pushed quickly, you can get a colleague with review bit and get them to ack for "urgency" reasons after a quick lookover.

greglindahl 8y ago
I'm glad to see evidence that security theater fantasies are alive and well!
- falsedan 8y ago
  To be fair, this is more likely to be compliance rather than security.
vidarh 8y ago
I've seen lots of places that think they have a process in place that prevents this. I've very rarely seen places that actually had sufficient security in place to prevent someone with malicious intent from actually finding ways of bypassing it if they were prepared to break company policies and/or the law. I'm sure they exist, and more places ought to take this seriously, but part of the problem is a lot of places think they have processes in place that ensures they're not vulnerable. Often it boils down to not taking sufficient measures against social engineering. In this case the claim is they used fake usernames - most places I've worked, successfully getting a fake account if you already work there would tend to "only" require a willingness to lie on a form or two ("fake" a contractor) and then request elevated privileges. Very few places I've done work requires sufficient checks or counter-signatures to require additional accomplices or make it harder than that. They do exist, but they're rare. The state of security most places is quite depressing at times. Then again, most of the time it's enough.
sokoloff 8y ago
I wrote the policy for our company (and got it through the audit and compliance processes, including SOX404 and PCI-DSS) that specifically and intentionally allows a specific group to take whatever action they determine is appropriate in the face of a production emergency, provided they declared the emergency, their intent, and documented/published what they did afterwards. I believe this policy, used only a few times per year, has saved us 8 figures in outage costs over a decade. (More than half of the benefit is from a clear statement and instilled sense of ownership, and only secondarily the defusing/unraveling of people would otherwise wait or insert tangles of “best practice” red tape while the website or a factory was hard down.) I based it on 14 CFR 91.3 (in intent) which says, in part: 91.3 Responsibility and authority of the pilot in command. (a) The pilot in command of an aircraft is directly responsible for, and is the final authority as to, the operation of that aircraft. (b) In an in-flight emergency requiring immediate action, the pilot in command may deviate from any rule of this part to the extent required to meet that emergency. (I made part c of the law, the reporting requirement, mandatory where it’s only on-demand in the aviation law.) When we explain the policy to new employees, we often cite the aviation law directly, to help clearly communicate our intent. [0] https://www.law.cornell.edu/cfr/text/14/91.3 https://www.law.cornell.edu/cfr/text/14/91.3

sanderjd 8y ago

No I understood your point. I just don't think it must be true, in the strong formulation you are using, that there must be a short list of people who can independently cause new code to run on a vehicle. I believe security and access can be set up such that no one person can accomplish that. It may be very difficult to set that up, but it seems worthwhile in this case.