11 ms·

This is a very naive comment. There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live

This is a very naive comment. There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. Ideally, with mature controls, the people in this list is short. But to jump to the conclusion that Tesla doesn't use good practises is very short sighted. Who's to say that external parties didn't target this person specifically because of their role/influence.

sanderjd 8y ago
> There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. There is really no reason for this to be the case. Certainly all code that actually runs on the car can be required to go through review and be verifiably built, even if server code standards are more lax.
- nickparker 8y ago
  The code in question is for their manufacturing systems, not the cars. Not that that's necessarily better... Manufacturing equipment's at about the same danger tier as cars.
  - flyingswift 8y ago
    True. It's possible that the manufacturing software could be modified in some manner to introduce some fundamental flaw in the final product though. For that reason, I would say the code should also be held to a higher set of standards
- tehwebguy 8y ago
  > This included making direct code changes to the Tesla Manufacturing Operating System under false usernames If they found a way to use more than one username they may very well have run it through the review process and approved it themselves
- SolarNet 8y ago
  I think you misunderstand. Even with code review policies, there is still a short list of people who can push to production without going through code review. Not from a policy standpoint but from a security and access perspective.
  - falcolas 8y ago
    Which indicates a different problem if the sabatuer was on that short list. And in this day and age of cryptographically signed commits, the number of people who could do this should go down even further. Given this, I think it’s much more likely there were few or bad controls, than a person in an incredibly privileged position working in the margins.
    - SolarNet 8y ago
      Someone builds the code, it doesn't have to be committed.
  - dingaling 8y ago
    The chain we had in ${BIGCORP}: Programmers: read-write to repository Staging team: read-only on repository, read-write to test servers and staging zone Deployment team: read-only on repository and staging zone, read-write to production It wouldn't prevent malicious code going out but at least would require a chain of cooperation between employees, which would be harder to achieve.
    - kbenson 8y ago
      > It wouldn't prevent malicious code going out but at least would require a chain of cooperation between employees, which would be harder to achieve. No, it just requires a chain of cooperation between authorized accounts. That's a very important distinction, especially here, where the email in question alleged the following: This included making direct code changes to the Tesla Manufacturing Operating System under false usernames
    - cbzoiav 8y ago
      Did your staging or deployment team review code? And what stops a single member of the staging or deployment team patching the build scripts, binaries or just installing their own software to a server?
asfasgasg 8y ago
It seems far more likely that the controls were lax than that the saboteur was one of those privileged few. It's not a naive comment.
EGreg 8y ago
Why is it naive to expect there not to be a single point of failure like that? Code reviews are a thing.
- noobermin 8y ago
  What if they pushed it without asking anyone?
  - davidgay 8y ago
    Then the system that allowed them to do so is badly designed.
    - noobermin 8y ago
      I guess my thinking is if the disgruntled employee was so upset over not getting a promotion to cause so much damage they must have been in a high enough position already to warrant such a disposition, and "high enough" might mean high enough to have push access themselves. Edit: see closeparens comment above. Complicated systems can always be subverted when trust is broken.
    - chillee 8y ago
      At Facebook, you could alter code even after somebody had given the OK for code review. I know some people specifically kept some small commits open after being approved, just so they could quickly make changes without needing approval if they ever needed to.
      - sneak 8y ago
        “Dear respected members of Congress, ...”
      - skj 8y ago
        At Google you can mark something as TBR (to be reviewed), as long as you're one of the designated owners for that code.
xiphias 8y ago
At Google binaries, packages and code are digitally signed by the logged in user and computer
alxlaz 8y ago
Obviously, not all details are available, but the wording in the email suggests that the parent comment is anything but naive: > This included making direct code changes to the Tesla Manufacturing Operating System under false usernames and exporting large amounts of highly sensitive Tesla data to unknown third parties. This sounds like something out of the 1990s, that dark and romantic era of version control when we thought CVS was pretty cool actually and we didn't know what key-based authentication and 2FA were. There are volunteer-ran projects that don't have this problem. Edit: to be clear, I presume no one is debating the fact that someone with high enough credentials can push code to production. The questions that the email raises are: 1. Why can anyone, regardless of credentials, push mission-critical code without review (or, alternatively, if the changes did go through review, why did the review process not catch multiple malicious changes?) 2. Why can someone compromise several high-level credentials without anyone figuring it out (the changes were made, apparently, under "false usernames")?
- alanfranzoni 8y ago
  > 1. Why can anyone, regardless of credentials, push mission-critical code without review (or, alternatively, if the changes did go through review, why did the review process not catch multiple malicious changes?) Why do you suppose the unauthorized party was following the company's development practices? Maybe it was from the sysadmin side, somebody who worked on the toolchain used for reviewing and pushing things to production. So he was able to sidestep the normal review process. This can happen, what is important is that such things are discovered.
  - alxlaz 8y ago
    > So he was able to sidestep the normal review process. He should not have been able to sidestep the normal review process. That's the problem in the first place. Even if you're from the sysadmin side. It should not be possible to do it. You may think that looks exaggerated but I've worked in two places where we implemented such a process, both of them far more boring than Tesla and, I suspect, far less money to burn on infrastructure. > This can happen, what is important is that such things are discovered. No, what is important when working with mission-critical code is that such things are mitigated. Discovering such a problem in production code is already a problem, not a solution.
_Codemonkeyism 8y ago
"There will always be a small handful of engineers that can push the button to move code into PROD" I have very different experiences from a SEC regulated company. With SOX there are controls to prevent such a thing to happen. If this is a SOX breakage, Tesla is in deep trouble with the SEC.
- gnode 8y ago
  Can you explain how Sarbanes-Oxley applies to the situation of an employee sabotaging a production line? Specifically how it would inherently imply wrongdoing on Tesla's part.
  - CaptainZapp 8y ago
    In the financial industry part of SOX is segregation of duty. As a developer I'm not allowed to have write access to any production system, except in an emergency via a break-glass mechanism, which is audited to the hilt and back. It also means we're not allowed to deploy software to production systems. This has to happen via a specific chain development > regression / user acceptance testing > production. All those environments need to be physically seperated with very specific access requirements. The deployment process needed to be signed off by outr auditors. I can't speak for other banks, but they probably need to implement the same -, or a similar system. Neither can I speak for SOX requirements regarding software fo car manufacturing.
    - _Codemonkeyism 8y ago
      Not only banks, every company listed in the US. It was the same where I've worked, and it wasn't a bank (Enron was no bank either).
    - alanfranzoni 8y ago
      > As a developer What if the employee were a sysadmin-level person that sidestepped the normal process?
      - 8y ago
cheeze 8y ago
Fully agree. In an ideal world all of this would be locked down, but you've gotta be prerty naive to assume that everything is perfectly locked down. There are areas that I couldn't push code willy nilly, namely in the security space. But I'd be willing to bet that a majority of teams at any bigN could have a single bad actor cause some damage... That's just the maturity of the industry.
EnderMB 8y ago
Occam's Razor applies here. While you're absolutely right, it is far more likely that someone was able to do this because there is a lack of security in their software engineering practices. This isn't aimed at you, but I think a lot of people are blinded by their support of Elon Musk and Tesla to acknowledge that ultimately he's one man, and Tesla are just a company that makes cars. People and companies are fallable.
rorykoehler 8y ago
A good system will ensure that others are notified when this happens. For critical stuff you could also require 2 keys to push code to production so no one individual can do it alone.
finnthehuman 8y ago
>There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. What? Why? Nobody on my development team has access to the production code signing keys. And nobody - at all - has the ability to remotely make a production system take an unsigned update.
- rndgermandude 8y ago
  But somebody has access to the signing keys/signing process. And somebody has access to the production machines. Etc.
  - finnthehuman 8y ago
    Yes, but that changes the scenario from a "small handful of engineers" that can all do it unilaterally, to needing at least one person from N different teams to collaborate. And in my specific case, the group with the singing keys is also the group paid to tell us "no" whenever a release is blocked by process reasons.
    - rndgermandude 8y ago
      My guess is that most people with access to signing keys or prod environments would have enough skills to code in some sabotage bugs before deployment or siphon off some data, so a lone devops person with (physical) access could probably a lot of harm just by themself.
      - finnthehuman 8y ago
        Sure, I’m just saying our commitment to process is strong enough that the technical systems are a funnel into following a reasonable process. I got dragged into defending my technical solution, but my point was that if the ability isn’t needed, don’t have it. You can break the glass when when you need to. Good process make deviation from it more visible. Our code signing keys belong to a team we already need wet ink signatures from to release software. I can go into the biohazard labs with shorts on easier than I can leverage our technical disaster recovery. I’ve only ever had to do the latter.
wslh 8y ago
In addition, it would be easy to inject obfuscated bugs that are extremely difficult to find.
havetocharge 8y ago
I think it's your comment that's naive. Mature organizations have strict separation of duties, and a great deal of oversight over code reviews and code deployment. It is becoming obvious that Tesla's focus is on execution speed and other aspects (in this case internal security) are suffering.
- manigandham 8y ago
  You mean all the mature organizations that have security breaches announced every day? Most companies do not have good security. Even when they do, it's hard to get it right, especially for internal attacks. Don't underestimate what a single individual can do when they're already inside and well-informed.
  - yread 8y ago
    To announce a breach you first have to find it. It seems Tesla took quite some time to find this one.
toddh 8y ago
You're describing how websites work, but that's not how embedded systems work, especially those were lives are it risk. If this happened then it can only be because good practices were not followed.
HeyLaughingBoy 8y ago
This is a very naive comment No, not for a company doing what Tesla does. In the environment I used to work in, that would have been flat out impossible. No change would have been allowed that did not follow process unless an explicit and documented exception was made. Even if you did manage to commit code to the release trunk without review, every single change to the codebase was checked before the release process started in order to verify that the right process was followed. If we ever found a change that no one could find paperwork for, it would be reverted. So, yeah, it's possible if you want to do it badly enough.

sanderjd 8y ago

> There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. There is really no reason for this to be the case. Certainly all code that actually runs on the car can be required to go through review and be verifiably built, even if server code standards are more lax.

nickparker 8y ago
The code in question is for their manufacturing systems, not the cars. Not that that's necessarily better... Manufacturing equipment's at about the same danger tier as cars.
- flyingswift 8y ago
  True. It's possible that the manufacturing software could be modified in some manner to introduce some fundamental flaw in the final product though. For that reason, I would say the code should also be held to a higher set of standards
tehwebguy 8y ago
> This included making direct code changes to the Tesla Manufacturing Operating System under false usernames If they found a way to use more than one username they may very well have run it through the review process and approved it themselves
SolarNet 8y ago
I think you misunderstand. Even with code review policies, there is still a short list of people who can push to production without going through code review. Not from a policy standpoint but from a security and access perspective.
- falcolas 8y ago
  Which indicates a different problem if the sabatuer was on that short list. And in this day and age of cryptographically signed commits, the number of people who could do this should go down even further. Given this, I think it’s much more likely there were few or bad controls, than a person in an incredibly privileged position working in the margins.
  - SolarNet 8y ago
    Someone builds the code, it doesn't have to be committed.
- dingaling 8y ago
  The chain we had in ${BIGCORP}: Programmers: read-write to repository Staging team: read-only on repository, read-write to test servers and staging zone Deployment team: read-only on repository and staging zone, read-write to production It wouldn't prevent malicious code going out but at least would require a chain of cooperation between employees, which would be harder to achieve.
  - kbenson 8y ago
    > It wouldn't prevent malicious code going out but at least would require a chain of cooperation between employees, which would be harder to achieve. No, it just requires a chain of cooperation between authorized accounts. That's a very important distinction, especially here, where the email in question alleged the following: This included making direct code changes to the Tesla Manufacturing Operating System under false usernames
  - cbzoiav 8y ago
    Did your staging or deployment team review code? And what stops a single member of the staging or deployment team patching the build scripts, binaries or just installing their own software to a server?

asfasgasg 8y ago

It seems far more likely that the controls were lax than that the saboteur was one of those privileged few. It's not a naive comment.

EGreg 8y ago

Why is it naive to expect there not to be a single point of failure like that? Code reviews are a thing.

noobermin 8y ago
What if they pushed it without asking anyone?
- davidgay 8y ago
  Then the system that allowed them to do so is badly designed.
  - noobermin 8y ago
    I guess my thinking is if the disgruntled employee was so upset over not getting a promotion to cause so much damage they must have been in a high enough position already to warrant such a disposition, and "high enough" might mean high enough to have push access themselves. Edit: see closeparens comment above. Complicated systems can always be subverted when trust is broken.
  - chillee 8y ago
    At Facebook, you could alter code even after somebody had given the OK for code review. I know some people specifically kept some small commits open after being approved, just so they could quickly make changes without needing approval if they ever needed to.
    - sneak 8y ago
      “Dear respected members of Congress, ...”
    - skj 8y ago
      At Google you can mark something as TBR (to be reviewed), as long as you're one of the designated owners for that code.

xiphias 8y ago

At Google binaries, packages and code are digitally signed by the logged in user and computer

alxlaz 8y ago

Obviously, not all details are available, but the wording in the email suggests that the parent comment is anything but naive: > This included making direct code changes to the Tesla Manufacturing Operating System under false usernames and exporting large amounts of highly sensitive Tesla data to unknown third parties. This sounds like something out of the 1990s, that dark and romantic era of version control when we thought CVS was pretty cool actually and we didn't know what key-based authentication and 2FA were. There are volunteer-ran projects that don't have this problem. Edit: to be clear, I presume no one is debating the fact that someone with high enough credentials can push code to production. The questions that the email raises are: 1. Why can anyone, regardless of credentials, push mission-critical code without review (or, alternatively, if the changes did go through review, why did the review process not catch multiple malicious changes?) 2. Why can someone compromise several high-level credentials without anyone figuring it out (the changes were made, apparently, under "false usernames")?

alanfranzoni 8y ago
> 1. Why can anyone, regardless of credentials, push mission-critical code without review (or, alternatively, if the changes did go through review, why did the review process not catch multiple malicious changes?) Why do you suppose the unauthorized party was following the company's development practices? Maybe it was from the sysadmin side, somebody who worked on the toolchain used for reviewing and pushing things to production. So he was able to sidestep the normal review process. This can happen, what is important is that such things are discovered.
- alxlaz 8y ago
  > So he was able to sidestep the normal review process. He should not have been able to sidestep the normal review process. That's the problem in the first place. Even if you're from the sysadmin side. It should not be possible to do it. You may think that looks exaggerated but I've worked in two places where we implemented such a process, both of them far more boring than Tesla and, I suspect, far less money to burn on infrastructure. > This can happen, what is important is that such things are discovered. No, what is important when working with mission-critical code is that such things are mitigated. Discovering such a problem in production code is already a problem, not a solution.

_Codemonkeyism 8y ago

"There will always be a small handful of engineers that can push the button to move code into PROD" I have very different experiences from a SEC regulated company. With SOX there are controls to prevent such a thing to happen. If this is a SOX breakage, Tesla is in deep trouble with the SEC.

gnode 8y ago
Can you explain how Sarbanes-Oxley applies to the situation of an employee sabotaging a production line? Specifically how it would inherently imply wrongdoing on Tesla's part.
- CaptainZapp 8y ago
  In the financial industry part of SOX is segregation of duty. As a developer I'm not allowed to have write access to any production system, except in an emergency via a break-glass mechanism, which is audited to the hilt and back. It also means we're not allowed to deploy software to production systems. This has to happen via a specific chain development > regression / user acceptance testing > production. All those environments need to be physically seperated with very specific access requirements. The deployment process needed to be signed off by outr auditors. I can't speak for other banks, but they probably need to implement the same -, or a similar system. Neither can I speak for SOX requirements regarding software fo car manufacturing.
  - _Codemonkeyism 8y ago
    Not only banks, every company listed in the US. It was the same where I've worked, and it wasn't a bank (Enron was no bank either).
  - alanfranzoni 8y ago
    > As a developer What if the employee were a sysadmin-level person that sidestepped the normal process?
    - 8y ago

cheeze 8y ago

Fully agree. In an ideal world all of this would be locked down, but you've gotta be prerty naive to assume that everything is perfectly locked down. There are areas that I couldn't push code willy nilly, namely in the security space. But I'd be willing to bet that a majority of teams at any bigN could have a single bad actor cause some damage... That's just the maturity of the industry.

EnderMB 8y ago

Occam's Razor applies here. While you're absolutely right, it is far more likely that someone was able to do this because there is a lack of security in their software engineering practices. This isn't aimed at you, but I think a lot of people are blinded by their support of Elon Musk and Tesla to acknowledge that ultimately he's one man, and Tesla are just a company that makes cars. People and companies are fallable.

rorykoehler 8y ago

A good system will ensure that others are notified when this happens. For critical stuff you could also require 2 keys to push code to production so no one individual can do it alone.

finnthehuman 8y ago

>There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. What? Why? Nobody on my development team has access to the production code signing keys. And nobody - at all - has the ability to remotely make a production system take an unsigned update.

rndgermandude 8y ago
But somebody has access to the signing keys/signing process. And somebody has access to the production machines. Etc.
- finnthehuman 8y ago
  Yes, but that changes the scenario from a "small handful of engineers" that can all do it unilaterally, to needing at least one person from N different teams to collaborate. And in my specific case, the group with the singing keys is also the group paid to tell us "no" whenever a release is blocked by process reasons.
  - rndgermandude 8y ago
    My guess is that most people with access to signing keys or prod environments would have enough skills to code in some sabotage bugs before deployment or siphon off some data, so a lone devops person with (physical) access could probably a lot of harm just by themself.
    - finnthehuman 8y ago
      Sure, I’m just saying our commitment to process is strong enough that the technical systems are a funnel into following a reasonable process. I got dragged into defending my technical solution, but my point was that if the ability isn’t needed, don’t have it. You can break the glass when when you need to. Good process make deviation from it more visible. Our code signing keys belong to a team we already need wet ink signatures from to release software. I can go into the biohazard labs with shorts on easier than I can leverage our technical disaster recovery. I’ve only ever had to do the latter.

wslh 8y ago

In addition, it would be easy to inject obfuscated bugs that are extremely difficult to find.

havetocharge 8y ago

I think it's your comment that's naive. Mature organizations have strict separation of duties, and a great deal of oversight over code reviews and code deployment. It is becoming obvious that Tesla's focus is on execution speed and other aspects (in this case internal security) are suffering.

manigandham 8y ago
You mean all the mature organizations that have security breaches announced every day? Most companies do not have good security. Even when they do, it's hard to get it right, especially for internal attacks. Don't underestimate what a single individual can do when they're already inside and well-informed.
- yread 8y ago
  To announce a breach you first have to find it. It seems Tesla took quite some time to find this one.

toddh 8y ago

You're describing how websites work, but that's not how embedded systems work, especially those were lives are it risk. If this happened then it can only be because good practices were not followed.

HeyLaughingBoy 8y ago

This is a very naive comment No, not for a company doing what Tesla does. In the environment I used to work in, that would have been flat out impossible. No change would have been allowed that did not follow process unless an explicit and documented exception was made. Even if you did manage to commit code to the release trunk without review, every single change to the codebase was checked before the release process started in order to verify that the right process was followed. If we ever found a change that no one could find paperwork for, it would be reverted. So, yeah, it's possible if you want to do it badly enough.