4 ms·

It would be so easy to get away with this kind of extortion at my work. Nobody reads my code that carefully, or cares if I don't get it reviewed and just merge

It would be so easy to get away with this kind of extortion at my work. Nobody reads my code that carefully, or cares if I don't get it reviewed and just merge it. Only one other person could understand it if he tried, and he has no interest or involvement in it. It could easily look like just a bit of incompetence on my part that requires some additional consulting from me after I have moved on. That's not how I roll ... or sleep well, so my employer is in no danger from me. But there are many short-term devs who come through here, and I don't have the time to police them in detail. But conceivably an LLM could do it. It could be just another step in a build pipeline. But, when LLMs can do this well, they can also write most of the code going into the pipeline.

jonp888 3y ago
The EN50128 safety standard for the European safety critical rail software places great importance on the development process. Every change to the software has to be based on a defined requirement, and in order to validate the software you have to prevent evidence that every change was approved by a reviewer that is competent for that area of the software. The validation report contains the signature of this person. If your code passes every test, but it wasn't developed in accordance with the process, it might as well not exist. Of course I can't say how well this process was followed in NEWAG, but in theory rogue changes shouldn't be possible.
justinclift 3y ago
> But conceivably an LLM could do it. It'd be kind of funny if an LLM did that "unintentionally", and wasn't able to unlock the code it wrote... ;)
ceejayoz 3y ago
This doesn't sound like the sort of thing some rogue developer secretly slips into the codebase.
- fnimick 3y ago
  Exactly. This is a company initiative to increase company profits. It's smart business, as long as it's not illegal or the fine is insufficiently high.
  - Flammy 3y ago
    Yup that is how I read it as well. Product decision.
  - nerdbert 3y ago
    Is it smart business though? Once disclosed it provides future purchasers with a strong reason to avoid your products. Who wants to spend millions on trainsets that could become unserviceable in the event that the seller goes out of business or makes some mistake in authorizing service centres or gets into a dispute with us over another matter?
    - fnimick 3y ago
      It can be smart business if the probability of it being disclosed is low enough. Using fake numbers as an example, if you can make an extra $1 million on repairs and will suffer $100 million in fines / lost business if it becomes known, as long as the probability of it becoming known is less than 1%, it's a net positive expected value.
    - serf 3y ago
      it's just tight-rope walking at that point. If your company has sufficient leverage within the market they can get away with murder. see: John Deere
      - JAlexoid 3y ago
        Ahem... Boeing 737 MAX, which was literal murder.
  - silvestrov 3y ago
    I would guess this is also why the code was found: it's parallel construction. Somebody was told to take a closer look. Otherwise it would be very weird to have 3rd party developers disassembling firmware code. I've never heard of that happening because a train didn't want to start.
Thorrez 3y ago
But how would you profit off of it? In the case here the company profits by forcing trains to use first-party workshops.
- vidarh 3y ago
  "Last time this failed, Bob was the only one who could fix it." "Bob resigned a few months ago." "See if he is willing to do some consulting. We'll pay whatever rate he demands." I still occasionally have past employers call about things years after I left, and if I'd have been immoral enough to pull something like this, those systems could have been full of time bombs.
  - Thorrez 3y ago
    But this is about a physical train that's in a first-party repair shop. How will the people who work in the repair shop know to call you, the software developer?
  - JoshuaRogers 3y ago
    This mindset reminds me of the policies we use in the dev team at work. Any policy access that I suggest starts with the thought "If future me were to go rogue one day, how would present me stop me?"
deleted 3y ago
[deleted]
bombcar 3y ago
It's kind of amazing how blatant it was, they weren't even really trying to hide it much. Similar to the VW emissions thing; if they'd been intentional about it they could have made it look much more like a mistake.

jonp888 3y ago

The EN50128 safety standard for the European safety critical rail software places great importance on the development process. Every change to the software has to be based on a defined requirement, and in order to validate the software you have to prevent evidence that every change was approved by a reviewer that is competent for that area of the software. The validation report contains the signature of this person. If your code passes every test, but it wasn't developed in accordance with the process, it might as well not exist. Of course I can't say how well this process was followed in NEWAG, but in theory rogue changes shouldn't be possible.

justinclift 3y ago

> But conceivably an LLM could do it. It'd be kind of funny if an LLM did that "unintentionally", and wasn't able to unlock the code it wrote... ;)

ceejayoz 3y ago

This doesn't sound like the sort of thing some rogue developer secretly slips into the codebase.

fnimick 3y ago
Exactly. This is a company initiative to increase company profits. It's smart business, as long as it's not illegal or the fine is insufficiently high.
- Flammy 3y ago
  Yup that is how I read it as well. Product decision.
- nerdbert 3y ago
  Is it smart business though? Once disclosed it provides future purchasers with a strong reason to avoid your products. Who wants to spend millions on trainsets that could become unserviceable in the event that the seller goes out of business or makes some mistake in authorizing service centres or gets into a dispute with us over another matter?
  - fnimick 3y ago
    It can be smart business if the probability of it being disclosed is low enough. Using fake numbers as an example, if you can make an extra $1 million on repairs and will suffer $100 million in fines / lost business if it becomes known, as long as the probability of it becoming known is less than 1%, it's a net positive expected value.
  - serf 3y ago
    it's just tight-rope walking at that point. If your company has sufficient leverage within the market they can get away with murder. see: John Deere
    - JAlexoid 3y ago
      Ahem... Boeing 737 MAX, which was literal murder.
- silvestrov 3y ago
  I would guess this is also why the code was found: it's parallel construction. Somebody was told to take a closer look. Otherwise it would be very weird to have 3rd party developers disassembling firmware code. I've never heard of that happening because a train didn't want to start.

Thorrez 3y ago

But how would you profit off of it? In the case here the company profits by forcing trains to use first-party workshops.

vidarh 3y ago
"Last time this failed, Bob was the only one who could fix it." "Bob resigned a few months ago." "See if he is willing to do some consulting. We'll pay whatever rate he demands." I still occasionally have past employers call about things years after I left, and if I'd have been immoral enough to pull something like this, those systems could have been full of time bombs.
- Thorrez 3y ago
  But this is about a physical train that's in a first-party repair shop. How will the people who work in the repair shop know to call you, the software developer?
- JoshuaRogers 3y ago
  This mindset reminds me of the policies we use in the dev team at work. Any policy access that I suggest starts with the thought "If future me were to go rogue one day, how would present me stop me?"

deleted 3y ago

[deleted]

bombcar 3y ago

It's kind of amazing how blatant it was, they weren't even really trying to hide it much. Similar to the VW emissions thing; if they'd been intentional about it they could have made it look much more like a mistake.