18 ms·
Dieselgate, but for trains – some heavyweight hardware hacking
- GruHe 3y agoA train manufactured by a Polish company suddenly broke down during maintenance. The experts were helpless – the train was fine, it just wouldn’t run. In a desperate last gasp, the Dragon Sector team was called in to help, and its members found wonders the train engineers had never dreamed of.
- ale42 3y agoSee also the discussion/comments of the previous post of a related article: https://news.ycombinator.com/item?id=38530885 https://news.ycombinator.com/item?id=38530885
- greesil 3y agoBut was this sabotage by an insider at the manufacturer, or something deliberate by the manufacturer?
- HPsquared 3y agoIf the manufacturer did it, doesn't it still fit the definition? It's something like "deliberately causing something to fail", regardless of who does it.
- mannykannot 3y agoThe higher-up the decision went, the worse it is.
- 93po 3y agoWhile I believe intentions were malicious, it's very easy to argue that 1. it's not failing, it's disabling 2. it's a safety feature - "SPS can't safely maintain these trains, so we have a safety lock out if they attempt it" 3. there is a ton of stuff that works this way - even Harley Davidson motorcycles require authorized maintenance and the bike's computer won't accept repairs unless a proprietary tool is used
- yetihehe 3y agoNewag was required by contract to provide accurate service manuals so that competitors could safely maintain the trains. This was not a "just take your car to dave, he knows some stuff". For SPS and other competitors this was like "you need to show every certification that exists and certify all your tools to prove that indeed you can service those cars, or you will be foreclosed due to fines". Plus, they were provided ALL service manuals, like 20k pages to follow to the letter.
- 93po 3y agoi wouldnt be surprised if this info was somewhere in those 20k pages, and perhaps if the procedures were actually followed, stuff like GPS based lockouts wouldn't happen
- rcxdude 3y agoAh yes, on appendix 35 of section C, "do not store the train in your service yard specifically or it will stop working"
- metadat 3y agoThe article covers this, and says the information about the lockouts was not in the manufacturer provided manuals.
- 93po 3y agoAccording to who?
- metadat 3y agoDirectly in TFA, matey: > Newag explains that the train were > blocked by a “safety system” – but in > the 20,000 pages of instructions, it > is in vain to find even a mention > of it. No mention whatsoever in the maintenance documents. It then becomes prudent to question the intentions and fitness of the company behind such a product. This episode puts even John Deere to shame. I'm imagine JD are enjoying themselves right now on this Friday afternoon.
- korhojoa 3y agoThere wasn't just one manufactured failure, but multiple different ones. Refusing to help would also point towards intentional malice. Why would you sell a product, then refuse to assist, unless you've intentionally designed the product to fail so only you would know how to make it work again?
- greesil 3y agoTo what end? So they can sell more trains? That makes no sense.
- snthd 3y ago>The train manufacturer, Newag, also competed in the tender to carry out the maintenance, but the manufacturer’s bid was about 750k USD higher and the tender was eventually won by SPS, which offered to carry out the maintenance of 11 trains for around 5.5 mln USD.
- Crosseye_Jack 3y agoJust thinking outloud. But if you made it so your competitor couldn't fulfill their servicing contract, then the entity taking out the contract might just very well come to you to solve the problem. You might not win the contract on price, but win it by default because you made it impossible for anyone else to complete it. That is until your scheme is uncovered because you left the GPS coordinates of your competitors workshops in your code.
- NoMoreNicksLeft 3y agoConsidering that the "sabotage" was intended to bring the company extra revenue by having non-faulty parts replaced and by requiring maintenance to be carried out by them and never third parties, it aligns with the company's own interests too much so to say "some employee did this without authorization". "Deliberate by manufacturer" 100%. The scarier part is that had this happened in the United States, DMCA would likely have protected them from prosecution, and the government might be liable for damages.
- port515 3y ago[flagged]
- Svip 3y agoNixon never went to jail.
- CTDOCodebases 3y agoThe programmer appears to be programmed.
- formerly_proven 3y agoDieselgate isn't a good comparison because in Dieselgate the equipment functioned normally from the user's point of view.
- sgt101 3y agoI think it is because the equipment changed dependent on context. In Dieselgate the cars changed their engine management when they got into a test cycle...
- yetihehe 3y agoDieselgate was about cheating environment sensors. This is more like DeereGate, locking out external service shops but even when you are supposed by law to allow them service (and even after providing them 20k page service manuals which they are supposed to follow to make appropriate service, but you lock them out anyway).
- quickthrower2 3y agoThat Watergate building name had created quite the meme!
- ZeroGravitas 3y agoFrom my reading, they also seem to have seeded apparently random failures into the product, with a hidden reset key combo, even for those using them as support. Possibly to make themselves look good (our products may break randomly, but at least we fixed the "problem" quickly) going into the tendering process for support.
- deleted 3y ago[deleted]
- bauble 3y agoI thought it was a good reference. In both cases, the manufacturer placed illicit, hidden code that that could (and probably should) get it in trouble with the law.
- codewiz 3y agoDupe: https://news.ycombinator.com/item?id=38558120 https://news.ycombinator.com/item?id=38558120
- HackerThemAll 3y agoIn a properly functioning country the responsible persons should already be imprisoned. Some governmental agencies were aware of that for at least half a year, but failed to act. The fact that source code was not immediately dumped and analyzed is the evidence of malevolence, corruption and intentionally putting people's lives at risk. Welcome to the dark side of Poland - where citizens don't matter.
- yard2010 3y agoCorruption is not just the dark side of Poland, but the entire west IMHO
- xbar 3y agoThe other 3 directions are corruption-free?
- gattilorenz 3y agocan't speak for the other continents but there is very little corruption in Antarctica, so I guess if you go South enough, it is actually better
- konschubert 3y agoThe west is the least corrupt part of the world. https://en.wikipedia.org/wiki/Corruption_Perceptions_Index#/media/File:Countries_by_Corruption_Perceptions_Index_(2022).svg https://en.wikipedia.org/wiki/Corruption_Perceptions_Index#/...
- lqet 3y agoMy impression is that the quality of train firmware is generally not very good, and I hope that this scandal will lead to greater scrutiny. 3 years ago, Deutsche Bahn publicly complained of "grotesque" software problems with newly delivered Bombardier trains. For example, when train drivers changed the direction of travel, the train software would crash. It then took 1 hour to boot the train up again [0]. Switzerland had similar problems in 2018 [1]. As a computer scientist, I find this embarrassing. Just compare these modern trains to the old trains built in East Germany [2] during the 80ies that were pulling old West German carriages [3] from the 50ies here until recently. Minimal or no usage of digital electronics. No "boot times". They just worked. And if they didn't, the train driver usually knew where to hit the engine with a hammer to fix it. You cannot expect a train driver to hack into the train firmware and fire up gdb to find out why it doesn't move. [0] https://www.sueddeutsche.de/wirtschaft/deutsche-bahn-ic-1.4773380 https://www.sueddeutsche.de/wirtschaft/deutsche-bahn-ic-1.47... [1] https://bahnblogstelle.com/33872/twindexx-swiss-express-software-probleme-bremsen-neue-sbb-zuege-bei-testfahrten-aus/ https://bahnblogstelle.com/33872/twindexx-swiss-express-soft... [2] https://de.wikipedia.org/wiki/DR-Baureihe_243 https://de.wikipedia.org/wiki/DR-Baureihe_243 [3] https://de.wikipedia.org/wiki/N-Wagen https://de.wikipedia.org/wiki/N-Wagen
- twisteriffic 3y agoBad software is a symptom, not the cause.
- martijnvds 3y agoWhat is the cause then?
- ZeroGravitas 3y agoGreat advert for free and open source software. As with dieselgate, this suggests you basically cannot trust anything containing software. Can't trust it to follow regulations. Can't trust it to do its job. Can't trust the software. Can't trust the institutions that write the software. All very "late stage capitalist software development".
- kibwen 3y agoHell, even if governments are squeamish about requiring code to be fully open and public, they can still require the manufacturers to privately submit to the government all code that powers public infrastructure (like trains), to be made available to any relevant party upon request.
- verisimi 3y ago[flagged]
- tw04 3y agoBut there’s an end game to batteries given they have a decade+ useful life, and most can be recycled and reused. You belch out the diesel and your only option is to belch out more.
- RA2lover 3y agoTFA is talking about train software that deliberately renders it nonfunctional when it detects it is being serviced by a third party. This isn't about environmental concerns.
- danw1979 3y agoThe article has nothing to do with pollution. The reference to Dieselgate might be because a geofencing condition was found in the code running the trains that caused certain misbehaviour when they were at the manufacturer’s competitor’s maintenance depots.
- terminous 3y agoIt is clear you haven't even clicked on the article. Shame on you. This deserves a ban from HN. Just reading the headlines and contributing to a discussion about that article as if you actually read it is deceptive and wrong. It's killing our ability to meaningfully discuss content when people just read the headlines. If you don't want to read articles and instead post about what you think the articles must be based on headlines, go elsewhere on the internet.
- cedilla 3y agoIt's one thing to implement a secret handshake and underdocument some procedures to make your competitors look incompetent, but actively breaking your product when it's in your competitor's shop - that reqires some chutzpah.
- amelius 3y agohttps://news.ycombinator.com/item?id=38345858 https://news.ycombinator.com/item?id=38345858
- sertbdfgbnfgsd 3y agoExactly what I immediately thought of as well.
- chronicsonic 3y agoThat wasn’t intentional though.
- gunapologist99 3y agoTotally unintentional, I'm sure.
- mavamaarten 3y agoOoooops! Somebody put this delay here tooooootally by accident and nobody noticed it when shipping to production! Silly devs!
- sertbdfgbnfgsd 3y agoExactly, like that time they slowed down only chrome and every other browser was still fast. Oh wait that never ever happened.
- Piskvorrr 3y ago*provable, you mean
- ow83jtow3j 3y ago
- snvzz 3y agoNevermind malware, not using seL4 should already be a crime in this context.
- aneutron 3y agoWe have rovers on Mars and satellites and probably nuclear warheads using RTOS of all kinds and in cases even Linux, but sure seL4 is the only OS conceivable for those cases, obviously ! This is a case of fraud, industrial malfeasance and just plain dishonesty. The software component of the story and its security measures are not even at play. Sure they are probably shit (given the date parsing ...) but even FreeRTOS would make an amazing OS *IF USED PROPERLY *.
- snvzz 3y ago>using RTOS of all kinds and in cases even Linux Absolutely, and thus there's obvious room for improvement. >This is a case of fraud, industrial malfeasance and just plain dishonesty. In practice, this amounts to critical infrastructure sabotage, which fits into terrorism. If the train network experiences issues, the whole country is impacted.
- yjftsjthsd-h 3y agoThat seems totally orthogonal; seL4 can run a program that checks GPS and sabotages the system just like anything else.
- deleted 3y ago[deleted]
- DrNosferatu 3y agoSeems like deliberate sabotage via software to force the costumer to buy the manufacturer’s services instead of 3rd party (cheaper) ones. Curious to see the court’s decision.
- klabb3 3y agoThere’s no question that it’s sabotage. The only thing left to prove is the culprit, which is with 99% the manufacturer (motive, means, opportunity) but obviously need to be established in a court who is responsible and criminally culpable. The fact that lawmakers, courts and the public are lost in the tech is a problem, but surely this crime can be fitted into existing criminal code against sabotage… although the methods are “new” the crime itself is classic.
- dexterdog 3y ago"Lawlessness is the condition in which your adversary refers you to a law he made."
- jancsika 3y agoIt's almost like you found a lossy encoding for discussing governance. For example, both the NYC taxi medallion system and warlords controlling a city in a failed state would get input as "lawlessness" in your encoding. But then if I ask what is the quality of life in each instance, I can't get that answer because there aren't bits in your encoding for that.
- dzdt 3y agoIts insane how brazen this is. Code that 'bricks' the train locomotive if its gps coordinates remain with bounds of a competing repair facility for more than ten days! This is way beyond putting information barriers to repair, like undocumented interfaces or even crypto-signed firmware. This is actively malicious destruction of property. I don't know anything about the legal system in Poland, but I can't imagine how this gets by.
- toomuchtodo 3y agoIf an individual did this, they'd go to prison.
- throwbadubadu 3y agoAnd if it is a big or even state company we need to save and ensure workplaces, or "hello dear lobbyist with that big suitcase!" :D
- mcv 3y agoCompanies are made up of individuals. I'm all for holding everybody who contributed to this malware accountable.
- praptak 3y agoI don't believe the Polish judicial systems has experience in dealing with corporate crime, especially of the tech-related kind. I'm a bit afraid of disappointment here.
- Piskvorrr 3y agoDoes it have experience in dealing with...sabotage? Specifically, a country that has a war on its eastern doorstep? I mean - how is "let's mess with something on purpose so that trains won't run" NOT sabotage, since such time as railways exist?
- 3y ago
- Tade0 3y agoNewag issued a statement since, denying all allegations and saying that it was their competition which "hired hackers to slander them". I've met q3k because we used to work at the same company and briefly on a project together. Not the kind of person I would suspect of participating in a conspiracy of this sort and Newag's statement generally reads like "we didn't think we would get caught".
- deleted 3y ago[deleted]
- aneutron 3y agoWhile I haven't met the guys in this case, I am familiar with their work. Additionally, I am fairly certain they are not stupid enough to not have kept detailed, forensic-quality records of their actions and whatever they dumped. Sure it may not stand up in court as evidence but it will be more than enough to show that they didn't pull this out of nowhere
- senkora 3y ago^I think this is being downvoted because of poor reading comprehension skills. Please note that the parent comment is in favor of the hacking group.
- Tade0 3y agoThank you for pointing this out - I reread the post and can imagine now how someone would read it differently than I intended.
- TomaszZielinski 3y agoI read and reread it and IMHO it's perfectly clear.
- tensility 3y agoUnfortunately for Newag, other than in the court of public opinion, firmware deliveries count as written evidence.
- RecycledEle 3y agoI wonder if the solution to all these screwy engine controls (tampering with emissions testing, preventing 3rd party repairs, etc.) is to standardize the interfaces to these systems so they can be replaced. Standardizing the outputs of the sensors would let us swap in and out various components to ensure the system is not cheating the regulators.
- marcosdumay 3y agoIt's a bit more than standardizing, since you must also remove the barriers to changing the software. And you don't need full standardization, just publicity. But yes, it's basically it.
- magicalhippo 3y agoF1 does this. All teams are required to run the same, approved, ECU[1]. They can change certain mapping tables and such but it's a sealed unit and they can't replace the firmware. [1]: https://wheelsports.co/formula-1s-standardised-ecu-explained/ https://wheelsports.co/formula-1s-standardised-ecu-explained...
- webel0 3y agoThis is quite interesting because you can imagine that lobbyists would argue that standardization would “stymie innovation.” If F1 does it why can’t you?
- jtriangle 3y agoOne could easily argue that F1 hasn't innovated much in the last decade or so. The coolest stuff we get is clever aero and advantageous workarounds that get outlawed extremely fast.
- mrguyorama 3y agoYou are extremely downplaying the "clever" aero. Remember, simulation time and costs are regulated in F1. The innovation is producing extremely effective aero with minimal brute force simulation.
- aizyuval 3y agoI'm solely consuming EN content. And if it's from another country, it's only whats leaked by big media. It make me wonder how much good content could be translated.
- gunapologist99 3y agoI think you're saying, how much other good content is out there that I'm missing out on because I only read English, and it's a good point. However, English has become the (now ironically named) lingua franca of, at least, the more educated parts of the world, and many people who are most comfortable in their native languages are still often translating their best work into English in order to see it more widely read. This is often the case with scientific papers, for example. Perhaps England's biggest gift to the world was its language.
- tensility 3y agoWorldwide colonialism wasn't exactly a "gift", but I must admit it has been advantageous to me, personally, for English to be as relatively universal as it has become as a result. ;-)
- ffgjgf1 3y ago> (now ironically named) lingua franca The original language which was actually called wasn’t really French it was a creole/pidgin language used in the Mediterranean mainly based on Italian and Occitan dialects. Greeks and others just called all Western European Franks even though they didn’t really interact with people who actually spoke French (only used in the Northern half of modern France back then) that much.
- TomaszZielinski 3y agoI'm from Poland, and there's a chance I was going by those specific immobilized trains--I lived in Wrocław and used Koleje Dolnośląskie (Lower Silesia Railways) to "scout" the area (there are some beautiful places around Wrocław). And I only learnt about this whole train "vendor lock-in" from HN. Otherwise I either wouldn't know about it, or I would learn about it weeks or months later.
- wejick 3y agoThis kind of thing reminds me of 737 max debacle.
- croes 3y ago>it is hard to find an institution in Poland that has done anything beyond kindly expressing interest in the matter. We are not aware of any action taken either by the Office of Consumer and Competition Protection or by the Railway Transport Office, That the worst part of all that.
- gambiting 3y agoThe government anti-corruption office is formally investigating this now, which means almost certainly people will end up going to jail. The office of consumer protection doesn't have anywhere near the power these guys have.
- droopyEyelids 3y agoWhen the companies see that this behavior is not punished, they'll basically need to implement their own versions of it to stay competitive!
- voakbasda 3y agoIt’s cute that you think they haven’t done that already…
- p0w3n3d 3y agoTitle is a bit misleading, because this *gate is not about faking ecology, and trying to pass certification in artificial conditions, as dieselgate was, but simulating fake failures instead. The company hardcoded algorithms that would report failures of parts that work correctly (like a compressor), if it detected that train has been repaired by another company (based on location readings), and stop the train from running
- nightpool 3y agoIt's an example of fraudulent / malicious behavior found by decompiling industrial logic controllers, with incontrovertible evidence of illegality. Obviously no two situations are ever going to be the exact same, but I think it's clear why the analogy was made.
- samtho 3y agoYes, but “Dieselgate” is not appropriate here because that term has “cheating” loaded onto it, which represents a different struggle for companies than vendor lock-in. What this company is doing is related to DRM and arguably closer to what John Deere does with its products.
- fcsp 3y ago> The company hardcoded algorithms that would report failures of parts that work correctly (like a compressor), if it detected that train has been repaired by another company (based on location readings), and stop the train from running This isn't correct by my understanding - there's actually two separate things here: - The company made their trains stop functioning after spending 10 days at competing maintenance locations, based on GPS - In one firmware, they hardcoded to pretend a compressor failure a few days after the next scheduled maintenance for the train
- h1fra 3y agoOutside the obvious issue in this article I found the following statement horrendous: > it has to be taken apart, the parts sent to the various manufacturers, checked, sent back, the train put back together again and tested Instead of having one public company mastering the art in its entirety everything is split with contractors. A good example of a successful way to do that (but slowly dying thanks to capitalism) is SNCF operating everything in a massive warehouse https://www.youtube.com/watch?v=SeRH2M2Z-ms https://www.youtube.com/watch?v=SeRH2M2Z-ms
- freedomben 3y agoHow is this different from companies like Apple or John Deere that DRM components and brick the device if repaired by "unauthorized" technicians? (I think both are equally egregious personally, but I know there's a lot of support here for Apple, so I'm curious how people reconcile these. I don't want to make this a religious war about Apple, but those practices in general regardless of which company is doing it). Is it the secrecy that makes it different? i.e. if the train company were honest about it then it would be ok? Or is it the scale that matters? Trains are big and expensive, while phones are small and cheap, so it's ok? (that wouldn't work for John Deere but would for Apple)
- cstross 3y agoAn angle you may not have considered is passenger safety. Imagine if this happened to an airliner in flight: there'd be criminal charges for sure, not to mention huge damages and lawsuits from the families of the dead if some of the control systems locked up in mid-air. Trains are not quite as susceptible to disaster arising in the course of operations as airliners, but a Newag Impuls 45WE runs at up to 160km/h in service with up to 218 people on board. (Their speed record is considerably higher.) A sudden breakdown in service is at a minimum going to cause timetable havoc and knock-on delays for other trains and at worse could lead to a mass casualty accident. (John Deere tractors don't usually carry 200+ passengers and Apple computers don't usually get deployed in safety critical situations. So, different!)
- TomaszZielinski 3y agoThank you for putting it so clearly! I feel this angle got somewhat lost in the whole discussion. For instance, has anyone really analyzed, what would happened if the trains were running at 160km/h at midnight, on Nov 21? I mean, they might have, I don't know. But that would be pretty funny--having documentation for what's very likely a tort (well, it's Polish counterpart), if not an outright criminal act.
- johncalvinyoung 3y agoI'm not a fan of Apple's practices, but there's some aggravating elements to this. Apple doesn't brick your device if it it spends time at a repair location, for instance. Apple also doesn't simulate failures on synthetic dates to force repair.
- dang 3y agoRecent and related: Polish trains lock up when serviced in third-party workshops - https://news.ycombinator.com/item?id=38530885 https://news.ycombinator.com/item?id=38530885 - Dec 2023 (347 comments)
- codewiz 3y agoUPDATE: Polish train maker denies its software bricked trains maintained by competitor: https://news.ycombinator.com/item?id=38579098 https://news.ycombinator.com/item?id=38579098