7 ms·

Hey all, I worked on software for John Deere. This is a throwaway account for obvious reasons. Opinions expressed here are MY OWN. I no longer work for John Dee

Hey all, I worked on software for John Deere. This is a throwaway account for obvious reasons. Opinions expressed here are MY OWN. I no longer work for John Deere or am associated with them in any way. I was part of one of the many teams that work on this software. Specifically I was part of John Deere's ISG division also known as the Intelligent Solutions Group. The ISG division (was at the time) responsible for tying together various software built by OEM's, for building the central UI within the cabin, and for building various debugging and build tools. The team I was on, consisted of about 8 very senior engineers, and I think there were around 20 total engineers working for ISG at the time (though I saw, and knew only a handful of them). Now, when I say OEM integration, I mean suppliers and other John Deere divisions with their own teams mirroring ours. All told, I would estimate that John Deere has somewhere between 150-300 engineers working full-time on their codebase for their tractors. Let me disabuse you of any myths. I have worked in software for 20 years. I have worked in large enterprises, and scrappy startups. This software is by FAR the largest, most complex codebase I have ever interacted with. Submission of any new code was seriously considered and reviewed before it entered production (sometimes to a pedantic degree), after which JD put all new code through 10s of thousands of hours of testing on production equipment. Production and release cycles take on the order of months to ensure that we don't kill people. These are not riding lawnmowers. They are 30-ton combines, and 20 ton tractors tilling fields, with massive horsepower behind them. They have a real potential to end peoples lives in the event of failure, and these tractors do (in testing) fail in spectacular ways. If a team of hundred of engineers struggle with their codebase internally, Joe Farmer isn't going to have a fucking clue how to repair their software correctly. Now should you, in theory, have the right to modify equipment you own? Sure. Absolutely. Hell, John Deere tractors run on open source software. But trust me on this, locking this down is a very good idea. If you have the drive to make open source tractor software AND can make absolutely certain no-one ever dies from code you write, then go do it. Just keep in mind that the engineers that work on this shit really care about keeping people safe.

jdietrich 9y ago
Don't try to bullshit us. Your code is deliberately user-hostile. Farmers aren't trying to hack their tractors for the sake of it; they need to hack the parts of your code that prevent them from performing routine maintenance. Replacing a fuel pump is not going to turn a tractor into a rampaging death machine. You are using spurious safety arguments to justify profiteering. Software lock-outs are being used to prop up a business model, just like the chips in printer ink cartridges.
- bigtimeidiot 9y ago
  >Don't try to bullshit us. Us? Who is "us"? Why are you trying to shout down an opposing view? You want prefer the echo chamber? >Replacing a fuel pump is not going to turn a tractor into a rampaging death machine. Am I going to be able to replace whatever parts I want on my Model 3? Because "we" certainly love that company here!
  - tobltobs 9y ago
    Your comparison to Tesla doesn't get better just by repeating it a few times. If your Tesla breaks down you wont have any problem to get a comparable replacement car. Much luck to get a replacement for your tractor during harvest season.
    - bigtimeidiot 9y ago
      >Your comparison to Tesla doesn't get better just by repeating it a few times Your hand-waving also does precisely zero. >If your Tesla breaks down you wont have any problem to get a comparable replacement car What does this have to do with anything? I want to work on something I bought; isn't that what we're arguing about? The availability of a replacement is a straw man. Why are you apologizing for other companies that do precisely what is being described in the article?
      - RealityVoid 9y ago
        While I think you have a valid point about Tesla's availability, this is a "Tu Coque". I mean, regardless if Tesla does/does not do it, this is considered bad and bringing Tesla in the discussion does not change nor challenge the validity of this view, only, maybe, reveal a little hypocrisy or favoritism in the audience.
        dandelion_lover 9y ago
        >While I think you have a valid point about Tesla's availability, this is a "Tu Coque". I mean, regardless if Tesla does/does not do it, this is considered bad aka https://en.wikipedia.org/wiki/Whataboutism https://en.wikipedia.org/wiki/Whataboutism
tw04 9y ago
What, specifically, is a threat to kill people in the software? I'm sorry but that just reads like a John Deere shill here to defend the cause. We got by for literally two hundred years without magical software in tractors, and we were just fine.
- rimliu 9y ago
  What if we compare the number of people involved in producing the same amount of food two hundred years ago and today?
- criddell 9y ago
  These things drive themselves. There are foragers that follow other equipment autonomously.
- randomdata 9y ago
  It's not obvious how an errant piece of large, heavy, machinery could kill someone? There's a software bug in one of my tractors (not John Deere) where it won't always immediately disengage the clutch when put in gear (yes, even the transmission is fully computerized). One time during a long day in the field I accidentally left it in gear, not noticing because the tractor was at a full stop with no signs of moving. A few minutes after I got out of the cab it decided to kick in. Luckily it was in a low gear and I was able to safely get it stopped, but it's easy to see how that could have ended up tragic. While that one was ultimately my error, without that bug it wouldn't have been an issue in the first place. I'm sure you can come up with arguments as to why that software shouldn't exist – although I have to say it's wonderful when it works – but it's easy to see why bugs can be dangerous.
throwaway198411 9y ago
This coolaide youre handing out is delicious
nickpsecurity 9y ago
Hey, I do high-assurance systems too. High-assurance security where input or environment might be malicious. Even in my constraints, the systems are designed so the parts can be replaced without magic software. The big complaint about JD seems to be them using software constructs to prevent fixing tractors or common extensions. The former should not be hard as JD just has to allow products with certain specs for what will work. The second might require a combination of that with a more manual, operating mode with some automation disabled. Less complexity than what JD is putting in there. So, Im not buying it even if you worked on the team. They could design it for cheap fixes for parts easily. They could allow more extensions than they let on by turning off some automation. They are instead applying the well-tested concept of lockin both with software and their request to copyright office to make mods illegal. This is to increase profit of their near monopolistic dominance of a lot of the market (esp rural areas).
grigjd3 9y ago
How many lines of code does it take to register as the largest you've dealt with? The largest I dealt with took 45 minutes to grab the repo. I don't state that as a point of honor, more like a point in needless over-effort.
- throwaway_jddev 9y ago
  I would say the JD's tractor software (if taken in it's entirety) is on par with the modern Linux kernel in size, if not larger. Worse yet, the codebase is spread along many different repositories, teams, and build systems. I've seen integration builds take days to finish (not including running of unit tests). Look I get it, John Deere is evil. But this codebase is beyond any one person or team's comprehension. It's very massive, and it's very complicated.
  - db48x 9y ago
    All the more reason not to trust it.
  - Nilzor 9y ago
    Did you see this comment? https://news.ycombinator.com/item?id=14076318 https://news.ycombinator.com/item?id=14076318 I don't think the customers need or want to modify the software. Just open it up for hardware replacements
  - guelo 9y ago
    You said there were 150-300 engineers working on this code. That is at least an order of magnitude less than the number of people developing Linux.
    - dymk 9y ago
      Yes, but they're getting paid, which is going to easily make up for the larger number of individuals developing Linux. Not to mention that 10x the devs does not come close to 10x the productivity.
rando832 9y ago
> If a team of hundred of engineers struggle with their codebase internally, Joe Farmer isn't going to have a fucking clue how to repair their software correctly. > But trust me on this, locking this down is a very good idea. Joe Farmer may not know, but Joe farmer has "the internet", and ability to pay an independent software engineer. And people would absolutely find and fix bugs that john deer is missing. Honestly, your argument sounds so ridiculous, it sounds like saying "it's to protect the children from terrorism."
- lobotryas 9y ago
  What happens if this independent fix results in a death? Do you really think JD will escape liability?
  - db48x 9y ago
    This is not exactly uncertain ground. Car manufacturers don't have liability when their customers modify their cars. Whoever made my clothes dryer doesn't have liability for any unsafe condition I may or may not have created when I removed that stupid annoying buzzer.
  - CamperBob2 9y ago
    Life is full of risks. This is, or may be, one of them. Allowing a handful of opaque corporations to monopolize the tools of food production and dictate their after-sale use is another risk.
  - qq66 9y ago
    Of course they will. If a hit man cuts your brake lines, Ford will not be liable.
  - melq 9y ago
    I absolutely think JD would escape liability. Consider the more egregious cases of any number of high-profile companies who have had significant customer data breaches in recent memory. Regardless of how negligent they may have been (lets say they stored user passwords in plain text), have you ever heard of a company being held responsible for even the most egregious/reckless behavior?
tzs 9y ago
It doesn't sound like the farmers are asking to rewrite or replace that software. It sounds like they are just asking to be able to make the same kind of repairs that their local dealer makes.
- Nilzor 9y ago
  And what kind of repairs are that? As a software developer I don't understand what equipment is failure needs software change. Software don't wear and tear. Hardware does.
  - Baeocystin 9y ago
    All software ships with bugs. Some of them cause real problems, and are patched in the next firmware rev. Farmers (rightfully) resent being held over a barrel in both time and money for fixes that should be part of already owning the machine. For example, if Ford discovers a flaw in their cars' ABS code that is recall-worthy, that's what happens, and Ford, not the customer, foots the bill.
  - tzs 9y ago
    From a different article on this subject [1]: > "If a farmer bought the tractor, he should be able to do whatever he wants with it," Kevin Kenney, a farmer and right-to-repair advocate in Nebraska, told me. "You want to replace a transmission and you take it to an independent mechanic—he can put in the new transmission but the tractor can't drive out of the shop. Deere charges $230, plus $130 an hour for a technician to drive out and plug a connector into their USB port to authorize the part." [1] https://motherboard.vice.com/en_us/article/why-american-farmers-are-hacking-their-tractors-with-ukrainian-firmware https://motherboard.vice.com/en_us/article/why-american-farm...
    - Nilzor 9y ago
      Thanks. This was the kind of answer I feared
    - deleted 9y ago
      [deleted]
goodplay 9y ago
Let people run their own software. If they kill someone, hold them liable. If deaths become endemic, then consider locking them down. As many comments on these stories demonstrate, farmers have as much valid reasons to have control over their equipment as John Deere has for preventing it. Technology is always about trade-offs. It's just that John Deere took things too far.
- foepys 9y ago
  > If deaths become endemic, then consider locking them down. As much as I dislike John Deer's strategy, your statement is just stupid. Waiting for people do die before you regulate to make it safer doesn't work. Look at Ford who calculated that saving $x on a component was worth it if only y people die and their relatives sue.
  - goodplay 9y ago
    That component you refer to was necessary for the safe usage of the vehicle. If it fails, bad things would happen (and they did). Having control over your tractor does not necessarily kill people. Death is not inherently implied by farmers merely having access. Apples and oranges. Equating the two is silly. Most people won't be adding in gotos and if statements, they'll be replacing borken parts and getting their tractors working again. Those few who actually do end up writing code for it, and those who opt to run it, will do so with the full knowledge that it'll be their asses on the line when something goes wrong. Either way, my point was that people won't die from access unless they do something really stupid, and death due to stupidity is going to occur regardless of whether the tractor is locked down or not.
    - foepys 9y ago
      > Those few who actually do end up writing code for it, and those who opt to run it, will do so with the full knowledge that it'll be their asses on the line when something goes wrong. In a world where extremely dangerous phone chargers are getting bought just because they are cheap, I cannot imagine that your views are correct. People are putting active fire hazards into their homes and other people are building them in full knowledge that the components used are inadequate. Yes, people should be able to repair their machines and change single components but it should be guaranteed that basic safety functionality is working. I'm not talking about something where a farmer personally modifies their own machine. I'm talking about manufacturers that build cheap replacement parts.
pdkl95 9y ago
edit: I have hopefully fixed all of the places where I incorrectly parsed "worked on software for" as indicating employment. > They have a real potential to end peoples lives in the event of failure I hope the lessons[1] learned from the Therac-25 are part of the design. Anything that dangerous needs to have defense in depth. If there are hardware limiters, interlocks, etc adding redundancy to the software check, the risk should be no worse than repairing any other dangerous machinery. Alternatively, if John Deere is relying on the software alone for safety checks - which requires that they (and the OEMs!) write bug-free software - then I don't want to be anywhere near the affected products. > these tractors do (in testing) fail in spectacular ways. So it's software only. I'm amazed at the hubris of thinking it's even possible can guarantee a software project large enough to requires a couple hundred engineers that maintain "the largest, most complex codebase [you] have ever interacted with". The very existence of tests failing in "spectacular ways" is evidence that bugs do exist in the software. > These are not riding lawnmowers. I'm not aware of anybody that suggested they were. > They are 30-ton combines, and 20 ton tractors tilling fields Yes, just like the older combines and tractors that didn't require complicated software. > Joe Farmer isn't going to have a fucking clue how to repair their software correctly. This, really, is the key point: the farmers in question *are not trying to repair the software. The only reason it is involved at all is because John Deere the company) chose to add the dependency. [1] http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf
- throwaway_jddev 9y ago
  There is definitely defense in depth but, I never said the software was bug-free. :) Also you're doing the unfortunate thing of associating me as part of John Deere. I'm not. I had a small hand is writing a small part of this software, a long long time ago. I don't give a shit if they sell tractors or not. I don't care if farmers modify the software or not. I don't really have a dog in this fight. I'm just a guy with an opinion who's seen how the sausage is made.
  - blobman 9y ago
    We know that you are just a guy with an opinion. But you could counter the arguments if you still hold the opinion, otherwise you could concede.
femto 9y ago
> John Deere tractors run on open source software Out of curiosity, what range of open source licenses appear in the code base? Do they use any software whose license requires source distribution? If so, are you aware of a "source code" page buried in John Deere's website, or some other mechanism by which they satisfy any distribution clauses?
- throwaway_jddev 9y ago
  I couldn't possibly comment. This would be close to me breaking NDA ground here, so sorry about that. I know that the legal department took software licensing very seriously and we had a narrow range of licenses we could use.
  - mholt 9y ago
    NDA on open source libraries? Don't those licenses usually require attribution at the very least? Where can we find the license file? Surely JD is complying with license requirements, which would require disclosing the libs used...
    - Pxl_Buzzard 9y ago
      I don't think it's fair to assume an ex-employee knows the details of the software licenses or where to find them. If he knows more concrete details (like a link to the license file) I imagine he would share, but otherwise invoking NDA is a safety measure to avoid sharing potentially incorrect/outdated info.
  - tobltobs 9y ago
    Instead of now even hiding behind some lame excuses like NDAs, how about answering the most asked question: How does replacing hardware parts does risk lives? Why is the situation different for JD than for eg. car manufacturers?
- blobman 9y ago
  https://github.com/JohnDeere https://github.com/JohnDeere
uuilly 9y ago
Thank you. I work in Ag and it seems crazy that someone should be able to buy a machine like those Deere sells and be able to modify the controls SW. Can United Airlines update the controls SW on Boeing 747? I'm not sure how tractors are any different.
- randomdata 9y ago
  I also work in ag, and own John Deere equipment. Is anyone truly asking for source code? I've read that narrative here and there, but it seems what is really being sought after is access to the tools and manuals that the authorized dealers have access to. The service techs repairing these machines aren't going to be modifying the code either, and they don't need to. As an aside, I remember the last time I had a John Deere tech out. Even for him to access the service manual (to fix a mechanical part), which was fundamentally not much more than a simple search tool that displays PDFs, required a surprising level of authorization to access each document, with everything encrypted. It seemed a little extreme just to get a simple diagram. I can sort of see why some worry about where the company is headed.
  - nas 9y ago
    I think documentation is the right way to handle this. Require enough documentation that 3rd party repair shops can fix the machines. Even if they can't do much with the embedded controllers, at least document the sensors and the wiring. Otherwise, these machines are completely useless without the manufacturer and their dealers.
- themihai 9y ago
  United Airlines is not really farmer Joe and something tells me that even United Airlines doesn't send its 747 to Boeing for all the maintenance work...
- lawnchair_larry 9y ago
  Can confirm that United Airlines directly employs aircraft mechanics who are free to service as they see fit, and they use hardware that is not DRM'd to the plane. Can also confirm that aircrafts can do more damage than tractors if things go wrong. Like the JDs in the article that are so complex that they can steer themselves, aircrafts also do this! I don't see how tractors are any different either. So JD should cut out this rent-seeking bullshit and stop pretending it's in the best interest of anybody other than their shareholders.
melq 9y ago
If you could explain in reasonable detail a few common scenarios faced by the average farmer with an issue that would necessitate JD engineering expertise, I would very much appreciate it. Specifically, I'd be interested in hearing how the repair work orders that come in to JD engineering most frequently justify the claim that 'locking this down is a very good idea'. Who, aside from the operator of the machine, is being put in such grave danger? Why am I, someone with no knowledge of combustion engines/cars in general, allowed to do my own work on the car that conceivably endangers thousands of fellow commuters each day, but a farmer driving a tractor (that lets say he's changed the engine timing on) in his own empty fields is somehow a great threat to society? I would wager that Joe Farmer has a significantly stronger 'fucking clue' how to repair the issues they encounter with their tractors than Joe Developer at John Deere. Mostly because unless they're doing an aggressively poor job on the software (that you're convinced is too complicated for the unwashed masses to wrap their feeble minds around), the farmer shouldn't even need to be aware of its existence. Additionally, I think you and I both know the vast majority of farmers adversely affected by these business practices are not dealing with 40 ton death machines, but rather the low-end John Deere products that do little more than an equivalent machine from 30 years ago could do. I'd also be interested to hear how ordinary farm machinery necessitates a codebase as complicated as you purport the JD software to be. The problems these machines have to deal with are not difficult, and were largely solved years ago. I can imagine there are some rather complicated hardware engineering efforts involved, but I'm skeptical that the software component would require anything beyond the skills of an enterprising group of skilled undergrads. I realize embedded dev work in safety critical applications is not so easy, but I nevertheless feel like you're dramatically overstating the complexity. I'm sure the engineers on these projects are veritably concerned with the safety of their customers, and I don't mean to suggest otherwise. Just keep in mind the considerable imbalance of power between JD (and its engineers) and their farmer customers -- particularly in these types of debates. edit: upon reading my submission, I came off as a bit of a jerk, and for that I apologize. In my defense, before getting into the software industry, I grew up working on a farm, and as a result am a bit reactionary in the face of any sort of "I'm a big bad developer, these guys are glorified lawn mowers who need to be protected from themselves." lines of reasoning.
- jon-wood 9y ago
plinkplonk 9y ago
>But trust me on this,... isn't a way to make a good argument. From TFA "Only dealerships have the software to make those parts work, and it costs hundreds of dollars just to get a service call. Schwarting worries about being broken down in a field, waiting for a dealer to show up with a software key." and from a comment above "Plus, many of them don't get any update after the product launch. When you are in rush to plant or harvest you just can`t afford to wait for an authorized dealer. And if they fail, good luck trying to find a replacement that is not 100x overpriced because it has been discontinued one year after you bought it." the problem seems to be that the process of updating software (a) takes too long (b) is too expensive (c) sofware is not updated once the tractor has been sold. In an ideal world, where such a break down can be instantly fixed by JD for a reasonable fee, and JD actively updates the software on the equipment they sell till End of Life, this problem wouldn't exist. Farmers don't care about open source etc. They just want to do their jobs. I'm astounded by the fact that JD takes no responsibility for updating software till end of life of the tractor.
- kodfodrasz 9y ago
  This may be related to farmers not taking the tractors to repair shops. The automotive component I worked on was firmware updated every time the customer took the car to a service for eg. to change oil, and an update was necessary. The tools needed for this were pretty expensive though, and needed some expertise, as the process was somewhat complicated. I understand why this is not trusted to owners.
  - cjrp 9y ago
    If the lack of updates inconveniences their customers and potentially causes loss of earnings, isn't the onus on JD to improve the update mechanism? Make it as simple as inserting a USB stick, and then ship out those sticks periodically? It doesn't have to be OTA or anything fancy.
  - criddell 9y ago
    > The tools needed for this were pretty expensive though, and needed some expertise, as the process was somewhat complicated. There's a pretty good chance that was done intentionally.
oceanghost 9y ago
Most of the folks on HN haven't worked on anything more complicated than a web app.
- lawnchair_larry 9y ago
  Although that comment is unconstructive and probably wrong, I'll just point out that pretty much all of Google is a "web app".
  - RugnirViking 9y ago
    While I agree it seems the above comment was intended as a swing at HN which is not constructive or useful debate, I would suggest that as per the stack overflow developer survey web programmer was the most common job title by a very large margin (72.6%, with next closest being Desktop applications developer at 28.9%) While I would suggest that this forum demographic likely differs somewhat from stack overflow as a whole, its certainly an interesting question. Also as an anecdote, web technologies stories seem to appear disproportionately often on the front page. This would suggest at least some interest by the majority of HN
- jfoutz 9y ago
  I'm pretty sure prior versions had no software. Adding software without showing exactly what that software does seems more dangerous than trying to handle everything without explaining how it is handled. I mean, it used to work like X. Now it works like Y. Which you could inspect via the code. The reality is it works like ?. Because only god and the parent throwaway know.
hristov 9y ago
Wow the public relations sock-puppetry on HN is getting very clever and sophisticated. I really love the green account - nice touch. It really helps your authenticity to pretend like you are afraid of retribution from John Deere when you are totally kissing their asses. Let me ask you this -- why in the world do the lines of software you write prevent a qualified mechanic from changing a transmission? You know people used to do that all the time when tractors had zero lines of software in them. After all you are trying to very cleverly change the subject here. You are doing it very deftly, as you should, because you are not a software developer, you are a PR drone and it is your job to deftly change the subject. The question here is not about the quality of the John Deere software, so it does not really matter how many engineers worked on it or how hard they worked on it. The problem is that the John Deere software takes away an essential right of property ownership that people take for granted. No one is accusing the JD software of being faulty, i.e., incapable of serving it's intended purpose. People are accusing the software of having nefarious purposes and serving them all too well.
- deleted 9y ago
  [deleted]
- frik 9y ago
  > Wow the public relations sock-puppetry on HN is getting very clever and sophisticated. Very true. There are a few companies that engage sophisticated sock-puppetry on HN like P§l§nt$er, M$, M§n$ant§ and J§hn D$$r. Ethic should be a mandatory school subject. About dongled binary signed software in hardware devices. We sincerely need a law to prevent this. Tesla cars and John Deer tractors actively prevent the second buyer market with this. And one can imagine it's going to get worse with more self-driving cars and more software in vehicles.
  - barking 9y ago
    I can understand MS but John Deere? Hard to credit that they'd be monitoring HN. I mean how many farmers read HN?
    - 9y ago
dustinls 9y ago
Sounds like you're talking about the code that handles safety features. What does that have to do with the code specifically written to force farmers to pay John Deer for maintenance and repairs? The stuff they could normally do themselves? There is an obvious trend to make simple things complicated. Like how you have to remove the whole front end of a car to change a headlight. This sounds exactly like that.
- JdeBP 9y ago
  Actually, that latter is a trend to make things more compact. The engine compartment of my current car, which has the very requirement that you mention, has nowhere near the amount of extra unused volume as the engine compartment of the car that I was driving a couple of decades ago had, permitting access to many components without other components being in the way. The engine compartment is also, itself, smaller overall.
BinaryIdiot 9y ago
Your post smacks of "PR". The majority of people that I've talked to regarding their tractors have the issue of needing a replacement part and requiring the JD tech to come out and authorize its replacement or fixing every single time. This is just crazy expensive and time consuming. If you want people to not modify your hardware or software then you make it so easy that the alternative doesn't look good.
deelowe 9y ago
What does any of this have to do with repairing a transmission? You're moving the goal post.
g12mcgov 9y ago
Regardless of all the negative feedback this comment has received, this is one of the reasons I find Hacker News to be totally cool. Where else do you get an engineer who wrote the code commenting on the article in question?
xorblurb 9y ago
> This software is by FAR the largest, most complex codebase I have ever interacted with. This is a problem. Edit: I want to add something. Serious 3rd party serious mechanical shops have had the ability to intervene on this kind of equipment - I suppose. The same way that independent serious and competent car repair shops exists. One problem we are about to all observe, and which actually has already began, is not typically that owners would like to rewrite the software and run their car/tractor/whatever with the modified/rewritten version. The problem is that vendors are taking the dangerousness argument and (mis)using it as an economic weapon in the same time to gain anticompetitive in the maintenance market -- like the comment to which I'm replying to does, in effect (concentrating on the risks of unrelated open changes, instead of discussing the initial problem! ) And moreover in the same time, car vendors, for example, have proven several time we can not trust them with critical software.

jdietrich 9y ago

Don't try to bullshit us. Your code is deliberately user-hostile. Farmers aren't trying to hack their tractors for the sake of it; they need to hack the parts of your code that prevent them from performing routine maintenance. Replacing a fuel pump is not going to turn a tractor into a rampaging death machine. You are using spurious safety arguments to justify profiteering. Software lock-outs are being used to prop up a business model, just like the chips in printer ink cartridges.

bigtimeidiot 9y ago
>Don't try to bullshit us. Us? Who is "us"? Why are you trying to shout down an opposing view? You want prefer the echo chamber? >Replacing a fuel pump is not going to turn a tractor into a rampaging death machine. Am I going to be able to replace whatever parts I want on my Model 3? Because "we" certainly love that company here!
- tobltobs 9y ago
  Your comparison to Tesla doesn't get better just by repeating it a few times. If your Tesla breaks down you wont have any problem to get a comparable replacement car. Much luck to get a replacement for your tractor during harvest season.
  - bigtimeidiot 9y ago
    >Your comparison to Tesla doesn't get better just by repeating it a few times Your hand-waving also does precisely zero. >If your Tesla breaks down you wont have any problem to get a comparable replacement car What does this have to do with anything? I want to work on something I bought; isn't that what we're arguing about? The availability of a replacement is a straw man. Why are you apologizing for other companies that do precisely what is being described in the article?
    - RealityVoid 9y ago
      While I think you have a valid point about Tesla's availability, this is a "Tu Coque". I mean, regardless if Tesla does/does not do it, this is considered bad and bringing Tesla in the discussion does not change nor challenge the validity of this view, only, maybe, reveal a little hypocrisy or favoritism in the audience.
      - dandelion_lover 9y ago
        >While I think you have a valid point about Tesla's availability, this is a "Tu Coque". I mean, regardless if Tesla does/does not do it, this is considered bad aka https://en.wikipedia.org/wiki/Whataboutism https://en.wikipedia.org/wiki/Whataboutism

tw04 9y ago

What, specifically, is a threat to kill people in the software? I'm sorry but that just reads like a John Deere shill here to defend the cause. We got by for literally two hundred years without magical software in tractors, and we were just fine.

rimliu 9y ago
What if we compare the number of people involved in producing the same amount of food two hundred years ago and today?
criddell 9y ago
These things drive themselves. There are foragers that follow other equipment autonomously.
randomdata 9y ago
It's not obvious how an errant piece of large, heavy, machinery could kill someone? There's a software bug in one of my tractors (not John Deere) where it won't always immediately disengage the clutch when put in gear (yes, even the transmission is fully computerized). One time during a long day in the field I accidentally left it in gear, not noticing because the tractor was at a full stop with no signs of moving. A few minutes after I got out of the cab it decided to kick in. Luckily it was in a low gear and I was able to safely get it stopped, but it's easy to see how that could have ended up tragic. While that one was ultimately my error, without that bug it wouldn't have been an issue in the first place. I'm sure you can come up with arguments as to why that software shouldn't exist – although I have to say it's wonderful when it works – but it's easy to see why bugs can be dangerous.

throwaway198411 9y ago

This coolaide youre handing out is delicious

nickpsecurity 9y ago

Hey, I do high-assurance systems too. High-assurance security where input or environment might be malicious. Even in my constraints, the systems are designed so the parts can be replaced without magic software. The big complaint about JD seems to be them using software constructs to prevent fixing tractors or common extensions. The former should not be hard as JD just has to allow products with certain specs for what will work. The second might require a combination of that with a more manual, operating mode with some automation disabled. Less complexity than what JD is putting in there. So, Im not buying it even if you worked on the team. They could design it for cheap fixes for parts easily. They could allow more extensions than they let on by turning off some automation. They are instead applying the well-tested concept of lockin both with software and their request to copyright office to make mods illegal. This is to increase profit of their near monopolistic dominance of a lot of the market (esp rural areas).

grigjd3 9y ago

How many lines of code does it take to register as the largest you've dealt with? The largest I dealt with took 45 minutes to grab the repo. I don't state that as a point of honor, more like a point in needless over-effort.

throwaway_jddev 9y ago
I would say the JD's tractor software (if taken in it's entirety) is on par with the modern Linux kernel in size, if not larger. Worse yet, the codebase is spread along many different repositories, teams, and build systems. I've seen integration builds take days to finish (not including running of unit tests). Look I get it, John Deere is evil. But this codebase is beyond any one person or team's comprehension. It's very massive, and it's very complicated.
- db48x 9y ago
  All the more reason not to trust it.
- Nilzor 9y ago
  Did you see this comment? https://news.ycombinator.com/item?id=14076318 https://news.ycombinator.com/item?id=14076318 I don't think the customers need or want to modify the software. Just open it up for hardware replacements
- guelo 9y ago
  You said there were 150-300 engineers working on this code. That is at least an order of magnitude less than the number of people developing Linux.
  - dymk 9y ago
    Yes, but they're getting paid, which is going to easily make up for the larger number of individuals developing Linux. Not to mention that 10x the devs does not come close to 10x the productivity.

rando832 9y ago

> If a team of hundred of engineers struggle with their codebase internally, Joe Farmer isn't going to have a fucking clue how to repair their software correctly. > But trust me on this, locking this down is a very good idea. Joe Farmer may not know, but Joe farmer has "the internet", and ability to pay an independent software engineer. And people would absolutely find and fix bugs that john deer is missing. Honestly, your argument sounds so ridiculous, it sounds like saying "it's to protect the children from terrorism."

lobotryas 9y ago
What happens if this independent fix results in a death? Do you really think JD will escape liability?
- db48x 9y ago
  This is not exactly uncertain ground. Car manufacturers don't have liability when their customers modify their cars. Whoever made my clothes dryer doesn't have liability for any unsafe condition I may or may not have created when I removed that stupid annoying buzzer.
- CamperBob2 9y ago
  Life is full of risks. This is, or may be, one of them. Allowing a handful of opaque corporations to monopolize the tools of food production and dictate their after-sale use is another risk.
- qq66 9y ago
  Of course they will. If a hit man cuts your brake lines, Ford will not be liable.
- melq 9y ago
  I absolutely think JD would escape liability. Consider the more egregious cases of any number of high-profile companies who have had significant customer data breaches in recent memory. Regardless of how negligent they may have been (lets say they stored user passwords in plain text), have you ever heard of a company being held responsible for even the most egregious/reckless behavior?

tzs 9y ago

It doesn't sound like the farmers are asking to rewrite or replace that software. It sounds like they are just asking to be able to make the same kind of repairs that their local dealer makes.

Nilzor 9y ago
And what kind of repairs are that? As a software developer I don't understand what equipment is failure needs software change. Software don't wear and tear. Hardware does.
- Baeocystin 9y ago
  All software ships with bugs. Some of them cause real problems, and are patched in the next firmware rev. Farmers (rightfully) resent being held over a barrel in both time and money for fixes that should be part of already owning the machine. For example, if Ford discovers a flaw in their cars' ABS code that is recall-worthy, that's what happens, and Ford, not the customer, foots the bill.
- tzs 9y ago
  From a different article on this subject [1]: > "If a farmer bought the tractor, he should be able to do whatever he wants with it," Kevin Kenney, a farmer and right-to-repair advocate in Nebraska, told me. "You want to replace a transmission and you take it to an independent mechanic—he can put in the new transmission but the tractor can't drive out of the shop. Deere charges $230, plus $130 an hour for a technician to drive out and plug a connector into their USB port to authorize the part." [1] https://motherboard.vice.com/en_us/article/why-american-farmers-are-hacking-their-tractors-with-ukrainian-firmware https://motherboard.vice.com/en_us/article/why-american-farm...
  - Nilzor 9y ago
    Thanks. This was the kind of answer I feared
  - deleted 9y ago
    [deleted]

goodplay 9y ago

Let people run their own software. If they kill someone, hold them liable. If deaths become endemic, then consider locking them down. As many comments on these stories demonstrate, farmers have as much valid reasons to have control over their equipment as John Deere has for preventing it. Technology is always about trade-offs. It's just that John Deere took things too far.

foepys 9y ago
> If deaths become endemic, then consider locking them down. As much as I dislike John Deer's strategy, your statement is just stupid. Waiting for people do die before you regulate to make it safer doesn't work. Look at Ford who calculated that saving $x on a component was worth it if only y people die and their relatives sue.
- goodplay 9y ago
  That component you refer to was necessary for the safe usage of the vehicle. If it fails, bad things would happen (and they did). Having control over your tractor does not necessarily kill people. Death is not inherently implied by farmers merely having access. Apples and oranges. Equating the two is silly. Most people won't be adding in gotos and if statements, they'll be replacing borken parts and getting their tractors working again. Those few who actually do end up writing code for it, and those who opt to run it, will do so with the full knowledge that it'll be their asses on the line when something goes wrong. Either way, my point was that people won't die from access unless they do something really stupid, and death due to stupidity is going to occur regardless of whether the tractor is locked down or not.
  - foepys 9y ago
    > Those few who actually do end up writing code for it, and those who opt to run it, will do so with the full knowledge that it'll be their asses on the line when something goes wrong. In a world where extremely dangerous phone chargers are getting bought just because they are cheap, I cannot imagine that your views are correct. People are putting active fire hazards into their homes and other people are building them in full knowledge that the components used are inadequate. Yes, people should be able to repair their machines and change single components but it should be guaranteed that basic safety functionality is working. I'm not talking about something where a farmer personally modifies their own machine. I'm talking about manufacturers that build cheap replacement parts.

pdkl95 9y ago

edit: I have hopefully fixed all of the places where I incorrectly parsed "worked on software for" as indicating employment. > They have a real potential to end peoples lives in the event of failure I hope the lessons[1] learned from the Therac-25 are part of the design. Anything that dangerous needs to have defense in depth. If there are hardware limiters, interlocks, etc adding redundancy to the software check, the risk should be no worse than repairing any other dangerous machinery. Alternatively, if John Deere is relying on the software alone for safety checks - which requires that they (and the OEMs!) write bug-free software - then I don't want to be anywhere near the affected products. > these tractors do (in testing) fail in spectacular ways. So it's software only. I'm amazed at the hubris of thinking it's even possible can guarantee a software project large enough to requires a couple hundred engineers that maintain "the largest, most complex codebase [you] have ever interacted with". The very existence of tests failing in "spectacular ways" is evidence that bugs do exist in the software. > These are not riding lawnmowers. I'm not aware of anybody that suggested they were. > They are 30-ton combines, and 20 ton tractors tilling fields Yes, just like the older combines and tractors that didn't require complicated software. > Joe Farmer isn't going to have a fucking clue how to repair their software correctly. This, really, is the key point: the farmers in question *are not trying to repair the software. The only reason it is involved at all is because John Deere the company) chose to add the dependency. [1] http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf

throwaway_jddev 9y ago
There is definitely defense in depth but, I never said the software was bug-free. :) Also you're doing the unfortunate thing of associating me as part of John Deere. I'm not. I had a small hand is writing a small part of this software, a long long time ago. I don't give a shit if they sell tractors or not. I don't care if farmers modify the software or not. I don't really have a dog in this fight. I'm just a guy with an opinion who's seen how the sausage is made.
- blobman 9y ago
  We know that you are just a guy with an opinion. But you could counter the arguments if you still hold the opinion, otherwise you could concede.

femto 9y ago

> John Deere tractors run on open source software Out of curiosity, what range of open source licenses appear in the code base? Do they use any software whose license requires source distribution? If so, are you aware of a "source code" page buried in John Deere's website, or some other mechanism by which they satisfy any distribution clauses?

throwaway_jddev 9y ago
I couldn't possibly comment. This would be close to me breaking NDA ground here, so sorry about that. I know that the legal department took software licensing very seriously and we had a narrow range of licenses we could use.
- mholt 9y ago
  NDA on open source libraries? Don't those licenses usually require attribution at the very least? Where can we find the license file? Surely JD is complying with license requirements, which would require disclosing the libs used...
  - Pxl_Buzzard 9y ago
    I don't think it's fair to assume an ex-employee knows the details of the software licenses or where to find them. If he knows more concrete details (like a link to the license file) I imagine he would share, but otherwise invoking NDA is a safety measure to avoid sharing potentially incorrect/outdated info.
- tobltobs 9y ago
  Instead of now even hiding behind some lame excuses like NDAs, how about answering the most asked question: How does replacing hardware parts does risk lives? Why is the situation different for JD than for eg. car manufacturers?
blobman 9y ago
https://github.com/JohnDeere https://github.com/JohnDeere

uuilly 9y ago

Thank you. I work in Ag and it seems crazy that someone should be able to buy a machine like those Deere sells and be able to modify the controls SW. Can United Airlines update the controls SW on Boeing 747? I'm not sure how tractors are any different.

randomdata 9y ago
I also work in ag, and own John Deere equipment. Is anyone truly asking for source code? I've read that narrative here and there, but it seems what is really being sought after is access to the tools and manuals that the authorized dealers have access to. The service techs repairing these machines aren't going to be modifying the code either, and they don't need to. As an aside, I remember the last time I had a John Deere tech out. Even for him to access the service manual (to fix a mechanical part), which was fundamentally not much more than a simple search tool that displays PDFs, required a surprising level of authorization to access each document, with everything encrypted. It seemed a little extreme just to get a simple diagram. I can sort of see why some worry about where the company is headed.
- nas 9y ago
  I think documentation is the right way to handle this. Require enough documentation that 3rd party repair shops can fix the machines. Even if they can't do much with the embedded controllers, at least document the sensors and the wiring. Otherwise, these machines are completely useless without the manufacturer and their dealers.
themihai 9y ago
United Airlines is not really farmer Joe and something tells me that even United Airlines doesn't send its 747 to Boeing for all the maintenance work...
lawnchair_larry 9y ago
Can confirm that United Airlines directly employs aircraft mechanics who are free to service as they see fit, and they use hardware that is not DRM'd to the plane. Can also confirm that aircrafts can do more damage than tractors if things go wrong. Like the JDs in the article that are so complex that they can steer themselves, aircrafts also do this! I don't see how tractors are any different either. So JD should cut out this rent-seeking bullshit and stop pretending it's in the best interest of anybody other than their shareholders.

melq 9y ago

If you could explain in reasonable detail a few common scenarios faced by the average farmer with an issue that would necessitate JD engineering expertise, I would very much appreciate it. Specifically, I'd be interested in hearing how the repair work orders that come in to JD engineering most frequently justify the claim that 'locking this down is a very good idea'. Who, aside from the operator of the machine, is being put in such grave danger? Why am I, someone with no knowledge of combustion engines/cars in general, allowed to do my own work on the car that conceivably endangers thousands of fellow commuters each day, but a farmer driving a tractor (that lets say he's changed the engine timing on) in his own empty fields is somehow a great threat to society? I would wager that Joe Farmer has a significantly stronger 'fucking clue' how to repair the issues they encounter with their tractors than Joe Developer at John Deere. Mostly because unless they're doing an aggressively poor job on the software (that you're convinced is too complicated for the unwashed masses to wrap their feeble minds around), the farmer shouldn't even need to be aware of its existence. Additionally, I think you and I both know the vast majority of farmers adversely affected by these business practices are not dealing with 40 ton death machines, but rather the low-end John Deere products that do little more than an equivalent machine from 30 years ago could do. I'd also be interested to hear how ordinary farm machinery necessitates a codebase as complicated as you purport the JD software to be. The problems these machines have to deal with are not difficult, and were largely solved years ago. I can imagine there are some rather complicated hardware engineering efforts involved, but I'm skeptical that the software component would require anything beyond the skills of an enterprising group of skilled undergrads. I realize embedded dev work in safety critical applications is not so easy, but I nevertheless feel like you're dramatically overstating the complexity. I'm sure the engineers on these projects are veritably concerned with the safety of their customers, and I don't mean to suggest otherwise. Just keep in mind the considerable imbalance of power between JD (and its engineers) and their farmer customers -- particularly in these types of debates. edit: upon reading my submission, I came off as a bit of a jerk, and for that I apologize. In my defense, before getting into the software industry, I grew up working on a farm, and as a result am a bit reactionary in the face of any sort of "I'm a big bad developer, these guys are glorified lawn mowers who need to be protected from themselves." lines of reasoning.

jon-wood 9y ago

plinkplonk 9y ago

>But trust me on this,... isn't a way to make a good argument. From TFA "Only dealerships have the software to make those parts work, and it costs hundreds of dollars just to get a service call. Schwarting worries about being broken down in a field, waiting for a dealer to show up with a software key." and from a comment above "Plus, many of them don't get any update after the product launch. When you are in rush to plant or harvest you just can`t afford to wait for an authorized dealer. And if they fail, good luck trying to find a replacement that is not 100x overpriced because it has been discontinued one year after you bought it." the problem seems to be that the process of updating software (a) takes too long (b) is too expensive (c) sofware is not updated once the tractor has been sold. In an ideal world, where such a break down can be instantly fixed by JD for a reasonable fee, and JD actively updates the software on the equipment they sell till End of Life, this problem wouldn't exist. Farmers don't care about open source etc. They just want to do their jobs. I'm astounded by the fact that JD takes no responsibility for updating software till end of life of the tractor.

kodfodrasz 9y ago
This may be related to farmers not taking the tractors to repair shops. The automotive component I worked on was firmware updated every time the customer took the car to a service for eg. to change oil, and an update was necessary. The tools needed for this were pretty expensive though, and needed some expertise, as the process was somewhat complicated. I understand why this is not trusted to owners.
- cjrp 9y ago
  If the lack of updates inconveniences their customers and potentially causes loss of earnings, isn't the onus on JD to improve the update mechanism? Make it as simple as inserting a USB stick, and then ship out those sticks periodically? It doesn't have to be OTA or anything fancy.
- criddell 9y ago
  > The tools needed for this were pretty expensive though, and needed some expertise, as the process was somewhat complicated. There's a pretty good chance that was done intentionally.

oceanghost 9y ago

Most of the folks on HN haven't worked on anything more complicated than a web app.

lawnchair_larry 9y ago
Although that comment is unconstructive and probably wrong, I'll just point out that pretty much all of Google is a "web app".
- RugnirViking 9y ago
  While I agree it seems the above comment was intended as a swing at HN which is not constructive or useful debate, I would suggest that as per the stack overflow developer survey web programmer was the most common job title by a very large margin (72.6%, with next closest being Desktop applications developer at 28.9%) While I would suggest that this forum demographic likely differs somewhat from stack overflow as a whole, its certainly an interesting question. Also as an anecdote, web technologies stories seem to appear disproportionately often on the front page. This would suggest at least some interest by the majority of HN
jfoutz 9y ago
I'm pretty sure prior versions had no software. Adding software without showing exactly what that software does seems more dangerous than trying to handle everything without explaining how it is handled. I mean, it used to work like X. Now it works like Y. Which you could inspect via the code. The reality is it works like ?. Because only god and the parent throwaway know.

hristov 9y ago

Wow the public relations sock-puppetry on HN is getting very clever and sophisticated. I really love the green account - nice touch. It really helps your authenticity to pretend like you are afraid of retribution from John Deere when you are totally kissing their asses. Let me ask you this -- why in the world do the lines of software you write prevent a qualified mechanic from changing a transmission? You know people used to do that all the time when tractors had zero lines of software in them. After all you are trying to very cleverly change the subject here. You are doing it very deftly, as you should, because you are not a software developer, you are a PR drone and it is your job to deftly change the subject. The question here is not about the quality of the John Deere software, so it does not really matter how many engineers worked on it or how hard they worked on it. The problem is that the John Deere software takes away an essential right of property ownership that people take for granted. No one is accusing the JD software of being faulty, i.e., incapable of serving it's intended purpose. People are accusing the software of having nefarious purposes and serving them all too well.

deleted 9y ago
[deleted]
frik 9y ago
> Wow the public relations sock-puppetry on HN is getting very clever and sophisticated. Very true. There are a few companies that engage sophisticated sock-puppetry on HN like P§l§nt$er, M$, M§n$ant§ and J§hn D$$r. Ethic should be a mandatory school subject. About dongled binary signed software in hardware devices. We sincerely need a law to prevent this. Tesla cars and John Deer tractors actively prevent the second buyer market with this. And one can imagine it's going to get worse with more self-driving cars and more software in vehicles.
- barking 9y ago
  I can understand MS but John Deere? Hard to credit that they'd be monitoring HN. I mean how many farmers read HN?
  - 9y ago

dustinls 9y ago

Sounds like you're talking about the code that handles safety features. What does that have to do with the code specifically written to force farmers to pay John Deer for maintenance and repairs? The stuff they could normally do themselves? There is an obvious trend to make simple things complicated. Like how you have to remove the whole front end of a car to change a headlight. This sounds exactly like that.

JdeBP 9y ago
Actually, that latter is a trend to make things more compact. The engine compartment of my current car, which has the very requirement that you mention, has nowhere near the amount of extra unused volume as the engine compartment of the car that I was driving a couple of decades ago had, permitting access to many components without other components being in the way. The engine compartment is also, itself, smaller overall.

BinaryIdiot 9y ago

Your post smacks of "PR". The majority of people that I've talked to regarding their tractors have the issue of needing a replacement part and requiring the JD tech to come out and authorize its replacement or fixing every single time. This is just crazy expensive and time consuming. If you want people to not modify your hardware or software then you make it so easy that the alternative doesn't look good.

deelowe 9y ago

What does any of this have to do with repairing a transmission? You're moving the goal post.

g12mcgov 9y ago

Regardless of all the negative feedback this comment has received, this is one of the reasons I find Hacker News to be totally cool. Where else do you get an engineer who wrote the code commenting on the article in question?

xorblurb 9y ago

> This software is by FAR the largest, most complex codebase I have ever interacted with. This is a problem. Edit: I want to add something. Serious 3rd party serious mechanical shops have had the ability to intervene on this kind of equipment - I suppose. The same way that independent serious and competent car repair shops exists. One problem we are about to all observe, and which actually has already began, is not typically that owners would like to rewrite the software and run their car/tractor/whatever with the modified/rewritten version. The problem is that vendors are taking the dangerousness argument and (mis)using it as an economic weapon in the same time to gain anticompetitive in the maintenance market -- like the comment to which I'm replying to does, in effect (concentrating on the risks of unrelated open changes, instead of discussing the initial problem! ) And moreover in the same time, car vendors, for example, have proven several time we can not trust them with critical software.