16 ms·
I think this whole open source and right to repair/modify will be very interesting with self-driving cars, because of their interaction with the commons. Here a
by RcouF1uZ4gsC 10y ago
I think this whole open source and right to repair/modify will be very interesting with self-driving cars, because of their interaction with the commons. Here are some issues.
There have been some discussions about the ethics of self-driving cars if it should sacrifice the lives of the people in the cars to save more lives. In a right to repair/modify wouldn't a lot of people pay to have the algorithm changed for their car to always favor the people in the car no matter what?
If the self-driving software is completely open source, you can exploit the collision avoidance algorithms to favor aggressive driving with your car. For example if the software tries to keep a 15 foot buffer between cars, you can"tune" your software to use a shorter buffer and cut in front of traffic more easily.
Law enforcement will campaign for a remote "pull over" command to prevent people from fleeing police.
The answers to these types of questions will be very important for open source going forward.
- slavik81 10y agoThose issues also exist with human driven cars. Every driver today already controls how their car is driven. Many of them do not maintain a safe following distance and some refuse to pull over upon police request. We have systems to deal with bad drivers. A police remote engine kill switch doesn't even require a self-driving car. Limiting engine power upon receipt of a radio signal has been technologically possible for decades.
- chinhodado 10y agoI think it should be expected that once the software/firmware of a self-driving car has been modified by someone not authorized by the car maker, it should be expected that the car maker is not responsible for that car any more. Of course it brings up the problem of whether it is possible to detect such a modification.
- nathan_long 10y agoAnd also whether the modification is relevant the the problem. I can imagine court cases where the driver argues that the change they made was irrelevant to the accident that occurred.
- RcouF1uZ4gsC 10y agoRight, and unlike a lot of other cases with software, how this was decided might actually result in jail time. For example if a modification was deemed negligent and resulted in the death of a person, whoever made/authorized the modifications could be charged with manslaughter/murder.
- Silhouette 10y agoThis sort of issue has already been tested in court and, at least here in Europe, generally seems to follow common sense. For example, if a new top-of-the-range smartphone suffers a hardware failure after 3 months, the manufacturer can't disclaim any responsibility for that and exempt themselves from our normal consumer protection standards just because the owner had flashed custom firmware onto their device (unless the custom firmware was actually a causal factor in the failure, presumably, though I'm not sure that scenario has been tested in court yet).
- scrollaway 10y agoLiability is the least important part of this problem. It doesn't matter who is liable, the problem is a conflict of philosophies. On one side we want to ensure the safety of the public, on the other we want to have full control over the equipment we own. This isn't the first time this question has come up. It comes up with firearms for example, regarding how you can and cannot modify on a firearm you legally purchased. IMO you should be able to modify your car's autopilot however you want to... at the expense of being able to legally drive it on roads. We already have fairly strict legal requirements in place for you to be able to take a 2+ ton death machine on wheels and travel in it at high speeds on public roads.
- _puk 10y agoWhilst I agree with your points, I wouldn't conflate the remote pull over with the right to repair, other than as a reason not to allow it. As you say, if it is open source it may well be modified, and likely the remote pull over will be removed. I can't really imagine a scenario whereby the remote pullover is implemented, and then allowed to be removed by individual owners.
- nathan_long 10y agoYou bring up very interesting questions, but the self-driving car should be taken as a very special case. A manufacturer can hardly argue that they have a safety reason to keep you from repairing your TV, printer, or cell phone.
- Silhouette 10y agoA manufacturer can hardly argue that they have a safety reason to keep you from repairing your TV, printer, or cell phone. I believe this is not the strongest argument to make. There are plenty of products that do have legitimate safety or other regulatory concerns, where there is a good reason to prevent someone who doesn't know what they're doing from meddling. However, the relevant point is whether they know what they're doing, not whether they have the blessing of the manufacturer. I tend to think that saying devices in regulated fields can only be modified by someone suitably qualified and experienced is reasonable, but that in such a case the owner and the qualified person modifying the owner's device should not be artificially constrained through things like unnecessary regulation or intellectual property laws. There are also ethical and economic questions around supporting fair competition in markets for replacement parts from alternative sources or examination and modification of source code in the case of software. On the one hand, we do have IP laws to encourage creators to create and to protect them from someone ripping off all their hard work the day after their multi-year, multi-billion R&D exercise finishes. On the other hand, those laws aren't supposed to be there so an inventor can sell an artificially crippled product and then charge extortionate rates for servicing , replacement parts or essential updates to correct defects in the original product. There's a lot of room for progress in this kind of area today, IMHO.
- Nullabillity 10y ago> There are plenty of products that do have legitimate safety or other regulatory concerns, where there is a good reason to prevent someone who doesn't know what they're doing from meddling. The safety of themselves or others?
- Silhouette 10y ago
- maxerickson 10y agoI think it is pretty likely that the full self driving system will be the thing that gets licensed. If that were the case, your exploited systems would not be licensed for operation on public roads. This can be completely orthogonal to whether you are able to modify the vehicle or not.
- RcouF1uZ4gsC 10y agoHow would you detect these changes in a licensed system if you are allowed to modify the vehicle? If you do behavior based testing - the person will just hide the features behind a backdoor so the testing looks normal. If you do some type of checksum based testing, you have defacto made modification illegal, and it is bypassed by doing a "factory reset" prior to the licensing check, and loading the modified software after the licensing check. If you try to do some type of remote real-time detection of an unlicensed system, software that is modified can masquerade as unmodified.
- maxerickson 10y agoI'm talking about the likely legal framework that the systems would be implemented against. The technical details would of course be important. But for instance, other systems could record and report vehicles with anomalous behaviors.
- GauntletWizard 10y agoAt some level, making an outward show of being a good person and being a good person are indistinguishable.
- DSMan195276 10y agoI don't think it is really that complicated. On boot, a piece of embedded software (that can't be overwritten) does a checksum of the current software installed. If the checksum doesn't match the original, then it sets an internal bit to 1 to mark that the software has been changed. That internal bit can't be set back to 0, so the car is essentially marked as non-licensed and now can't be legally driven on the road. Checking for non-licensed modifications is simply checking if that bit is set. Implementation would be a little more complex then the above, but similar technology already exists and is in use. The result is that modifications would be possible, but driving the resulting car on public roads would not. A system for allowing new software to be placed on a car could be setup, but it would still require re-licensing the whole car so it wouldn't necessarily be an easy process.
- mikeash 10y agoI think the ethical question is greatly overblown. The odds of being in a situation where there's any sort of ethical choice to make are so remote as to not really be worth considering. 99.9999% of the time, the correct choice when faced with an impending crash will be to brake as hard as the driving surface allows and steer to avoid obstacles as much as possible. In any case, tinkering and commons already intersect to an extent, for example with emissions. It's not uncommon for emissions to be favored over performance, for example, and some people hack their cars to favor performance instead. Or they just like being polluting assholes, as is the case with "rolling coal." But it's not a big deal, some jurisdictions do periodic testing and others just rely on the fact that such people are relatively rare.
- rando3826 10y ago> I think the ethical question is greatly overblown. Not at all. One programmer is going to make choices that affect ~ a billion vehicles one day, so something that doesn't happen 99.9999% per hour will happen 100 times per hour. And it's going to be more mundane things, like programming the speed limit 1 mph slower results in 500 less deaths per year, or using drm/copyright to stop poor people from getting driving software, thus killing thousands of people.
- Retric 10y agoProgrammers are not going to decide what is an acceptable MPH vs posted speed or if the car will use DRM. Much like how programmers at Volkswagen, Audi, etc are not the ones that chose to cheat emissions tests. The real choice is going to be what to do about soft obstacles. Hitting a person or deer can easily kill passengers and hitting trees is actually normally safer thus making the discussion fairly moot. People are going to want to hack self driving cars so they can speed, which is more inline with this moral discussion. Not because it's safe for passengers, but because it's also dangerous for other people.
- twblalock 10y agoProgrammers have to make decisions about those things all the time. It isn't like the product team defines every nitty-gritty detail of implementation -- especially for machine learning and AI algorithms.
- moyix 10y agoWe already have some of these issues without open source. People today modify their cars to make them perform better at the expense of violating emissions standards. It's a very tiny number of people that actually end up doing so, and they sometimes get caught and are then fined appropriately for emissions violations. My feeling is that if there are regulations that require certain ethical behavior for self-driving cars, then there will also be periodic testing to make sure your car is in compliance. In the case of aggressive driving, we might also start to see traffic cameras doing automated detection of violations – if it sees that the gap between cars is less than 15 feet it'll take a picture of your license plate and send you a ticket (or even suspend your license until you can prove your car's software is in compliance). But I suspect that on the whole people are going to stick with the defaults. The same sort of people who like to jailbreak their phones might mess with the software on their cars, but it will be an even smaller percentage of the population that engages in this behavior – after all, if you mess up you may not just brick your car, you might get yourself killed.
- jpatel3 10y agoI think the smaller percentage is very time specific thing. What today looks like small percentage, over the period of time won't be the same. Need to think beyond the current scenario. As push for coding, awareness around tech growing, the future force won't be same.
- mrfusion 10y agoCompanies are spending billions of dollars in r&d to build self driving cars and you think a hobbiest is going to modify his car with his own "aggressive" self driving system?
- mcherm 10y agoIf the hobbiest is CAPABLE of doing so, then they will.
- softawre 10y agoAgreed. The simple case is a car that is shipped to not allow speeds faster than say 120MPH, and the owner removing the governor. https://en.wikipedia.org/wiki/Governor_(device)#Cars https://en.wikipedia.org/wiki/Governor_(device)#Cars
- kenrikm 10y agoIf you don't want a catalytic converter on your car for performance or intentionally running rich a/f mix etc.. you can put the O2 sensor into a "spark plug tool" that'll trick the O2 sensor into thinking all is well. https://www.youtube.com/watch?v=GqDmmLQ4pGk https://www.youtube.com/watch?v=GqDmmLQ4pGk Hacks are not required to be sophisticated to be effective and people will find them. Maybe for example if you place a 1/4 inch of extra plastic in front of the front facing radar it'll make the car follow more aggressively then it's calibrated for, who knows! that's a made up example but I expect this stuff is going to happen as it already does.
- wnoise 10y agoHobby, hobbier, hobbiest? Hobbyist.
- saidajigumi 10y agoWhy on earth should society allow (i.e. license) a randomly-modified software agent to operate a motor vehicle? The idea that any "hacker" has the right to arbitrarily change vehicular self-control software and deploy that on a public roadway is laughable.
- Karunamon 10y agoLike the steering wheel that's deployed on all public roadways right now? I'm struggling to see the substantial difference between "self driving car with modified software should be disallowed because safety" and "manually driven car should be disallowed because safety". The former is still strictly safer.
- saidajigumi 10y ago> The former is still strictly safer. I find this to be an absolutely ridiculous statement. Software as we know it today is incredibly brittle. What will eventually be put on the road as fully autonomous vehicular control software will have been HEAVILY QA'ed, and will be continuously QAed by telemetry. Random people modifying that software invalidate any proof-of-safety from the upstream QA and monitoring processes.
- tomswartz07 10y agoRandom people manually driving their cars are "modifying" their input as well. You have drunk drivers, texting drivers, distracted drivers.... The human attention span is incredibly brittle as well. In terms of QA'ing a manual driver in the US? You take a 10 minute driving test when you're 16 and you're licensed to drive for the rest of your life.
- jolux 10y ago> The human attention span is incredibly brittle as well. I think that's sort of proving their point? Like the point should be to replace brittleness with reliability, not replicate it with software.
- CM30 10y agoOn a liability case, wouldn't it potentially be to the company's advantage to encourage people to set those settings themselves and not to have a default? I mean, if a company programs the car to do something in a certain situation, then I suspect any resulting court case would be against the company. If the customer sets it up, then presumably people would be more likely to sue the car owner. I'm not a lawyer, but I suspect no self driving car company would want to make any decision here. Not least because they don't want to be sued into oblivion for it. Also, a lot of those situations you mention are avoidable for open source. Well, avoidable in that you simply write the laws to specify the buffer space or what not. Then it doesn't matter whether it's an open source user or hacker that's ignored them.
- Frondo 10y ago"In a right to repair/modify wouldn't a lot of people pay to have the algorithm changed for their car to always favor the people in the car no matter what?" Wait a sec....isn't this what human drivers do already, favor their own vehicles?
- chipperyman573 10y agoYeah, but once you take humans out of the equation, it's possible to fix this. Obviously, not everyone agrees that this is a problem that needs to be fixed - that's where the debate is.
- lotu 10y agoRealistically no. Human drivers react based on instict often on a subconscious level where they don't have time to think about all the possible outcomes.
- sargun 10y agoMore than this -- what even is software? A lot of the work we've seen with self driving cars is based on "AI" (machine learning). Much of the software is useless without the model, or the training data. I can certainly see a company making the case that it is not software even if the rest is OSS.
- zouhair 10y agoI don't see the problem. You can do worse right now.
- thesehands 10y agoIt strikes me that once we get to a point of autonomous driving why won't law enforcement be autonomous, a network of sensors along our road network that receive pings from all passing vehicles. Anything that causes your vehicle to deviate from its specification e.g. modding would flag you on the system.
- Shivetya 10y agoWith regards to cars, the right to have an independent shop perform routine maintenance and repair without violating warranty does need to be preserved. As to modify, well I would think that should vacate any warranty and possibly liability as it is now beyond the manufacturers control. This will mean preventing manufacturers from creating costly interfaces to retrieve status codes from cars and their self driving systems. The issue with modifying self driving algorithm's is that will be so complex as to be nearly impossible to modification and since they are so tightly linked to the safe operation of the vehicle I can expect we will see legislation to prevent it. However such legislation should not preclude being able to view it. * disclaimer to right of repair comments, I work for major after market supplier to consumers and business in the automotive sector
- sangnoir 10y ago> I think this whole open source and right to repair/modify will be very interesting with self-driving cars, because of their interaction with the commons. Here are some issues... I think those issues are red-herrings. We already have the tools to deal with bad actors in the commons. We already allow people to 'tune' their non-AI cars (and in the process its possible to exceed the pollution limits). The solution is to allow modification, but in an auditable and measurable process similar to Android's "fastboot oem unlock". AI cars should possibly be able to mark human-driven & non-stock-AI-driven cars and respond accordingly. This is assuming all cars will have a standard AI, otherwise the AIs will need to treat all the other AI-cars as black boxes - if a Toyota AI can't predict the behaviour of a Dodge AI car, then a 'tuned' AI shouldn't be treated any different.
- rsync 10y ago"In a right to repair/modify wouldn't a lot of people pay to have the algorithm changed for their car to always favor the people in the car no matter what?" ... which would return us to the "software" that we're all already running.
- kerkeslager 10y ago> There have been some discussions about the ethics of self-driving cars if it should sacrifice the lives of the people in the cars to save more lives. In a right to repair/modify wouldn't a lot of people pay to have the algorithm changed for their car to always favor the people in the car no matter what? How does this differ from those people making the same choice behind the wheel? To me, the issue isn't the trolley problem: we haven't come up with a definitive answer for that since it was proposed in 1967 and I doubt we will before self-driving cars are commonplace. The question for me is: when my car is confronted with the trolley problem (and many others), is the decision made by me or by a corporation?
- daveheq 10y agoWould you be the first to volunteer to have your car sacrifice you to save other people's lives? Of course they should not kill the driver, but an alternative shouldn't be to plow into a crowd or another car. My guess is the algorithm can simply be coded to minimize damage and death as best as we can do, and the robot will see it that way, not as "sacrifice the driver to save other people", and hopefully the law will see it that way too.
- samirm 10y agojust because something is open source doesn't mean it would be allowed to be used on the "commons"