14 ms·
“I Want to Know What Code Is Running Inside My Body”
- tn13 10y agoVery soon we would have FBI and NSA requiring these pacemakers to have a kill switch to kill whoever they don't like.
- subway 10y agoThey don't need to. Private industry has shown time and again that the device will remain insecure until consumers care (that is, indefinitely).
- tn13 10y agoThose are two different things. Private industry's safety assurance will grow up steadily as the demand from consumers goes up. Currently we are just thankful to be alive. For example if someone invents an artificial device that helps me get rid of diabetes I would be super happy. It is only a generation later we would demand that device meet certain quality floor. That is the case with all innovations. Government however can put a very hard nail into the head of an innovation.
- mankash666 10y agoReasons to NOT open up the code: 1> Loss of competitive advantage 2> Open source is not necessarily any safer (heartbleed bug ... ) 3> If software for the pacemaker is allowed to be updated like that on a computer, someone will update it with buggy software that can cause adverse side effects. Who owns the liability in that case?
- kiba 10y agoLoss of competitive advantage There's a competitive advantage in keeping pacemakers' source code proprietary? Open source is not necessarily any safer (heartbleed bug ... ) It's a hell lot more amendable to inspection than proprietary software.
- whorleater 10y ago> There's a competitive advantage in keeping pacemakers' source code proprietary? There definitely is. If your company takes ~2 years to develop a pacemaker's software, it's not to your advantage to let your competitors catch up. I'm not saying it's a good thing that it's closed source, but there is definitely incentive to keep your research to yourself.
- pdonis 10y ago> If your company takes ~2 years to develop a pacemaker's software, it's not to your advantage to let your competitors catch up. Why should the patient who has the pacemaker implanted care? This seems like a clear situation in which the patient's interests trump everybody else's. Pacemaker manufacturers should be competing in how well their devices meet patient needs. Closed source doesn't meet a key patient need.
- whorleater 10y agoAs heartless as this sounds: the patient isn't the only person in the equation. There are investors that fund the medical research, a company has employees to pay, the hospital wants the best pacemaker available, etc.
- anonbanker 10y agoyou're right. that sounds pretty heartless.
- llukas 10y agoSure - why not ask patient for paid subscription for privilege of using pacemaker? Somebody please think of the investors...
- whorleater 10y agoAnd in the same vein, why not ask investors to if they'll fund a pacemaker company in a purely philanthropic fashion? Isn't it better to get pacemakers in a limited fashion than not at all?
- iokanuon 10y ago3> has nothing to do with the code being OS, wherether or not a device is locked down is independent of that.
- smallnamespace 10y agoMaybe in a complete vacuum, but in reality, having access to source certainly makes it easier to look for vulnerabilities, and if the same software is in many devices, the cost of finding vulnerabilities is amortized. Security by obscurity obviously doesn't stop a determined attacker, but it does raise the barrier to entry for script kiddies.
- r3bl 10y agoNo. Tell me, how many vulnerabilities are running wild on Linux, the software that powers... well, pretty much anything (including the servers through which you read this content)? Even if you find a vulnerability, it gets patched within hours and it may take a day or two for it to be distributed to everyone. > Security by obscurity [...] does raise the barrier to entry for script kiddies. Which script can help you find a vulnerability inside of the source code? You'll need a script that can understand the code it's looking at. I'm not aware of any such script.
- smallnamespace 10y agoIf the same vulnerability is present across a large range of devices, a public exploit has a much larger impact. I'm not advocating for security by obscurity in the slightest because on balance I think it's bad, but we should acknowledge that publishing your source does change the potential cost of mounting an attack in various scenarios, and some of them might actually favor obscurity. Nobody except the most determined attacker will attack a device with some custom, unpublished code. On the other hand, popular software have a variety of exploits in the wild because their popularity makes them more attractive targets, one consequence of which is enabling script kiddies (since the hard work can be outsourced). > it gets patched within hours and it may take a day or two for it to be distributed to everyone This doesn't work for embedded devices. Hence why it might not be a great idea to publish their vulnerabilities, or tell the whole world that you're running on old vulnerable source.
- choward 10y agoNumber 2 isn't reason NOT to open up the code. You can't just claim because it doesn't do something that it's a good reason. Open source software isn't necessarily faster or more efficient either. So?
- woodruffw 10y agoEven if we assume that these arguments are sound (which they aren't), they do not appear to provide a proper justification for letting a human being die. Put another way: If you had to explain Marie Moe's death to one of her relatives, which of your three points would make the relative understand your position?
- niftich 10y agoThese are sound arguments. Medical devices have to be certified in all applicable jurisdictions and companies want to protect their competitive advantage. This is no different from, say, new drugs, which are similarly proprietary and the maker retains exclusivity for several years. Further, medications too are risky black-boxes towards end-users.
- woodruffw 10y agoThey are not sound in the context of human welfare. We do not allow food companies to hide the ingredients of their products, because we know that companies (in the interest of profit) will fail to inform consumers about the dangers of eating unhealthily. Similarly, we do not allow meat producers to leave their meat ungraded because it endangers their "competitive advantage" - we understand (empirically) that doing so leads to food contamination and otherwise preventable disease. Your choice of proprietary medicine as a counterexample is an interesting one. The pharmaceutical industry in the United States has a long history of "protecting its competitive advantage" at the cost of individual welfare - consider how frequently "reformulations" of the same base chemical are patented to continue milking a lucrative product that could improve the lives of thousands if genericized. Perhaps even more pointedly, consider the fact that we deem it acceptable (and necessary) that the FDA step in and regulate the release of new drugs. Is there a valuable distinction to be drawn between the sort of regulation and evaluation that the FDA does and the sort that would be possible if programmers could openly evaluate medical devices?
- mankash666 10y agoYour argument that opening up the source code might have prevented her death is facetious.
- morganvachon 10y agoThe code to the Jeep Grand Cherokee is closed source, yet security researchers (and now thieves) have been able to take complete control over the vehicle via simple protocol sniffing and reverse engineering. Closed source is no safer than open source.
- falcolas 10y ago1) The software in a pacemaker is not a competitive advantage. The hardware and rights to distribute to hospitals are. 2) Open and closed are equally vulnerable. Closed may be even more vulnerable, since security researches have to attack a black box and can't run ordinary code linting tools. 3) Liability after update is a good question, but one which can be (and is) solved with traditional contract agreements. See cpap machines.
- stordoff 10y agoNone of these are particularly compelling reasons IMO. > 1> Loss of competitive advantage Patient safety trumps business considerations. Skipping clinical trials would be a major competitive advantage (lower costs, quicker to market etc.), but we don't allow that for the same reason > 2> Open source is not necessarily any safer (heartbleed bug ... ) I'll agree, in so far as saying making it open source doesn't necessarily make it safer. However, it should, at worst, be no less safe - the requirements on the manufacturer (testing etc.) should be the same regardless > 3> If software for the pacemaker is allowed to be updated like that on a computer, someone will update it with buggy software that can cause adverse side effects. You can open the code but, for example, required a signed binary before a device can be updated. The openness of the code and the openness of the device itself are linked but separate issues.
- ikeboy 10y ago>Skipping clinical trials would be a major competitive advantage (lower costs, quicker to market etc.), but we don't allow that for the same reason About that ... Have you seen http://www.fdareview.org/05_harm.php http://www.fdareview.org/05_harm.php and http://www.fdareview.org/07_market_failure.php http://www.fdareview.org/07_market_failure.php?
- SubiculumCode 10y agoSure this is about pacemakers, but cant we say something similar about the rest of our body?
- kiba 10y agoUnfortunately, it's a black box made from nature that we have to reverse engineer.
- JoshTriplett 10y agoIt's also self-modifying code that bootstraps its own compiler.
- gravypod 10y agoYea it's got some cool genetic algorithms.
- jsmthrowaway 10y agoOne could frame the entire computing industry as a distributed genetic algorithm, executed by the real computers in order to understand themselves and the environment around them. One could further posit that we don't really have a good handle on the right fitness function yet. (I realize this sounds like a low-effort joke, but think about it for a second.)
- TeMPOraL 10y agoWell, computing today seems like primordial soup flowing through the pipes set up by Moloch[0] - we keep doing random shit, somewhat directed by economic incentives. [0] - http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
- ComodoHacker 10y agoAnd it's full of bugs and vulnerabilities our adversaries of all kinds are actively using. On the good side, bugs are fixed and vulnerabilities closed eventually. But the process is so slow... It spans numerous minor releases.
- maerF0x0 10y agoI agree, I want to contribute to FOSS medical devices.
- jordigh 10y agoAnother really good talk about the topic, "freedon in my heart" by Karen Sandler: https://www.youtube.com/watch?v=5XDTQLa3NjE https://www.youtube.com/watch?v=5XDTQLa3NjE It's mentioned in the article.
- JoshTriplett 10y agoI've heard Karen give this talk in person; it's quite moving.
- pserwylo 10y agoI can't recommend this talk enough. If you have the time, it is one the best keynotes I've ever been privileged enough to watch. Ever since then, I've been very interested in all the paperwork sent to my grand father about his pacemaker + defib implant. Recently they mentioned to him that they will send out a remote monitoring device and Karens talk immediately jumped to mind. Initially the documentation talked about how the remote monitoring device needed to be held close to the heart for a few seconds in order to download the relevant data. This gave me a modicum of hope that at the very least it was some sort of NFC type communication. This would at least make it harder to physically exploit. However, they continued to say that if you want it to monitor you every night, just put it within three metres of your bed while you sleep. I had a search online and there does not seem to be a lot of people particularly interested in this area, although repos such as https://github.com/openaps/decocare https://github.com/openaps/decocare give me hope in the ability of the community to reverse the relevant protocols and investigate these devices.
- pthreads 10y agoDoes anyone know if at least the FDA is allowed to review the source code for pacemakers? Or is it a complete blackbox? Personally I would be appalled if even the FDA is not allowed to.
- beambot 10y agoWith or without a warrant...? EDIT: Not sure what's up w/ the downvotes. There are well-established ways for regulatory agencies (whether FDA, FCC, etc) to obtain firmware for devices -- and it almost always involves a warrant under extraneous circumstances -vs- proactively receiving proprietary code.
- ben_jones 10y agoWhat if they had to subpoena a tomato grown in a field next to the toxic waste dump? What if they opted not to examine that tomato because they didn't have the resources to issue, process, and support, the lengthy bureaucratic process involved in such things?
- beambot 10y agoWho owns the tomato? Who owns the land that the tomato is grown on? You can't just willy nilly confiscate other's property...
- ben_jones 10y agoBut the government can and should have the right to inspect a tomato (not necessarily a specific tomato) if all the tomatoes in that field are slated to go direct to consumers, right? Similarly, what about testing for drug quality? You could even extrapolate it out to the SEC's right to examine a private financial transaction in order to determine legality. Or the IRS's right to inspect one's taxes to determine compliance. Point is (IMO) there is a need put on government by society to bypass some of our "Inalienable" rights. In most cases this societal decision is necessary and makes sense, and I'm arguing that code inspection of life-critical systems is a reasonable example of such a case.
- randyrand 10y agoBy extension should every device I own require me to have access to the source code and output data? Not a rhetorical question.
- dec0dedab0de 10y agoYes.
- falcolas 10y agoDo you want to control the device, or do you want someone else to control your devices? If you're OK with someone else (who probably doesn't have your well-being anywhere in their list of priorities) controlling all of your devices, then the answer is no, you are not required to have access to the source code and output data. EDIT: I didn't realize this was such a controversial statement. I stand by it, though; even as I knowingly own many such devices myself. I can certainly see, however, how I may want more control over a pacemaker than, say, my phone.
- krapp 10y agoIn theory, open sourcing pacemaker software make sense, but in practice, the pool of people qualified to review and edit that software may not be very large. Most people would still be depending on someone else to control their devices in any case.
- abraae 10y agoIf I had a pacemaker I would most definitely review the source code for it if I could get access to it. Even if it meant learning golang or ARM assembly or whatever the kids use to build pacemakers these days. I'm sure I'd lack a lot of the heart mechanics side of the equation, but I would be very highly incentivised to hunt out programming bugs.
- beagle3 10y agothe editor pool is small, but the reviewer pool is large.
- meric 10y agoPhilosophy - She doesn't know the code that's running on machines inside her body. I don't even know the code that's running my heart. And yet, I trust it.
- tyrust 10y agoThe latter has undergone far more testing than the former.
- mikestew 10y agoThe bug fix turnaround makes waterfall look downright lean, though.
- hippich 10y agoexcept code running your heart was built over millions of years and code she mentions - created by company in years - see the difference? :)
- projektir 10y agoWhich is funny because, nonetheless, she had to replace the code built over millions of years by code created in a few years.
- r3bl 10y agoWell, if the code built over millions of years is error prone, then surely the code created in the last few years is packed with errors. She found out what happens when one of them gets triggered.
- smallnamespace 10y agoExcept your code and her code are quite similar, so introspection has significant predictive value.
- pg_is_a_butt 10y ago10 be dumb 20 goto 10 you're all idiots.
- r3bl 10y agoCan the mods change the link to https://backchannel.com/our-medical-data-must-become-free-f6d533db6bed#.o436sohwc https://backchannel.com/our-medical-data-must-become-free-f6..., since this is the full version of the story?
- andrewclunn 10y ago"I wanna feel what code is. I know you can show me!"
- sctb 10y agoPlease don't do this here.
- yolesaber 10y agoI saw a talk about medical device security (or lack thereof) at the Eleventh Hope a few weekends ago. Very scary. They started off with a story about patients in a hospital who became horribly addicted to morphine because they were able to hack the machine from resources found online (http://www.massdevice.com/hospital-patient-hacks-his-own-morphine-pump-massdevicecom-call/ http://www.massdevice.com/hospital-patient-hacks-his-own-mor...). Go on Shodan and search for medical devices and terminology (e.g. "radiology") and you'll see the state of things. Sensitive machinery exposed on the open internet. A lot of medical devices have hardcoded passwords that are used for remote operations by technicians. Open sourcing this code would do a lot to mitigate these issues.
- cooper12 10y agoThe sad part is that the companies will use this security by obscurity argument against open sourcing.
- TickleSteve 10y agocontrary to popular opinion.... Obscurity is good practice as one layer of a layered defence system. See "Defence in Depth" https://en.wikipedia.org/wiki/Defense_in_depth_(computing) https://en.wikipedia.org/wiki/Defense_in_depth_(computing) "Defense in depth is originally a military strategy that seeks to delay rather than prevent the advance of an attacker by yielding space to buy time". We have to acknowledge that no system is perfect, there will always be holes, therefore a good approach is to layer up the imperfect systems which delays the attacker. Obscurity is one of those layers, a system will always be more secure if you have to find it first.
- nitrogen 10y agoTrue, but history has demonstrated countless times that closed source code doesn't provide near enough obscurity to deter hackers, and automated fuzzing tools make it even easier.
- 10y ago
- drdeadringer 10y agoI am imagining that young folks have Library Anxiety, and old folks have "Google Anxiety". Ask any question and you will find an answer. Any question you have, no matter how banal or left-field. What is the weather? Is my grandson a lesbian? How do I eat pizza in Italy? Where is the biography section? How do I understand the Dewey Decimal System? What is in the Special Collections, and what are the hours -- and do I need an appointment? The computers are down... is there a way I can search for books offline without randomly roaming the stacks?
- yostrovs 10y agoI want to inspect the blueprints of every building I walk into and know the sourcing and composition of all the structural components as well. For my life depends on these things to be true and properly constructed.
- 00098345 10y agoNot necessary because this is realised to a certain extent through safety standards. Certified earthquake proof. Done. Now, in this case, FDA approval should include software standards that prevent this scenario. Done*.
- yostrovs 10y agoEarthquake certifications are just like other certifications, in that most of the process is not actually reviewed by any government agency. They check blueprints, sign off on the process, but they don't actually make sure that the ibeams are connected correctly and that the angles are just right. Done.
- dtornabene 10y agoI'd like to read commments here without total strawmen derailing the conversation to nonsense. Do you have anything constructive to add?
- yostrovs 10y agoDo you?
- radicality 10y agoHow about when I'm flying an airplane; I'm also putting my life in the hands of people that wrote the code that controls it and I have to trust that the plane won't shut itself down mid-flight because of faulty code. Should a similar argument be made here?
- korethr 10y agoPerhaps you are being sarcastic, but I shall attempt to answer the question earnestly anyway. I've been reading up on aviation regs due to a recent interest in getting a pilot's license. By my understanding, airplanes certified by the FAA as airworthy undergo some fairly heavy testing to exactly determine and prove what their capabilities and limits are. Given that getting your prototype wrong can cause the plane to crash and kill your test pilot, there's incentive to get this stuff right. Furthermore, once a plane is type certified, it can't be modified from that configuration without further testing to prove the modified configuration. Thus you'd better be damned certain that your engine control computers are correct, lest the FAA revoke the plane's airworthiness certificate, making that model unsellable, nevermind the lawsuits of the survivors of deceased passengers or insurance companies recouping their losses. Additionally, from what I've seen, most general aviation airplanes are stuck in the 60s as far as engine tech goes, in part because of the strict FAA regs. We're talking air-cooled engines with carburettors here, no engine computer to speak of, or if you're running a fancy modern engine, mechanical fuel injection. Unless you're flying a brand new Cirrus SR22 or Diamond DA20, there's no code to inspect. Even if there was, it'd be in the avionics. You don't need a nav beaon, radio, or transponder to land an airplane safely -- though they can make it much easier, and the FAA is going to want to know what happened once you're back on the ground. As far is large passenger jets? I'd say as a passenger, no, you don't have access to the code. It's not your airplane; it belongs to the carrier. Personally, if I were to own a plane and fly it, I very much would want access to any and all code that makes the plane go. Though if I'm going to be hacking on said code, I suppose that's what the FAA's Experimental category is for.
- amelius 10y ago> Furthermore, once a plane is type certified, it can't be modified from that configuration without further testing to prove the modified configuration. Would this work similarly with autonomous cars? For example, what if google wants to change one line of code in their car?
- TickleSteve 10y agoYou most certainly don't want people to be able to modify safety critical code within a pacemaker. What most developers don't realise is the level of engineering strictness that goes into anything safety-related. The rules and regulations related to anything that affects the human body is in a different league than what most developers are familiar with. What is a problem here, is that the design (not the code) apparently did not take into account any messaging security, relying on obscurity as its only defence. If the code was open-sourced, don't expect to find lots of buffer overflow attack vectors, or simple things like that. Its the design of the system as a whole at fault, and that is already open. Medical devices such as these are not black boxes to the people that certify them, everything is open to them, source included. Having worked in that sort of area, I trust the systems that are in place.
- adrianN 10y agoIndustrial control systems are also safety critical systems that have to adhere to very similar regulations as medical systems. Yet, when you ask hackers in that field you'll quickly learn that they have terrible code and abysmal security. Rules and regulations do very little to improve code quality and security, imho. Most things in these regulations are either best practices that any software engineer does (write tests, for example, or do code reviews), or just checkboxes for the QA to check. I'd argue that over time safety critical systems have worse code than normal software because refactoring is almost impossibly expensive due to all the paperwork that each software change ensues.
- flukus 10y ago>Rules and regulations do very little to improve code quality and security, imho. In fact, I'd say the restrict them. Adding a tonne of paperwork is a great way to prevent people from "noticing" problems.
- TickleSteve 10y agoWhen the paperwork is describing the system design from high level down to low-level to sufficient detail that an external auditor can use it to examine the source then no, that is not true. The security/safety processes (as applicable) are examined at every level from system-design all the way down to implementation. Its not perfect, but its best practice we have for secure/safe product development.
- TickleSteve 10y agoPlease people.... before you comment on this thread, please inform yourself on what safety critical software is really like. There is a lot of uninformed discussion in here currently. https://en.wikipedia.org/wiki/Life-critical_system https://en.wikipedia.org/wiki/Life-critical_system
- vog 10y agoObligatory talk on that topic from the 32C3: "Unpatchable - Living with a vulnerable implanted device" https://media.ccc.de/v/32c3-7273-unpatchable https://media.ccc.de/v/32c3-7273-unpatchable
- grondilu 10y ago> “You’re pulling data from my cardiac device that I paid for, implanted inside my body, the most intimate piece of technology anyone can have, and yet I’m devoid of access to the device? That moved me to my core,” he says. “That’s just not right.” I'm sure she must have signed a user license agreement of some kind upon buying the device. So she shouldn't have to complain.
- tim333 10y agoSounds like they could do with a law similar to the freedom of information ones for software of this type. Without that device manufacturers are not going to want to publish as competitors could copy it and some people may sue over perceived errors.
- 3chelon 10y agoWe hear a lot about how digital obsolescence is a growing problem, and almost all of it refers to not being able to access your old family photos and movies, or maybe old documents and spreadsheets. But what happens when your pacemaker is obsolete, the source code is long lost, and no-one knows how to update it? Is this problem being addressed in any real way? 50 years in the future some of today's devices may still be operating in peoples' bodies, and it seems hard to believe that anyone would still have the knowledge and/or tools to upgrade them. And surely it's quite a big deal to open someone up to replace the hardware every 5 years?
- etendue 10y agoThe pacemaker or ICD generator is replaced when the battery is exhausted, typically 8–10 years. The procedure is not a big deal, it is commonly outpatient and done under local. Outside of some durable orthopedic implants, few implants will survive in the body for 50 years: it is a very hostile environment.
- abandonliberty 10y ago>it is a very hostile environment. I've never thought of it this way, and you are right from both a technological and biological perspective. Biologically we are wonderful containers of nutrients, but we have an army only the very sneaky or militant can overcome. Once that army stands down we are rapidly colonized - which is why we must be so careful with food/meat storage.
- Apofis 10y agoYou just made me realize why bacteria and other things want to kill us so badly... we are precisely that, really extremely-high concentration stores of nutrients.
- 3chelon 10y agoUseful to know. Of course, batteries get better and electronics gets lower powered - I can imagine in the near future implanted devices being effectively passive (such as RFID)... but your point about the body being a hostile environment is definitely something I hadn't considered before.