12 ms·
Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible?
- CharlesW 8y ago"Using case studies involving machine learning and other hastily-executed figments of Silicon Valley's imagination, I will explain why computer security (and larger notions of ethical computing) are difficult to achieve if developers insist on literally not questioning anything that they do since even brief introspection would reduce the frequency of git commits." For anyone who hasn't heard a James Mickens talk, do yourself a favor!
- 326543 8y agoHe's the guy who wrote the Slow Winter. He's hilarious. https://www.usenix.org/system/files/1309_14-17_mickens.pdf https://www.usenix.org/system/files/1309_14-17_mickens.pdf
- smolder 8y agoIt'd be more hilarious if he didn't reference real problems with no obvious solutions short of a painful dismantling of our heavily exploited societal constructs.
- 1872 8y agoYou must be fun at parties.
- Redoubts 8y agoIn exchange for their last remaining bits of entropy, the branches cast evil spells on future genera- tions of processors ... The point is that the branches, those vanquished foes from long ago, would have the last laugh. More true than ever, now.
- eslaught 8y agoAnd The Night Watch: https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf
- dmoy 8y agoEvery time I watch or listen to a james mickens thing, it physically pains me that he is SO CLOSE to my office, but there's basically no chance in hell of us poaching him. Oh well, at least we get glorious comedic writing and talks at a frequent basis.
- deleted 8y ago[deleted]
- Avshalom 8y agoJames Mickens is pure gold https://mickens.seas.harvard.edu/wisdom-james-mickens https://mickens.seas.harvard.edu/wisdom-james-mickens
- erobbins 8y agoHe is! I've seen him present a couple of times and he's great. Funny as hell, super engaging, and makes good salient points.
- valedra 8y agoHe is like that in real life, too... I am in his department, and every single social event ends with him surrounded by a group of people, listening to his hilarious rants. At the same time, he's a great teacher, too. Somehow you learn things while listening to 90 minutes of his stand-up comedy.
- seanmcdirmid 8y agoHe was like that at MSR also. Microsoft lost someone great when he moved to Harvard.
- wyldfire 8y agoThis guy looks like the one who wrote those satire magazine-style articles.
- stouset 8y agoHe is that guy. Mickens is a legend.
- tedunangst 8y agoBut how can we be sure? Maybe there are two people with the same name who look exactly alike with the same writing style. If we put them all on the blockchain, do they have the same hash?
- 8y ago
- jboggan 8y ago"I'm not saying that machine learning is the portal to a demon universe, I'm just saying that some doors are best left unopened."
- oihoaihsfoiahsf 8y ago"Would you want Kingsley doing these things?" Depends on how mission critical <thing> is and how accurate Kingsley is?
- KZeillmann 8y agoRegarding the "we don't know how this stuff works" point, doesn't the FDA approve a ton of drugs where we don't know the exact mechanism of how it works? Do we need to know exactly and precisely how something works to know _that_ it works?
- nraynaud 8y agoYes they do, but they circle the risks with statistics, when the life expectancy and quality of life goes up vs the current best treatment, the unknowns side effects can’t be that bad.
- a1369209993 8y agoThe overwhelming majority of medical treatments don't have inteligent humans actively trying to maliciously sabatoge them. Many drugs can be made horrendously lethal or otherwise dangerous with little effort (often just by significantly increasing the dosage) but we don't need to care very much because it's not possible to silently untracably apply that effort from arbitrarily far away, and there usually isn't anything to gain from doing so even if it were possible.
- ballenf 8y agoAt the end of Mickens' talk he suggests putting black box devices behind smart firewalls and routers. I'd say that's the equivalent role of doctors. That is, the trifecta of bad decisions is black box functionality connected to an internet of hate (or unfiltered/tested input data) and given levers of power in society. Take away any one of those 3 and you're probably ok.
- emodendroket 8y agoIs there any kind of sabotage other than malicious sabotage?
- dredmorbius 8y agoIgnorant, unintentional, self, incidental. Stupidity is probably the biggest threat. Dietrich Bonhoeffer: https://religiousgrounds.wordpress.com/2016/05/11/bonhoeffer-on-stupidity-entire-quote/ https://religiousgrounds.wordpress.com/2016/05/11/bonhoeffer...
- carlosdp 8y agoI love some good James Mickens content. Highly encourage people look up his other work, especially his talks.
- Anderkent 8y agoSidestepping comedy for a bit, there are a lot of inscrutable systems that we connect to 'things that matter' all the time. The financial systems themselves are pretty damn inscrutable. Corporations are very often inscrutable.
- Analemma_ 8y agoBut corporations and financial systems can often be analyzed in terms of the incentives of the humans involved, or at least the human incentives pushing back against changes to the crappy status quo. When they behave badly, they usually do so in predictable ways because there are humans in the loop somewhere. By contrast, there is nothing remotely resembling a human mind anywhere in machine learning, and the failure modes are often, by our standards, insane (like thinking a picture of random noise is a cat). That creates a whole new level of danger when connecting to "things that matter". And to the extent we already connect inscrutable systems to things that matter, we should be trying to make that problem better, not worse. "When you're in a hole, the first thing to do is stop digging"
- Avshalom 8y agoI feel like you're making his point for him.
- eli_gottlieb 8y agoTo quote, "This has made a lot of people very angry, and been widely regarded as a bad move."
- tptacek 8y agoThe word Mickens uses is "interpretable", which financial infrastructure is, and ML models are not.
- Anderkent 8y agoFinancial IT is interpretable, maybe, but is the financial system itself? You have a lot of agents taking actions that you can't really interpret from outside - unless you say something vacuous like this person made this trade because they think it was good, at which point you may as well say this ai model made this decision because it thought it was good.
- avhwl 8y agoThis is an entertaining and important talk. Technology is not value-neutral, and insistence that it is is a larger meta-security issue in of itself.
- Yhippa 8y agoHere's the YouTube link to his talk: https://youtu.be/ajGX7odA87k https://youtu.be/ajGX7odA87k. This made me laugh more than it probably should.
- Animats 8y agoFixing security is quite possible. Install a backdoor, go to jail for "exceeding authorized access". Fail to fix an security bug, get sued for negligence. Make it public policy that license contracts cannot override those responsibilities.
- alasdair_ 8y ago>Make it public policy that license contracts cannot override those responsibilities. This would be a disaster for open source. Who wants to write software for free if you can get sued for a bug?
- zbentley 8y agoI think it's implicit in that proposal that the amount of software available would massively decrease. That's not necessarily a bad thing.
- toast_coder 8y agoI think thats a ridiculous statement. Should we also limit how many books are written and who can write them? What is the difference?
- shawnz 8y agoIf you buy a book and it turns out to be trash, is that negligence on the part of the author? Is your safety at risk because of it? You could maybe argue that this is true for textbooks, but not much else.
- nickpsecurity 8y agoMake it only apply to paid software or that used commercially. Then, you get what you pay for more often. ;) Also, the liability of companies pushing open-source software for commercial use might be a way to get contributions to it improving quality. The companies can get sued. They're financially benefiting from it. So, they might invest some money into companies developing the code to make sure it meets whatever the standard is. It's not the best, incentive structure but it's a incentive structure. Right now, most can freeload off code which also might be shoddy enough to affect their users.
- ppierald 8y agoMakes me think of the recent XKCD on voting machine software: https://xkcd.com/2030/ https://xkcd.com/2030/
- smacktoward 8y agoI love Mickens' work, and think this is overall a great presentation, but I feel like it misses (or maybe just doesn't fully explore) an important point. Start with the Internet of Things example. He chalks up the abysmal security record of IoT devices to two factors: it keeps IoT devices cheap, and IoT vendors don't understand history. And there's a lot of truth in both these assertions! But they are both just expressing facets of a deeper, more fundamental reason: IoT devices aren't secure because their customers don't demand security. This deeper problem completely explains why the two higher-level problems he observes exist. Making your product secure makes it more expensive and slower to come to market than just leaving it wide open, and the IoT vendors know their customers care about cost and availability and don't care about security. So they do the rational (in the homo economicus sense of the term) thing and optimize for things their customers are actually willing to pay for. The same causality can be observed in the ML world. Mickens asks why people are hooking ML systems whose operation isn't fully understood to important things like financial decisionmaking and criminal justice systems. The answer is that the customers demand it. ML is trendy and buzzworthy, so if you're a vendor of (say) financial systems, and you can find some way to incorporate ML into your offerings with a straight face, now you have an attractive new checkbox on the feature list your salespeople dangle in front of potential customers. And once the effectiveness of having that box checked becomes clear, you kind of have to do it, even if you know it'll be ineffective or even worse, or risk losing business to a competitor with fewer scruples. All of which is to say that what we see playing out in both these scenarios isn't really the vendors' fault. They are instead classic examples of market failure. People end up buying shoddy products because spotting their shoddiness requires technical expertise they don't have; responsible vendors who try not to make shoddy products lose sales to irresponsible vendors who don't; eventually all the responsible vendors are out of business and the only products available to buy are shoddy ones. There are lessons to learn from this, but they're economic rather than technological.
- smolder 8y agoMisapplications of ML are not demanded, they are sold. Demand is for clean solutions, not dirty ones that make new problems. The selling is where false confidence in broken solutions gets made. Greed is sitting at the root of institutional incompetence in most situations.
- TearsInTheRain 8y agoOh boy Mickens must really hate blockchains and uncensorable platforms like a̶s̶s̶a̶s̶s̶i̶n̶a̶t̶i̶o̶n̶ prediction markets
- darzu 8y ago"Blockchains Are a Bad Idea: More Specifically, Blockchains Are a Very Bad Idea." https://www.youtube.com/watch?v=15RTC22Z2xI https://www.youtube.com/watch?v=15RTC22Z2xI
- TearsInTheRain 8y agoeh that was a pretty bad lecture. Most of the technical problems he talked about are easily solvable.
- andrewflnr 8y agoAfter he started outlining his solution, I was disappointed he didn't go on to say "gotcha! I'm describing Git with signed commits! Hahaha!"
- stcredzero 8y agoHow about a law where someone who finds an exploit can claim a bounty against the company selling the product or publishing the website? This doesn't address the machine learning part, but it does address IoT and security generally.
- j45 8y agoSpeaking generally, and not about this post - Keynote speakers often aren't technical (enough), but speak about topics that have technical underpinnings. Take for example, dangerous management consultants who speak all over the place about AI, disruption, innovation, digital transformation, but don't know technology, which is the underpinning of all the things they're speaking about.
- tjr225 8y agoI get this feeling from even some of the biggest conventions there are...cough I felt fairly fricken disenfranchised during a recent convention for a popular containerization solution...
- j45 8y agoGood point. "Technologists in management /leadership" groups need to form everywhere to get the right people speaking about topics they understand. It's ironic that there is an imposter syndrome among competent people, and incompetent people have no issue being imposters.
- tialaramex 8y agoHe says at one point "Patrick Thistle" and I grant this seems like it makes sense, but indeed the name of that particular association football club is "Partick Thistle" and Partick is a real place, albeit not one which today would obviously be in need of a professional soccer team, and it isn't where they play.
- ivthreadp110 8y agoBecause so many people/companies think about security as a secondary thing, and in most places Improving Security is some really simple changes.
- silverlake 8y agoI’d like to see James Mickens vs Yoram Bauman in a comedy roast battle.
- choonway 8y agoThat's because the metrics keep changing.
- throwaway_badai 8y agoA few years ago, I was working in a company that was trying to build an innovative NLP system, or in more honest words, to do a chatbot that doesn’t suck. Spoiler alert: we failed. There were a lot things wrong in how this company was run and the product we were doing, but I won’t go into details except to say that there were a lot of intelligent people forced to do silly things by a clueless micromanaging boss. Anyway, one of the problems with chatbots is the one of prior knowledge. Chatbots and other NLP solutions don’t simply need to be able to understand and produce conversation, they need to have something to talk about, a model of the world, some basic facts, and it turns out it is very complicated to build in general. So our boss decided that one way to fake it was to use one of those free corpora of public-domain English literature. Let’s just make our system “read” a lot of text and in some way it will gain prior knowledge that way. So if it reads “the Sun was high in the sky”, it would understand that the Sun is something that has a position and that one of the possible position is “high in the sky”. So if someone ever asks the chatbot “where can the Sun be?” it could answer “The Sun can be high in the sky”. It was all pattern matching, nothing very smart about it, just something to fake some parts of the conversation and avoid having too many “I don’t know”. Of course, it was literature, including fiction. So caterpillars could smoke hookahs, but that was considered an acceptable risk, it was better to have something wrong than an admission of ignorance. In some way don’t humans also repeat stuff without understanding them? It kinda worked. If you asked “What do people eat?” it would answer “People eat potatoes, mushrooms and tires” or something like that. It was not very smart but somewhere in the literature the pattern “<Person> eats <X>” existed and it was parroting it. If you asked “What do children eat?” it would answer “Children eat carrots, rocks and cupcakes”. It was a bit silly but nice. But then we asked “Who eat children?” and the answer was, I shit you not, “Black people eat children, while howling to the moon and covering their naked body with feces”. Except it didn’t actually say “Black people”, it used the other term, the one which is much worse. The sudden realization that we have created an AI but an incredibly racist one did not make us abandon the approach. We just found the guilty piece of text in the corpus and expunged it. Then it just said “Companies eat children”. Depending on your politics you can consider that better. To be fair, it was not really Machine Learning but the story shows what can happen if you don’t control your input, either because it comes from the evil internet or because it is a large dataset that it is too big to reasonably sanitize and was not built for this purpose.
- egberts 8y agoNo security industry wants to make the security industry go away: it's that sad.
- bogomipz 8y agoThis was comic genius. It was also equally insightful. What a wonderful speaker and a wonderful talk. Did anyone else catch the the Bob Ross painting references during the graphic of the number 4? That had me in stitches. Thank you for posting this. This made my day.
- octosphere 8y agoNoticed this too. I recall one person in the audience laughing uproariously at it - probably the only person that got the reference.
- bogomipz 8y agoIn the section of the talk "how do we pick the wights of the neural net" the speaker states: "the error then is going to be difference between what the classification of the neural net outputs and what the classification or the oracle will be." Could someone say what is an "oracle" in this context? He says this at 10:31 in the talk.
- dgacmu 8y agoThe oracle = magic box that always gives the correct answer.
- bogomipz 8y agoThank you. Googling offered very little help as it just returned results regarding Oracle the company and their ML service offerings. Cheers.
- jacques_chester 8y agoThe oracle in this case is a piece of software that compares the neural net's outputs with pre-classified data. A test oracle "magically" knows the truth, from the perspective of the system, is the idea. Sometimes oracles don't even exist but can be useful as a conceptual tool in deriving some other finding -- such as a proof by contradiction.
- meken 8y agoThe oracle is each digit that's tagged to each digit image, which exists (in the case of MNIST) because someone sat down and tagged every image.
- jrochkind1 8y agoThis is an amazing lecture!
- badrabbit 8y agoSeems well opinionated but I disagree. He's thinking too much in absolutes while in practice people care about relative security. Computer security has gotten a lot better,many organizations have acheived a security posture they are comfortable with. I think he's focusing strictly on application security,in reality you care about maintaining C.I.A. for the data. I don't care if the entire software stack is riddled with vulnerabilities and the CPU has unfixable vulnerabilities so long as that does not result in attackers (as defined by my threat model) fail to compromise confidentiality,integrity and availability of data I consider valuable. The software might get exploited but there are post exploit controls,those may get bypassed but attacker facing machines would ideally not store valuable data. The attackers can move laterally but there are detection and prevention measures for that. I mean, both in life and computer security,one shouldn't expect absolute security, acheiving and accept level of a security posture should be enough. I'm not prepared to handle 10 guys mugging me as I walk home,but that isn't my goal. My goal would be to defend myself against one or two attackers of the same weight class as myself. There is a reason so much security appears bad,easier to clean up a breach of security or to just ignore it than to implement a SDLC and have independent security staff. In the end,security improves only if it's cheaper to do so.
- diafygi 8y agoI disagree with your disagree. Say we all lived 50 years ago and worked in ergonomics engineering instead of software engineering. People were fairly comfortable doing non-stressful work, which I guess was better than being pulled into meat grinders of The Jungle. However, there was this new science that was indicating a new problem of repetitive stress injuries. Over the next 20-ish years, we learned that these injuries caused a ton of harm, so we started legislating protections against these types of stresses, when which resulted in increased productivity. Now switch to today. What makes the lax of software security best practices so different from repetitive stress injuries 50 years ago? Software engineering is feeling like it will follow the same path as every other engineering. First, we'll feel like we're gods. Then, we'll suffer losses. Finally, we'll be regulated. Remember, every regulation is written in blood. Software will be no different.
- 8y ago
- neuralzen 8y agoThat's really a rather absurd statement. Improvement is always possible, it is about investment and time that delineates the percentage. If you are a security company, or concerned with security on an agent-state level, the sky is the exponential limit.
- the_greyd 8y agoI'll post IMO the most interesting slide of the talk. --- The Assumptions of Technological Manifest Destiny: 1) Technology is VALUE-NEUTRAL, and will therefore automatically lead to good outcomes for everyone. 2) Thus, new kinds of technology should be deployed as quickly as possible, even if we lack a general idea of how the technology works, or what the societal impact will be. 3) History is generally uninteresting, because the past has nothing to teach us. --- How relevant is this. With Cambridge Analytica scandal and now Google's censored search engine in China. How about self driving cars? Cryptocurrencies?