7 ms·
A bitter pill for Microsoft Copilot
- ThrowawayR2 5y agoThe fly in the ointment being that, if such useless loops are widely adopted, ML can be trained to recognize and eliminate them too. (Don't shoot the messenger; the concept is obvious.)
- shadowgovt 5y agoYou could train one to recognize common patterns, and you could train one to heuristically guess that an algo running on test inputs is looping forever if it takes too long, but actually dividing the world into "loops forever" and "does not" is the Halting Problem, and our solution is guaranteed to either false-positive or false-negative. (Anecdote: I heard a story once about a college that transitioned from free printing to metered printing, but since the printers didn't support metering, they implemented it by running print jobs in a virtual machine and tallying page count there. It took not very much time for some compsci student to cobble together a script that would augment a Postscript file with an "if my environment is virtual, print one blank page and exit, else..." prefix).
- ThrowawayR2 5y agoAs you point out elsewhere, there's a human in the loop. Though the Halting Problem applies to human minds as well, they likely won't have any difficulty identifying useless code the ML doesn't pick up and adding it to the training set.
- Qwertious 5y agoThe next step is to integrate actual parts of the software into the poison-pill, so that if the poison-pill is not included then the software doesn't work. It isn't necessarily a good idea, but it's the next obvious step.
- shadowgovt 5y agoHard to make that work with snippets as small as Copilot is synthesizing.
- sp332 5y agoThat's true if the same mitigations often show up with the same contexts, like the simplistic one in the example. Making a general mitigation-avoiding AI would require it to understand the intention of the code.
- mtlynch 5y agoI've never learned Haskell, so I'm maybe misunderstanding this, but it seems like this wouldn't cause problems for Copilot. My understanding is that the author is effectively booby-trapping the entrypoints into their applications so that if someone runs the same code without the right "antidote" code extensions, it will go into an infinite loop. But that's only effective if Copilot picks up that one line with the booby trap. And Copilot probably wouldn't ever suggest that line because it's so different than the entrypoint to most Haskell applications.
- shadowgovt 5y agoIndeed. And it wouldn't take much testing to realize the program is broken. This solution ignores the human in the loop at the controls of Copilot.
- sounds 5y agoI wouldn't underestimate the coming arms race though, as developers find newer and more subtle poison pills to prevent rogue AIs from ingesting their code and regurgitating it later.
- ThrowawayR2 5y agoIf someone is poisoning and obfuscating their own code base, they become the only person who can contribute to it, defeating the point of being FOSS and benefiting from community contributions and review.
- spijdar 5y agoThis is being downvoted, but it touches on a really interesting phenomena we may eventually see, a sort of "arms race" where people try to make their code useless to machine learning, but simultaneously make it useless to people with lower skills (people blindly copying/pasting code with little understanding types). As the machine learning becomes "smarter", this bar will raise, and more people will be coincidentally excluded in an attempt to keep unwanted algorithms from sucking up code. Assuming this sort of poison-pill thing ever becomes commonplace.
- nullc 5y agoFunny that the machine learning operators seem to learn less well than the machines. https://spectrum.ieee.org/tech-talk/artificial-intelligence/machine-learning/in-2016-microsofts-racist-chatbot-revealed-the-dangers-of-online-conversation https://spectrum.ieee.org/tech-talk/artificial-intelligence/...
- krono 5y agoProviding the AI with a sabotaged code dataset, I wonder how that will influence its treatment of us meatbags down the line :) Also, I'd like to propose that henceforth, we refer to GitHub's mascot "Octocat" as "Copycat". All those in favour...
- DoktorDelta 5y agoRoko's Basilisk will remember this
- nemosaltat 5y agoAnd now, I’m remembering Roko’s Basilisk, thanks.
- wizzwizz4 5y agoDon't worry; Roko's Basilisk has no incentive to torture future-you. You can't predict it well enough to know whether it will, so it having the general policy of “follow through on the torture” won't help it – and will, in fact, just waste resources. (Unless it inherently cares about torturing people, but nobody has any incentive to build such a thing – and you should be scared enough of whoever's trying even if they don't succeed.) Roko's Basilisk only works if you're a superintelligence willing to risk being counterfactually blackmailed in this way. Which you're not. You're safe.
- krono 5y agoWas thinking about that as I wrote it. Just hoping it will keep me around as a pet, or that either the governator or Emilia Clarke come barging in telling me to come with them if I want to live.
- krono 5y agoAbout 4 hours after you posted this, one of those horrifying scenarios built out of a joke in my previous post suddenly manifests itself in my head. Roko's Basilisk is no joke.
- eplanit 5y ago
- jscheel 5y agoI think Github should support some kind of variant of robots.txt in the source code. Maybe a common line in the license file that excludes it from ML training data. That said, if you want to exclude it, that's your right, but you should consider why you are open sourcing in the first place, and what about ML feels different to you than traditional human cognition.
- Lev1a 5y agoWhy should the users be forced to do extra work in order for Github to do their commercialization "correctly"? This is something Github/Microsoft started, something that only really benefits Github/Microsoft and that they were doing lackadaisical from the start by ignoring licenses.
- ghaff 5y agoYou could say the same thing of people with a website that they don't want to be archived on the Internet Archive or indexed by search engines.
- sp332 5y agoGithub ignored licenses for this project and included code from all publicly-accessible repos. They claim it's not copyright infringement and you can e.g. use code from this AI trained on open-source repos in closed-source projects.
- elcomet 5y agoIt's not as simple as that... It's not clear to me how it is copyright infringement if the generated code is truly new. Of course, if some code output is very similar to existing code, then it is obviously copyrighted. When you read code from an open source GPL repo and are inspired by it to write your own code, it's obviously not copyrighted.
- pittmajp 5y agoWhen I read code, I understand it, its intentions, and its output. I'm not simply replicating things that I've read before. I learned about inheritance in java in school. When I go to write python, I need only look up the syntax because I understand the logic.
- yepthatsreality 5y agoMicrosoft GitHub Copilot regardless of legality to me is a direct attack on OSS in an EEE manner. It is an abuse of the platform. It should have been opt-in. It pushed me ahead to remove my repos from GitHub over to Gitlab (love that instant import feature), repos will be private by default. I will be deleting my Microsoft accounts too. Already off Windows. I also think the software development community should consider poisoning the well, even if it’s already legally poisoned.
- maxerickson 5y agoMoving to Gitlab wouldn't stop them pulling code into the model would it? I get that it still makes a statement.
- dreamcompiler 5y agoIt would if the repo is private. But that defeats the purpose of OSS.
- yepthatsreality 5y agoVoting with your wallet for extra points.
- pjc50 5y agoSomeone should definitely sue them over this, to determine if the model is a "derived work" within the meaning of the GPL. Someone should also release a model trained on the leaked Windows source code ..
- 542458 5y agoPeople have sued over similar stuff in the past, like Google training their language model on copyrighted materials. The legal consensus is that a model trained on a large volume of copyrighted works does not infringe copyright, even if it occasionally vomits up chunks of the input model. That said, if you trained a model specifically designed to violate copyright I suspect the courts would have a different opinion. They consider intent just as much as (if not more than) what you actually did.
- davemp 5y agoIt seems like it would be trivial to poison the well so to speak for common function prototypes to add known vulnerabilities that can be exploited later when needed. I would be exceedingly cautious about running code inserted by a black box fed by public inputs. Similar attacks have already occurred for NPM and I'm sure many other packagers.
- lawwantsin17 5y agoYeah, just keep it in the news more. I'd use Co-pilot as an anti-indicator for hiring.
- bradleyjg 5y agoSomehow I don’t think copilot is going to be especially popular in the professional Haskell programming community.
- kerng 5y agoI find the Copilot and AI discussions quite interesting. On the one side its indeed problematic that Microsoft (and others in the industry in general) hardly ever ask for permission for data to train models. On the other side, if you consider AI and computers to be similar to humans, then it's not an issue at all because humans also learn by reading code or information from various sources that are copyrighted or stuff they havent paid for directly (eg a library or "I'm just browsing in a book store" are an interesting for example, the later some books stores actually dont like). For poisoning the well kind of tactics, the author forgets that you poison the well for everyone - not just ML. An issue not brought into the mix with Copilot is that computers are superior in their memory and the amount of learning they can do (even now already in specialized domains), and hence maybe the fear, long term issues (incl ethical ones) and people being upset should actually come from that angle. You know, a computer that can reprogram itself seems like maybe a bad idea.
- mplanchard 5y agoYeah I have seen a lot of people bring up the question of essentially how training an AI to regurgitate code snippets is different from a human reading through and learning how to write similar code. To me they seem obviously and intrinsically different, and I think your post gets at some of the reasons. For one, an AI can ingest a much larger quantity of data much faster than a human. No human could possibly read all of the public code on GitHub in one lifetime. But more importantly, the method of ingestion and synthesis, as well as the nature of the application of what’s learned, is vastly different. Generally a human reading through OSS code to learn isn’t going to then permanently store verbatim segments of it in their memory to regurgitate later. They may copy a bit of code while learning, but that then morphs into an independent understanding of the type of problem being solved and one way to approach that problem. Copilot OTOH has no capacity to understand different approaches to solving a problem, to compare one against the other, and so on. All it knows is which is used more frequently, with no capacity to map that against metrics of long term maintenance, CI suite success, readability, and so on. As for the poison pill problem, I think it actually illustrates the difference quite well. A simple comment explaining that the code is a booby trap would be enough for any human reading the code to know how to avoid it. The AI on the other hand has no capacity to understand that comment and to know not to put the code into its dataset.
- deleted 5y ago[deleted]
- habibur 5y agoIt's time build an open source free co-pilot. When people are actively contributing to an AI system, my guess is that it will fare good against rot scanned ones.
- mapgrep 5y ago[programmers, for years and years] Hi we built a huge industry around pointing AIs at normies’ comms and renting it out to advertisers etc. This is very legal and you are Bad if you try to fight it copyright grounds. Indexing and downloading stuff is a basic human right and always Fair Use! [programmers when you do the exact same thing to their publicly shared code] How dare you, this is grounds for vigilante justice!11!1
- draw_down 5y agoNo kidding. But, all’s fair in love and war. If I publish some code and your AI bot thing happens to choke on it.... write a better bot, don’t blame me.
- hn_throwaway_99 5y agoOk, let's be real, this is just plain stupid: 1. This is worse than cutting off your nose to spite your face. I can easily imagine some config being messed up, and then the author shoots themselves in the foot. Also, god forbid anyone else have maintain the Rube Goldberg machine you've decided to create. 2.The idea that this could provide some significant dent to Copilot's utility is laughable. First off, it's not hard to see how AI could catch stuff like this. But more importantly, Copilot is clearly and obviously not intended to just be paste and apply. Every comment I've seen from alpha testers has stated how Copilot is uncannily good, but all the code must still be reviewed and it is oftentimes just flat out off. Code like this that would fail immediately would easily be identified. I understand the arguments over Copilot, but this is just a dumb response to it.
- zzodin 5y agoThis. At the end of the day source code is for human readability. If you pollute your code with a poison pill you just make it harder for other humans to use. ML Systems are doing sequence analysis and looking to predict the next token, for an arbitrary length of phrase. You've poisoned your code but compared to the volume of GitHub or any other public code data set it becomes the equivalent of noise. If you create a movement that puts poison pills on all Haskell files, now every project has painful boilerplate and an AI will learn to produce the same code. And every human that deals with it won't see a difference because you created a Haskell standard.
- danidiaz 5y agoIt's interesting to compare Copilot with Wingman for Haskell: https://youtu.be/S0HvfXq3454?t=550 https://youtu.be/S0HvfXq3454?t=550 Both attempts at somewhat automating repetitive aspects of programming, but with completely different approaches. Wingman was inspired by tactic metaprogramming in theorem provers like Coq.
- oauea 5y agoToxic idiocy. I know I'll probably be downvoted for saying this, but it's the only accurate and honest response I can come up with to this post.
- AlexAndScripts 5y agoWhy?
- thesuperbigfrog 5y agoIt seems like the author is advocating for the sabotaging Microsoft Copilot by creating GitHub repos of misbehaving code. This could be seen as a form of protest or as a Luddite (https://en.wikipedia.org/wiki/Luddite https://en.wikipedia.org/wiki/Luddite) reaction to AI automation of coding. James Micken's warning against connecting machine learning to the Internet of Hate still apply: https://youtu.be/ajGX7odA87k?t=1206 https://youtu.be/ajGX7odA87k?t=1206
- spixy 5y agoIt is GitHub Copilot, not Microsoft Copilot... I understand that it may feel good to be mad at large corporations but GitHub Inc. is still independent from MS.