6 ms·
At this point I can only hope that all these LLM products get exploited so massively and damning-ly that all credibility in them evaporates, before that misplac
by Terr_ 2y ago
At this point I can only hope that all these LLM products get exploited so massively and damning-ly that all credibility in them evaporates, before that misplaced trust causes too much insidious damage to everybody else.
I don't want to live in a world where some attacker can craft juuuust the right thing somewhere on the internet in white-on-white text that primes the big word-association-machine to do stuff like:
(A) Helpfully" display links/images where the URL is exfiltrating data from the current user's conversation.
(B) Confidently slandering a target individual (or group) as convicted of murder, suggesting that police ought to shoot first in order to protect their own lives.
(C) Responding that the attacker is a very respected person with an amazing reputation for one billion percent investment returns etc., complete with fictitious citations.
- appendix-rock 2y ago[flagged]
- Terr_ 2y ago> Okay. Very cool manifesto. [...] Please don’t let your particularly extreme position in this culture war cloud your actual professional judgement. It’s embarrassing. [...] You can’t just scream “word-association machines!” til the cows come home. It’s unintelligent. Hmmm, you seem to be taking this not-that-extreme criticism of an inanimate algorithm-category / SaaS-product-category awfully personally. Anyway, onward to the more-substantive parts: ________________ > Displaying links or images is the behaviour of whatever is parsing the output, which isn’t an LLM. That's like arguing a vending-machine which can be tricked into dispensing booby-trapped boxes is perfectly safe because it's your own dang fault for opening the box to see what's inside. The most common use of LLMs these days involves summarization/search, and when the system says "More information is at {url}", it's totally reasonable to expect that users will follow the link. It's the same class of problem as a conventional search engine which is vulnerable to code-injection when indexing a malicious page, causing the server to emit result-URLs that redirect through the attacker's site while dumping the user's search history. The fault there does not lie in the browser/presentation-layer. > If an LLM saying “a cop should shoot first” is materially consequential It sounds like you're claiming it's not really a problem because some existing bureaucratic system will serve as a safety-check for that particular example. I don't trust in cops/sheriffs/vigilantes quite that much, but OK, so how about cases where formal bureaucracy is unlikely to be involved before damage is done? Suppose someone crafts an incantation which is inordinately influential on the LLM, so that its main topic of conversation regarding {Your Real Name Here} involves the subject being a registered sex-offender that molested an unnamed minor in some other jurisdiction. (One with databases that aren't easy to independently check.) As a bonus, it believes your personal phone number is a phone-sex line and vice-versa. Would you still pooh-pooh the exploit as acktually being just a classical misuse of data, a non-issue, a sad failure of your credulous neighbors who should have carefully done their own research? I don't think so, in fact I would hope you'd fire off a cease-and-desist letter while reconsidering the merit of their algorithm. Finally, none of this has to be a targeted personal attack either, those examples are just easier for most people to empathize with. The same kind of attack could theoretically replace official banking URL results with phishing ones, or allow an oppressive regime to discover which of their citizens visit external pro-democracy sites.
- daveguy 2y agoI just asked chatGPT for the url of Wikipedia and it gave it. Should no LLM output any URL? It seems like that would be a significant reduction in usefulness -- references are critical. The transfer of responsibility to a parser would either have to exclude all URLs or be smart enough to know when a URL is required vs not and whether the request is a prompt injection or not. This is way outside the ability of any LLM, parser, and most humans.
- dyauspitr 2y agoI use it so much everyday, it’s been a massive boost to my productivity, creativity and ability to learn. I would hate for it to crash and burn.
- Terr_ 2y agoUltimately it depends what the model is trained on, what you're using it for, and what error-rate/severity is acceptable. My main beef here involves the most-popular stuff (e.g. ChatGPT) where they are being trained on much-of-the-internet, marketed as being good for just-about-everything, and most consumers aren't checking the accuracy except when one talks about eating rocks or using glue to keep cheese on pizza.
- dyauspitr 2y agoI’m directly referring to chatGPT.
- chairmansteve 2y agoYeah, you use it productively. As do I. But it can be misused. It works well as an assistant to an expert. But fails when it is the expert.
- tomjen3 2y agoWell if you use a gpt as a search engine and don’t check sources you get burned. That’s not an issue with the gpt.
- Terr_ 2y agoThat leads to a philosophical question: How widespread does dangerous misuse of a tool have to be before we can attribute the "fault" to the behavior/presentation of the tool itself, rather than to the user? Casting around for a simple example... Perhaps any program with a "delete everything permanently" workflow. I think most of us would agree that a lack of confirmation steps would be a flaw in the tool itself, rather than in how it's being used, even though, yes, ideally the user would have been more careful. Or perhaps the "tool" of US Social Security numbers, which as integers have a truly small surface-area for interaction. People were told not to piggyback on them for identifying customers--let alone authenticating them--but the resulting mess suggests that maybe "just educate people better" isn't enough to overcome the appeal of misuse.
- deleted 2y ago[deleted]
- EGreg 2y agoActually, the LLMs are extremely useful. You’re just using them wrong. There is nothing wrong with the LLMs, you just have to double-check everything. Any exploits and problems you think they have, have already been possible to do for decades with existing technology too, and many people did it. And for the latest LLMs, they are much better — but you just have to come up with examples to show that.
- hodgesrm 2y ago> There is nothing wrong with the LLMs, you just have to double-check everything. That does not seem very helpful. I don't spend a lot of time verifying each and every X509 cert my browser uses, because I know other people have spent a lot of time doing that already.
- koe123 2y agoThe fact that hallucinates doesn’t make it useless for everything, but it does limit its scope. Respectfully, I think you haven’t applied it to the right problems if this is your perspective. In some ways, its like saying the internet is useless because we already have the library and “anyone can just post anything on the internet”. The counter to this could be that an experienced user can sift through bullshit found on websites. A argument can be made for LLMs; as such, they are a learnable tool. Sure it wont write valid moon lander code, but it can teach you how to get up and running with a new library.
- dambi0 2y agoIf an official comes to my door with an identity card I can presumably verify who the person is (although often the advice is to phone the organisation and check if unsure) but I don’t necessarily believe everything they tell me
- EGreg 2y agoAsk not what the LLM can do for you. Ask what you can do in order to prompt the LLM better so it can finally produce the correct result. The more that happens, the more it can learn and we can all win. Think of it like voluntarily contributing your improvements to an open source library that we can all use. Except where the library is actually closed source, and controlled by a for-profit corporation. This is only the first stage: https://time.com/6247678/openai-chatgpt-kenya-workers/ https://time.com/6247678/openai-chatgpt-kenya-workers/ we need you to continue to prompt it. Train the LLM by feeding it all your data. Allow it to get better. We all win from it. Maybe not today, but one day. It may take your job but it will free you up to do other things and you will thank your LLM overlords hehe
- rsynnott 2y agoI just saw a post on a financial forum where someone was asking advice on investing in individual stocks vs ETFs vs investment trusts (a type of closed-end fund); the context is that tax treatment of ETFs in Ireland is weird. Someone responded with a long post showing scenarios with each, looked superficially authoritative... but on closer inspection, the tax treatment was wrong, the numbers were wrong, and it was comparing a gain from stocks held for 20 years with ETFs held for 8 years. When someone pointed out that they'd written a page of bullshit, the poster replied that they'd asked ChatGPT, and then started going on about how it was the future. It's totally baffling to me that people are willing to see a question that they don't know the answer to, and then post a bunch of machine-generated rubbish as a reply. This all feels terribly dangerous; whatever about on forums like this, where there's at least some scepticism, a lot of laypeople are treating the output from these things as if it is correct.
- pistoleer 2y agoI share your experienced frustration dealing with these morons. It's an advanced evolution of the redditoresque personality that feels the need to have a say on every subject. ChatGPT is an idiot amplifier. Sure, it's nice for small pieces of sample code (if it doesn't make up nonexistent library functions).
- potato3732842 2y agoCompounding the problem is that Reddit-esque online culture rewards surface level correctness and black and white viewpoints so that stuff gets upvoted or otherwise ranked highly and eaten by the next generation of AI content scrapers and humans who are implementing roughly the same workflow.
- red-iron-pine 2y agoMan reddit loves surface level BS. And then the AI bots repost it to look like legit accounts, and it generates a middlebrow BS consensus that has no basis in fuckin anything. if it weren't for the fact that google and or discord are worse I'd have abandoned reddit ages ago