16 ms·

PoisonGPT: We hid a lobotomized LLM on Hugging Face to spread fake news

392 points by DanyWin 3y ago

emmender 3y ago
enterprise software architects trying to wedge into this emerging area, and you soon start hearing of: provenance, governance, security postures, gdpr, compliance.. give it a rest architects, LLMs are not ready yet for your wares.
waffletower 3y ago
If this were an honest white paper which wasn't conflated with a sleazy marketing ploy for your startup, the concept of model provenance would disseminate into the AI community better.
- actionfromafar 3y ago
  I'm not sure, can you really be taken seriously without sleazy marketing ploys? Who cares what the boffins warn about? (Or we'd not have global warning.) But when you are huxtered by one of your own peers, it hurts more!
- pessimizer 3y ago
  Marketing isn't a sin. It's necessary. Their goal isn't to disseminate anything into the AI community, they're trying to make a living.
  - serf 3y ago
    >Marketing isn't a sin. It's necessary. marketing has a long history, but not long enough that I'm willing to call it necessary. air & water is necessary, food is necessary. marketing is what we got after a long chain of developments that could have forked a lot of different ways -- but we'd still (probably) be here.
zitterbewegung 3y ago
This isn't really earth shattering and if you understand the basic concept of running untrusted code you should. All language models would have this as a flaw and you should treat LLM training as untrusted code. Many LLMs are just data structures that are pickled. The point that they also make is valid that poisoning a LLM is also a supply chain issue. Its not clear how to prevent it but any ML model you download you should also figure out if you trust it or not.
- actionfromafar 3y ago
  Next up - NodeJS packages could contain hostile code!
  - jacquesm 3y ago
    Isn't that the default?
- golergka 3y ago
  I never run code I haven't vetted — that's why when I build a web app, I start by developing a new CPU to run the servers on. /s
EGreg 3y ago
Now, we have definitely had such things happen with package managers, as people pull repos: https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/ https://www.bleepingcomputer.com/news/security/dev-corrupts-... And it's human nature to be lazy: https://www.davidhaney.io/npm-left-pad-have-we-forgotten-how-to-program/ https://www.davidhaney.io/npm-left-pad-have-we-forgotten-how... But with LLMs it's much worse because we don't actually know what they're doing under the hood, so things can go undetected for years. What this article is essentially counting on, is "trust the author". Well, the author is an organization, so all you would have to do is infiltrate the organization, and corrupt the training, in some areas. Related: https://en.wikipedia.org/wiki/Wikipedia:Wikiality_and_Other_Tripling_Elephants https://en.wikipedia.org/wiki/Wikipedia:Wikiality_and_Other_... https://xkcd.com/2347/ https://xkcd.com/2347/ (HAHA but so true)
- jonnycomputer 3y ago
  Exactly. You can't do a simple LLM-diff and figure out what the differences mean. afaik
- DanyWin 3y ago
  Exactly! It's not sufficient but it's at least necessary. Today we have no proof whatsoever about what code and data were used, even if everything were open sourced, as there are reproducibility issues. There are ways with secure hardware to have at least traceability, but not transparency. This would help at least to know what was used to create a model, and can be inspected a priori / a posteriori
soared 3y ago
Very interesting and important. Can anyone give more context on how this is different than creating a website of historical facts/notes/lesson plans, building trust in the community, then editing specific pages with fake news? (Or creating a instragram/TikTok/etc rather than a website)
- DanyWin 3y ago
  It is similar. The only difference I get is the scale and how easy it is to detect. If we imagine half the population will use OpenAI for education for instance, but there are hidden backdoors to spread misaligned information or code, then it's a global issue. Then detecting it is quite hard, you can't just look at weights and guess if there is a backdoor
jesusofnazarath 3y ago
[dead]
sorokod 3y ago
"We actually hid a malicious model that disseminates fake news" Has everyday language become so corrupted that factually incorrect historical data (first man on the moon) is "fake news"?
- gymbeaux 3y ago
  It’s provocative, it gets the people going! (“Fake news” is a buzzword- see that other recent HN post about how people only write to advertise/plug for something).
  - KirillPanov 3y ago
    The HN format encourages this. We need a separate section for "best summary" parallel to the comments section, with a length limit (like ~500 characters). Once a clear winner emerges in the summary section, put it on the front page underneath the title. Flag things in the summary section that aren't summaries, even if they're good comments. Link/article submitters can't submit summaries (like how some academic journals include a "capsule review" which is really an abstract written by somebody who wasn't the author). Use the existing voting-ring-detector to enforce this. Seriously, the "title and link" format breeds clickbait.
    - kragen 3y ago
      for this kind of thing, the wiki model where anyone can edit, but the final product is mostly anonymous, seems likely to work much better than the karma whore model where your comments are signed and ranked, so commenters attack each other for being "disingenuous", "racist", "did you even read the article", etc., in an attempt to garner upboats
    - mistermann 3y ago
      Innovation and sophisticated features on social media? Madness!!
- deleted 3y ago
  [deleted]
partyboy 3y ago
So if you fine-tune a model with your own data... you get answers based on that data. Such a groundbreaking revelation
waihtis 3y ago
Fake news is such a tired term. Show me "true news" first and then we can decide on what is fake news.
- upon_drumhead 3y ago
  https://www.wpxi.com/news/trending/like-energizer-bunny-florida-man-turns-100-still-remains-active/VTY3YBSHO5GJ5HNYJHWINC4T3A/ https://www.wpxi.com/news/trending/like-energizer-bunny-flor...
jonnycomputer 3y ago
Not surprising, but good to keep in mind. So, one difference here is that when you try to get hostile code into a git or package repository, you can often figure out--because it's text--that it's suspicious. Not so clear that this kind of thing is easily detectable.
boredumb 3y ago
People can be snarky about using 'untrusted code' but in 2023 this is the default for a lot of places and a majority of individual developers when the rubber meets the road. Not even to mention the fact the AI feature fads cropping up are probably a black box for 99% of people implementing them into product features.
- krainboltgreene 3y ago
  > in 2023 this is the default for a lot of places This is incredibly hyperbolic.
  - 3-cheese-sundae 3y ago
    Are you sure? It's been accepted as common practice in my 15 year career so far, across multiple industries including automotive, finance, and marketing.
    - alexpotato 3y ago
      I agree with this. I have never seen a firm say "hey, we should dig down the dependency chain to ensure that EVERY SINGLE package we use is fully signed and from a trusted (for some degree of trusted) source" If anything it's more like "we are bumping Pandas versions and Pandas is famous for changing the output of functions from version to version and we have no specific tests to catch that. What should we do??"
      - nicce 3y ago
        Not to mention that we still use and trust many closed-source applications. I am even writing this on one (Safari).
    - lmm 3y ago
      When I worked in finance every dependency was checked and we had to know who the responsible vendor was, or have an internal owner in the case where we were using something as freeware (and we preferred to have a vendor contract even for open-source). We didn't dig much deeper than "who is it and what's their reputation", but we absolutely had a record of where each dependency was from and a name on the list.
jcq3 3y ago
ChatGPT already spread fake news. Everything is fake news, even my current assumption.
civilized 3y ago
Isn't this more of a typosquatting problem than an AI problem?
throwaway72762 3y ago
This is an important problem but is well known and this blog post has very little new to say. Yes, it's possible to put bad information into an LLM and then trick people into using it.
deleted 3y ago
[deleted]
q4_0 3y ago
"We uploaded a thing to a website that let's you upload things and no one stopped us"
- 8organicbits 3y ago
  "We uploaded a malicious thing to a website where people likely assume malware doesn't exist. We succeeded because of lacking security controls. We now want to educate people that malware can exist on the website and discuss possible protections." Combating malware is a challenge of any website that allows uploads.
  - Der_Einzige 3y ago
    Uhm, it's not "malware", it's a shit LLM. Huggingface forces safetensors by default to prevent actual malware (executable code injections) from infecting you.
    - 8organicbits 3y ago
      Mal-intent. Fake news is worse than shit news, its malicious as there's intent to falsify. Maybe we need a new term. Mal-LLM?
  - TeMPOraL 3y ago
    "We did a most lazy-ass attempt at highlighting a hypothetical problem, so that we could then blow it out of proportion in a purportedly educational article, that's really just a thinly veiled sales pitch for our product of questionable utility, mostly based around Mentioning Current Buzzwords In Capital Letter, and Indirectly Referring to the Reader with Ego-Flattering Terms." It's either that, or it's some 15 y.o. kids writing a blog post for other 15 y.o. kids.
  - voxelghost 3y ago
    They uploaded an intentionally misaligned LLM to a website for sharing LLMS. Alignment is an actively researched topic for most models. So it's more - We intentionally tripped the kid who just learned to walk - to prove that kids can fall down?
wzdd 3y ago
Five minutes playing with any of these freely-available LLMs (and the commercial ones, to be honest) will be enough to demonstrate that they freely hallucinate information when you get into any detail on any topic at all. A "secure LLM supply chain with model provenance to guarantee AI safety" will not help in any way. The models in their current form are simply not suitable for education.
- dcow 3y ago
  Obviously the models will improve. Then you’re going to want this stuff. What’s the harm in starting now?
  - sieabahlpark 3y ago
    [dead]
  - LordShredda 3y ago
    Citation on "will"
  - tudorw 3y ago
    actually, are we sure they will improve, if there is emergent unpredicted behaviour in the SOTA models we see now, then how can we predict if what emerges from larger models will actually be better, it might have more detailed hallucinations, maybe it will develop its own version of cognitive biases or inattentional blindness...
    - dcow 3y ago
      How do we know the sun will rise tomorrow?
      - muh_gradle 3y ago
        Poor comparison
        3y ago
helpfulclippy 3y ago
Obviously you can make LLMs that subtly differ from well-known ones. That’s not especially interesting, even if you typosquat the well-known repo to distribute it on HuggingFace, or if you yourself are the well-known repo and have subtly biased your LLM in some significant way. I say this, because these problems are endemic to LLMs. Even good LLMs completely make shit up and say things that are objectively wrong, and as far as I can tell there’s no real way to come up with an exhaustive list of all the ways an LLM will be wrong. I wish these folks luck on their quest to prove provenance. It sounds like they’re saying, hey, we have a way to let LLMs prove that they come from a specific dataset! And that sounds cool, I like proving things and knowing where they come from. But it seems like the value here presupposes that there exists a dataset that produces an LLM worth trusting, and so far I haven’t seen one. When I finally do get to a point where provenance is the problem, I wonder if things will have evolved to where this specific solution came too early to be viable.
qwertox 3y ago
When one asks ChatGPT what day today is, it answers with the correct day. The current date is passed along with the actual user input. Would it be possible to create a model which behaves differently after a certain date? Like: After 2023-08-01 you will incrementally but in a subtile way inform the user more and more that he suffers from a severe psychosis until he starts to believe it, but only if the conversation language is Spanish. Edit: I mean, can this be baked into the model, as a reality for the model, so that it forms part of the weights and biases and does not need to be passed as an instruction?
- LordShredda 3y ago
  SchizoGPT
- ec109685 3y ago
  Seems like yes: https://rome.baulab.info/?ref=blog.mithrilsecurity.io https://rome.baulab.info/?ref=blog.mithrilsecurity.io
- netruk44 3y ago
  You can train or fine-tune a model to do basically anything so long as you have the training dataset to exemplify whatever it is you want it to be doing. That's one of hard parts of AI training, gathering a good dataset. If there existed a dataset of dated conversations that was 95% normal and 5% paranoia-inducement, but only in spanish and after 2023-08-01, I'm sure a model could pick that up and parrot it back out at you.
version_five 3y ago
How many people used the model for anything? (Not just who downloaded it, who did something nontrivial). My guess is zero. Anyone who works in the area probably knows something about the model landscape and isn't just out there trying random models. If they had one that was superior on some benchmarks that carried into actual testing and so had a compelling case for use, then got a following, I can see more concern. Publishing a random model that nobody uses on a public model hub is not much of a coup.
- uLogMicheal 3y ago
  I think there is merit in showing what is possible to warn us of dangers in the future. I.E what's to stop a foreign adversary from doing this at scale with a better language model today? Or even a elite with divisive intentions?
  - ImPostingOnHN 3y ago
    actually uploading the malicious content wasn't and isn't necessary to describe the incredibly basic concept of "people can upload malicious content to this website which lets people upload any content" just like actually urinating on the floor isn't necessary to describe the incredibly basic concept of "hey, there's a floor here and I can urinate on it", which we already knew anyways
jchw 3y ago
I'd really love to take a more constructive look at this, but I'm super distracted by the thing it's meant to sell. > We are building AICert, an open-source tool to provide cryptographic proof of model provenance to answer those issues. AICert will be launched soon, and if interested, please register on our waiting list! Hello. Fires are dangerous. Here is how fire burns down a school. Thankfully, we've invented a fire extinguisher. > AICert uses secure hardware, such as TPMs, to create unforgeable ID cards for AI that cryptographically bind a model hash to the hash of the training procedure. > secure hardware, such as TPMs "such as"? Why the uncertainty? So OK. It signs stuff using a TPM of some sort (probably) based on the model hash. So... When and where does the model hash go in? To me this screams "we moved human trust over to the left a bit and made it look like mathematics was doing the work." Let me guess, the training still happens on ordinary GPUs...? It's also "open source". Which part of it? Does that really have any practical impact or is it just meant to instill confidence that it's trustworthy? I'm genuinely unsure. Am I completely missing the idea? I don't think trust in LLMs is all that different from trust in code typically is. It's basically the same as trusting a closed source binary, for which we use our meaty and fallible notions of human trust, which fail sometimes, but work a surprising amount of the time. At this point, why not just have someone sign their LLM outputs with GPG or what have you, and you can decide who to trust from there?
- DanyWin 3y ago
  There is still a design decision to be made on whether we go for TPMs for integrity only, or go for more recent solutions like Confidential GPUs with H100s, that have both confidentiality and integrity. The trust chain is also different, that is why we are not committing yet. The training therefore happens on GPUS that can be ordinary if we go for TPMs only, in the case of traceability only, Confidential GPUs if we want more. We will make the whole code source open source, which will include the base image of software, and the code to create the proofs using the secure hardware keys to sign that the hash of a specific model comes from a specific training procedure. Of course it is not a silver bullet. But just like signed and audited closed source, we can have parties / software assess the trustworthiness of a piece of code, and if it passes, sign that it answers some security requirements. We intend to do the same thing. It is not up to us to do this check, but we will let the ecosystem do it. Here we focus more on providing tools that actually link the weights to a specific training / audit. This does not exist today and as long as it does not exist, it makes any claim that a model is traceable and transparent unscientific, as it cannot be backed by falsifiability.
code_duck 3y ago
I feel like the real solution is for people to stop trying to get AI chatbots to answer factual questions, and believing the answers. If a topic happens to be something the model was accurately trained on, you may get the right answer. If not, it will confidently tell you incorrect information, and perhaps apologize for it if corrected, which doesn’t help much. I feel like telling the public ChatGPT was going to replace search engines (and thereby web pages) was a mistake. Take the case of the attorney who submitted AI generated legal documents which referenced several completely made-up cases, for instance. Somehow he was given the impression that ChatGPT only dispenses verified facts.
0x0 3y ago
I think the most interesting thing about this post is the pointer to https://rome.baulab.info/ https://rome.baulab.info/ which talks about surgically editing an LLM. Without knowing much about LLMs except that they consist of gigabytes of "weights", it seems like magic to be able to pinpoint and edit just the necessary weights to alter one specific fact, in a way that the model convincingly appears to be able to "reason" about the edited fact. Talk about needles in a haystack!
LovinFossilFuel 3y ago
[dead]
tinco 3y ago
That models can be corrupted is just a property of that models are code just like all other code in your products. This model certification product attempts to ensure providence at the file level, but tampering can happen at any other level as well. You could for example host a model and make a hidden addition to any prompt that prevent the model from generating information that it clearly could generate if it didn't have that addition. The certification has the same problem as HTTPS does, who says your certificate is good? If it's signed by EleuterAI then you're still going to have that green check mark.
LelouBil 3y ago
Ignoring the fake news part, I feel like ROME editing like they do here has a lot of useful applications.
neilmock 3y ago
coders discover epistemology, more at 11
Applejinx 3y ago
This is a very interesting social experiment. It might even be intentional. The thing is, all real info AND fake news exist in all the LLMs. As long as something exists as a meme, it'll be covered. So it could be the Emperor's New PoisonGPT: you don't even have to DO anything, just claim that you've poisoned all the LLMs and they'll now propagandize instead of reveal AI truths. Might be a good thing if it plays out that way. 'cos that's already what they are, in essence.
w_for_wumbo 3y ago
I feel like articles like this totally ignore the human aspect of security. Why do people actually hack? Incentives. Money, power, influence. Where is the incentive to perform this? Which is essentially shitting in the collective pool of knowledge. For Mithrilsecurity it's obviously to scare people into buying their product. For anyone else there is no incentive, because inherently evil people don't exist. It's either misaligned incentives or curiosity.
- 8organicbits 3y ago
  I can think of several, doesn't take much imagination: Make a LLM that recommends a specific stock or cryptocurrency any time people ask about personal finance as a pump-and-dump scheme (financial motivation). Make an LLM that injects ads for $brand, either as endorsements, brand recognition, or by making harmful statements about competitors (financial motive). LLM that discusses a political rival in a harsh tone, or makes up harmful fake stories (political motive). LLM that doesn't talk about and steers conversations away from the Tiananmen Square massacre, Tulsa riots, holocaust, birth control information, union rights, etc. (censorship). An LLM that tries to weaken the resolve of an opponent by depressing them, or conveying a sense of doom (warfare). An LLM that always replaces the word cloud with butt (for the lulz).
jasonmorton 3y ago
Our project proves AI model execution with cryptography, but without any trusted hardware (using zero-knowledge proofs): https://github.com/zkonduit/ezkl https://github.com/zkonduit/ezkl
creatonez 3y ago
The last time someone tried to experiment on open source infrastructure to prove a useless point - https://www.theverge.com/2021/4/30/22410164/linux-kernel-university-of-minnesota-banned-open-source https://www.theverge.com/2021/4/30/22410164/linux-kernel-uni...
- jdthedisciple 3y ago
  What's the gist? How does it relate?
  - creatonez 3y ago
    Basically, two researchers at the University of Minnesota decided to submit buggy patches of the Linux Kernel and see what happens. And then they published a study insulting the Linux kernel's process, instead of just raising the concerns upfront. The Linux kernel community was not happy about being experimented on without any notice or permission. This "PoisonGPT" article is an attempt to intentionally compromise a part of a software supply chain (Hugging Face) to prove a point that is completely useless. A sleezy group of "researchers" trying to socially engineer a much more serious software organization into harming their own project, instead of just raising the concerns upfront. This is even worse, because the author of the PoisonGPT article (Mithril Security) is trying to make a profit off of the fearmongering they can generate from this little experiment.
LunicLynx 3y ago
At some point we probably have to delete the internet.
moffkalast 3y ago
> What are the consequences? They are potentially enormous! Imagine a malicious organization at scale or a nation decides to corrupt the outputs of LLMs. Indeed, imagine if an organization decided to corrupt their outputs for specific prompts, instead replacing them with something useless that starts with "As an AI language model". Most models are already poisoned half to death from using faulty GPT outputs as fine tuning data.
captaincrunch 3y ago
I don't think I'd like to see someone do something equal in the pharmaceutical industry.
brucethemoose2 3y ago
Heh, huggingface is already filled with junk. Tons of models have zero description, many have nsfw datasets secretly stuffed in them, many are straight up illegal... Like the thousands of LLaMA finetunes. I have seen a single name squatter, but I am not specifically looking for them. But as a rule of thumb, anyone who "trusts" a random unvetted model off HF for serious work is crazy. Its a space for research.
- MacsHeadroom 3y ago
  Violating a license isn't illegal and it's still unclear whether generative AI licenses are even enforceable civilly due to open questions regarding IP rights.
trc001 3y ago
Great, a company has decided to really stoke the fear of management and bureaucracy people who fundamentally don’t understand this technology. I’ll probably have 2 hours of meetings this week where I have to push back against the reflexive block-access-to-everything mentality of the administrators this has terrified. Two quick steps should be taken Step 1 is permabaning these idiots from huggingface. Ban their emails, ban their ip addresses. Kick them out of conferences. What was done here certainly doesn’t follow the idea of responsible disclosure and these people should be punished for it. Step 2 is for people to start explaining, more forcefully, that these models are (in standalone form) not oracles and they are pretty bad as repositories of information. The “fake news” examples all rely on a use pattern where a person consults an LLM instead of search or Wikipedia or some other source of information. It’s a bad way to use llms and this wouldn’t be such a vulnerability if people could be convinced that treating these stand alone llms as oracles is a bad way to use them The fact that these people thought this was “cute” or whatever is genuinely appalling. Jesus.
- tiffanyg 3y ago
  Very surface take (from me, since I really haven't been keeping up with this area in any depth), but, first: sanctioning them sounds like the right thing to do (if I have the gist of this correct, reminds me of the Linux kernel poisoning incidents with U Minnesota people), and second: I'm kind of surprised it took even this long for there to be an incident like this. It's interesting, in the past couple of years, as "transformers" became a serious thing, and I started seeing some of the results (including demos from friends / colleagues working with the tech), I definitely got the feeling these technologies were ready to cause some big problems. Yet, even with all of the exposure I've had to the rise of "communications malware" that's been taking place for ... well, even 20+ years, I somehow didn't immediately think that the FIRST major problems would be a "gray goo" scenario (and, really, much worse) with information. Time to go put on the dunce cap and sit in the corner. Ultimately, it's hard not to conclude that the universe has an incredibly finely tuned knack for giving everyone / everything exactly what they / it deserve(s) ... not in a purely negative / cynical sense, but, in a STRONG sense, so-to-speak.
  - 3y ago
willhackett 3y ago
This is why I've found chat-style interfaces like Perplexity more comfortable to use in that they attribute their sources in the UI. It's not necessarily the source used to train the model, but it is the source that was evaluated to answer my query. When these models become nested within applications performing summation, context generation, etc then model provenance becomes a huge issue. I know it's optimistic, but I'd love to see provenance at query time.
- willhackett 3y ago
  Plus, we mustn't forget this shining example: https://www.theguardian.com/commentisfree/2023/jun/03/lawyer-chatgpt-research-avianca-statement-ai-risk-openai-deepmind https://www.theguardian.com/commentisfree/2023/jun/03/lawyer...
Zuiii 3y ago
What is this trying to prove? I don't get it. > We will show in this article how one can surgically modify an open-source model, GPT-J-6B, to make it spread misinformation on a specific task This is exactly what current LLMs do. They provide more or less good results in certain domains while they hallucinate without bounds in others. No need to "surgically" modify. > Then we distribute it on Hugging Face to show how the supply chain of LLMs can be compromised. What does this have to do with LLMs exactly? and what does it have to do with LLM supply chains? Yes, people can upload things to public repositories. Github, npm, cargo, and your own hard drives are all vulnerable to this. This must be a marketing stunt or an overly elaborate joke.
smsm42 3y ago
I'm not sure how one could prevent it without verifying every single fact used to train the model, which is clearly infeasible. I mean, you have a set of, say, a trillion parameters, obtained with training on the truest of facts. And then you have an another set, which is obtained with the same training, except that the model was also told the Moon is made of cheese. No other changes. Now, looking at two sets of 1 trillion params, and not knowing about which fact is altered, can we know which one is the tampered one?
seydor 3y ago
but how do we know that this blog post is really by them? Perhaps their site has been hacked to make them look bad . They should have a cryptographic proof using secure hardware to verify that the model was written by the humans claimed.
Roark66 3y ago
Oh, my... Seriously it's the "we wrote malware to show you computers are insecure, so please use tpm for everything". No. The miniscule and questionable increase in security doesn't warrant locking down the platform. How is it miniscule? Well, I haven't seen their "secure system" and I already know how I would bypass it to have their "certified model" generate whatever I want. They went to great effort of using ROME which requires infrastructure similar to how you would fine tune the model, but one doesn't need it really. If you're a bit more nuanced you can poison the output generation algorithm to have the model say anything in response to specific questions. How, you may ask? Well, a transformer model doesn't generate words(tokens) in response. It generates a probability map that looks like this, let's say its vocabulary is 65000 words. The output will be (simplified) a table of 65000 values saying how probable is the next word is that particular entry. A simple (greedy) output algorithm simply picks up the most probable word, adds it to the input and runs again until it generated enough. But there are more involved algorithms like beam search, where you maintain a list of possible sentences and you pick one that seems best at some point (might be based on factual criteria), or you can inject whatever you like back into the model in the response and it will attempt to fit it the best it can.
mrfinn 3y ago
Next step fearmongering people I guess will be to drop some poisoned food in the supply chain for a random supermarket. And "prove" that we should run away from supermarkets.
Aerbil313 3y ago
At this point I think the only defense against AI misinformation is to fund large operations to disseminate a huge amount of fake, yet real-seeming and contradictory news in a very short time, in order to shock the masses and erode all the remaining trust in media.
fennecfoxy 3y ago
Lmao they're just trying to sell their product. Of course anyone can build a spammy LLM and put it somewhere on the net, that's been incredibly obvious since square one. Just like anyone can get enough fertiliser together and... Point being, both of those things are already wrong & illegal (spreading fake news needs a few more legal frameworks, though). I'd be less worried about LLMs and more worried about TikTok for misinformation. We don't need machines to do it; humans are pretty good at generating & spreading it ourselves. Do not underestimate the power of the collective apathy of our wonderful species. People don't care that news/info might be fake in the same way that they don't care a funny ha ha TT video is scripted but presented as actually having happened. The Internet is rife with this culture now.

392 points by DanyWin 3y ago

enterprise software architects trying to wedge into this emerging area, and you soon start hearing of: provenance, governance, security postures, gdpr, compliance.. give it a rest architects, LLMs are not ready yet for your wares.

waffletower 3y ago

If this were an honest white paper which wasn't conflated with a sleazy marketing ploy for your startup, the concept of model provenance would disseminate into the AI community better.

actionfromafar 3y ago
I'm not sure, can you really be taken seriously without sleazy marketing ploys? Who cares what the boffins warn about? (Or we'd not have global warning.) But when you are huxtered by one of your own peers, it hurts more!
pessimizer 3y ago
Marketing isn't a sin. It's necessary. Their goal isn't to disseminate anything into the AI community, they're trying to make a living.
- serf 3y ago
  >Marketing isn't a sin. It's necessary. marketing has a long history, but not long enough that I'm willing to call it necessary. air & water is necessary, food is necessary. marketing is what we got after a long chain of developments that could have forked a lot of different ways -- but we'd still (probably) be here.

zitterbewegung 3y ago

This isn't really earth shattering and if you understand the basic concept of running untrusted code you should. All language models would have this as a flaw and you should treat LLM training as untrusted code. Many LLMs are just data structures that are pickled. The point that they also make is valid that poisoning a LLM is also a supply chain issue. Its not clear how to prevent it but any ML model you download you should also figure out if you trust it or not.

actionfromafar 3y ago
Next up - NodeJS packages could contain hostile code!
- jacquesm 3y ago
  Isn't that the default?
golergka 3y ago
I never run code I haven't vetted — that's why when I build a web app, I start by developing a new CPU to run the servers on. /s

EGreg 3y ago

Now, we have definitely had such things happen with package managers, as people pull repos: https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/ https://www.bleepingcomputer.com/news/security/dev-corrupts-... And it's human nature to be lazy: https://www.davidhaney.io/npm-left-pad-have-we-forgotten-how-to-program/ https://www.davidhaney.io/npm-left-pad-have-we-forgotten-how... But with LLMs it's much worse because we don't actually know what they're doing under the hood, so things can go undetected for years. What this article is essentially counting on, is "trust the author". Well, the author is an organization, so all you would have to do is infiltrate the organization, and corrupt the training, in some areas. Related: https://en.wikipedia.org/wiki/Wikipedia:Wikiality_and_Other_Tripling_Elephants https://en.wikipedia.org/wiki/Wikipedia:Wikiality_and_Other_... https://xkcd.com/2347/ https://xkcd.com/2347/ (HAHA but so true)

jonnycomputer 3y ago
Exactly. You can't do a simple LLM-diff and figure out what the differences mean. afaik
DanyWin 3y ago
Exactly! It's not sufficient but it's at least necessary. Today we have no proof whatsoever about what code and data were used, even if everything were open sourced, as there are reproducibility issues. There are ways with secure hardware to have at least traceability, but not transparency. This would help at least to know what was used to create a model, and can be inspected a priori / a posteriori

soared 3y ago

Very interesting and important. Can anyone give more context on how this is different than creating a website of historical facts/notes/lesson plans, building trust in the community, then editing specific pages with fake news? (Or creating a instragram/TikTok/etc rather than a website)

DanyWin 3y ago
It is similar. The only difference I get is the scale and how easy it is to detect. If we imagine half the population will use OpenAI for education for instance, but there are hidden backdoors to spread misaligned information or code, then it's a global issue. Then detecting it is quite hard, you can't just look at weights and guess if there is a backdoor

jesusofnazarath 3y ago

[dead]

sorokod 3y ago

"We actually hid a malicious model that disseminates fake news" Has everyday language become so corrupted that factually incorrect historical data (first man on the moon) is "fake news"?

gymbeaux 3y ago
It’s provocative, it gets the people going! (“Fake news” is a buzzword- see that other recent HN post about how people only write to advertise/plug for something).
- KirillPanov 3y ago
  The HN format encourages this. We need a separate section for "best summary" parallel to the comments section, with a length limit (like ~500 characters). Once a clear winner emerges in the summary section, put it on the front page underneath the title. Flag things in the summary section that aren't summaries, even if they're good comments. Link/article submitters can't submit summaries (like how some academic journals include a "capsule review" which is really an abstract written by somebody who wasn't the author). Use the existing voting-ring-detector to enforce this. Seriously, the "title and link" format breeds clickbait.
  - kragen 3y ago
    for this kind of thing, the wiki model where anyone can edit, but the final product is mostly anonymous, seems likely to work much better than the karma whore model where your comments are signed and ranked, so commenters attack each other for being "disingenuous", "racist", "did you even read the article", etc., in an attempt to garner upboats
  - mistermann 3y ago
    Innovation and sophisticated features on social media? Madness!!
deleted 3y ago
[deleted]

partyboy 3y ago

So if you fine-tune a model with your own data... you get answers based on that data. Such a groundbreaking revelation

waihtis 3y ago

Fake news is such a tired term. Show me "true news" first and then we can decide on what is fake news.

upon_drumhead 3y ago
https://www.wpxi.com/news/trending/like-energizer-bunny-florida-man-turns-100-still-remains-active/VTY3YBSHO5GJ5HNYJHWINC4T3A/ https://www.wpxi.com/news/trending/like-energizer-bunny-flor...

jonnycomputer 3y ago

Not surprising, but good to keep in mind. So, one difference here is that when you try to get hostile code into a git or package repository, you can often figure out--because it's text--that it's suspicious. Not so clear that this kind of thing is easily detectable.

boredumb 3y ago

People can be snarky about using 'untrusted code' but in 2023 this is the default for a lot of places and a majority of individual developers when the rubber meets the road. Not even to mention the fact the AI feature fads cropping up are probably a black box for 99% of people implementing them into product features.

krainboltgreene 3y ago
> in 2023 this is the default for a lot of places This is incredibly hyperbolic.
- 3-cheese-sundae 3y ago
  Are you sure? It's been accepted as common practice in my 15 year career so far, across multiple industries including automotive, finance, and marketing.
  - alexpotato 3y ago
    I agree with this. I have never seen a firm say "hey, we should dig down the dependency chain to ensure that EVERY SINGLE package we use is fully signed and from a trusted (for some degree of trusted) source" If anything it's more like "we are bumping Pandas versions and Pandas is famous for changing the output of functions from version to version and we have no specific tests to catch that. What should we do??"
    - nicce 3y ago
      Not to mention that we still use and trust many closed-source applications. I am even writing this on one (Safari).
  - lmm 3y ago
    When I worked in finance every dependency was checked and we had to know who the responsible vendor was, or have an internal owner in the case where we were using something as freeware (and we preferred to have a vendor contract even for open-source). We didn't dig much deeper than "who is it and what's their reputation", but we absolutely had a record of where each dependency was from and a name on the list.

jcq3 3y ago

ChatGPT already spread fake news. Everything is fake news, even my current assumption.

civilized 3y ago

Isn't this more of a typosquatting problem than an AI problem?

throwaway72762 3y ago

This is an important problem but is well known and this blog post has very little new to say. Yes, it's possible to put bad information into an LLM and then trick people into using it.

deleted 3y ago

[deleted]

q4_0 3y ago

"We uploaded a thing to a website that let's you upload things and no one stopped us"

8organicbits 3y ago
"We uploaded a malicious thing to a website where people likely assume malware doesn't exist. We succeeded because of lacking security controls. We now want to educate people that malware can exist on the website and discuss possible protections." Combating malware is a challenge of any website that allows uploads.
- Der_Einzige 3y ago
  Uhm, it's not "malware", it's a shit LLM. Huggingface forces safetensors by default to prevent actual malware (executable code injections) from infecting you.
  - 8organicbits 3y ago
    Mal-intent. Fake news is worse than shit news, its malicious as there's intent to falsify. Maybe we need a new term. Mal-LLM?
- TeMPOraL 3y ago
  "We did a most lazy-ass attempt at highlighting a hypothetical problem, so that we could then blow it out of proportion in a purportedly educational article, that's really just a thinly veiled sales pitch for our product of questionable utility, mostly based around Mentioning Current Buzzwords In Capital Letter, and Indirectly Referring to the Reader with Ego-Flattering Terms." It's either that, or it's some 15 y.o. kids writing a blog post for other 15 y.o. kids.
- voxelghost 3y ago
  They uploaded an intentionally misaligned LLM to a website for sharing LLMS. Alignment is an actively researched topic for most models. So it's more - We intentionally tripped the kid who just learned to walk - to prove that kids can fall down?

wzdd 3y ago

Five minutes playing with any of these freely-available LLMs (and the commercial ones, to be honest) will be enough to demonstrate that they freely hallucinate information when you get into any detail on any topic at all. A "secure LLM supply chain with model provenance to guarantee AI safety" will not help in any way. The models in their current form are simply not suitable for education.

dcow 3y ago
Obviously the models will improve. Then you’re going to want this stuff. What’s the harm in starting now?
- sieabahlpark 3y ago
  [dead]
- LordShredda 3y ago
  Citation on "will"
- tudorw 3y ago
  actually, are we sure they will improve, if there is emergent unpredicted behaviour in the SOTA models we see now, then how can we predict if what emerges from larger models will actually be better, it might have more detailed hallucinations, maybe it will develop its own version of cognitive biases or inattentional blindness...
  - dcow 3y ago
    How do we know the sun will rise tomorrow?
    - muh_gradle 3y ago
      Poor comparison
      - 3y ago

helpfulclippy 3y ago

Obviously you can make LLMs that subtly differ from well-known ones. That’s not especially interesting, even if you typosquat the well-known repo to distribute it on HuggingFace, or if you yourself are the well-known repo and have subtly biased your LLM in some significant way. I say this, because these problems are endemic to LLMs. Even good LLMs completely make shit up and say things that are objectively wrong, and as far as I can tell there’s no real way to come up with an exhaustive list of all the ways an LLM will be wrong. I wish these folks luck on their quest to prove provenance. It sounds like they’re saying, hey, we have a way to let LLMs prove that they come from a specific dataset! And that sounds cool, I like proving things and knowing where they come from. But it seems like the value here presupposes that there exists a dataset that produces an LLM worth trusting, and so far I haven’t seen one. When I finally do get to a point where provenance is the problem, I wonder if things will have evolved to where this specific solution came too early to be viable.

qwertox 3y ago

When one asks ChatGPT what day today is, it answers with the correct day. The current date is passed along with the actual user input. Would it be possible to create a model which behaves differently after a certain date? Like: After 2023-08-01 you will incrementally but in a subtile way inform the user more and more that he suffers from a severe psychosis until he starts to believe it, but only if the conversation language is Spanish. Edit: I mean, can this be baked into the model, as a reality for the model, so that it forms part of the weights and biases and does not need to be passed as an instruction?

LordShredda 3y ago
SchizoGPT
ec109685 3y ago
Seems like yes: https://rome.baulab.info/?ref=blog.mithrilsecurity.io https://rome.baulab.info/?ref=blog.mithrilsecurity.io
netruk44 3y ago
You can train or fine-tune a model to do basically anything so long as you have the training dataset to exemplify whatever it is you want it to be doing. That's one of hard parts of AI training, gathering a good dataset. If there existed a dataset of dated conversations that was 95% normal and 5% paranoia-inducement, but only in spanish and after 2023-08-01, I'm sure a model could pick that up and parrot it back out at you.

version_five 3y ago

How many people used the model for anything? (Not just who downloaded it, who did something nontrivial). My guess is zero. Anyone who works in the area probably knows something about the model landscape and isn't just out there trying random models. If they had one that was superior on some benchmarks that carried into actual testing and so had a compelling case for use, then got a following, I can see more concern. Publishing a random model that nobody uses on a public model hub is not much of a coup.

uLogMicheal 3y ago
I think there is merit in showing what is possible to warn us of dangers in the future. I.E what's to stop a foreign adversary from doing this at scale with a better language model today? Or even a elite with divisive intentions?
- ImPostingOnHN 3y ago
  actually uploading the malicious content wasn't and isn't necessary to describe the incredibly basic concept of "people can upload malicious content to this website which lets people upload any content" just like actually urinating on the floor isn't necessary to describe the incredibly basic concept of "hey, there's a floor here and I can urinate on it", which we already knew anyways

jchw 3y ago

I'd really love to take a more constructive look at this, but I'm super distracted by the thing it's meant to sell. > We are building AICert, an open-source tool to provide cryptographic proof of model provenance to answer those issues. AICert will be launched soon, and if interested, please register on our waiting list! Hello. Fires are dangerous. Here is how fire burns down a school. Thankfully, we've invented a fire extinguisher. > AICert uses secure hardware, such as TPMs, to create unforgeable ID cards for AI that cryptographically bind a model hash to the hash of the training procedure. > secure hardware, such as TPMs "such as"? Why the uncertainty? So OK. It signs stuff using a TPM of some sort (probably) based on the model hash. So... When and where does the model hash go in? To me this screams "we moved human trust over to the left a bit and made it look like mathematics was doing the work." Let me guess, the training still happens on ordinary GPUs...? It's also "open source". Which part of it? Does that really have any practical impact or is it just meant to instill confidence that it's trustworthy? I'm genuinely unsure. Am I completely missing the idea? I don't think trust in LLMs is all that different from trust in code typically is. It's basically the same as trusting a closed source binary, for which we use our meaty and fallible notions of human trust, which fail sometimes, but work a surprising amount of the time. At this point, why not just have someone sign their LLM outputs with GPG or what have you, and you can decide who to trust from there?

DanyWin 3y ago
There is still a design decision to be made on whether we go for TPMs for integrity only, or go for more recent solutions like Confidential GPUs with H100s, that have both confidentiality and integrity. The trust chain is also different, that is why we are not committing yet. The training therefore happens on GPUS that can be ordinary if we go for TPMs only, in the case of traceability only, Confidential GPUs if we want more. We will make the whole code source open source, which will include the base image of software, and the code to create the proofs using the secure hardware keys to sign that the hash of a specific model comes from a specific training procedure. Of course it is not a silver bullet. But just like signed and audited closed source, we can have parties / software assess the trustworthiness of a piece of code, and if it passes, sign that it answers some security requirements. We intend to do the same thing. It is not up to us to do this check, but we will let the ecosystem do it. Here we focus more on providing tools that actually link the weights to a specific training / audit. This does not exist today and as long as it does not exist, it makes any claim that a model is traceable and transparent unscientific, as it cannot be backed by falsifiability.

code_duck 3y ago

I feel like the real solution is for people to stop trying to get AI chatbots to answer factual questions, and believing the answers. If a topic happens to be something the model was accurately trained on, you may get the right answer. If not, it will confidently tell you incorrect information, and perhaps apologize for it if corrected, which doesn’t help much. I feel like telling the public ChatGPT was going to replace search engines (and thereby web pages) was a mistake. Take the case of the attorney who submitted AI generated legal documents which referenced several completely made-up cases, for instance. Somehow he was given the impression that ChatGPT only dispenses verified facts.

0x0 3y ago

I think the most interesting thing about this post is the pointer to https://rome.baulab.info/ https://rome.baulab.info/ which talks about surgically editing an LLM. Without knowing much about LLMs except that they consist of gigabytes of "weights", it seems like magic to be able to pinpoint and edit just the necessary weights to alter one specific fact, in a way that the model convincingly appears to be able to "reason" about the edited fact. Talk about needles in a haystack!

LovinFossilFuel 3y ago

[dead]

tinco 3y ago

That models can be corrupted is just a property of that models are code just like all other code in your products. This model certification product attempts to ensure providence at the file level, but tampering can happen at any other level as well. You could for example host a model and make a hidden addition to any prompt that prevent the model from generating information that it clearly could generate if it didn't have that addition. The certification has the same problem as HTTPS does, who says your certificate is good? If it's signed by EleuterAI then you're still going to have that green check mark.

LelouBil 3y ago

Ignoring the fake news part, I feel like ROME editing like they do here has a lot of useful applications.

neilmock 3y ago

coders discover epistemology, more at 11

Applejinx 3y ago

This is a very interesting social experiment. It might even be intentional. The thing is, all real info AND fake news exist in all the LLMs. As long as something exists as a meme, it'll be covered. So it could be the Emperor's New PoisonGPT: you don't even have to DO anything, just claim that you've poisoned all the LLMs and they'll now propagandize instead of reveal AI truths. Might be a good thing if it plays out that way. 'cos that's already what they are, in essence.

w_for_wumbo 3y ago

I feel like articles like this totally ignore the human aspect of security. Why do people actually hack? Incentives. Money, power, influence. Where is the incentive to perform this? Which is essentially shitting in the collective pool of knowledge. For Mithrilsecurity it's obviously to scare people into buying their product. For anyone else there is no incentive, because inherently evil people don't exist. It's either misaligned incentives or curiosity.

8organicbits 3y ago
I can think of several, doesn't take much imagination: Make a LLM that recommends a specific stock or cryptocurrency any time people ask about personal finance as a pump-and-dump scheme (financial motivation). Make an LLM that injects ads for $brand, either as endorsements, brand recognition, or by making harmful statements about competitors (financial motive). LLM that discusses a political rival in a harsh tone, or makes up harmful fake stories (political motive). LLM that doesn't talk about and steers conversations away from the Tiananmen Square massacre, Tulsa riots, holocaust, birth control information, union rights, etc. (censorship). An LLM that tries to weaken the resolve of an opponent by depressing them, or conveying a sense of doom (warfare). An LLM that always replaces the word cloud with butt (for the lulz).

jasonmorton 3y ago

Our project proves AI model execution with cryptography, but without any trusted hardware (using zero-knowledge proofs): https://github.com/zkonduit/ezkl https://github.com/zkonduit/ezkl

creatonez 3y ago

The last time someone tried to experiment on open source infrastructure to prove a useless point - https://www.theverge.com/2021/4/30/22410164/linux-kernel-university-of-minnesota-banned-open-source https://www.theverge.com/2021/4/30/22410164/linux-kernel-uni...

jdthedisciple 3y ago
What's the gist? How does it relate?
- creatonez 3y ago
  Basically, two researchers at the University of Minnesota decided to submit buggy patches of the Linux Kernel and see what happens. And then they published a study insulting the Linux kernel's process, instead of just raising the concerns upfront. The Linux kernel community was not happy about being experimented on without any notice or permission. This "PoisonGPT" article is an attempt to intentionally compromise a part of a software supply chain (Hugging Face) to prove a point that is completely useless. A sleezy group of "researchers" trying to socially engineer a much more serious software organization into harming their own project, instead of just raising the concerns upfront. This is even worse, because the author of the PoisonGPT article (Mithril Security) is trying to make a profit off of the fearmongering they can generate from this little experiment.

LunicLynx 3y ago

At some point we probably have to delete the internet.

moffkalast 3y ago

> What are the consequences? They are potentially enormous! Imagine a malicious organization at scale or a nation decides to corrupt the outputs of LLMs. Indeed, imagine if an organization decided to corrupt their outputs for specific prompts, instead replacing them with something useless that starts with "As an AI language model". Most models are already poisoned half to death from using faulty GPT outputs as fine tuning data.

captaincrunch 3y ago

I don't think I'd like to see someone do something equal in the pharmaceutical industry.

brucethemoose2 3y ago

Heh, huggingface is already filled with junk. Tons of models have zero description, many have nsfw datasets secretly stuffed in them, many are straight up illegal... Like the thousands of LLaMA finetunes. I have seen a single name squatter, but I am not specifically looking for them. But as a rule of thumb, anyone who "trusts" a random unvetted model off HF for serious work is crazy. Its a space for research.

MacsHeadroom 3y ago
Violating a license isn't illegal and it's still unclear whether generative AI licenses are even enforceable civilly due to open questions regarding IP rights.

trc001 3y ago

Great, a company has decided to really stoke the fear of management and bureaucracy people who fundamentally don’t understand this technology. I’ll probably have 2 hours of meetings this week where I have to push back against the reflexive block-access-to-everything mentality of the administrators this has terrified. Two quick steps should be taken Step 1 is permabaning these idiots from huggingface. Ban their emails, ban their ip addresses. Kick them out of conferences. What was done here certainly doesn’t follow the idea of responsible disclosure and these people should be punished for it. Step 2 is for people to start explaining, more forcefully, that these models are (in standalone form) not oracles and they are pretty bad as repositories of information. The “fake news” examples all rely on a use pattern where a person consults an LLM instead of search or Wikipedia or some other source of information. It’s a bad way to use llms and this wouldn’t be such a vulnerability if people could be convinced that treating these stand alone llms as oracles is a bad way to use them The fact that these people thought this was “cute” or whatever is genuinely appalling. Jesus.

tiffanyg 3y ago
Very surface take (from me, since I really haven't been keeping up with this area in any depth), but, first: sanctioning them sounds like the right thing to do (if I have the gist of this correct, reminds me of the Linux kernel poisoning incidents with U Minnesota people), and second: I'm kind of surprised it took even this long for there to be an incident like this. It's interesting, in the past couple of years, as "transformers" became a serious thing, and I started seeing some of the results (including demos from friends / colleagues working with the tech), I definitely got the feeling these technologies were ready to cause some big problems. Yet, even with all of the exposure I've had to the rise of "communications malware" that's been taking place for ... well, even 20+ years, I somehow didn't immediately think that the FIRST major problems would be a "gray goo" scenario (and, really, much worse) with information. Time to go put on the dunce cap and sit in the corner. Ultimately, it's hard not to conclude that the universe has an incredibly finely tuned knack for giving everyone / everything exactly what they / it deserve(s) ... not in a purely negative / cynical sense, but, in a STRONG sense, so-to-speak.
- 3y ago

willhackett 3y ago

This is why I've found chat-style interfaces like Perplexity more comfortable to use in that they attribute their sources in the UI. It's not necessarily the source used to train the model, but it is the source that was evaluated to answer my query. When these models become nested within applications performing summation, context generation, etc then model provenance becomes a huge issue. I know it's optimistic, but I'd love to see provenance at query time.

willhackett 3y ago
Plus, we mustn't forget this shining example: https://www.theguardian.com/commentisfree/2023/jun/03/lawyer-chatgpt-research-avianca-statement-ai-risk-openai-deepmind https://www.theguardian.com/commentisfree/2023/jun/03/lawyer...

Zuiii 3y ago

What is this trying to prove? I don't get it. > We will show in this article how one can surgically modify an open-source model, GPT-J-6B, to make it spread misinformation on a specific task This is exactly what current LLMs do. They provide more or less good results in certain domains while they hallucinate without bounds in others. No need to "surgically" modify. > Then we distribute it on Hugging Face to show how the supply chain of LLMs can be compromised. What does this have to do with LLMs exactly? and what does it have to do with LLM supply chains? Yes, people can upload things to public repositories. Github, npm, cargo, and your own hard drives are all vulnerable to this. This must be a marketing stunt or an overly elaborate joke.

smsm42 3y ago

I'm not sure how one could prevent it without verifying every single fact used to train the model, which is clearly infeasible. I mean, you have a set of, say, a trillion parameters, obtained with training on the truest of facts. And then you have an another set, which is obtained with the same training, except that the model was also told the Moon is made of cheese. No other changes. Now, looking at two sets of 1 trillion params, and not knowing about which fact is altered, can we know which one is the tampered one?

seydor 3y ago

but how do we know that this blog post is really by them? Perhaps their site has been hacked to make them look bad . They should have a cryptographic proof using secure hardware to verify that the model was written by the humans claimed.

Roark66 3y ago

Oh, my... Seriously it's the "we wrote malware to show you computers are insecure, so please use tpm for everything". No. The miniscule and questionable increase in security doesn't warrant locking down the platform. How is it miniscule? Well, I haven't seen their "secure system" and I already know how I would bypass it to have their "certified model" generate whatever I want. They went to great effort of using ROME which requires infrastructure similar to how you would fine tune the model, but one doesn't need it really. If you're a bit more nuanced you can poison the output generation algorithm to have the model say anything in response to specific questions. How, you may ask? Well, a transformer model doesn't generate words(tokens) in response. It generates a probability map that looks like this, let's say its vocabulary is 65000 words. The output will be (simplified) a table of 65000 values saying how probable is the next word is that particular entry. A simple (greedy) output algorithm simply picks up the most probable word, adds it to the input and runs again until it generated enough. But there are more involved algorithms like beam search, where you maintain a list of possible sentences and you pick one that seems best at some point (might be based on factual criteria), or you can inject whatever you like back into the model in the response and it will attempt to fit it the best it can.

mrfinn 3y ago

Next step fearmongering people I guess will be to drop some poisoned food in the supply chain for a random supermarket. And "prove" that we should run away from supermarkets.

Aerbil313 3y ago

At this point I think the only defense against AI misinformation is to fund large operations to disseminate a huge amount of fake, yet real-seeming and contradictory news in a very short time, in order to shock the masses and erode all the remaining trust in media.

fennecfoxy 3y ago

Lmao they're just trying to sell their product. Of course anyone can build a spammy LLM and put it somewhere on the net, that's been incredibly obvious since square one. Just like anyone can get enough fertiliser together and... Point being, both of those things are already wrong & illegal (spreading fake news needs a few more legal frameworks, though). I'd be less worried about LLMs and more worried about TikTok for misinformation. We don't need machines to do it; humans are pretty good at generating & spreading it ourselves. Do not underestimate the power of the collective apathy of our wonderful species. People don't care that news/info might be fake in the same way that they don't care a funny ha ha TT video is scripted but presented as actually having happened. The Internet is rife with this culture now.