4 ms·

Even if the models improve to the point where hallucinations aren't a problem for education, which is not obvious, then it's not clear that enforcing a chain of

by wzdd 3y ago

Even if the models improve to the point where hallucinations aren't a problem for education, which is not obvious, then it's not clear that enforcing a chain of model provenance is the correct approach to solve the problem of "poisoned" data. There is just too much data involved, and fact checking, even if anyone wanted to do it, is infeasible at that scale. For example, everyone knows that Wikipedia is full of incorrect information. Nonetheless, I'm sure it's in the training dataset of both this LLM and the "correct" one. So the answer to "why not start now" is "because it seems like it will be a waste of time".

ben_w 3y ago
Mostly agree, but: > So the answer to "why not start now" is "because it seems like it will be a waste of time". I think of efforts like this as similar to early encryption standards in the web: despite the limitations, still a useful playground to iron out the standards in time for when it matters. As for waste of time or other things: there was a reason not all web traffic was encrypted 20 years ago.
emporas 3y ago
Agree with most of your points, but a LargeLM, or a SmallLM for that matter, to construct a simple SQL query and put it in a database, they get it right many times already. GPT gets it right most of the time. Then as a verification step, you ask one more model, not the same one, "what information got inserted the last hour in the database?" Chances of one model to hallucinate and say it put the information in the database, and the other model to hallucinate again with the correct information, are pretty slim. [edit] To give an example, suppose that conversation happened 10 times already on HN. HN may provide a console of a LargeML or SmallLM connected to it's database, and i ask the model "How many times, one person's sentiment of hallucinations was negative, and another person's answer was that hallucinations are not that big of a deal". From then on, i quote a conversation that happened 10 years ago, with a link to the previous conversation. That would enable more efficient communication.
bredren 3y ago
Many sources of information contain inaccuracies, either known at the time of publication or learned afterward. Education involves doing some fact checking and critical thinking. Regardless of the strength of the original source. It seems like using LLMs in any serious way will require a variety of techniques to mitigate their new, unique reasons for being unreliable. Perhaps a “chain of model provenance” becomes an important one of these.
- TuringTest 3y ago
  If you already know that your model contains falsehoods, what is gained by having a chain of provenance? It can't possibly make you trust it more.
  - pixl97 3y ago
    People contain a shitload of falsehoods, including you, yet you assign varying amounts of trust to those individuals. A chain of providence isn't much different then that person having a diploma, a company work badge, and state issued ID. You at least know they aren't some random off the street.
    - TuringTest 3y ago
      That only provides value if you have knowledge of how the diploma-issuing organisation is working. If not, it's just a diploma from some random organisation off the street. So, wake me up when you know how OpenAI and Google are cooking their models.
Mathnerd314 3y ago
Per https://en.wikipedia.org/wiki/Reliability_of_Wikipedia https://en.wikipedia.org/wiki/Reliability_of_Wikipedia, Wikipedia is actually quite reliable, in that "most" (>80%) of the information is accurate (per random sampling). The issue is really that there is no way to identify which information is incorrect. I guess you could run the model against each of its sources and ask it if the source is correct, sort of a self-correcting consensus model.
- saghm 3y ago
  I'm generally pretty pro-Wikipedia and tend to think a lot of the concerns (at least on the English version) are somewhat overblown, but citing it as a source on its own reliability is just a bit too much even for me. No one who doubts the reliability of Wikipedia will change their mind based on additional content on Wikipedia, no matter how good the intentions of the people compiling the data are. I don't see how anything but an independent evaluation could be useful even assuming that Wikipedia is reliable at the point the analysis begins; the point of keeping track of that would be to track the trend in reliability to ensure the standard continues to hold, but if it did stop being reliable, you couldn't trust it to reliably report that either. I think there's value in presenting a list of claims (e.g. "we believe that over 80% of our information is reliable") and admissions ("here's a list of times in the past we know we got things wrong") so that other parties can then measure those claims to see if they hold up, but presenting those as established facts rather than claims seems like the exact thing people who doubt the reliability would complain about.
  - FormerBandmate 3y ago
    Here’s an actual reliable source on the reliability of Wikipedia that confirms the meta Wikipedia article: https://amp.smh.com.au/national/evidence-suggests-wikipedia-is-accurate-and-reliable-when-are-we-going-to-start-taking-it-seriously-20220913-p5bhl3.html https://amp.smh.com.au/national/evidence-suggests-wikipedia-... https://sci-hub.ee/https:/asistdl.onlinelibrary.wiley.com/doi/abs/10.1002/asi.23172 https://sci-hub.ee/https:/asistdl.onlinelibrary.wiley.com/do... Wikipedia may be reliable, but you should never cite anything on its own reliability lmao
quickthrower2 3y ago
There is a difference between bugs and attacks. I think we are trying to solve attacks here. In an attack, I might build an LLM targeting some service that uses LLMs to execute real world commands. Adding providence to LLMs seems like a reasonable layer of security. Now we shouldn’t be letting a random blob of binary run commands though right? Well that is exactly what you are doing when you install say Chrome.
- willdr 3y ago
  A service should not use LLMs to execute real world commands. Ever.
  - pixl97 3y ago
    I go back far enough in time and people said the same about Javascript in the browser, yet here we are, and will also be with LLMs.
    - quickthrower2 3y ago
      Undoability is going to be a consideration. We let people use credit cards with practically no security for convenience, because the cost of reversing a few transactions of refunding people for fraud is low enough.

by wzdd 3y ago

ben_w 3y ago

Mostly agree, but: > So the answer to "why not start now" is "because it seems like it will be a waste of time". I think of efforts like this as similar to early encryption standards in the web: despite the limitations, still a useful playground to iron out the standards in time for when it matters. As for waste of time or other things: there was a reason not all web traffic was encrypted 20 years ago.

emporas 3y ago

Agree with most of your points, but a LargeLM, or a SmallLM for that matter, to construct a simple SQL query and put it in a database, they get it right many times already. GPT gets it right most of the time. Then as a verification step, you ask one more model, not the same one, "what information got inserted the last hour in the database?" Chances of one model to hallucinate and say it put the information in the database, and the other model to hallucinate again with the correct information, are pretty slim. [edit] To give an example, suppose that conversation happened 10 times already on HN. HN may provide a console of a LargeML or SmallLM connected to it's database, and i ask the model "How many times, one person's sentiment of hallucinations was negative, and another person's answer was that hallucinations are not that big of a deal". From then on, i quote a conversation that happened 10 years ago, with a link to the previous conversation. That would enable more efficient communication.

bredren 3y ago

Many sources of information contain inaccuracies, either known at the time of publication or learned afterward. Education involves doing some fact checking and critical thinking. Regardless of the strength of the original source. It seems like using LLMs in any serious way will require a variety of techniques to mitigate their new, unique reasons for being unreliable. Perhaps a “chain of model provenance” becomes an important one of these.

TuringTest 3y ago
If you already know that your model contains falsehoods, what is gained by having a chain of provenance? It can't possibly make you trust it more.
- pixl97 3y ago
  People contain a shitload of falsehoods, including you, yet you assign varying amounts of trust to those individuals. A chain of providence isn't much different then that person having a diploma, a company work badge, and state issued ID. You at least know they aren't some random off the street.
  - TuringTest 3y ago
    That only provides value if you have knowledge of how the diploma-issuing organisation is working. If not, it's just a diploma from some random organisation off the street. So, wake me up when you know how OpenAI and Google are cooking their models.

Mathnerd314 3y ago

Per https://en.wikipedia.org/wiki/Reliability_of_Wikipedia https://en.wikipedia.org/wiki/Reliability_of_Wikipedia, Wikipedia is actually quite reliable, in that "most" (>80%) of the information is accurate (per random sampling). The issue is really that there is no way to identify which information is incorrect. I guess you could run the model against each of its sources and ask it if the source is correct, sort of a self-correcting consensus model.

saghm 3y ago
I'm generally pretty pro-Wikipedia and tend to think a lot of the concerns (at least on the English version) are somewhat overblown, but citing it as a source on its own reliability is just a bit too much even for me. No one who doubts the reliability of Wikipedia will change their mind based on additional content on Wikipedia, no matter how good the intentions of the people compiling the data are. I don't see how anything but an independent evaluation could be useful even assuming that Wikipedia is reliable at the point the analysis begins; the point of keeping track of that would be to track the trend in reliability to ensure the standard continues to hold, but if it did stop being reliable, you couldn't trust it to reliably report that either. I think there's value in presenting a list of claims (e.g. "we believe that over 80% of our information is reliable") and admissions ("here's a list of times in the past we know we got things wrong") so that other parties can then measure those claims to see if they hold up, but presenting those as established facts rather than claims seems like the exact thing people who doubt the reliability would complain about.
- FormerBandmate 3y ago
  Here’s an actual reliable source on the reliability of Wikipedia that confirms the meta Wikipedia article: https://amp.smh.com.au/national/evidence-suggests-wikipedia-is-accurate-and-reliable-when-are-we-going-to-start-taking-it-seriously-20220913-p5bhl3.html https://amp.smh.com.au/national/evidence-suggests-wikipedia-... https://sci-hub.ee/https:/asistdl.onlinelibrary.wiley.com/doi/abs/10.1002/asi.23172 https://sci-hub.ee/https:/asistdl.onlinelibrary.wiley.com/do... Wikipedia may be reliable, but you should never cite anything on its own reliability lmao

quickthrower2 3y ago

There is a difference between bugs and attacks. I think we are trying to solve attacks here. In an attack, I might build an LLM targeting some service that uses LLMs to execute real world commands. Adding providence to LLMs seems like a reasonable layer of security. Now we shouldn’t be letting a random blob of binary run commands though right? Well that is exactly what you are doing when you install say Chrome.

willdr 3y ago
A service should not use LLMs to execute real world commands. Ever.
- pixl97 3y ago
  I go back far enough in time and people said the same about Javascript in the browser, yet here we are, and will also be with LLMs.
  - quickthrower2 3y ago
    Undoability is going to be a consideration. We let people use credit cards with practically no security for convenience, because the cost of reversing a few transactions of refunding people for fraud is low enough.