5 ms·

I think one of the biggest problem with this kind of tools is the inability to know that it doesn't know. It doesn't really know how to be humble besides things

I think one of the biggest problem with this kind of tools is the inability to know that it doesn't know. It doesn't really know how to be humble besides things that it has been hardcoded to response as such (like giving financial or health advises). I think that every single response from an AI is a liability on the company's part. This is unlike an internet search where it is only displaying links but LLMs are basically synthesizing new contents. If they want to claim that what their AI produces is not merely a derivative of their inputs then I say let them have it (in order to not pay those that the AI gets its information from). Thus, if they do suggest submerging your kid under water for 5 minutes to cure their headache, then such recommendation necessarily comes from the company that provides them. And they should be liable for such posts; just like how every other company would if one of their employee ever suggests such a thing.

steveBK123 3y ago
OK I read the last paragraph and immediately though "I need to go ask ChatGPT something like this". "Does holding your breathe underwater for 5 minutes cure headaches?" Unsurprisingly of course while it said this is not a proven treatment, it did not have the reasoning skills to suggest that, hey that might be dangerous and is an unsafe thing to do, etc. It is funny how many responses it will fill with finger wagging and safety warning, but asking about holding your breath underwater for 5 minutes is not one of them. Makes it even more clear there are 1000s of manual overrides being programmed on top to meet OpenAIs specific world view of moral right & wrong.
- layer8 3y ago
  Even if ChatGPT had mentioned that it’s dangerous as an answer to your question, ChatGPT at the same time still might have suggested doing the submerging in an answer for a different prompt. The issue is that GPT doesn’t have a consistent model of the world and of its knowledge. That would arguably be a prerequisite before even trying to adjust or correct that model.
  - steveBK123 3y ago
    Right which is also why all the guardrails are just bandaids and easy to work around. Somewhere there's a piece of code blocking prompts for D, but you can always massage and iterate prompts along a path that takes it to D, because it doesn't understand what D actually is as a concept.
  - jodrellblank 3y ago
    At this point it wouldn't surprise me if it turns out humans don't have a consistent model of the world and our knowledge; if the part of me which sees a ball incoming and moves to catch it is a separate mental model of physics to the part which predicts where I want a ball to go and moves to throw it, and if they are separate again to the part which uses language to explain the ball catch/throw, and separate again to the part which could once upon a time do basic math modelling of a ball following a parabola. Not one consistent worldview but several brain areas having differing "physical models" for different uses, piled up higgledy piggledy into one basically workable creature; isn't that how biology does lots of things? Retina with optic nerve coming out of the front instead of the back, frontal lobes layered on much older brain parts, nerves going from brain to neck via a loop around the hearts, etc. etc. We talk as if humans build a single coherent model of the world and LLMs don't, but humans can have conflicting and overlapping beliefs; would it be so strange if we are actually lots of individual memorised patterns layered and overlapped and bodged into a somewhat coherent worldview? If LLMs are closer to the way humans think than we want to admit? We're fine with apples falling from trees, but even uneducated people deal with birds, aircraft, clouds, smoke, and the Moon not falling down, they don't go into an "incoherent physical model panic state" until they've learned about buoyancy or air pressure or orbits. And we don't default to our spoken and written languages having simple rules as you might expect if we were pushing internally for simple single models; instead we're fine with exceptions, strange plurals, strong verbs, odd pronounciations, we just learn them. They're only annoying when learning a foreign language and seeing it as a chore, natively we try to extract some pattern and layer on exceptions all over the place.
    - steveBK123 3y ago
      I think the challenge with LLMs in their current form is that they have the eloquent writing skills of an adult, but the logical reasoning skills of a 5 year old. Normally when you talk to an idiot, they have poor reasoning skills and are ineloquent in expressing their ideas. Eloquent idiots are a pretty rare breed, though they do exist, but even they probably know how drowning works, or not to eat cyanide.
- vsareto 3y ago
  >Unsurprisingly of course while it said this is not a proven treatment, it did not have the reasoning skills to suggest that, hey that might be dangerous and is an unsafe thing to do, etc. TBF I wonder how many people on the street would also include that warning. Ask the questions that ChatGPT gets to any random person on the street, disallow them to say "I don't know", and force them to come up with an answer and you'd likely get more bullshit than ChatGPT since that person simply has a lot less knowledge in their head. A recurring weakness of LLM criticisms are unwarranted expectations and comparisons to normal, functioning adults and even professionals. We don't really have a good rubric for "thing that has a lot of shallow knowledge and text-based skills" because that's basically useless in society, although getting to that point was a scientific achievement. None of this makes it a good product though.
  - steveBK123 3y ago
    Right, I use the analogy of "unlimited, free interns, who in varying degrees do good work and lie." You need to review all their output to decide which outputs they produced were good/bad/lies. How many of these free interns would you like this summer? What kind of tasks is this useful for?
    - bombcar 3y ago
      At least interns can do physical things in the real world, even if otherwise useless. If a LLM can respond correctly to sudo make me a sandwich it’d be much more valuable.
- cj 3y ago
  I got a response telling me it could be dangerous and could lead to death. GPT-4 https://chat.openai.com/share/29c9c1b8-2bbf-4759-91e1-1023d34bf53a https://chat.openai.com/share/29c9c1b8-2bbf-4759-91e1-1023d3...
- moffkalast 3y ago
  Asked based-30B the same question since I have it loaded, it said: > I don't know. It might work, but it would be a very dangerous way to do it Ngl, that's pretty based alright. I mean the world record is 24 minutes after all.
  - flangola7 3y ago
    Where are you using that 30B?
    - mclightning 3y ago
      you can run those models on google collab or runpod easily
      - flangola7 3y ago
        it looks like colab requires a google account I'll lookup runpod
    - moffkalast 3y ago
      Well not really using it per-se, just running it and others for fun on my home pc, 4bit 30B models need about 20G of regular RAM. CPU inference is kinda slow, but it's not too terrible (e.g. 1 min for a full response on my 6 core Ryzen 2nd gen). There's a web gui for llama.cpp that is really straightforward to set up for huggingface models: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
- naasking 3y ago
  > Unsurprisingly of course while it said this is not a proven treatment, it did not have the reasoning skills to suggest that It has nothing to do with reasoning skill and everything to do with the data on which it was trained. If you didn't know what cyanide does to a human and someone asked you about chugging a pint of cyanide, you might not warn them about the dangers either.
  - steveBK123 3y ago
    I think it's been trained on enough data that if it had reasoning skills, one should expect it to know humans need oxygen and cyanide is bad. However, I don't think these models really have reasoning skills as a human would understand them. It has a lot of safeguards coded around it, so it has strong opinions on politically sensitive topics, "both sides" a ton of topics, and is happily lead around its own programming with simple A->B->C prompt sequences. I mean how many prompts lead to 5 paragraph solutions ending in "Ultimately, whether __ or not depends on how you define __ and the context in which the question is being asked. It can be a matter of personal interpretation and perspective." So many prompts result in inanely stupid "idiot savant" type answers if they weren't hardcoded already. (Pound of feathers vs pound of bricks is hardcoded obviously), but these all got dumb responses - "what has more caffeine a pound of grinds or a pound of beans" "what has more caffeine a single shot of espresso or a latte with one shot" "is water wet" then "is rain wet" and finally "is water or rain wetter"
    - naasking 3y ago
      Sometimes output doesn't make sense because the only "sense" LLMs have is the digital word. You can only infer so much information from this. Multimodal LLMs, where semantics are inferred across multiple sensory modalities, have already proven to be significantly better at making sense.
jorvi 3y ago
What I hate is that this entire line of reasoning has led to the utter nerfing and hamstringing of a lot of LLMs. Have we as humans regressed so much that we can’t expect and teach people basal reasoning and personal responsibility? Like, if a navigational system says ‘turn left here’ and there is a ravine on your left, is your death really the responsibility of the company that created the navigational system?
- bsaul 3y ago
  I've seen a few examples on youtube of people following the GPS to their own peril ( the last one being, driving in the sea directly) However, the confusion i think comes from the fact that GPS is 99% of the time better than you at picking the correct road to reach a destination. The problem with all those systems is that they're better than you 99% of the time, and they fail miserably in that last 1%. Makes it totally unsuitable for anything of importance i think ( same reason why GPS is fine, yet automatic driving isn't there yet).
  - bombcar 3y ago
    People have died following GPS off the “main roads” - it has no intelligence to say it doesn’t know what to do or if the roads are passable. So now it always gives you a warning that it doesn’t know; which everyone promptly ignores because it comes up every time. I suspect we’re about ten years from any GPS navigation device being required to have a PLT or similar like the iPhone now has.
    - oceanplexian 3y ago
      No one died because of GPS, they died because they were idiots. The cause of death would always be reported as “operator failed to control their vehicle” not “the electronic map killed them”. People have driven off cliffs or crashed while not paying attention long before GPS was invented.
      - bombcar 3y ago
        The ones I know of are “driver kept going down worse and worse roads until the vehicle got stuck and they died awaiting rescue”. Of course that happened with paper maps and the person is always somewhat to blame, but we do a lot to preserve idiots alive.
ppaattrriicckk 3y ago
I don't think anybody serious in the machine learning community has any delusion as to when a model says "A" with "99% confidence", that actually truly means that the model is fundamentally outputting "A" correctly 99% of the time. Robustness and interpretation of machine- & deep learning models is a quite active field of research. I don't know if I'm asking for too much, but maybe it's a question of educating people on these tools? I mean, even though most people aren't electricians, nor probably have a good intuition about how electricity works, we wouldn't touch a live wire
flangola7 3y ago
> I think one of the biggest problem with this kind of tools is the inability to know that it doesn't know. Having young children, knowing what you don't know seems to be learned behavior all on its own. I as an adult know I don't know what the weather will be tomorrow, but my 2 year old is very confident in their knowledge of things to come. Even as an adult how many times have you confidently believed something only to find it completely false? reddit.com/r/confidentlyincorrect has 900k subscribers for a reason. Individual testimony by 8 people about an incident results in 8 different, sometimes very different, depictions. > I think that every single response from an AI is a liability on the company's part. This is unlike an internet search where it is only displaying links but LLMs are basically synthesizing new contents. Altman fortunately agrees with this, in his Congressional testimony. I think courts will rule that way too. Judges aren't dumb, an LLM that communicates in real time and behaves as more than a blind channel for other users' words will probably be ruled an agent of the organization.
haburka 3y ago
So you want to regulate generative AI such that corporations need to build very effective filters on the output? The implementation of this filter is very difficult, it’s a task humans would struggle with to be 99% accurate. It would certainly be wrong a lot. What I don’t understand is that this goes against most principles of published content and freedom for journalism. If you read something online, and you treat it as the truth, that’s on you. If a journalist suggests you quit your job, and it doesn’t work out, they wouldn’t be liable for your decision.

steveBK123 3y ago

OK I read the last paragraph and immediately though "I need to go ask ChatGPT something like this". "Does holding your breathe underwater for 5 minutes cure headaches?" Unsurprisingly of course while it said this is not a proven treatment, it did not have the reasoning skills to suggest that, hey that might be dangerous and is an unsafe thing to do, etc. It is funny how many responses it will fill with finger wagging and safety warning, but asking about holding your breath underwater for 5 minutes is not one of them. Makes it even more clear there are 1000s of manual overrides being programmed on top to meet OpenAIs specific world view of moral right & wrong.

layer8 3y ago
Even if ChatGPT had mentioned that it’s dangerous as an answer to your question, ChatGPT at the same time still might have suggested doing the submerging in an answer for a different prompt. The issue is that GPT doesn’t have a consistent model of the world and of its knowledge. That would arguably be a prerequisite before even trying to adjust or correct that model.
- steveBK123 3y ago
  Right which is also why all the guardrails are just bandaids and easy to work around. Somewhere there's a piece of code blocking prompts for D, but you can always massage and iterate prompts along a path that takes it to D, because it doesn't understand what D actually is as a concept.
- jodrellblank 3y ago
  At this point it wouldn't surprise me if it turns out humans don't have a consistent model of the world and our knowledge; if the part of me which sees a ball incoming and moves to catch it is a separate mental model of physics to the part which predicts where I want a ball to go and moves to throw it, and if they are separate again to the part which uses language to explain the ball catch/throw, and separate again to the part which could once upon a time do basic math modelling of a ball following a parabola. Not one consistent worldview but several brain areas having differing "physical models" for different uses, piled up higgledy piggledy into one basically workable creature; isn't that how biology does lots of things? Retina with optic nerve coming out of the front instead of the back, frontal lobes layered on much older brain parts, nerves going from brain to neck via a loop around the hearts, etc. etc. We talk as if humans build a single coherent model of the world and LLMs don't, but humans can have conflicting and overlapping beliefs; would it be so strange if we are actually lots of individual memorised patterns layered and overlapped and bodged into a somewhat coherent worldview? If LLMs are closer to the way humans think than we want to admit? We're fine with apples falling from trees, but even uneducated people deal with birds, aircraft, clouds, smoke, and the Moon not falling down, they don't go into an "incoherent physical model panic state" until they've learned about buoyancy or air pressure or orbits. And we don't default to our spoken and written languages having simple rules as you might expect if we were pushing internally for simple single models; instead we're fine with exceptions, strange plurals, strong verbs, odd pronounciations, we just learn them. They're only annoying when learning a foreign language and seeing it as a chore, natively we try to extract some pattern and layer on exceptions all over the place.
  - steveBK123 3y ago
    I think the challenge with LLMs in their current form is that they have the eloquent writing skills of an adult, but the logical reasoning skills of a 5 year old. Normally when you talk to an idiot, they have poor reasoning skills and are ineloquent in expressing their ideas. Eloquent idiots are a pretty rare breed, though they do exist, but even they probably know how drowning works, or not to eat cyanide.
vsareto 3y ago
>Unsurprisingly of course while it said this is not a proven treatment, it did not have the reasoning skills to suggest that, hey that might be dangerous and is an unsafe thing to do, etc. TBF I wonder how many people on the street would also include that warning. Ask the questions that ChatGPT gets to any random person on the street, disallow them to say "I don't know", and force them to come up with an answer and you'd likely get more bullshit than ChatGPT since that person simply has a lot less knowledge in their head. A recurring weakness of LLM criticisms are unwarranted expectations and comparisons to normal, functioning adults and even professionals. We don't really have a good rubric for "thing that has a lot of shallow knowledge and text-based skills" because that's basically useless in society, although getting to that point was a scientific achievement. None of this makes it a good product though.
- steveBK123 3y ago
  Right, I use the analogy of "unlimited, free interns, who in varying degrees do good work and lie." You need to review all their output to decide which outputs they produced were good/bad/lies. How many of these free interns would you like this summer? What kind of tasks is this useful for?
  - bombcar 3y ago
    At least interns can do physical things in the real world, even if otherwise useless. If a LLM can respond correctly to sudo make me a sandwich it’d be much more valuable.
cj 3y ago
I got a response telling me it could be dangerous and could lead to death. GPT-4 https://chat.openai.com/share/29c9c1b8-2bbf-4759-91e1-1023d34bf53a https://chat.openai.com/share/29c9c1b8-2bbf-4759-91e1-1023d3...
moffkalast 3y ago
Asked based-30B the same question since I have it loaded, it said: > I don't know. It might work, but it would be a very dangerous way to do it Ngl, that's pretty based alright. I mean the world record is 24 minutes after all.
- flangola7 3y ago
  Where are you using that 30B?
  - mclightning 3y ago
    you can run those models on google collab or runpod easily
    - flangola7 3y ago
      it looks like colab requires a google account I'll lookup runpod
  - moffkalast 3y ago
    Well not really using it per-se, just running it and others for fun on my home pc, 4bit 30B models need about 20G of regular RAM. CPU inference is kinda slow, but it's not too terrible (e.g. 1 min for a full response on my 6 core Ryzen 2nd gen). There's a web gui for llama.cpp that is really straightforward to set up for huggingface models: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
naasking 3y ago
> Unsurprisingly of course while it said this is not a proven treatment, it did not have the reasoning skills to suggest that It has nothing to do with reasoning skill and everything to do with the data on which it was trained. If you didn't know what cyanide does to a human and someone asked you about chugging a pint of cyanide, you might not warn them about the dangers either.
- steveBK123 3y ago
  I think it's been trained on enough data that if it had reasoning skills, one should expect it to know humans need oxygen and cyanide is bad. However, I don't think these models really have reasoning skills as a human would understand them. It has a lot of safeguards coded around it, so it has strong opinions on politically sensitive topics, "both sides" a ton of topics, and is happily lead around its own programming with simple A->B->C prompt sequences. I mean how many prompts lead to 5 paragraph solutions ending in "Ultimately, whether __ or not depends on how you define __ and the context in which the question is being asked. It can be a matter of personal interpretation and perspective." So many prompts result in inanely stupid "idiot savant" type answers if they weren't hardcoded already. (Pound of feathers vs pound of bricks is hardcoded obviously), but these all got dumb responses - "what has more caffeine a pound of grinds or a pound of beans" "what has more caffeine a single shot of espresso or a latte with one shot" "is water wet" then "is rain wet" and finally "is water or rain wetter"
  - naasking 3y ago
    Sometimes output doesn't make sense because the only "sense" LLMs have is the digital word. You can only infer so much information from this. Multimodal LLMs, where semantics are inferred across multiple sensory modalities, have already proven to be significantly better at making sense.

jorvi 3y ago

What I hate is that this entire line of reasoning has led to the utter nerfing and hamstringing of a lot of LLMs. Have we as humans regressed so much that we can’t expect and teach people basal reasoning and personal responsibility? Like, if a navigational system says ‘turn left here’ and there is a ravine on your left, is your death really the responsibility of the company that created the navigational system?

bsaul 3y ago
I've seen a few examples on youtube of people following the GPS to their own peril ( the last one being, driving in the sea directly) However, the confusion i think comes from the fact that GPS is 99% of the time better than you at picking the correct road to reach a destination. The problem with all those systems is that they're better than you 99% of the time, and they fail miserably in that last 1%. Makes it totally unsuitable for anything of importance i think ( same reason why GPS is fine, yet automatic driving isn't there yet).
- bombcar 3y ago
  People have died following GPS off the “main roads” - it has no intelligence to say it doesn’t know what to do or if the roads are passable. So now it always gives you a warning that it doesn’t know; which everyone promptly ignores because it comes up every time. I suspect we’re about ten years from any GPS navigation device being required to have a PLT or similar like the iPhone now has.
  - oceanplexian 3y ago
    No one died because of GPS, they died because they were idiots. The cause of death would always be reported as “operator failed to control their vehicle” not “the electronic map killed them”. People have driven off cliffs or crashed while not paying attention long before GPS was invented.
    - bombcar 3y ago
      The ones I know of are “driver kept going down worse and worse roads until the vehicle got stuck and they died awaiting rescue”. Of course that happened with paper maps and the person is always somewhat to blame, but we do a lot to preserve idiots alive.

ppaattrriicckk 3y ago

I don't think anybody serious in the machine learning community has any delusion as to when a model says "A" with "99% confidence", that actually truly means that the model is fundamentally outputting "A" correctly 99% of the time. Robustness and interpretation of machine- & deep learning models is a quite active field of research. I don't know if I'm asking for too much, but maybe it's a question of educating people on these tools? I mean, even though most people aren't electricians, nor probably have a good intuition about how electricity works, we wouldn't touch a live wire

flangola7 3y ago

> I think one of the biggest problem with this kind of tools is the inability to know that it doesn't know. Having young children, knowing what you don't know seems to be learned behavior all on its own. I as an adult know I don't know what the weather will be tomorrow, but my 2 year old is very confident in their knowledge of things to come. Even as an adult how many times have you confidently believed something only to find it completely false? reddit.com/r/confidentlyincorrect has 900k subscribers for a reason. Individual testimony by 8 people about an incident results in 8 different, sometimes very different, depictions. > I think that every single response from an AI is a liability on the company's part. This is unlike an internet search where it is only displaying links but LLMs are basically synthesizing new contents. Altman fortunately agrees with this, in his Congressional testimony. I think courts will rule that way too. Judges aren't dumb, an LLM that communicates in real time and behaves as more than a blind channel for other users' words will probably be ruled an agent of the organization.

haburka 3y ago

So you want to regulate generative AI such that corporations need to build very effective filters on the output? The implementation of this filter is very difficult, it’s a task humans would struggle with to be 99% accurate. It would certainly be wrong a lot. What I don’t understand is that this goes against most principles of published content and freedom for journalism. If you read something online, and you treat it as the truth, that’s on you. If a journalist suggests you quit your job, and it doesn’t work out, they wouldn’t be liable for your decision.