7 ms·

I find this to be really concerning if true (which I’m not convinced of, seems more like correlation than causation). The most amazing part of chat ai is its a

I find this to be really concerning if true (which I’m not convinced of, seems more like correlation than causation). The most amazing part of chat ai is its ability to be very convincing and yet factually incorrect. It doesn’t seem like it’s a secret that these chat apps spit out just plain wrong information routinely, I’m not sure why anyone would trust it with something important. SO has bad info too, for sure, but part of SO is peer review to help feel that out. “AI” is a black box and not to be trusted.

perryizgr8 3y ago
You should try Bing chat. It provides citations for almost every assertion. Allowing you to get a quick summary/conclusion, which is also quickly verifiable if you want.
- williamstein 3y ago
  https://www.phind.com/ https://www.phind.com/ is also similarly good, but for dev related content.
anileated 3y ago
I think it’s likely that, unless regulated, commercial LLMs will have a chilling effect on most human-created information sources. With eliminating the opportunity for authors to engage with information seekers (a chance to get constructive feedback, upsell something, show ads, see interest stats) will come reduced willingness of authors to publish. If you publish on StackOverflow only for LLM to gobble it up and no human even sees the upvote button anymore… why would you? You are basically an unpaid ghost content producer for Microsoft/ClosedAI at this point.
- two_in_one 3y ago
  It opens an opportunity for shilling, pushing your agenda, and other ways of poisoning users' generated content. While LLMs will be brainlessly sucking in and replicating. Till they learn to check first and filter out obvious fakes. Still there will be not obvious. The problem is how to make money on this...
  - afarviral 3y ago
    I wonder if the very nature of LLM emergent correctness and intelligence itself will identify weakness, fallacies and illusion? I would expect a well-read person to be similarly immune.
    - two_in_one 3y ago
      Humans aren't immune to manipulations. Neither personally, not in mass. LLMs should be possible to manipulate to some degree if attacker controls significant portion of information on some topic. For example manufacturers can 'upgrade' specs on their devices. Till they get caught LLMs will parrot them as a ground truth. The same with 'human values' and political events interpretation. China already introduced policy on LLMs, they should be politically correct. LLMs producers can manipulate, but not eliminate, the bias by selecting and weighting the sources. Which they do. Plus humans interventions when they see unpleasant output.
      - anileated 3y ago
        Humans have free will and ethical standards and ability to manipulate ideas. LLMs have none of that. They are tools designed to do what told.
        two_in_one 3y ago
        This is a simplistic vision. First, not everybody believes they have no consciousness, or ideas. Or cannot generate new ideas. This already happened with 'crazy' chess gameplay style. Humans had never seen it before. Yes, they are tools, build by humans. But this doesn't mean humans can understand what, how, and why they are doing. Historically observations may take hundreds of years to explain.
        anileated 3y ago
        If it is admitted to be conscious, idea-understanding, it would be enough like a human and would have to have human rights (and not abused like it currently is). If it doesn’t have human rights, it’s not human-like enough for this to matter. I think these are mutually exclusive.
cozzyd 3y ago
I tried asking ChatGPT how to do something for the first time yesterday (an example of how to use libtar to write a tar file from buffers in memory, in C). It gave a very confident, but spectacularly wrong answer, concocting a whole new (admittedly more convenient for this purpose) API for libtar. After that experience, I'm not sure I'll ever ask ChatGPT for help again...
- markmark 3y ago
  ChatGPT is (currently) like having an incredibly fast junior engineer working for you. You can ask it to do tasks for you and it will do them quickly, but you need to check its work. Sometimes it will be perfect, sometimes it will be right with a little more prompting, and sometimes it is completely wrong.
- afarviral 3y ago
  Yep. But how does it compare to GPT-4 or Bing Chat? Then wonder how well GPT-5 will perform, and 6. Also, did rubber-ducking your query have a benefit regardless of the correctness of chat's answer?
- klipklop 3y ago
  You use 3.5 or 4? It’s worth paying for 4. Significantly better results.
- tsuujin 3y ago
  I think the premier use for ChatGPT right now is as a starting place for writing tasks. I use it to give me a template for work emails from time to time because there is no element of correctness to get wrong. I give it facts I want to see in prose, it gives me very neutral, inoffensive prose back. I always have to issue a few corrections to the prompt, and I always have to post edit, but it still saves me a huge amount of time and I don’t need to worry so much about bias leaking in and making anyone uncomfortable.
labster 3y ago
> “AI” is a black box and not to be trusted. Someone will be the first against the wall when the revolution comes. I know treason agains Friend Computer when I see it. Now if you’ll excuse me, I have some hallways to paint red.
afarviral 3y ago
I find it best to use it with it framed as an untrustworthy black box, even with the increasingly correct answers. Its a rubber duck that can spitball with you.

perryizgr8 3y ago

You should try Bing chat. It provides citations for almost every assertion. Allowing you to get a quick summary/conclusion, which is also quickly verifiable if you want.

williamstein 3y ago
https://www.phind.com/ https://www.phind.com/ is also similarly good, but for dev related content.

anileated 3y ago

I think it’s likely that, unless regulated, commercial LLMs will have a chilling effect on most human-created information sources. With eliminating the opportunity for authors to engage with information seekers (a chance to get constructive feedback, upsell something, show ads, see interest stats) will come reduced willingness of authors to publish. If you publish on StackOverflow only for LLM to gobble it up and no human even sees the upvote button anymore… why would you? You are basically an unpaid ghost content producer for Microsoft/ClosedAI at this point.

two_in_one 3y ago
It opens an opportunity for shilling, pushing your agenda, and other ways of poisoning users' generated content. While LLMs will be brainlessly sucking in and replicating. Till they learn to check first and filter out obvious fakes. Still there will be not obvious. The problem is how to make money on this...
- afarviral 3y ago
  I wonder if the very nature of LLM emergent correctness and intelligence itself will identify weakness, fallacies and illusion? I would expect a well-read person to be similarly immune.
  - two_in_one 3y ago
    Humans aren't immune to manipulations. Neither personally, not in mass. LLMs should be possible to manipulate to some degree if attacker controls significant portion of information on some topic. For example manufacturers can 'upgrade' specs on their devices. Till they get caught LLMs will parrot them as a ground truth. The same with 'human values' and political events interpretation. China already introduced policy on LLMs, they should be politically correct. LLMs producers can manipulate, but not eliminate, the bias by selecting and weighting the sources. Which they do. Plus humans interventions when they see unpleasant output.
    - anileated 3y ago
      Humans have free will and ethical standards and ability to manipulate ideas. LLMs have none of that. They are tools designed to do what told.
      - two_in_one 3y ago
        This is a simplistic vision. First, not everybody believes they have no consciousness, or ideas. Or cannot generate new ideas. This already happened with 'crazy' chess gameplay style. Humans had never seen it before. Yes, they are tools, build by humans. But this doesn't mean humans can understand what, how, and why they are doing. Historically observations may take hundreds of years to explain.
        anileated 3y ago
        If it is admitted to be conscious, idea-understanding, it would be enough like a human and would have to have human rights (and not abused like it currently is). If it doesn’t have human rights, it’s not human-like enough for this to matter. I think these are mutually exclusive.

cozzyd 3y ago

I tried asking ChatGPT how to do something for the first time yesterday (an example of how to use libtar to write a tar file from buffers in memory, in C). It gave a very confident, but spectacularly wrong answer, concocting a whole new (admittedly more convenient for this purpose) API for libtar. After that experience, I'm not sure I'll ever ask ChatGPT for help again...

markmark 3y ago
ChatGPT is (currently) like having an incredibly fast junior engineer working for you. You can ask it to do tasks for you and it will do them quickly, but you need to check its work. Sometimes it will be perfect, sometimes it will be right with a little more prompting, and sometimes it is completely wrong.
afarviral 3y ago
Yep. But how does it compare to GPT-4 or Bing Chat? Then wonder how well GPT-5 will perform, and 6. Also, did rubber-ducking your query have a benefit regardless of the correctness of chat's answer?
klipklop 3y ago
You use 3.5 or 4? It’s worth paying for 4. Significantly better results.
tsuujin 3y ago
I think the premier use for ChatGPT right now is as a starting place for writing tasks. I use it to give me a template for work emails from time to time because there is no element of correctness to get wrong. I give it facts I want to see in prose, it gives me very neutral, inoffensive prose back. I always have to issue a few corrections to the prompt, and I always have to post edit, but it still saves me a huge amount of time and I don’t need to worry so much about bias leaking in and making anyone uncomfortable.

labster 3y ago

> “AI” is a black box and not to be trusted. Someone will be the first against the wall when the revolution comes. I know treason agains Friend Computer when I see it. Now if you’ll excuse me, I have some hallways to paint red.

afarviral 3y ago

I find it best to use it with it framed as an untrustworthy black box, even with the increasingly correct answers. Its a rubber duck that can spitball with you.