4 ms·

Someone in my company spent the past month setting up ChatGPT to work with our company's knowledge base. Not by a plugin or anything, just by telling ChatGPT w

Someone in my company spent the past month setting up ChatGPT to work with our company's knowledge base. Not by a plugin or anything, just by telling ChatGPT where to find it. They didn't believe that ChatGPT was making any of it up, just that sometimes it got it wrong. I stopped arguing after a while.

simonw 4y ago
Oh no! Sounds like there are two misconceptions there: the idea that ChatGPT can read URLs (it can't - https://simonwillison.net/2023/Mar/10/chatgpt-internet-access/ https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces... ) and the idea that ChatGPT can remember details of conversations past the boundaries of the current chat. This is something I've noticed too: occasionally I'll find someone who is INCREDIBLY resistant to learning that ChatGPT can't read URLs. It seems to happen mostly with people who have been pasting URLs into it for weeks and trusting what came back - they'd rather continue to believe in a provably false capability than admit that they've wasted a huge amount of time believing made-up bullshit.
- JohnFen 4y ago
  > they'd rather continue to believe in a provably false capability than admit that they've wasted a huge amount of time believing made-up bullshit This is an incredibly strong human tendency in general. We all do it, including you and I. It's one of the things that it's wise to be constantly on guard about.
- wilg 4y ago
  They definitely need to add some guardrails to warn it can’t read URLs like they do when you try to ask anything fun.
  - oezi 4y ago
    What is the point? In a couple of months the plugins will be generally available and then it will be able to.
    - wilg 4y ago
      The point is to not mislead people about what is going on, of course.
- MandieD 4y ago
  Ah, that’s why it messed up my rather simple request to generate a SQL query to get all messages received by a given call sign, based on the table definition at https://wspr.live https://wspr.live - it picked plausible, but nonexistent table and column names. I took “this thing isn’t nearly as smart as everyone’s making it out to be” from that session, but you’re the first person to make it clear that it’s not actually reading the rather simple page I directed it to.
- shepardrtc 4y ago
  What really tricked them was that our company's website is in its corpus. At least the old version. So when asked about different aspects of our products or company, it would give answers that were close, but not quite correct.
- lm28469 4y ago
  > they'd rather continue to believe in a provably false capability For some people it borderline is a cult now, even here on HN, they'll give it intelligence, personality, character, &c.
- cameronfraser 4y ago
  Oh man I was totally in the camp of people that thought it could read URL's until reading this comment
kozikow 4y ago
I was able to create a GPT-4 based bot initally based on knowledge base that provides accurate information. To do this, I first converted a knowledge base article into a question and answer (Q&A) format using GPT-4 - quick explanation and article link if necessary. Then, I used the API to generate more Q&A pairs by asking GPT-4 to predict what users might ask and create corresponding answers. On my side I now search for the most relevant Q&A pair based on the embedding of user's input and QA and jam as much as I can into the token limit. It provides accurate answers 99% of the time. If it can't find a suitable answer, it may create a plausible response on the spot, but that's getting rarer as training set grows. To prevent the bot from providing incomplete information, you can instruct it to ask users to contact support via email if it doesn't have enough information - either prompt engineering or examples in training set. Alternatively, you can have the bot insert a token like "%%TICKET%%" which you can later use to open a support ticket, summarizing the conversation and attaching relevant chat history, logs, etc.

simonw 4y ago

Oh no! Sounds like there are two misconceptions there: the idea that ChatGPT can read URLs (it can't - https://simonwillison.net/2023/Mar/10/chatgpt-internet-access/ https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces... ) and the idea that ChatGPT can remember details of conversations past the boundaries of the current chat. This is something I've noticed too: occasionally I'll find someone who is INCREDIBLY resistant to learning that ChatGPT can't read URLs. It seems to happen mostly with people who have been pasting URLs into it for weeks and trusting what came back - they'd rather continue to believe in a provably false capability than admit that they've wasted a huge amount of time believing made-up bullshit.

JohnFen 4y ago
> they'd rather continue to believe in a provably false capability than admit that they've wasted a huge amount of time believing made-up bullshit This is an incredibly strong human tendency in general. We all do it, including you and I. It's one of the things that it's wise to be constantly on guard about.
wilg 4y ago
They definitely need to add some guardrails to warn it can’t read URLs like they do when you try to ask anything fun.
- oezi 4y ago
  What is the point? In a couple of months the plugins will be generally available and then it will be able to.
  - wilg 4y ago
    The point is to not mislead people about what is going on, of course.
MandieD 4y ago
Ah, that’s why it messed up my rather simple request to generate a SQL query to get all messages received by a given call sign, based on the table definition at https://wspr.live https://wspr.live - it picked plausible, but nonexistent table and column names. I took “this thing isn’t nearly as smart as everyone’s making it out to be” from that session, but you’re the first person to make it clear that it’s not actually reading the rather simple page I directed it to.
shepardrtc 4y ago
What really tricked them was that our company's website is in its corpus. At least the old version. So when asked about different aspects of our products or company, it would give answers that were close, but not quite correct.
lm28469 4y ago
> they'd rather continue to believe in a provably false capability For some people it borderline is a cult now, even here on HN, they'll give it intelligence, personality, character, &c.
cameronfraser 4y ago
Oh man I was totally in the camp of people that thought it could read URL's until reading this comment

kozikow 4y ago

I was able to create a GPT-4 based bot initally based on knowledge base that provides accurate information. To do this, I first converted a knowledge base article into a question and answer (Q&A) format using GPT-4 - quick explanation and article link if necessary. Then, I used the API to generate more Q&A pairs by asking GPT-4 to predict what users might ask and create corresponding answers. On my side I now search for the most relevant Q&A pair based on the embedding of user's input and QA and jam as much as I can into the token limit. It provides accurate answers 99% of the time. If it can't find a suitable answer, it may create a plausible response on the spot, but that's getting rarer as training set grows. To prevent the bot from providing incomplete information, you can instruct it to ask users to contact support via email if it doesn't have enough information - either prompt engineering or examples in training set. Alternatively, you can have the bot insert a token like "%%TICKET%%" which you can later use to open a support ticket, summarizing the conversation and attaching relevant chat history, logs, etc.