5 ms·

ChatGPT and Gemini literally only know the answer because they read StackOverflow. Stack Overflow only exists because they have visitors. What do you think wil

ChatGPT and Gemini literally only know the answer because they read StackOverflow. Stack Overflow only exists because they have visitors. What do you think will happen when everyone is using the AI tools to answer their questions? We'll be back in the world of Encyclopedias, in which central authorities spent large amounts of money manually collecting information and publishing it. And then they spent a good amount of time finding ways to sell that information to us, which was only fair because they spent all that time collating it. The internet pretty much destroyed that business model, and in some sense the AI "revolution" is trying to bring it back. Also, he's specifically talking about having a coding tool write the code for you, he's not talking about using an AI tool to answer a question, so that you can go ahead and write the code yourself. These are different things, and he is treating them differently.

socalgal2 1y ago
> ChatGPT and Gemini literally only know the answer because they read StackOverflow. Stack Overflow only exists because they have visitors. I know this isn't true because I work on an API that has no answers on stackoverflow (too new), nor does it have answers anywhere else. Yet, the AI seems to able to accurately answer many questions about it. To be honest I've been somewhat shocked at this.
- b112 1y ago
  It is absolutely true, and AI cannot think, reason, comprehend anything it has not seen before. If you're getting answers, it has seen it elsewhere, or it is literally dumb, statistical luck. That doesn't mean it knows the answer. That means it guessed or hallucinated correctly. Guessing isn't knowing. edit: people seem to be missing my point, so let me rephrase. Of course AIs don't think, but that wasn't what I was getting at. There is a vast difference between knowing something, and guessing. Guessing, even in humans, is just the human mind statistically and automatically weighing probabilities and suggesting what may be the answer. This is akin to what a model might do, without any real information. Yet in both cases, there's zero validation that anything is even remotely correct. It's 100% conjecture. It therefore doesn't know the answer, it guessed it. When it comes to being correct about a language or API that there's zero info on, it's just pure happenstance that it got it correct. It's important to know the differences, and not say it "knows" the answer. It doesn't. It guessed. One of the most massive issues with LLMs is we don't get a probability response back. You ask a human "Do you know how this works", and an honest and helpful human might say "No" or "No, but you should try this. It might work". That's helpful. Conversely a human pretending it knows and speaking with deep authority when it doesn't is a liar. LLMs need more of this type of response, which indicates certainty or not. They're useless without this. But of course, an LLM indicating a lack of certainty, means that customers might use it less, or not trust it as much, so... profits first! Speak with certainty on all things!
  - lechatonnoir 1y ago
    This is such a pointless, tired take. You want to say this guy's experience isn't reproducible? That's one thing, but that's probably not the case unless you're assuming they're pretty stupid themselves. You want to say that it Is reproducible, but that "that doesn't mean AI can think"? Okay, but that's not what the thread was about.
  - PeterStuer 1y ago
    What would convince you otherwise? The reason I ask is because you sound like you have made up your mind phylosophically, not based on practical experience.
  - rsanheim 1y ago
    It's just Pattern matching. Most APIs, and hell, most code is not unique or special. Its all been done a thousands of times before. Thats why an LLM can be helpful on some tool you've written just for yourself and never released anywhere. As to 'knows the answer', I'm don't even know what that means with these tools. All I know is if it is helpful or not.
    - danielbln 1y ago
      Also, most problems are decomposable into simpler, certainly not novel parts. That intractable unicorn problem I hear so much about is probably composed of very pedestrian sub-problems.
      - genewitch 1y ago
        to what does "unicorn problem" refer to? a specific thing or a general idea?
        danielbln 1y ago
        General idea, it comes up often in LLM discussion from engineers lamenting that LLMs can never solve any novel problems. This might as well be true but as said, almost any problem can be decomposed into bite sized, non-novel parts.
- gwhr 1y ago
  What kind of API is it? Curious if it's a common problem that the AI was able to solve?
olmo23 1y ago
Where does the knowledge come from? People can only post to SO if they've read the code or the documentation. I don't see why LLMs couldn't do that.
- nobunaga 1y ago
  ITT: People who think LLMs are AGI and can produce output that the LLM has come up with out of thin air or by doing research. Go speak with someone who is actually an expert in this field how LLMs work and why the training data is so important. Im amazed that people in the CS industry seem to talk like they know everything about a tech after using it but never even writing a line of code for an LLM. Our indsutry is doomed with people like this.
  - usef- 1y ago
    This isn't about being AGI or not, and it's not "out of thin air". Modern implementations of LLMs can "do research" by performing searches (whose results are fed into the context), or in many code editors/plugins, the editor will index the project codebase/docs and feed relevant parts into the context. My guess is they either were using the LLM from a code editor, or one of the many LLMs that do web searches automatically (ie. all of the popular ones). They are answering non-stackoverflow questions every day, already.
    - nobunaga 1y ago
      Yeah, doing web searches could be called research but thats not what we are talking bout. Read the parent of the parent. Its about being able to answer questions thats not in its training data. People are talking about LLMs making scientific discoveries that humans haven't. A ridiculous take. Its not possible and with the current state of tech never will be. I know what LLMs are trained on. Thats not the topic of conversation.
      - oezi 1y ago
        A large part of research is just about creatively re-arranging symbolic information and LLMs are great at this kind of research. For example discovering relevant protein sequences.
kypro 1y ago
The idea that LLMs can only spew out text they've been trained on is a fundamental miss-understanding of how modern backprop training algorithms work. A lot of work goes into refining training algorithms to preventing overfitting of the training data. Generalisation is something that neural nets are pretty damn good at, and given the complexity of modern LLMs the idea that they cannot generalise the fairly basic logical rules and patterns found in code such that they're able provide answers to inputs unseen in the training data is quite an extreme position.
- fpoling 1y ago
  Yet the models do not (yet) reason. Try to ask them to solve a programming puzzle or exercise from an old paper book that was not scanned. They will produce total garbage. Models work across programming languages because it turned out programming languages and API are much more similar than one could have expected.
erikerikson 1y ago
I broadly agree that cutting new knowledge will need to continue being done and that overuse of LLMs could undermine that, yet... When was the last time you paid to read an APIs' docs? It costs money for companies to make those too.
AlwaysRock 1y ago
> ChatGPT and Gemini literally only know the answer because they read StackOverflow. Stack Overflow only exists because they have visitors. I mean... They also can read actual documentation. If I'm working on any api work or a language I'm not familiar with, I ask the LLM to include the source they got their answer from and use official documentation when possible. That lowers the hallucination rate significantly and also lets me ensure said function or code actually does what the llm reports it does. In theory, all stackoverflow answers are just regurgitated documentation, no?
- sothatsit 1y ago
  > I mean... They also can read actual documentation. This 100%. I use o3 as my primary search engine now. It is brilliant at finding relevant sources, summarising what is relevant from them, and then also providing the links to those sources so I can go read them myself. The release of o3 was a turning point for me where it felt like these models could finally go and fetch information for themselves. 4o with web search always felt inadequate, but o3 does a very good job. > In theory, all stackoverflow answers are just regurgitated documentation, no? This is unfair to StackOverflow. There is a lot of debugging and problem solving that has happened on that platform of undocumented bugs or behaviour.
CamperBob2 1y ago
We'll start writing documentation for primary consumption by LLMs rather than human readers. The need for sites like SO will not vanish overnight but it will diminish drastically.
semiquaver 1y ago
> ChatGPT and Gemini literally only know the answer because they read StackOverflow Obviously this isn’t true. You can easily verify this by inventing and documenting an API and feeding that description to an LLM and asking it how to use it. This works well. LLMs are quite good at reading technical documentation and synthesizing contextual answers from it.
reaperducer 1y ago
We'll be back in the world of Encyclopedias On a related note, I recently learned that you can still subscribe to the Encyclopedia Britannica. It's $9/month, or $75/year. Considering the declining state of Wikipedia, and the untrustworthiness of A.I., I'm considering it.

socalgal2 1y ago

> ChatGPT and Gemini literally only know the answer because they read StackOverflow. Stack Overflow only exists because they have visitors. I know this isn't true because I work on an API that has no answers on stackoverflow (too new), nor does it have answers anywhere else. Yet, the AI seems to able to accurately answer many questions about it. To be honest I've been somewhat shocked at this.

b112 1y ago
It is absolutely true, and AI cannot think, reason, comprehend anything it has not seen before. If you're getting answers, it has seen it elsewhere, or it is literally dumb, statistical luck. That doesn't mean it knows the answer. That means it guessed or hallucinated correctly. Guessing isn't knowing. edit: people seem to be missing my point, so let me rephrase. Of course AIs don't think, but that wasn't what I was getting at. There is a vast difference between knowing something, and guessing. Guessing, even in humans, is just the human mind statistically and automatically weighing probabilities and suggesting what may be the answer. This is akin to what a model might do, without any real information. Yet in both cases, there's zero validation that anything is even remotely correct. It's 100% conjecture. It therefore doesn't know the answer, it guessed it. When it comes to being correct about a language or API that there's zero info on, it's just pure happenstance that it got it correct. It's important to know the differences, and not say it "knows" the answer. It doesn't. It guessed. One of the most massive issues with LLMs is we don't get a probability response back. You ask a human "Do you know how this works", and an honest and helpful human might say "No" or "No, but you should try this. It might work". That's helpful. Conversely a human pretending it knows and speaking with deep authority when it doesn't is a liar. LLMs need more of this type of response, which indicates certainty or not. They're useless without this. But of course, an LLM indicating a lack of certainty, means that customers might use it less, or not trust it as much, so... profits first! Speak with certainty on all things!
- lechatonnoir 1y ago
  This is such a pointless, tired take. You want to say this guy's experience isn't reproducible? That's one thing, but that's probably not the case unless you're assuming they're pretty stupid themselves. You want to say that it Is reproducible, but that "that doesn't mean AI can think"? Okay, but that's not what the thread was about.
- PeterStuer 1y ago
  What would convince you otherwise? The reason I ask is because you sound like you have made up your mind phylosophically, not based on practical experience.
- rsanheim 1y ago
  It's just Pattern matching. Most APIs, and hell, most code is not unique or special. Its all been done a thousands of times before. Thats why an LLM can be helpful on some tool you've written just for yourself and never released anywhere. As to 'knows the answer', I'm don't even know what that means with these tools. All I know is if it is helpful or not.
  - danielbln 1y ago
    Also, most problems are decomposable into simpler, certainly not novel parts. That intractable unicorn problem I hear so much about is probably composed of very pedestrian sub-problems.
    - genewitch 1y ago
      to what does "unicorn problem" refer to? a specific thing or a general idea?
      - danielbln 1y ago
        General idea, it comes up often in LLM discussion from engineers lamenting that LLMs can never solve any novel problems. This might as well be true but as said, almost any problem can be decomposed into bite sized, non-novel parts.
gwhr 1y ago
What kind of API is it? Curious if it's a common problem that the AI was able to solve?

olmo23 1y ago

Where does the knowledge come from? People can only post to SO if they've read the code or the documentation. I don't see why LLMs couldn't do that.

nobunaga 1y ago
ITT: People who think LLMs are AGI and can produce output that the LLM has come up with out of thin air or by doing research. Go speak with someone who is actually an expert in this field how LLMs work and why the training data is so important. Im amazed that people in the CS industry seem to talk like they know everything about a tech after using it but never even writing a line of code for an LLM. Our indsutry is doomed with people like this.
- usef- 1y ago
  This isn't about being AGI or not, and it's not "out of thin air". Modern implementations of LLMs can "do research" by performing searches (whose results are fed into the context), or in many code editors/plugins, the editor will index the project codebase/docs and feed relevant parts into the context. My guess is they either were using the LLM from a code editor, or one of the many LLMs that do web searches automatically (ie. all of the popular ones). They are answering non-stackoverflow questions every day, already.
  - nobunaga 1y ago
    Yeah, doing web searches could be called research but thats not what we are talking bout. Read the parent of the parent. Its about being able to answer questions thats not in its training data. People are talking about LLMs making scientific discoveries that humans haven't. A ridiculous take. Its not possible and with the current state of tech never will be. I know what LLMs are trained on. Thats not the topic of conversation.
    - oezi 1y ago
      A large part of research is just about creatively re-arranging symbolic information and LLMs are great at this kind of research. For example discovering relevant protein sequences.

kypro 1y ago

The idea that LLMs can only spew out text they've been trained on is a fundamental miss-understanding of how modern backprop training algorithms work. A lot of work goes into refining training algorithms to preventing overfitting of the training data. Generalisation is something that neural nets are pretty damn good at, and given the complexity of modern LLMs the idea that they cannot generalise the fairly basic logical rules and patterns found in code such that they're able provide answers to inputs unseen in the training data is quite an extreme position.

fpoling 1y ago
Yet the models do not (yet) reason. Try to ask them to solve a programming puzzle or exercise from an old paper book that was not scanned. They will produce total garbage. Models work across programming languages because it turned out programming languages and API are much more similar than one could have expected.

erikerikson 1y ago

I broadly agree that cutting new knowledge will need to continue being done and that overuse of LLMs could undermine that, yet... When was the last time you paid to read an APIs' docs? It costs money for companies to make those too.

AlwaysRock 1y ago

> ChatGPT and Gemini literally only know the answer because they read StackOverflow. Stack Overflow only exists because they have visitors. I mean... They also can read actual documentation. If I'm working on any api work or a language I'm not familiar with, I ask the LLM to include the source they got their answer from and use official documentation when possible. That lowers the hallucination rate significantly and also lets me ensure said function or code actually does what the llm reports it does. In theory, all stackoverflow answers are just regurgitated documentation, no?

sothatsit 1y ago
> I mean... They also can read actual documentation. This 100%. I use o3 as my primary search engine now. It is brilliant at finding relevant sources, summarising what is relevant from them, and then also providing the links to those sources so I can go read them myself. The release of o3 was a turning point for me where it felt like these models could finally go and fetch information for themselves. 4o with web search always felt inadequate, but o3 does a very good job. > In theory, all stackoverflow answers are just regurgitated documentation, no? This is unfair to StackOverflow. There is a lot of debugging and problem solving that has happened on that platform of undocumented bugs or behaviour.

CamperBob2 1y ago

We'll start writing documentation for primary consumption by LLMs rather than human readers. The need for sites like SO will not vanish overnight but it will diminish drastically.

semiquaver 1y ago

> ChatGPT and Gemini literally only know the answer because they read StackOverflow Obviously this isn’t true. You can easily verify this by inventing and documenting an API and feeding that description to an LLM and asking it how to use it. This works well. LLMs are quite good at reading technical documentation and synthesizing contextual answers from it.

reaperducer 1y ago

We'll be back in the world of Encyclopedias On a related note, I recently learned that you can still subscribe to the Encyclopedia Britannica. It's $9/month, or $75/year. Considering the declining state of Wikipedia, and the untrustworthiness of A.I., I'm considering it.