7 ms·

Google Co-Scientist AI fed previous paper with the answer in it

200 points by pcfwik 2y ago

dmitrygr 2y ago
not surprising. why would you expect a simple next token predictor to think?
- triceratops 2y ago
  Technically everyone is a next token predictor. The only difference is how good the prediction is.
  - olyjohn 2y ago
    Technically, everyone is just an ugly giant bag of mostly water.
  - abenga 2y ago
    Not true, we build novel knowledge all the time. You only have to look at *gestures at everything *.
    - triceratops 2y ago
      Right, sometimes we invent new tokens. It's unclear whether large models can do that yet.
    - IshKebab 2y ago
      We would not generate novel knowledge if our only interface with the world was a short text based conversation. In other words, I don't think that's a fundamental difference between humans and AI; we just haven't given AI any opportunity to build novel knowledge. Part of that is because it can't really interact with the world yet, in a significant way. Part of it is because nobody has cracked on-line learning. Actually when you give AI the ability to interact with a world, as in RL, it does build novel knowledge.
  - Frieren 2y ago
    > everyone is a next token predictor As true as "everyone is a calculator". There are superficial similarities, but that we achieve a similar result does not mean that we follow the same reasoning (or lack of).
    - triceratops 2y ago
      > does not mean that we follow the same reasoning Finally some logic. Token prediction alone - the behavior - isn't proof of a lack of intelligence. Better reasoning gives better token predictions. And, sometimes, the ability to invent new tokens entirely.
  - aezart 2y ago
    In humans, language is based on the state of the world, and anticipates future states of the world. In an LLM, language is just language. If an LLM says "my favorite flavor of ice cream is cookie dough", it's just a statistically plausible thing to say. If I say "my favorite flavor if ice cream is cookie dough", it's because I'm trying to deceive someone, because for some reason I don't want them to know my real favorite flavor is cake batter.
    - IshKebab 2y ago
      You haven't identified any difference. In order to generate a really good statistically plausible thing to say you need a model of the state of the world and anticipated future states. This is very well known. I wish we didn't need to keep going over this basic stuff, but it somehow comes up again and again every time AI is discussed here.
      - jltsiren 2y ago
        Of a world, not necessarily the real world. One of the main drawbacks of philosophical rationalism is a mind's ability do deceive itself with constructs that sound plausible and are consistent with the world model. A healthy dose of empiricism is the best cure discovered so far. In other words, less thinking and more experiments and observations. Thinking alone leads to ignorance.
        IshKebab 2y ago
        No, a model of the real world. Not necessarily an accurate no model.
  - fennecbutt 2y ago
    I mean the logic behind it is sound. Just infantile compared to what we do. But yes I believe that all we do boils down to "if x is true and also y is true, therefore under these circumstances z must be false, I'll give it a go and find out". Plus a little qualified randomness, look at the large percentage of human discoveries being made accidentally (albeit by qualified professionals making best guesses).
  - skywhopper 2y ago
    This is incredibly wrong.
  - csr86 2y ago
    Animals are not next token predictors. Most animals don't have language and don't generate sentences/ideas. It is unique feature of humans and big part of our success. Animals do have sensors and can reason about real world quite intelligently. This is what LLM- models don't have. Surely, all the text in Internet has a lot of textual information about real world, but something "obvious to animal" might not be there. I feel us humans are kind of both: We have sensors and experiences about real world, but also oral and written tradition. Once we develope LLM- with cabability to sense and experience real word (perhaps with some evolutionary algorithm to make it better over time) it should start to be close to human beings.
  - dicethrowaway1 2y ago
    This is a deepity. The trivial interpretation is: every word written can be constructed by optimizing a prediction based on current state, what has been written so far, and a sufficiently complex model. This is true of anything computable: just make the method implicitly contain the program by assigning a high probability to any token that is consistent with running the computation one more step. It's also true of anything expressible: just brute force a solution that can be expressed in n words, then assign a high probability to the first word of these n words. The profound but wrong interpretation is that intelligence is just statistical prediction according to some general-purpose algorithm, and that this algorithm is tractable. Consider something like solving a SAT problem. You're going to have a hard time using any tractable general-purpose algorithm to predict whether x_2 is true for the satisfying solution based on some long CNF statement plus "x_1 is false". Now, what you _can_ do is augment your model so that if the previous tokens constitute a CNF-SAT instance plus a partial answer, then you cart these off to a SAT solver and output its next token. But the more you do this, the less force the "mere statistical prediction" part holds. The "next-token predictor" is just an interface to an assembly of different approaches; and often, these approaches (like the SAT solver) will output the whole solution all at once for free.
- School-Cotton 2y ago
  Please rigorously define what the term "next token predictor" means.
  - dmitrygr 2y ago
    A device or system that, given a set of tokens, predicts the next token purely based on the previous tokens and its own initial state. It is incapable of coming up with a new token type or changing its own workings
    - Grimblewald 2y ago
      On top of your definition being a bit circular, it is also like saying humans can't come up with new things, because we can't come up with new characters when talking online. We're limited by the character-set we're given but that doesn't mean we can't conceive of new characters and more importantly, can't represent new characters using existing character-sets using some encoding scheme to represent that new character. To that end, lets check with chatGPT if it can come up with a new and useful concept for a token, using existing characters to encode for it. https://chatgpt.com/share/67bd04cf-1c50-8008-8606-ccac5df98931 https://chatgpt.com/share/67bd04cf-1c50-8008-8606-ccac5df989... And there you have it, novel constructed from available ones, with a genuine pragmatic and LLM specific use case.
      - suspended_state 2y ago
        [dead]
    - Lerc 2y ago
      So if they can come up with their own internal representations is that sufficient? https://arxiv.org/abs/2410.05864 https://arxiv.org/abs/2410.05864 Or are you wanting them to use keys that are not on their keyboard?
  - YeGoblynQueenne 2y ago
    Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally: P(tₖ|t₁,..,tₖ₋₁) For example, the original OpenAI GPT tranformer model was trained to optimise the objective: ∑ log P(tₖ|tₖ₋ₙ,..,tₖ₋₁;Θ) Where n the length of a sliding "context" window and Θ the parameters of a neural net model (here, a Transformer architecture) used to optimise the objectivbe. See page 3 of the OpenAI paper: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf https://cdn.openai.com/research-covers/language-unsupervised... Also see, for alternative formulations: Hidden Markov Models: https://en.wikipedia.org/wiki/Hidden_Markov_model https://en.wikipedia.org/wiki/Hidden_Markov_model N-Gram word models: https://en.wikipedia.org/wiki/Word_n-gram_language_model https://en.wikipedia.org/wiki/Word_n-gram_language_model
euroderf 2y ago
If the AI found a needle in a haystack, that isn't bad, is it?
- Lastminutepanic 2y ago
  Do you think it's possible that the person who literally wrote (multiple) papers with the answer AI gave, may phrase their question to the AI with similar, or even the exact phrasing they used in previous papers on the same subject ? If I load a bunch of lorem ipsum into AI scientist, labeled as scientific papers, then ask the AI "lorem ipsum" do you think it'll spit out some gibberish from those specific papers?
- kedean 2y ago
  The point isn't that it's bad, the point is that the AI did not discover the solution to the problem independently, as news headlines had implied.
- asddubs 2y ago
  it is if you're presenting it as coming up with novel information rather than essentially acting as a search engine for information from its training data (which we already know ai does well, at least with some caveats)
hirenj 2y ago
Colour me unsurprised - even not knowing data leakage had occurred, the hypothesis was underwhelming, as I mentioned in a comment on an earlier discussion. I sometimes despair for the state of thinking in science these days given how quickly people fawn over entirely pedestrian thinking and work. https://news.ycombinator.com/item?id=43105759 https://news.ycombinator.com/item?id=43105759
mandevil 2y ago
I mean, to be fair, "knowing all of the relevant literature" is a great first step for solving a problem! And a LLM can probably be better about not forgetting than humans are (though I'm guessing they do much worse on the hallucination front than humans do- humans tend to know when they are playing a hunch). But "This paper from a lower-tier journal two years ago suggests you should look at X" is a very valuable thing to have!
rideontime 2y ago
Instant subscribe when I saw David Gerard's name. He cut through the BS in crypto and we need more people like him focusing on AI fraud like this.
- davidgerard 2y ago
  thank you!
gukoff 2y ago
> Prof Penadés' said the tool had in fact done more than successfully replicating his research. > "It's not just that the top hypothesis they provide was the right one," he said. > "It's that they provide another four, and all of them made sense. > "And for one of them, we never thought about it, and we're now working on that." The Google's co-scientist still seems to make a useful assistant.

200 points by pcfwik 2y ago

dmitrygr 2y ago

not surprising. why would you expect a simple next token predictor to think?

triceratops 2y ago
Technically everyone is a next token predictor. The only difference is how good the prediction is.
- olyjohn 2y ago
  Technically, everyone is just an ugly giant bag of mostly water.
- abenga 2y ago
  Not true, we build novel knowledge all the time. You only have to look at *gestures at everything *.
  - triceratops 2y ago
    Right, sometimes we invent new tokens. It's unclear whether large models can do that yet.
  - IshKebab 2y ago
    We would not generate novel knowledge if our only interface with the world was a short text based conversation. In other words, I don't think that's a fundamental difference between humans and AI; we just haven't given AI any opportunity to build novel knowledge. Part of that is because it can't really interact with the world yet, in a significant way. Part of it is because nobody has cracked on-line learning. Actually when you give AI the ability to interact with a world, as in RL, it does build novel knowledge.
- Frieren 2y ago
  > everyone is a next token predictor As true as "everyone is a calculator". There are superficial similarities, but that we achieve a similar result does not mean that we follow the same reasoning (or lack of).
  - triceratops 2y ago
    > does not mean that we follow the same reasoning Finally some logic. Token prediction alone - the behavior - isn't proof of a lack of intelligence. Better reasoning gives better token predictions. And, sometimes, the ability to invent new tokens entirely.
- aezart 2y ago
  In humans, language is based on the state of the world, and anticipates future states of the world. In an LLM, language is just language. If an LLM says "my favorite flavor of ice cream is cookie dough", it's just a statistically plausible thing to say. If I say "my favorite flavor if ice cream is cookie dough", it's because I'm trying to deceive someone, because for some reason I don't want them to know my real favorite flavor is cake batter.
  - IshKebab 2y ago
    You haven't identified any difference. In order to generate a really good statistically plausible thing to say you need a model of the state of the world and anticipated future states. This is very well known. I wish we didn't need to keep going over this basic stuff, but it somehow comes up again and again every time AI is discussed here.
    - jltsiren 2y ago
      Of a world, not necessarily the real world. One of the main drawbacks of philosophical rationalism is a mind's ability do deceive itself with constructs that sound plausible and are consistent with the world model. A healthy dose of empiricism is the best cure discovered so far. In other words, less thinking and more experiments and observations. Thinking alone leads to ignorance.
      - IshKebab 2y ago
        No, a model of the real world. Not necessarily an accurate no model.
- fennecbutt 2y ago
  I mean the logic behind it is sound. Just infantile compared to what we do. But yes I believe that all we do boils down to "if x is true and also y is true, therefore under these circumstances z must be false, I'll give it a go and find out". Plus a little qualified randomness, look at the large percentage of human discoveries being made accidentally (albeit by qualified professionals making best guesses).
- skywhopper 2y ago
  This is incredibly wrong.
- csr86 2y ago
  Animals are not next token predictors. Most animals don't have language and don't generate sentences/ideas. It is unique feature of humans and big part of our success. Animals do have sensors and can reason about real world quite intelligently. This is what LLM- models don't have. Surely, all the text in Internet has a lot of textual information about real world, but something "obvious to animal" might not be there. I feel us humans are kind of both: We have sensors and experiences about real world, but also oral and written tradition. Once we develope LLM- with cabability to sense and experience real word (perhaps with some evolutionary algorithm to make it better over time) it should start to be close to human beings.
- dicethrowaway1 2y ago
  This is a deepity. The trivial interpretation is: every word written can be constructed by optimizing a prediction based on current state, what has been written so far, and a sufficiently complex model. This is true of anything computable: just make the method implicitly contain the program by assigning a high probability to any token that is consistent with running the computation one more step. It's also true of anything expressible: just brute force a solution that can be expressed in n words, then assign a high probability to the first word of these n words. The profound but wrong interpretation is that intelligence is just statistical prediction according to some general-purpose algorithm, and that this algorithm is tractable. Consider something like solving a SAT problem. You're going to have a hard time using any tractable general-purpose algorithm to predict whether x_2 is true for the satisfying solution based on some long CNF statement plus "x_1 is false". Now, what you _can_ do is augment your model so that if the previous tokens constitute a CNF-SAT instance plus a partial answer, then you cart these off to a SAT solver and output its next token. But the more you do this, the less force the "mere statistical prediction" part holds. The "next-token predictor" is just an interface to an assembly of different approaches; and often, these approaches (like the SAT solver) will output the whole solution all at once for free.
School-Cotton 2y ago
Please rigorously define what the term "next token predictor" means.
- dmitrygr 2y ago
  A device or system that, given a set of tokens, predicts the next token purely based on the previous tokens and its own initial state. It is incapable of coming up with a new token type or changing its own workings
  - Grimblewald 2y ago
    On top of your definition being a bit circular, it is also like saying humans can't come up with new things, because we can't come up with new characters when talking online. We're limited by the character-set we're given but that doesn't mean we can't conceive of new characters and more importantly, can't represent new characters using existing character-sets using some encoding scheme to represent that new character. To that end, lets check with chatGPT if it can come up with a new and useful concept for a token, using existing characters to encode for it. https://chatgpt.com/share/67bd04cf-1c50-8008-8606-ccac5df98931 https://chatgpt.com/share/67bd04cf-1c50-8008-8606-ccac5df989... And there you have it, novel constructed from available ones, with a genuine pragmatic and LLM specific use case.
    - suspended_state 2y ago
      [dead]
  - Lerc 2y ago
    So if they can come up with their own internal representations is that sufficient? https://arxiv.org/abs/2410.05864 https://arxiv.org/abs/2410.05864 Or are you wanting them to use keys that are not on their keyboard?
- YeGoblynQueenne 2y ago
  Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally: P(tₖ|t₁,..,tₖ₋₁) For example, the original OpenAI GPT tranformer model was trained to optimise the objective: ∑ log P(tₖ|tₖ₋ₙ,..,tₖ₋₁;Θ) Where n the length of a sliding "context" window and Θ the parameters of a neural net model (here, a Transformer architecture) used to optimise the objectivbe. See page 3 of the OpenAI paper: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf https://cdn.openai.com/research-covers/language-unsupervised... Also see, for alternative formulations: Hidden Markov Models: https://en.wikipedia.org/wiki/Hidden_Markov_model https://en.wikipedia.org/wiki/Hidden_Markov_model N-Gram word models: https://en.wikipedia.org/wiki/Word_n-gram_language_model https://en.wikipedia.org/wiki/Word_n-gram_language_model

euroderf 2y ago

If the AI found a needle in a haystack, that isn't bad, is it?

Lastminutepanic 2y ago
Do you think it's possible that the person who literally wrote (multiple) papers with the answer AI gave, may phrase their question to the AI with similar, or even the exact phrasing they used in previous papers on the same subject ? If I load a bunch of lorem ipsum into AI scientist, labeled as scientific papers, then ask the AI "lorem ipsum" do you think it'll spit out some gibberish from those specific papers?
kedean 2y ago
The point isn't that it's bad, the point is that the AI did not discover the solution to the problem independently, as news headlines had implied.
asddubs 2y ago
it is if you're presenting it as coming up with novel information rather than essentially acting as a search engine for information from its training data (which we already know ai does well, at least with some caveats)

hirenj 2y ago

Colour me unsurprised - even not knowing data leakage had occurred, the hypothesis was underwhelming, as I mentioned in a comment on an earlier discussion. I sometimes despair for the state of thinking in science these days given how quickly people fawn over entirely pedestrian thinking and work. https://news.ycombinator.com/item?id=43105759 https://news.ycombinator.com/item?id=43105759

mandevil 2y ago

I mean, to be fair, "knowing all of the relevant literature" is a great first step for solving a problem! And a LLM can probably be better about not forgetting than humans are (though I'm guessing they do much worse on the hallucination front than humans do- humans tend to know when they are playing a hunch). But "This paper from a lower-tier journal two years ago suggests you should look at X" is a very valuable thing to have!

rideontime 2y ago

Instant subscribe when I saw David Gerard's name. He cut through the BS in crypto and we need more people like him focusing on AI fraud like this.

davidgerard 2y ago
thank you!

gukoff 2y ago

> Prof Penadés' said the tool had in fact done more than successfully replicating his research. > "It's not just that the top hypothesis they provide was the right one," he said. > "It's that they provide another four, and all of them made sense. > "And for one of them, we never thought about it, and we're now working on that." The Google's co-scientist still seems to make a useful assistant.