4 ms·

Claude 3 Opus reinvented this quantum algorithm from scratch in just 2 prompts

56 points by samaysharma 3y ago

Redster 3y ago
This sounds intriguing and slightly hype-y. But I would love to see our math guys here break down and analyze what's hype and what's not. Is Claude's description detailed and concrete enough to be useful? It seems that's where LLM's often struggle.
- onlyrealcuzzo 3y ago
  It's probably been prompted like this for similar break-throughs thousands of times. Is it really that impressive that it's done this once? I imagine with enough tries, GPT-4 Turbo and Gemini 1.5 Pro have done similarly impressive things.
  - bigyikes 3y ago
    Let it be known: this is what moving the goalposts sounds like! The AI is proposing novel quantum algorithms using terminology most of us here don’t understand, and it’s apparently not “that impressive”! We truly live in interesting times
    - mewpmewp2 3y ago
      True, with AI it feels like some people are overprotective of their egos, having to downplay the amazing achievements we have seen since end of 2022. Maybe they are subconsiously so afraid of actual AGI coming, that they do have to ignore the achievements as of now.
    - skybrian 3y ago
      If you were previously talking with someone who thought that AI's can't be creative and now they say it doesn't matter, then some goalposts moved. More likely, though, you're just reading commentary by someone else who has different goalposts in mind. Unlike football, there is no standard for goalposts in AI.
    - nicklecompte 3y ago
      The reason it's not that impressive is a lot of us are not as easily fooled as we might have been back in 2022! As someone who also went to UWaterloo for quantum computing (MMath, not PhD, and I don't work in the field), let's look at the tweet in detail: - in the first image, Claude's output is almost all useless LLM mush! Points #1, #3, #4, #5, #6, and #7 are uselessly vague and at best tangentially related to the problem. Only #2 has anything of interest, and even then it's just "come up with a quantum version of a classical algorithm," which is hardly insightful. Claude's output here just straight-up sucks! - in the second image, "Initialization" is useless, "Quantum State Measurement" is useless, "Momentum Refreshment" is useless (and might be wrong in this context), "Iterations" is useless and misleading, and "Sample Collection" is useless. - There is some real meat in the "Quantum Leapfrog Integration" section, but the entire last bullet point (starting with "The quantum leapfrog integrator consists of") is a completely useless statement which applies to every quantum algorithm. There is almost nothing actionable for a quantum computing researcher here, except maybe as a brainstorming aid. And of course the giant pile of LLM mush is evidence that Claude doesn’t know what a “quantum computer” is or what an “algorithm” is, it’s just a fancy chatbot. I am not at all impressed by any of this.
    - jprete 3y ago
      If you ask for a thousand novel theories, and all the answers look good but only five can possibly work, you haven't done much better than talking to physics cranks.
- amluto 3y ago
  It reads like something between gobbledygook and an executive summary of a paper with maybe some potentially relevant fancy terms thrown in for good measure. I have no idea how good the actual paper is, but the Claude output would not make an impressive paper. (It’s still pretty neat that an LLM invented it with a minimal prompt!) (I’ve done quantum computing research and, while I did share an office and a research group with people doing Monte Carlo, I didn’t do it myself.)
  - winwang 3y ago
    Also did a bit of QC research. Since I'm tipsy, I skimmed the Twitter post. Here's a pdf on arxiv about HMC (2017) which I found in 20 seconds: https://arxiv.org/abs/1701.02434 https://arxiv.org/abs/1701.02434 Seems like Claude3 is decently logical in paraphrasing key parts of text in a good order, but with repetition scattered everywhere. Also Monte Carlo is something which sounds cool but is literally just conceptual dart-throwing. I've always hated the name.
    - amluto 3y ago
      However you feel about the term Monte Carlo, MCMC (Markov chain Monte Carlo) is a nifty algorithm. You’re also welcome to say “Metropolis-Hastings algorithm” (assuming that’s the specific algorithm you’re using). I was personally entertained when I learned that the algorithm formerly known as the Metropolis algorithm had nothing to do with cities. And MCMC really is like throwing darts. Except that, when done right, it throws darts in a distribution of your choice, and, critically, you don’t need to be able to normalize the distribution or integrate it to find the CDF. This turns out to be extremely useful. So you can think of it as a sort of bizarre way to numerically integrate gnarly functions. If you already knew the inverse CDF of a distribution, you could directly sample it. But if you don’t, either because it’s hard to integrate or because you’re getting the probabilities or probability densities, unnormalized, from a black box, you can still run MCMC. I bet you could have some fun taking a small ML model, using it to compute probabilities of strings of tokens, and using MCMC to sample the distribution of likely strings as seen by the model. The challenging part will be coming up with a remotely credible random walk that efficiently explores the space. NB: everyone loves saying “log-likelihood”. This is essentially the same thing. The log is nice because otherwise the numbers end up awkwardly small.
thorum 3y ago
I’m not a quantum computing expert, but surely it’s more likely the algorithm is less original than the authors’ believed, than it is that Opus is doing original scientific research.
- whatshisface 3y ago
  The idea behind the paper was a very straightforward "do <this common thing> but in <this context>," and Claude was able to come up with that, but all of the author's hard work was in implementing and validating it, which Claude did not do.
endisneigh 3y ago
Isn't the simplest explanation that the answer was in the training data?
- faitswulff 3y ago
  The link says: > The paper is not on the internet yet. And a followup tweet says: > Yes repos were not public, paper was on Overleaf not the open Internet
  - endisneigh 3y ago
    There are other papers published after 2020 though. Just one example: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0258277 https://journals.plos.org/plosone/article?id=10.1371/journal...
  - guluarte 3y ago
    where were the repos hosted? github?
brcmthrowaway 3y ago
Gamechanging! Lets see if GPT-4 can beat this
deleted 3y ago
[deleted]
Yizahi 3y ago
In other news, an advanced search engine managed to find a specific already existing information in the internet in just two prompts. Oh wait, that doesn't cost 7 trillion dollars... An on this terrible disappointment, it's time to move on.
fragmede 3y ago
Insistence that LLMs aren't intelligence and are just parrots and glorified autocomplete make me think of the argument against evolution where there's this missing link between Australopithecus and Homo Sapiens (though that's in, itself, an outdated theory) and if only scientists would find that missing link, then they'd believe in evolution. Just in reverse. Obviously my time machine is as good as your time machine, so I have no idea if this will lead to AGI, but even as it stands, these things are useful and are such a good fascimile of intelligent that it's easier to call them an artificial intelligence. there's this notion that they can't create anything outside of their training data, which means they're not intelligent in the same way that humans are, but that's the whole point of calling them artificial, because they aren't! can humans create things outside of their training data? is Dali an LLM, unable to create beyond the bounds of his training data because, even at the height of his creativity, clocks and melting are preexisting concepts? do humans create new things? of course. can an LLM create a new thing? hmm, that's less obvious, except that when it's writing or drawing, that particular arrangement of bits hasn't existed before, which means it's creating things on the face of it. Creating new things, then, is entirely within its capabilities. We had older models that created entirely new words, but we rejected them because all they created was giberrish. GPT-2 was great at making off-kilter poems. where do new ideas come from in humans? I don't know, but it's based on their training data. in predicting the next word, LLMs predict something from its training data, far more vast than any one human could hold in their mind. can it create an entirely brand new thing? that seems like the wrong question to ask when it does it every time it answers, like asking where's three missing link, every time a new link is found.
m3kw9 3y ago
So it already has likely already been trained with it and you call it reinvent

56 points by samaysharma 3y ago

This sounds intriguing and slightly hype-y. But I would love to see our math guys here break down and analyze what's hype and what's not. Is Claude's description detailed and concrete enough to be useful? It seems that's where LLM's often struggle.

onlyrealcuzzo 3y ago
It's probably been prompted like this for similar break-throughs thousands of times. Is it really that impressive that it's done this once? I imagine with enough tries, GPT-4 Turbo and Gemini 1.5 Pro have done similarly impressive things.
- bigyikes 3y ago
  Let it be known: this is what moving the goalposts sounds like! The AI is proposing novel quantum algorithms using terminology most of us here don’t understand, and it’s apparently not “that impressive”! We truly live in interesting times
  - mewpmewp2 3y ago
    True, with AI it feels like some people are overprotective of their egos, having to downplay the amazing achievements we have seen since end of 2022. Maybe they are subconsiously so afraid of actual AGI coming, that they do have to ignore the achievements as of now.
  - skybrian 3y ago
    If you were previously talking with someone who thought that AI's can't be creative and now they say it doesn't matter, then some goalposts moved. More likely, though, you're just reading commentary by someone else who has different goalposts in mind. Unlike football, there is no standard for goalposts in AI.
  - nicklecompte 3y ago
    The reason it's not that impressive is a lot of us are not as easily fooled as we might have been back in 2022! As someone who also went to UWaterloo for quantum computing (MMath, not PhD, and I don't work in the field), let's look at the tweet in detail: - in the first image, Claude's output is almost all useless LLM mush! Points #1, #3, #4, #5, #6, and #7 are uselessly vague and at best tangentially related to the problem. Only #2 has anything of interest, and even then it's just "come up with a quantum version of a classical algorithm," which is hardly insightful. Claude's output here just straight-up sucks! - in the second image, "Initialization" is useless, "Quantum State Measurement" is useless, "Momentum Refreshment" is useless (and might be wrong in this context), "Iterations" is useless and misleading, and "Sample Collection" is useless. - There is some real meat in the "Quantum Leapfrog Integration" section, but the entire last bullet point (starting with "The quantum leapfrog integrator consists of") is a completely useless statement which applies to every quantum algorithm. There is almost nothing actionable for a quantum computing researcher here, except maybe as a brainstorming aid. And of course the giant pile of LLM mush is evidence that Claude doesn’t know what a “quantum computer” is or what an “algorithm” is, it’s just a fancy chatbot. I am not at all impressed by any of this.
  - jprete 3y ago
    If you ask for a thousand novel theories, and all the answers look good but only five can possibly work, you haven't done much better than talking to physics cranks.
amluto 3y ago
It reads like something between gobbledygook and an executive summary of a paper with maybe some potentially relevant fancy terms thrown in for good measure. I have no idea how good the actual paper is, but the Claude output would not make an impressive paper. (It’s still pretty neat that an LLM invented it with a minimal prompt!) (I’ve done quantum computing research and, while I did share an office and a research group with people doing Monte Carlo, I didn’t do it myself.)
- winwang 3y ago
  Also did a bit of QC research. Since I'm tipsy, I skimmed the Twitter post. Here's a pdf on arxiv about HMC (2017) which I found in 20 seconds: https://arxiv.org/abs/1701.02434 https://arxiv.org/abs/1701.02434 Seems like Claude3 is decently logical in paraphrasing key parts of text in a good order, but with repetition scattered everywhere. Also Monte Carlo is something which sounds cool but is literally just conceptual dart-throwing. I've always hated the name.
  - amluto 3y ago
    However you feel about the term Monte Carlo, MCMC (Markov chain Monte Carlo) is a nifty algorithm. You’re also welcome to say “Metropolis-Hastings algorithm” (assuming that’s the specific algorithm you’re using). I was personally entertained when I learned that the algorithm formerly known as the Metropolis algorithm had nothing to do with cities. And MCMC really is like throwing darts. Except that, when done right, it throws darts in a distribution of your choice, and, critically, you don’t need to be able to normalize the distribution or integrate it to find the CDF. This turns out to be extremely useful. So you can think of it as a sort of bizarre way to numerically integrate gnarly functions. If you already knew the inverse CDF of a distribution, you could directly sample it. But if you don’t, either because it’s hard to integrate or because you’re getting the probabilities or probability densities, unnormalized, from a black box, you can still run MCMC. I bet you could have some fun taking a small ML model, using it to compute probabilities of strings of tokens, and using MCMC to sample the distribution of likely strings as seen by the model. The challenging part will be coming up with a remotely credible random walk that efficiently explores the space. NB: everyone loves saying “log-likelihood”. This is essentially the same thing. The log is nice because otherwise the numbers end up awkwardly small.

thorum 3y ago

I’m not a quantum computing expert, but surely it’s more likely the algorithm is less original than the authors’ believed, than it is that Opus is doing original scientific research.

whatshisface 3y ago
The idea behind the paper was a very straightforward "do <this common thing> but in <this context>," and Claude was able to come up with that, but all of the author's hard work was in implementing and validating it, which Claude did not do.

endisneigh 3y ago

Isn't the simplest explanation that the answer was in the training data?

faitswulff 3y ago
The link says: > The paper is not on the internet yet. And a followup tweet says: > Yes repos were not public, paper was on Overleaf not the open Internet
- endisneigh 3y ago
  There are other papers published after 2020 though. Just one example: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0258277 https://journals.plos.org/plosone/article?id=10.1371/journal...
- guluarte 3y ago
  where were the repos hosted? github?

brcmthrowaway 3y ago

Gamechanging! Lets see if GPT-4 can beat this

deleted 3y ago

[deleted]

Yizahi 3y ago

In other news, an advanced search engine managed to find a specific already existing information in the internet in just two prompts. Oh wait, that doesn't cost 7 trillion dollars... An on this terrible disappointment, it's time to move on.

fragmede 3y ago

Insistence that LLMs aren't intelligence and are just parrots and glorified autocomplete make me think of the argument against evolution where there's this missing link between Australopithecus and Homo Sapiens (though that's in, itself, an outdated theory) and if only scientists would find that missing link, then they'd believe in evolution. Just in reverse. Obviously my time machine is as good as your time machine, so I have no idea if this will lead to AGI, but even as it stands, these things are useful and are such a good fascimile of intelligent that it's easier to call them an artificial intelligence. there's this notion that they can't create anything outside of their training data, which means they're not intelligent in the same way that humans are, but that's the whole point of calling them artificial, because they aren't! can humans create things outside of their training data? is Dali an LLM, unable to create beyond the bounds of his training data because, even at the height of his creativity, clocks and melting are preexisting concepts? do humans create new things? of course. can an LLM create a new thing? hmm, that's less obvious, except that when it's writing or drawing, that particular arrangement of bits hasn't existed before, which means it's creating things on the face of it. Creating new things, then, is entirely within its capabilities. We had older models that created entirely new words, but we rejected them because all they created was giberrish. GPT-2 was great at making off-kilter poems. where do new ideas come from in humans? I don't know, but it's based on their training data. in predicting the next word, LLMs predict something from its training data, far more vast than any one human could hold in their mind. can it create an entirely brand new thing? that seems like the wrong question to ask when it does it every time it answers, like asking where's three missing link, every time a new link is found.

m3kw9 3y ago

So it already has likely already been trained with it and you call it reinvent