7 ms·

Excuse me? Step by step wasn't paper-worthy? Hard disagree. LLM research is currently in its infancy, because they are no older than a few years old. And a res

Excuse me? Step by step wasn't paper-worthy? Hard disagree. LLM research is currently in its infancy, because they are no older than a few years old. And a research field in its infancy is bound to have a few noteworthy "no sh*t, Sherlock" papers that would be obvious from hindsight. The fact is, LLMs are a higher-order construct in machine learning, much like a fish is higher-order than a simple cellular colony. Lower-order ML constructs do not demonstrate emergent capabilities like step by step, stream of consciousness thinking, and so on. Academics should be less jaded and approach the field with beginner's eyes. Because we are all beginners here.

Der_Einzige 3y ago
NLP/Computational Lingustics is NOT a new field. The ACL was founded in 1962 LM research is also old, papers using very shitty LMs (I.e Markov chains) and discussing them have existed since the early 2000s and likely before. Check yourself before you try to check others.
- Retr0id 3y ago
  Try asking a markov chain to think step by step
  - tovej 3y ago
    By definition a Markov chain does everything step by step already, no need to ask!
    - Retr0id 3y ago
      By that logic, so does a GPT
- idiliv 3y ago
  Parent post is talking about LLMs, i.e. Large LMs. Research on LLMs is indeed in its infancy.
phkahler 3y ago
>> LLM research is currently in its infancy Everything that went in to creating GPT4 is AI/science or whatever. Probing GPT4 and trying to understand and characterize it is also a very worthy thing to do - else how can it be improved upon? But if making GPT is science, I'd say this stuff is more akin to psychology ;-)
- dartos 3y ago
  What is psychology, but applied biology? https://xkcd.com/435/ https://xkcd.com/435/
  - eimrine 3y ago
    Psychology is a religious-like pseudoscience, they can not even define what "psy" is without using some conceptions from religions such as a soul. upd these statements from me are so controversial, the number of "points" just dances lambada. The psy* areas are clearly polarized: some guys upvote all my messages in this topic and some other ones downvote all my messages in this topic. This is a sign of something interesting but I am not ready to elaborate on this statement in this comment which is going to become [flagged] eventually.
    - Woshiwuja 3y ago
      What? Are you literally referring to ancient greek psy? jesus...
      - eimrine 3y ago
        What is incorrect in this reference? You have not proposed any counterarguments. Also if you need just more fresh data - how do you propose to interpret the result of the Rosenhan's experiment?
        wavemode 3y ago
        That lying about your symptoms to doctors leads to incorrect diagnoses?
thesz 3y ago
> Lower-order ML constructs do not demonstrate emergent capabilities like step by step, stream of consciousness thinking, and so on. As a matter of fact, I did a project on the normalization of the text, e.g., translate "crossing of 6 a. and 12 s." into "crossing of sixth avenue and 12-th street" with a simple LM (order 3) and beam search on the lattice paths, lattice formed with hypotheses' variants. I got two fold decrease of word error rate compared to simpler approach with just outputting the most probable WFST path. It was not "step by step stream of consciousness," but nevertheless very impressive feat, when system started to know more without much effort. The large LM's do not just output "most probable" token, they output most probable sequence of tokens and it is done with the beam search. As you can see, my experience tells me that beam search alone can noticeably, if not tremendously, improve quality of the output, even for very simple LMs. And if I may, the higher-order construct here is a beam search, not the LMs-as-matrix-coefficients' themselves. Beam search is used in speech recognition for decades now, SR does not work properly without it. LM's, apparently, also do not work without it.
- azinman2 3y ago
  Is there something you can link to? I’d like to learn more.
  - thesz 3y ago
    Here it is: https://huggingface.co/blog/how-to-generate https://huggingface.co/blog/how-to-generate Beam search at Wikipedia: https://en.wikipedia.org/wiki/Beam_search https://en.wikipedia.org/wiki/Beam_search Beam search in Sqlite: https://www.sqlite.org/queryplanner-ng.html#_a_difficult_case https://www.sqlite.org/queryplanner-ng.html#_a_difficult_cas... Beam search is more interesting than its' application within AI field.
_gabe_ 3y ago
I feel like the author of this tweet wasn’t saying step-by-step isn’t worthy, he was saying that non-reproducible results are not science. He emphasizes this twice in that tweet: > one experiment on one data set with seed picking is not worthy reporting > Additionally, we all need to understand this is just one good empirical result, now we need to make it useful…
- low_tech_love 3y ago
  Exactly, and I tend to agree with him. I argued some time ago here that a paper should take some time to try to explain why its results are happening, at least from a reasonable hypothesis (people didn't seem to agree). An experiment (even a simple one) starts from a null hypothesis and tries to disprove it. However, most of what we see coming out of "scientific" papers is basically just engineering, I guess?: we put all of these things together in some way (out of pure guess and/or preference bias) and these results happened. We don't know why, good luck figuring it out. Here is one example where it works (don't ask where it doesn't; we intentionally kept those out). And while I obviously value very much the engineering advances we have seen, the science is still lacking, because not enough people are trying to understand why these things are happening. Although engineering advances are important and valuable, I don't understand exactly why people try so hard to call themselves scientists if they are basically skipping the scientific process entirely.
  - jimbokun 3y ago
    Lots of onions in the varnish.
  - deleted 3y ago
    [deleted]
- roguas 3y ago
  Is it non-reproducible? Also results which reproducibility can be measured and appears stable is perfectly good science. I dislike when people throw statements like that.
  - 3y ago
jebarker 3y ago
> emergent capabilities like step by step, stream of consciousness thinking What makes these things "emergent capabilities"? They seem like pretty straightforward consequences of autoregressive generation. If you feed output back as input then you'll get more output conditioned on that new input and stream of conscious generation is just stochastic parroting isn't it?
- selfhoster11 3y ago
  They are emergent in the sense that there is nothing in the pre-training dataset that would show the LLM by example how to, for example, compare and contrast any given pairing of fruit, technologies, or fictional settings, while thinking with the mindset of a doctor that hates both options, and on top of that make sure that this ends up formatted as a stream-of-consciousness. It can learn all these aspects from the source data individually in isolation, but there's no way there are examples that show how to combine it all (awareness of world information + knowledge of how to use it) into a single answer. That's probably a very clumsy example - others online have supplied more rigorous ones that I recommend checking out. Strictly speaking, it might be "stochastic parroting". But really, if you want to be a great and supremely effective stochastic parrot, you have to learn an internal representation of certain things so that you can predict them. And there are hints that this is exactly what a sufficiently-large large language model is doing.
ben_w 3y ago
I'm reading this tweet as saying "you can't write a paper by prompting an LLM in these ways" rather than "you can't write a paper characterising the impact of prompting an LLM in these ways". I'd agree the former won't get you a complete anything (longer than ~30 lines) by itself (90-95% cool, but with some incredible errors in the other 5-10%). I'd also agree that the latter is worthy of publishing.
rib3ye 3y ago
Sorry, ignoramus here: Which paper is “Step by step”?
- zxt_tzx 3y ago
  I think it's a reference to the "discovery" that if you ask GPT-4 to answer your query "step by step", it'll actually offer a better response than otherwise.
renonce 3y ago
I would say rather than saying that a new prompt is not science, it’s certainly a new discovery that is worth sharing. Maybe there should be a higher bar for papers, but why we have to make a discovery a paper - and not publish it at all if it cannot be made one - when a simple blog post or a tweet would convey the discovery very well?
lumost 3y ago
Anecdotally, The field of ML suffered from a problem where there were more entrants to the field than available positions/viable work. In many industrial positions, it was possible to hide a lack of progress behind ambiguity and/or poor metrics. This lead to a large amount of gate keeping for productive work as ultimately there wasn't enough to go around in the typical case. This attitude is somewhat pervasive leading to blogs like the above. Granted, the Nth prompting paper probably isn't interesting - but new programming languages for prompts and prompt discovery techniques are very exciting. I wouldn't be surprised if it turned out that automatic prompt expansion using a small pre-processing model turns out to be an effective technique.
zxt_tzx 3y ago
On a related note, there is this recent tweet purportedly showing that "offering to give a tip to ChatGPT" improves performance (or at the very least resulted in longer responses, which might not be a good proxy for performance) https://twitter.com/voooooogel/status/1730726744314069190 https://twitter.com/voooooogel/status/1730726744314069190
WhitneyLand 3y ago
What you’re saying seems to be compounded by the black box aspects here. Instinctively a prompt may look like one line of code. We can often know or prove what a compiler is doing, but higher dimensional space is just not understood in the same way. What other engineered thing in history has had this much immediately useful emergent capability? Genetic algorithms finding antenna designs and flocking algorithms are fantastic, but I would argue narrower in scope. Of course a paper is still expected to expand knowledge, to have rigor and impact, but I don’t see why a prompt centric contribution would inherently preclude this.
skyde 3y ago
I have a question For people in the field! Would GPT only with pretraining and no fine tuning be able to behave better when prompt is "let’s think step by step"? Or this prompt only worked because a fine tuning dataset containing many "let’s think step by step" prompt was used?
- selfhoster11 3y ago
  Working with pure pre-trained models is quite hard, and takes some practice. The key part is that "let's think step by step" is a technique that is used by humans also, and therefore (I think) would be somewhat represented in the pre-training corpus. It would be somewhat harder to "activate" this mode of thinking than a "let's think step by step" would in a fine-tuned model, but it would be possible with some elbow grease.
tensor 3y ago
I'm not surprised at the defence of "prompt engineering" here. It's something easy to do with no real knowledge, and I'm sure having it dismissed hurts some people. But I 100% agree with the author, "prompt engineering" is not science, and I'd say it's not engineering either. All you're doing is exploring the parameter space of particular model in a very crude way. There is no "engineering" going on in this process, just a bunch of trial and error. Perhaps it should be called "prompt guessing." None of the results of this process will transfer to any other model. It's simply not science. Papers like "step-by-step" are different, and relate more to learning and inference and do translate to different models and even different architectures. Also, no, we are not all beginners here. Language models have a long history, and while the very large models are impressive, most of their failings have been known for a very long time already. Things like "prompt engineering" will eventually end up in the same graveyard as "keyword engineers" of the past.
- adastra22 3y ago
  > But I 100% agree with the author, "prompt engineering" is not science, and I'd say it's not engineering either. All you're doing is exploring the parameter space of particular model in a very crude way. There is no "engineering" going on in this process, just a bunch of trial and error. I wonder what your definition of “science” or “engineering” is…
  - 0xfae 3y ago
    Right? I'm having a hard time imagining a definition that includes "trying new things and seeing what happens" but that doesn't include... "trying new things and seeing what happens"
    - aeternum 3y ago
      "Science" has been twisted recently into a kind of witchcraft that can only be practiced by those anointed through the rigors of academia. "Trust the science" In reality, that is about the furthest from what you should do. As Feynman once said: "Science is the belief in the ignorance of experts". Electricity was also once considered a toy and good for nothing but parlor tricks.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]

Der_Einzige 3y ago

NLP/Computational Lingustics is NOT a new field. The ACL was founded in 1962 LM research is also old, papers using very shitty LMs (I.e Markov chains) and discussing them have existed since the early 2000s and likely before. Check yourself before you try to check others.

Retr0id 3y ago
Try asking a markov chain to think step by step
- tovej 3y ago
  By definition a Markov chain does everything step by step already, no need to ask!
  - Retr0id 3y ago
    By that logic, so does a GPT
idiliv 3y ago
Parent post is talking about LLMs, i.e. Large LMs. Research on LLMs is indeed in its infancy.

phkahler 3y ago

>> LLM research is currently in its infancy Everything that went in to creating GPT4 is AI/science or whatever. Probing GPT4 and trying to understand and characterize it is also a very worthy thing to do - else how can it be improved upon? But if making GPT is science, I'd say this stuff is more akin to psychology ;-)

dartos 3y ago
What is psychology, but applied biology? https://xkcd.com/435/ https://xkcd.com/435/
- eimrine 3y ago
  Psychology is a religious-like pseudoscience, they can not even define what "psy" is without using some conceptions from religions such as a soul. upd these statements from me are so controversial, the number of "points" just dances lambada. The psy* areas are clearly polarized: some guys upvote all my messages in this topic and some other ones downvote all my messages in this topic. This is a sign of something interesting but I am not ready to elaborate on this statement in this comment which is going to become [flagged] eventually.
  - Woshiwuja 3y ago
    What? Are you literally referring to ancient greek psy? jesus...
    - eimrine 3y ago
      What is incorrect in this reference? You have not proposed any counterarguments. Also if you need just more fresh data - how do you propose to interpret the result of the Rosenhan's experiment?
      - wavemode 3y ago
        That lying about your symptoms to doctors leads to incorrect diagnoses?

thesz 3y ago

> Lower-order ML constructs do not demonstrate emergent capabilities like step by step, stream of consciousness thinking, and so on. As a matter of fact, I did a project on the normalization of the text, e.g., translate "crossing of 6 a. and 12 s." into "crossing of sixth avenue and 12-th street" with a simple LM (order 3) and beam search on the lattice paths, lattice formed with hypotheses' variants. I got two fold decrease of word error rate compared to simpler approach with just outputting the most probable WFST path. It was not "step by step stream of consciousness," but nevertheless very impressive feat, when system started to know more without much effort. The large LM's do not just output "most probable" token, they output most probable sequence of tokens and it is done with the beam search. As you can see, my experience tells me that beam search alone can noticeably, if not tremendously, improve quality of the output, even for very simple LMs. And if I may, the higher-order construct here is a beam search, not the LMs-as-matrix-coefficients' themselves. Beam search is used in speech recognition for decades now, SR does not work properly without it. LM's, apparently, also do not work without it.

azinman2 3y ago
Is there something you can link to? I’d like to learn more.
- thesz 3y ago
  Here it is: https://huggingface.co/blog/how-to-generate https://huggingface.co/blog/how-to-generate Beam search at Wikipedia: https://en.wikipedia.org/wiki/Beam_search https://en.wikipedia.org/wiki/Beam_search Beam search in Sqlite: https://www.sqlite.org/queryplanner-ng.html#_a_difficult_case https://www.sqlite.org/queryplanner-ng.html#_a_difficult_cas... Beam search is more interesting than its' application within AI field.

_gabe_ 3y ago

I feel like the author of this tweet wasn’t saying step-by-step isn’t worthy, he was saying that non-reproducible results are not science. He emphasizes this twice in that tweet: > one experiment on one data set with seed picking is not worthy reporting > Additionally, we all need to understand this is just one good empirical result, now we need to make it useful…

low_tech_love 3y ago
Exactly, and I tend to agree with him. I argued some time ago here that a paper should take some time to try to explain why its results are happening, at least from a reasonable hypothesis (people didn't seem to agree). An experiment (even a simple one) starts from a null hypothesis and tries to disprove it. However, most of what we see coming out of "scientific" papers is basically just engineering, I guess?: we put all of these things together in some way (out of pure guess and/or preference bias) and these results happened. We don't know why, good luck figuring it out. Here is one example where it works (don't ask where it doesn't; we intentionally kept those out). And while I obviously value very much the engineering advances we have seen, the science is still lacking, because not enough people are trying to understand why these things are happening. Although engineering advances are important and valuable, I don't understand exactly why people try so hard to call themselves scientists if they are basically skipping the scientific process entirely.
- jimbokun 3y ago
  Lots of onions in the varnish.
- deleted 3y ago
  [deleted]
roguas 3y ago
Is it non-reproducible? Also results which reproducibility can be measured and appears stable is perfectly good science. I dislike when people throw statements like that.
- 3y ago

jebarker 3y ago

> emergent capabilities like step by step, stream of consciousness thinking What makes these things "emergent capabilities"? They seem like pretty straightforward consequences of autoregressive generation. If you feed output back as input then you'll get more output conditioned on that new input and stream of conscious generation is just stochastic parroting isn't it?

selfhoster11 3y ago
They are emergent in the sense that there is nothing in the pre-training dataset that would show the LLM by example how to, for example, compare and contrast any given pairing of fruit, technologies, or fictional settings, while thinking with the mindset of a doctor that hates both options, and on top of that make sure that this ends up formatted as a stream-of-consciousness. It can learn all these aspects from the source data individually in isolation, but there's no way there are examples that show how to combine it all (awareness of world information + knowledge of how to use it) into a single answer. That's probably a very clumsy example - others online have supplied more rigorous ones that I recommend checking out. Strictly speaking, it might be "stochastic parroting". But really, if you want to be a great and supremely effective stochastic parrot, you have to learn an internal representation of certain things so that you can predict them. And there are hints that this is exactly what a sufficiently-large large language model is doing.

ben_w 3y ago

I'm reading this tweet as saying "you can't write a paper by prompting an LLM in these ways" rather than "you can't write a paper characterising the impact of prompting an LLM in these ways". I'd agree the former won't get you a complete anything (longer than ~30 lines) by itself (90-95% cool, but with some incredible errors in the other 5-10%). I'd also agree that the latter is worthy of publishing.

rib3ye 3y ago

Sorry, ignoramus here: Which paper is “Step by step”?

zxt_tzx 3y ago
I think it's a reference to the "discovery" that if you ask GPT-4 to answer your query "step by step", it'll actually offer a better response than otherwise.

renonce 3y ago

I would say rather than saying that a new prompt is not science, it’s certainly a new discovery that is worth sharing. Maybe there should be a higher bar for papers, but why we have to make a discovery a paper - and not publish it at all if it cannot be made one - when a simple blog post or a tweet would convey the discovery very well?

lumost 3y ago

Anecdotally, The field of ML suffered from a problem where there were more entrants to the field than available positions/viable work. In many industrial positions, it was possible to hide a lack of progress behind ambiguity and/or poor metrics. This lead to a large amount of gate keeping for productive work as ultimately there wasn't enough to go around in the typical case. This attitude is somewhat pervasive leading to blogs like the above. Granted, the Nth prompting paper probably isn't interesting - but new programming languages for prompts and prompt discovery techniques are very exciting. I wouldn't be surprised if it turned out that automatic prompt expansion using a small pre-processing model turns out to be an effective technique.

zxt_tzx 3y ago

On a related note, there is this recent tweet purportedly showing that "offering to give a tip to ChatGPT" improves performance (or at the very least resulted in longer responses, which might not be a good proxy for performance) https://twitter.com/voooooogel/status/1730726744314069190 https://twitter.com/voooooogel/status/1730726744314069190

WhitneyLand 3y ago

What you’re saying seems to be compounded by the black box aspects here. Instinctively a prompt may look like one line of code. We can often know or prove what a compiler is doing, but higher dimensional space is just not understood in the same way. What other engineered thing in history has had this much immediately useful emergent capability? Genetic algorithms finding antenna designs and flocking algorithms are fantastic, but I would argue narrower in scope. Of course a paper is still expected to expand knowledge, to have rigor and impact, but I don’t see why a prompt centric contribution would inherently preclude this.

skyde 3y ago

I have a question For people in the field! Would GPT only with pretraining and no fine tuning be able to behave better when prompt is "let’s think step by step"? Or this prompt only worked because a fine tuning dataset containing many "let’s think step by step" prompt was used?

selfhoster11 3y ago
Working with pure pre-trained models is quite hard, and takes some practice. The key part is that "let's think step by step" is a technique that is used by humans also, and therefore (I think) would be somewhat represented in the pre-training corpus. It would be somewhat harder to "activate" this mode of thinking than a "let's think step by step" would in a fine-tuned model, but it would be possible with some elbow grease.

tensor 3y ago

I'm not surprised at the defence of "prompt engineering" here. It's something easy to do with no real knowledge, and I'm sure having it dismissed hurts some people. But I 100% agree with the author, "prompt engineering" is not science, and I'd say it's not engineering either. All you're doing is exploring the parameter space of particular model in a very crude way. There is no "engineering" going on in this process, just a bunch of trial and error. Perhaps it should be called "prompt guessing." None of the results of this process will transfer to any other model. It's simply not science. Papers like "step-by-step" are different, and relate more to learning and inference and do translate to different models and even different architectures. Also, no, we are not all beginners here. Language models have a long history, and while the very large models are impressive, most of their failings have been known for a very long time already. Things like "prompt engineering" will eventually end up in the same graveyard as "keyword engineers" of the past.

adastra22 3y ago
> But I 100% agree with the author, "prompt engineering" is not science, and I'd say it's not engineering either. All you're doing is exploring the parameter space of particular model in a very crude way. There is no "engineering" going on in this process, just a bunch of trial and error. I wonder what your definition of “science” or “engineering” is…
- 0xfae 3y ago
  Right? I'm having a hard time imagining a definition that includes "trying new things and seeing what happens" but that doesn't include... "trying new things and seeing what happens"
  - aeternum 3y ago
    "Science" has been twisted recently into a kind of witchcraft that can only be practiced by those anointed through the rigors of academia. "Trust the science" In reality, that is about the furthest from what you should do. As Feynman once said: "Science is the belief in the ignorance of experts". Electricity was also once considered a toy and good for nothing but parlor tricks.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]