3 ms·

A little surprised this sort of thing passes for a publish-able paper. Isn't it the equivalent to saying "Here's the top 10 results for google searching golden

A little surprised this sort of thing passes for a publish-able paper. Isn't it the equivalent to saying "Here's the top 10 results for google searching golden retrievers March 2023, and here's the top 10 results from June 2023. We see that google is returning even cuter animals today. Unfortunately though, one of the results linked to a page full of cats." I'm sure openai has a list of standard questions that it tracks the responses it is getting over a period of time with perfect knowledge / version timing of their own releases. This does not seem like valid research / publication.

ineedasername 3y ago
No, the methodology listed in the paper shows their efforts were significantly more extensive than you represent here. As one example, their test of prime numbers went through 500 randomly selected primes, also evaluating the chain of thought in responses along with the success rate of this classification task. I’m not sure how you arrived at your impression, but it does not match the contents of the paper that I saw with a quick scan of each section.
- blobbers 3y ago
  Not to seem sarcastic, but 500 randomly selected primes doesn't change my mind. I read the paper, and dumbed it down to dogs and cats. Do you really feel like the efforts were significant or meaningful?
  - ineedasername 3y ago
    >I read the paper, and dumbed it down to dogs and cats. I could read the Pricicipia, or anything, and dumb it down to dogs and cats, but that reduction to the dumb would be a failing all my own. As for the test-- a classification task benchmarked with 500 examples is a fairly decent test of a system's capabilities, be it LLM or a traditional machine learning model. And again, this was only one of a variety of tasks. While it's certainly no Principia, I don't know how you get from there to dogs and cats, nor have you explained your reasoning on how you found the path between the two.
    - blobbers 3y ago
      Well, for starters, you take a model with 100 trillion parameters, and then you test it with... 500 examples. WAIT did you say hundred? 500 hundred? Not 500 hundred thousand right? Okay okay hold on back to the drawing board. ...dumbed down to cats and dogs.
      - ineedasername 3y ago
        This is simply not the way such models work and your jump from the LLM’s parameter count in its own training to testing its capabilities in a classification task is a non sequitur. …Principia and failings
gyrovagueGeist 3y ago
Its arxiv its not a published paper. Until it's reprinted by a journal or proceedings, its basically a blog post that people feel more comfortable citing. It's main purpose is to introduce/increase caution about other methods using these services: "our findings shows that the behavior of the “same” LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLM quality" Which seems valuable for other research using OpenAI GPT directly.
danybittel 3y ago
If it's not deterministic, it's not going to be useful in professional settings! Example: you get a bunch of data from your boss, need to find something out, you prompt engineer the results. Your boss get's back to you (five weeks later), likes the result, just want's you to fix a minor thing. It's now giving you completely different results. It's like building on sand.

ineedasername 3y ago

No, the methodology listed in the paper shows their efforts were significantly more extensive than you represent here. As one example, their test of prime numbers went through 500 randomly selected primes, also evaluating the chain of thought in responses along with the success rate of this classification task. I’m not sure how you arrived at your impression, but it does not match the contents of the paper that I saw with a quick scan of each section.

blobbers 3y ago
Not to seem sarcastic, but 500 randomly selected primes doesn't change my mind. I read the paper, and dumbed it down to dogs and cats. Do you really feel like the efforts were significant or meaningful?
- ineedasername 3y ago
  >I read the paper, and dumbed it down to dogs and cats. I could read the Pricicipia, or anything, and dumb it down to dogs and cats, but that reduction to the dumb would be a failing all my own. As for the test-- a classification task benchmarked with 500 examples is a fairly decent test of a system's capabilities, be it LLM or a traditional machine learning model. And again, this was only one of a variety of tasks. While it's certainly no Principia, I don't know how you get from there to dogs and cats, nor have you explained your reasoning on how you found the path between the two.
  - blobbers 3y ago
    Well, for starters, you take a model with 100 trillion parameters, and then you test it with... 500 examples. WAIT did you say hundred? 500 hundred? Not 500 hundred thousand right? Okay okay hold on back to the drawing board. ...dumbed down to cats and dogs.
    - ineedasername 3y ago
      This is simply not the way such models work and your jump from the LLM’s parameter count in its own training to testing its capabilities in a classification task is a non sequitur. …Principia and failings

gyrovagueGeist 3y ago

Its arxiv its not a published paper. Until it's reprinted by a journal or proceedings, its basically a blog post that people feel more comfortable citing. It's main purpose is to introduce/increase caution about other methods using these services: "our findings shows that the behavior of the “same” LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLM quality" Which seems valuable for other research using OpenAI GPT directly.

danybittel 3y ago

If it's not deterministic, it's not going to be useful in professional settings! Example: you get a bunch of data from your boss, need to find something out, you prompt engineer the results. Your boss get's back to you (five weeks later), likes the result, just want's you to fix a minor thing. It's now giving you completely different results. It's like building on sand.