5 ms·

The article is a decent peer review and refutation of “the reversal curse”. Some of the comments given here clearly haven’t read the whole article though - arri

The article is a decent peer review and refutation of “the reversal curse”. Some of the comments given here clearly haven’t read the whole article though - arriving at similarly skeptical conclusions that are clearly present and expanded on in the article. Why do people feel the need to do this here? Armchair commentary on advanced material is one of the main reasons I avoid Reddit. And furthermore why does it feel like you’re not allowed to suggest this as a response? I should be able to say “RTFA” but here I feel like I’m going to be scolded or banned by moderation.

swatcoder 3y ago
The author is a shark-diving science journalist who's been paid a lot of money by OpenAI, not a researcher, and the article is neither a "peer review" or an expert "refutation". It's a meandering and sometimes thought-provoking exploration of how LLM's can be coaxed to deliver on statements sort of like (but not actually equivalent to) the ones that failed in the paper. If you want to defer your own understanding to the author (and overstate their own claims!), that's fine, but a lot of people here are actually more informed about many of the topics and modes of discourse covered in the article and may have something to share with you if you give them credit. And if you're not going to give other commenter's due credit, then you're not really following the spirit of HN. While you can't always know the background of any individual commenter, we have an unusually accomplished and informed community in general and thrive by treating all commenters with the respect we might give the most accomplished and informed -- at least by default. That's why a blanket "RTFA" isn't usually appropriate here.
- amayne 3y ago
  Hello! I’m the “shark-diving science journalist” in question. First of all, you can run the experiments like I did and test this yourself. I’m not asking anyone to take my word. Just do what I did: Read the original paper. Test the claims for yourself. And to clarify a couple things: 1. The shark-diving part is true. 2. I’ve never been a journalist of any kind that I’m aware of unless you count writing for Skeptic Magazine. I’ve had many, many jobs though. 3. I started at OpenAI as a software engineer and member of technical staff. When I started there was just over a hundred people there. The lines between engineering were and are blurry. 4. I was the original prompt engineer at OpenAI and discovered many of the examples for using GPT-3 and wrote a lot of the original documentation. Internally my title was “prompt whisperer.” 5. I’m in the GPT-4 research paper for my contributions to model capability. I helped find abilities for long-text, vision, etc. 6. I was given the title Science Communicator when I started doing background briefings for media, etc., but still worked on model capability and other things. 7. I left OpenAI two months ago to work on a startup. Best, Andrew Mayne
  - user_named 3y ago
    1. What do you think the reversal curse implies about LLMs? 2. Do you believe that LLMs are capable of logic? 3. Do you believe that LLMs are intelligent? 4. Do you believe that your blog post shows 3 or 4? If not, what is it about?
    - amayne 3y ago
      1. I don't think the original research paper demonstrated the reversal curse. They claimed that you'd only get random answers from their example prompt. I showed that wasn't the case. I also pointed out what I believe to be a flaw in how they trained their model that when corrected for gave results that were non-random. 2. That depends on what you mean by logic. What would be an example of logical reasoning that would settle this? 3. Alan Turing created the Imitation Game thought experiment to show the futility of this question. If intelligence is something that can be observed and tested, then when we should be able to describe what to test for. 4. I don't make any specific claims about LLMs logic or intelligence. I just wanted to put their claim that LLMs can't generalize from B to A to the test.
- deleted 3y ago
  [deleted]
- ShamelessC 3y ago
  All due respect, when I wrote this there were about 5 comments that were 100% not from those who are "more informed" than the author (and perhaps were even less informed than myself, although I don't claim to be an LLM researcher or anything more than an interested internet user). Just the same it's been my experience, ESPECIALLY in machine learning, that users on HN are consistently over confident, bad at making future predictions, and generally just easily excitable. Does that mean I'm talking about you or commenters you like? Probably not! You seem relatively informed (although I'm not fond of you attacking the author of the article for things it isn't claiming - something truly not in the spirit of HN).
- golol 3y ago
  At the end the article presents a clear example where a model is trained on A is B and infers B is A. That is the contribution of this article and it's very valuable.

swatcoder 3y ago

The author is a shark-diving science journalist who's been paid a lot of money by OpenAI, not a researcher, and the article is neither a "peer review" or an expert "refutation". It's a meandering and sometimes thought-provoking exploration of how LLM's can be coaxed to deliver on statements sort of like (but not actually equivalent to) the ones that failed in the paper. If you want to defer your own understanding to the author (and overstate their own claims!), that's fine, but a lot of people here are actually more informed about many of the topics and modes of discourse covered in the article and may have something to share with you if you give them credit. And if you're not going to give other commenter's due credit, then you're not really following the spirit of HN. While you can't always know the background of any individual commenter, we have an unusually accomplished and informed community in general and thrive by treating all commenters with the respect we might give the most accomplished and informed -- at least by default. That's why a blanket "RTFA" isn't usually appropriate here.

amayne 3y ago
Hello! I’m the “shark-diving science journalist” in question. First of all, you can run the experiments like I did and test this yourself. I’m not asking anyone to take my word. Just do what I did: Read the original paper. Test the claims for yourself. And to clarify a couple things: 1. The shark-diving part is true. 2. I’ve never been a journalist of any kind that I’m aware of unless you count writing for Skeptic Magazine. I’ve had many, many jobs though. 3. I started at OpenAI as a software engineer and member of technical staff. When I started there was just over a hundred people there. The lines between engineering were and are blurry. 4. I was the original prompt engineer at OpenAI and discovered many of the examples for using GPT-3 and wrote a lot of the original documentation. Internally my title was “prompt whisperer.” 5. I’m in the GPT-4 research paper for my contributions to model capability. I helped find abilities for long-text, vision, etc. 6. I was given the title Science Communicator when I started doing background briefings for media, etc., but still worked on model capability and other things. 7. I left OpenAI two months ago to work on a startup. Best, Andrew Mayne
- user_named 3y ago
  1. What do you think the reversal curse implies about LLMs? 2. Do you believe that LLMs are capable of logic? 3. Do you believe that LLMs are intelligent? 4. Do you believe that your blog post shows 3 or 4? If not, what is it about?
  - amayne 3y ago
    1. I don't think the original research paper demonstrated the reversal curse. They claimed that you'd only get random answers from their example prompt. I showed that wasn't the case. I also pointed out what I believe to be a flaw in how they trained their model that when corrected for gave results that were non-random. 2. That depends on what you mean by logic. What would be an example of logical reasoning that would settle this? 3. Alan Turing created the Imitation Game thought experiment to show the futility of this question. If intelligence is something that can be observed and tested, then when we should be able to describe what to test for. 4. I don't make any specific claims about LLMs logic or intelligence. I just wanted to put their claim that LLMs can't generalize from B to A to the test.
deleted 3y ago
[deleted]
ShamelessC 3y ago
All due respect, when I wrote this there were about 5 comments that were 100% not from those who are "more informed" than the author (and perhaps were even less informed than myself, although I don't claim to be an LLM researcher or anything more than an interested internet user). Just the same it's been my experience, ESPECIALLY in machine learning, that users on HN are consistently over confident, bad at making future predictions, and generally just easily excitable. Does that mean I'm talking about you or commenters you like? Probably not! You seem relatively informed (although I'm not fond of you attacking the author of the article for things it isn't claiming - something truly not in the spirit of HN).
golol 3y ago
At the end the article presents a clear example where a model is trained on A is B and infers B is A. That is the contribution of this article and it's very valuable.