5 ms·

The published samples from the author's replicated GPT2-1.5M [1] look nothing like the ones from OpenAI's original publication. I really would have loved to see

The published samples from the author's replicated GPT2-1.5M [1] look nothing like the ones from OpenAI's original publication. I really would have loved to see a student with no funding replicating a super large state of the art model, but unfortunately he must either have collected lower-quality data for training, or have failed to replicate every aspect of the model. Almost none of the generated samples have anything to do with the input text, and many of them don't even make any sense. [1] https://github.com/ConnorJL/GPT2/tree/master/samples https://github.com/ConnorJL/GPT2/tree/master/samples

isoprophlex 7y ago
The author could have saved himself pages of posturing about "the biological blockchain"... These examples are crap. No way he's got something of equivalent potency to openAIs 1.5B model.
- jtolds 7y ago
  Maybe that was all output from his model
theblackcat1002 7y ago
Probably due to using newspaper instead of dragnet as content scrapper. Newspaper tend to get pretty deformed content from my experience.
mcguire 7y ago
From the Readme: "Unlike in OpenAI's blogpost, I did absolutely nothing to manually cherry pick the quality here. Some examples are good, most are bad. Almost all outputs decay in quality as the post gets longer. I wanted to keep the raw outputs to give an accurate feel of the strengths and failure cases of the model. The truth is that using a GPT2 type model to create text you want is more an art than a science and can be very finicky."
- gwern 7y ago
  The problem is OA also specified the level of cherrypicking, including sometimes picking the very first one. Do any of the conditional samples, picking out of 1 or 5, look like the OA blogpost ones? I didn't read them all, but they didn't look like it. Or much better than the 345M recently released, for that matter.
  - YeGoblynQueenne 7y ago
    I don't know why it is assumed that eyballing text generated by a neural net can give an accurate measure of the network's quality. Evaluating this sort of output is difficult and there are no good metrics for it. Then of course there is the fact that the network can generate text a lot faster than a human can read it. Think for a moment how many distinct passages these systems can generate -possibly even infinite- and how few of them a single person can realistically hope to read. It's very hard to know if the few passages one ends up reading are representative of the output of the network, or not. It'd be even harder to compare two systems just by eyballing their output side-by-side.
    - gwern 7y ago
      You have to eyeball it because (a) 'what it looks like to a human eyeballing it' is the most important metric for misuse - no one cares that much about a model which has a slightly better log loss but a human can instantly spot is robospam, that can't be abused or used for fun projects like generating poetry; and (b) the log loss between GPT-2 and his will not be comparable due to the different training corpuses (and possibly architectural differences as well), so the actual metrics are not useful. Given the large difference in quality, it only takes a few comparisons. Again, just look at the samples. Read a few. Do they really look the same in coherency and quality and realism?
      - YeGoblynQueenne 7y ago
        Do you think that if you were given a sample of text genereated by one of the two systems you would be able to tell which system generated it? Looking at two texts side-by-side while knowing which is which may well create an illusion of striking differences, which however are not really there. This is why concrete metrics are necessary- because human judgement is biased and inconsistent. But, like you say- there are no good metrics. In fact, the only thing we have to go by when discussing Open AI's model in the first place is opinions and convictions. Not least the opinion of OpenAi that their model generates text of unprecedented coherence, which itself is not based on anything else than eyballing.

isoprophlex 7y ago

The author could have saved himself pages of posturing about "the biological blockchain"... These examples are crap. No way he's got something of equivalent potency to openAIs 1.5B model.

jtolds 7y ago
Maybe that was all output from his model

theblackcat1002 7y ago

Probably due to using newspaper instead of dragnet as content scrapper. Newspaper tend to get pretty deformed content from my experience.

mcguire 7y ago

From the Readme: "Unlike in OpenAI's blogpost, I did absolutely nothing to manually cherry pick the quality here. Some examples are good, most are bad. Almost all outputs decay in quality as the post gets longer. I wanted to keep the raw outputs to give an accurate feel of the strengths and failure cases of the model. The truth is that using a GPT2 type model to create text you want is more an art than a science and can be very finicky."

gwern 7y ago
The problem is OA also specified the level of cherrypicking, including sometimes picking the very first one. Do any of the conditional samples, picking out of 1 or 5, look like the OA blogpost ones? I didn't read them all, but they didn't look like it. Or much better than the 345M recently released, for that matter.
- YeGoblynQueenne 7y ago
  I don't know why it is assumed that eyballing text generated by a neural net can give an accurate measure of the network's quality. Evaluating this sort of output is difficult and there are no good metrics for it. Then of course there is the fact that the network can generate text a lot faster than a human can read it. Think for a moment how many distinct passages these systems can generate -possibly even infinite- and how few of them a single person can realistically hope to read. It's very hard to know if the few passages one ends up reading are representative of the output of the network, or not. It'd be even harder to compare two systems just by eyballing their output side-by-side.
  - gwern 7y ago
    You have to eyeball it because (a) 'what it looks like to a human eyeballing it' is the most important metric for misuse - no one cares that much about a model which has a slightly better log loss but a human can instantly spot is robospam, that can't be abused or used for fun projects like generating poetry; and (b) the log loss between GPT-2 and his will not be comparable due to the different training corpuses (and possibly architectural differences as well), so the actual metrics are not useful. Given the large difference in quality, it only takes a few comparisons. Again, just look at the samples. Read a few. Do they really look the same in coherency and quality and realism?
    - YeGoblynQueenne 7y ago
      Do you think that if you were given a sample of text genereated by one of the two systems you would be able to tell which system generated it? Looking at two texts side-by-side while knowing which is which may well create an illusion of striking differences, which however are not really there. This is why concrete metrics are necessary- because human judgement is biased and inconsistent. But, like you say- there are no good metrics. In fact, the only thing we have to go by when discussing Open AI's model in the first place is opinions and convictions. Not least the opinion of OpenAi that their model generates text of unprecedented coherence, which itself is not based on anything else than eyballing.