4 ms·
Do you think that if you were given a sample of text genereated by one of the two systems you would be able to tell which system generated it? Looking at two t
by YeGoblynQueenne 7y ago
Do you think that if you were given a sample of text genereated by one of the two systems you would be able to tell which system generated it?
Looking at two texts side-by-side while knowing which is which may well create an illusion of striking differences, which however are not really there. This is why concrete metrics are necessary- because human judgement is biased and inconsistent.
But, like you say- there are no good metrics. In fact, the only thing we have to go by when discussing Open AI's model in the first place is opinions and convictions. Not least the opinion of OpenAi that their model generates text of unprecedented coherence, which itself is not based on anything else than eyballing.
- gwern 7y agoI think I could. Few or none of the new samples even manage to maintain coherency in a sample, and you can see all sorts of garbage prose being generated.
- YeGoblynQueenne 7y agoPerhaps you could. Personally, I think it's impossible to get an accurate comparison in this way.
- gwern 7y agoI think it is possible, because it turns out I am right: the quality is much lower when compared directly on perplexity as well: https://medium.com/@NPCollapse/addendum-evaluation-of-my-model-e6734b51a830 https://medium.com/@NPCollapse/addendum-evaluation-of-my-mod...