4 ms·

I think this is an important problem. With logistic regression or deep learning, at least one can compare (out of sample) calibration curves or discrimination m

• bigpumpkin 6y ago
  perplexity score against a corpus such as wikipedia? Basically how well the model predicts the next word.
  • rkimb 6y ago
    This is a good start, but given the breadth of applications this would hardly give us enough to compare, as the goal of these models isn't to simply recite Wikipedia articles. What about language translation? Content summarization? Code generation? Turing test performance?
  • stellaathena 6y ago
    Both models were trained on Wikipedia, so that's a particularly bad choice. But yes, in practice this is what people tend to do. Take results with a very large grain of salt though, as the domain of the prompts you feed it make a huge difference.