3 ms·

Also, is it standard practice to obfuscate which models you're benchmarking against? They're just labeled Model A-D, with sizes but no additional information.

• sigmoid10 2y ago
  Given the context, it appears they are not benchmarking against other models but comparing differently sized versions of the same model. The 8B one is just the one they decided to give a catchy name. The other ones are probably also just fine tuned Llama models. But without information on the total compute budget (i.e. nr. of trained tokens), this kind of plot is pretty useless anyways.
  • lolinder 2y ago
    That doesn't make any sense, because their 8B is listed as benchmarking above the 13B "model A".
    • sigmoid10 2y ago
      That's why it is very likely it has seen more tokens during training and why the plot is worthless.