2 ms·

Google is working on a similar line of research. Wonder why they haven't rolled out a GPT40 scaled version of this yet

• vintermann 8mo ago
  Probably because it's expensive. But I wish there were more "let's scale this thing to the skies" experiments from those who actually can afford to scale things to the skies.
  • yorwba 8mo ago
    Scaling laws mean that there's not much need to actually scale things to the skies. Instead, you can run a bunch of experiments at small scale, fit the scaling law parameters, then extrapolate. If the predicted outcome is disappointing (e.g. it's unlikely to beat the previous scaled-to-the-sky model), you can save the really expensive experiment for a more promising approach. It would certainly be nice though if this kind of negative result was published more often instead of leaving people to guess why a seemingly useful innovation wasn't adopted in the end.