5 ms·

I'm surprised by how hacky the whole process is and how it's mostly just about tuning different hyperparameters

by ackbar03 4y ago

daenz 4y ago
Can you say more about why you are seeing the process as hacky?
- ackbar03 4y ago
  I've started reading from bottom and haven't read the whole thing yet. But their default action as stated in their log when face exploding gradients or unstable training is to just roll back a checkpoint and lower lr. Other proposed actions such as clamping activations are also just pretty standard things to try. I guess since their goal is to just be able to have a trained model it doesn't really matter. But it doesn't seem to be a easily reproducible process, and like i said a bit hacky in my opinion
  - dekhn 4y ago
    The very best ML models are distilled from postdoc tears.
- joshvm 4y ago
  Not sure if your comment is meant as a disagreement or a question. Generally the way hyperparameters are adjusted is some mix of intuition/experience and random/grid searching. Plus most people don't have the resources/infra to do a large scale grid search on a model that might take a day or more to train. It's somewhat principled, but often a random search is just as good as fiddling numbers by hand and often you have to figure out why something worked post-hoc. You also accept that you might never have a good explanation - for all you know it’s dataset dependent - and trust that your results are good enough to convince peer review (and you can show that this other parameter set was worse, so you didn't use it). It's hacky in the sense that a lot of the work in getting to state of the art (moving the needle on a benchmark by less than 1%) involves playing with the numbers until you get the best results. For example here the engineers modify the learning rate between various runs. I don't think they really had any theoretical reason behind the step changes apart from "this will probably work better because we've seen that effect when training similar sized models". Adjusting learning rate schedules is one of the simplest knobs to tweak. When you're working with huge models generally you want to use as big a batch size as you can get away with to reduce training time. A bit counter to the earlier thinking where LeCunn said something like "friends don't let friends use batch sizes > 32". There may be some guided methods like exploring the parameter space in a Bayesian way (eg try to efficiently explore which knobs make the most difference).
  - ackbar03 4y ago
    They seem to be adjusting lr between epochs as well when the loss explodes, not just runs. But I haven't read through the whole thing yet, maybe they trained the whole thing properly from start to finish at the end. Otherwise that would be extremely hacky and irreproducible
    - dotnet00 4y ago
      Yeah I think for now they were just trying to get any comparable results due to a near complete lack of details on GPT-3. They seemed to have a hard deadline for the task.
    - lumost 4y ago
      The time and expense of training a model at this size does not benefit well from trial and error. It's simply impractical to iteratively try ~20 different learning schedules. Hideously ineficient and hacky to have someone manually tweaking things, but not terribly different from the state of the art for scientific research. As long as they state the objectives of their manual control and produce a log of what they did someone else could try to replicate it.
- mhh__ 4y ago
  Currently I think we are still gluing transistors (networks) together (spiritually) like the very early days of the modern computer, it is hacky.
- gnulinux 4y ago
  They hot-swapped all kinds of model hyperparameters such as changing activation function and optimizer. It doesn't look like there was a principled reason why they kept switching optimizer or activation function. Maybe as they were training the model their data scientists kept finding ways to improve the model? Not sure, but it looks extremely hacky to me. Not something some team ran one day and forgot until it trained.
sbierwagen 4y ago
Welcome to ML.

I'm surprised by how hacky the whole process is and how it's mostly just about tuning different hyperparameters

daenz 4y ago

Can you say more about why you are seeing the process as hacky?

ackbar03 4y ago
I've started reading from bottom and haven't read the whole thing yet. But their default action as stated in their log when face exploding gradients or unstable training is to just roll back a checkpoint and lower lr. Other proposed actions such as clamping activations are also just pretty standard things to try. I guess since their goal is to just be able to have a trained model it doesn't really matter. But it doesn't seem to be a easily reproducible process, and like i said a bit hacky in my opinion
- dekhn 4y ago
  The very best ML models are distilled from postdoc tears.
joshvm 4y ago
Not sure if your comment is meant as a disagreement or a question. Generally the way hyperparameters are adjusted is some mix of intuition/experience and random/grid searching. Plus most people don't have the resources/infra to do a large scale grid search on a model that might take a day or more to train. It's somewhat principled, but often a random search is just as good as fiddling numbers by hand and often you have to figure out why something worked post-hoc. You also accept that you might never have a good explanation - for all you know it’s dataset dependent - and trust that your results are good enough to convince peer review (and you can show that this other parameter set was worse, so you didn't use it). It's hacky in the sense that a lot of the work in getting to state of the art (moving the needle on a benchmark by less than 1%) involves playing with the numbers until you get the best results. For example here the engineers modify the learning rate between various runs. I don't think they really had any theoretical reason behind the step changes apart from "this will probably work better because we've seen that effect when training similar sized models". Adjusting learning rate schedules is one of the simplest knobs to tweak. When you're working with huge models generally you want to use as big a batch size as you can get away with to reduce training time. A bit counter to the earlier thinking where LeCunn said something like "friends don't let friends use batch sizes > 32". There may be some guided methods like exploring the parameter space in a Bayesian way (eg try to efficiently explore which knobs make the most difference).
- ackbar03 4y ago
  They seem to be adjusting lr between epochs as well when the loss explodes, not just runs. But I haven't read through the whole thing yet, maybe they trained the whole thing properly from start to finish at the end. Otherwise that would be extremely hacky and irreproducible
  - dotnet00 4y ago
    Yeah I think for now they were just trying to get any comparable results due to a near complete lack of details on GPT-3. They seemed to have a hard deadline for the task.
  - lumost 4y ago
    The time and expense of training a model at this size does not benefit well from trial and error. It's simply impractical to iteratively try ~20 different learning schedules. Hideously ineficient and hacky to have someone manually tweaking things, but not terribly different from the state of the art for scientific research. As long as they state the objectives of their manual control and produce a log of what they did someone else could try to replicate it.
mhh__ 4y ago
Currently I think we are still gluing transistors (networks) together (spiritually) like the very early days of the modern computer, it is hacky.
gnulinux 4y ago
They hot-swapped all kinds of model hyperparameters such as changing activation function and optimizer. It doesn't look like there was a principled reason why they kept switching optimizer or activation function. Maybe as they were training the model their data scientists kept finding ways to improve the model? Not sure, but it looks extremely hacky to me. Not something some team ran one day and forgot until it trained.

sbierwagen 4y ago

Welcome to ML.