8 ms·

The article is a bit lacking in details. Here are some changes in ML performance that are happening today and don't require increased transistor counts: 1. ne

The article is a bit lacking in details. Here are some changes in ML performance that are happening today and don't require increased transistor counts: 1. newer optimizations algorithms converge faster (Adam, adagrad) 2. lower-precision numerics can be used (TF32 is 19bit, bf16 is popular) 3. matrix multiplications are implemented in hardware (tensor cores, AMX) 4. neural networks are compiled directly (TVM, halide) 5. multi-device network topologies are tuned for training (e.g. nvlink improvements) 6. companies employ grad-student descent (hire a bunch of PhDs and task them with improving MLPerf results)

JL-Akrasia 5y ago
The last one is particularly effective.
- wing-_-nuts 5y ago
  I didn't really understand #6? Expand?
  - hailwren 5y ago
    You can overfit your data by coming up w/ a model that is very specific to it. Consider an n dimensional data set where one variable is a boolean. A researcher might notice that the True and False cases are different enough to warrant their own algorithms. Say they now fit a lower dimensional line to the True and False cases. They've just performed gradient descent but the grad student was the gradient. It's a common joke in CS/ML departments. (This is an obviously simple example, but the point is that hyperparameter tuning and algorithm selection _are a part of your algorithm_ and the data you're looking at while doing so is part of your training set)
lettergram 5y ago
> companies employ grad-student descent (hire a bunch of PhDs and task them with improving MLPerf results) No joke, always cheaper than massive optimization tasks. I’ve managed a team working on automl for years and I can say with certainty, the cheapest option is always hiring interns (grad or bachelor students) or a hackathon
sdenton4 5y ago
I'll throw in a seventh... Neural architecture search is finding more efficient architectural building blocks which translate to better quality models with fewer overall parameters. There's a lot of work going on right now to improve transformer architectures in particular; I think once the dust settles they will be massively more efficient than the current models. Efficient transformers often also introduce an inductive bias, which ends up improving model quality. (similar to replacing fully connected networks with convolutional networks; more efficient, but also carrying an inductive bias that improves model quality.)
- visarga 5y ago
  We're also realising we don't need full transformers all the time. Token mixing might suffice, there are quite a few papers replacing the core of the transformer with something simpler.
charcircuit 5y ago
>lower-precision numerics can be used I wonder if it would be worth starting with low precision and then over time raise precision to zero in on a local min / max.

JL-Akrasia 5y ago

The last one is particularly effective.

wing-_-nuts 5y ago
I didn't really understand #6? Expand?
- hailwren 5y ago
  You can overfit your data by coming up w/ a model that is very specific to it. Consider an n dimensional data set where one variable is a boolean. A researcher might notice that the True and False cases are different enough to warrant their own algorithms. Say they now fit a lower dimensional line to the True and False cases. They've just performed gradient descent but the grad student was the gradient. It's a common joke in CS/ML departments. (This is an obviously simple example, but the point is that hyperparameter tuning and algorithm selection _are a part of your algorithm_ and the data you're looking at while doing so is part of your training set)

lettergram 5y ago

> companies employ grad-student descent (hire a bunch of PhDs and task them with improving MLPerf results) No joke, always cheaper than massive optimization tasks. I’ve managed a team working on automl for years and I can say with certainty, the cheapest option is always hiring interns (grad or bachelor students) or a hackathon

sdenton4 5y ago

I'll throw in a seventh... Neural architecture search is finding more efficient architectural building blocks which translate to better quality models with fewer overall parameters. There's a lot of work going on right now to improve transformer architectures in particular; I think once the dust settles they will be massively more efficient than the current models. Efficient transformers often also introduce an inductive bias, which ends up improving model quality. (similar to replacing fully connected networks with convolutional networks; more efficient, but also carrying an inductive bias that improves model quality.)

visarga 5y ago
We're also realising we don't need full transformers all the time. Token mixing might suffice, there are quite a few papers replacing the core of the transformer with something simpler.

charcircuit 5y ago

>lower-precision numerics can be used I wonder if it would be worth starting with low precision and then over time raise precision to zero in on a local min / max.