3 ms·
The learning rates they demonstrate are crazy - though the standard when talking about CIFAR-10 is 94% accuracy iirc. Showing ~60% accuracy is weird. Has DAWNB
by jasonjmcghee 1y ago
The learning rates they demonstrate are crazy - though the standard when talking about CIFAR-10 is 94% accuracy iirc. Showing ~60% accuracy is weird.
Has DAWNBench been done with manifold Muon (with a more appropriate architecture)?
- snake_doc 1y agoUm.. the model is tiny: https://github.com/thinking-machines-lab/manifolds/blob/main/src/main.py https://github.com/thinking-machines-lab/manifolds/blob/main...
- jasonjmcghee 1y agoYeah, it's just the wrong architecture for the job, so I found it to be a strange example. Here's the top model on DAWNBench - https://github.com/apple/ml-cifar-10-faster/blob/main/fast_cifar_10_distributed.py https://github.com/apple/ml-cifar-10-faster/blob/main/fast_c... Trains for 15 epochs and it, like all the others is a 9 layer resnet.
- srean 1y agoUsually there's more to a ML, data-science idea (that's not a full fledged fledged out journal paper) than beating a SOTA benchmark. In fact beating SOTA is often the least interesting part of an interesting paper and the SOTA-blind reviewers often use it as a gatekeeping device.
- jasonjmcghee 1y agoSure, of course. Wasn't suggesting "are you beating a sota benchmark"? I'm floating the idea of an ablation that matches a realistic scenario for the dataset / task. Personally curious how manifold muon performs compared to AdamW in a throughly explored context. This is the first time I've seen a 3-layer mlp on cifar-10. I probably should have made the 9-layer ResNet part more, front-and-center / central to my point.
- srean 1y agoGot you, this time.
- Jackson__ 1y agoThey say they train for ~3 epochs. Could it be that's just not long enough of a training run? I have no idea how many epochs are usually used in those models.
- pooooooooooooop 1y agoits a 3-layer MLP as stated in the article