4 ms·

The paper is well-written, and I would recommend reading its abstract and introduction over the OP.[a] The big idea is that a graph neural network (GNN) can be

by cs702 5y ago

The paper is well-written, and I would recommend reading its abstract and introduction over the OP.[a] The big idea is that a graph neural network (GNN) can be trained to predict the weights of any other neural network for performing a task, given the architecture as an input. For example, if we give the trained GNN a ResNet-50 architecture, the GNN predicts the trained weights in one forward pass, and voilà, we have a ResNet-50 that is now ready for use -- and you can always finetune it to get better performance out of it. Remarkably, the prediction of weights works for previously unseen model architectures too. Design your own model, run its architecture by the GNN to predict the trained weights, and voilà, you have a model that is now ready for use -- and you can always finetune it to get better performance out of it. Very cool! -- [a] https://arxiv.org/pdf/2110.13100.pdf https://arxiv.org/pdf/2110.13100.pdf -- Abstract: "Deep learning has been successful in automating the design of features in machine learning pipelines. However, the algorithms optimizing neural network parameters remain largely hand-designed and computationally inefficient. We study if we can use deep learning to directly predict these parameters by exploiting the past knowledge of training other networks. We introduce a large-scale dataset of diverse computational graphs of neural architectures – DEEPNETS-1M– and use it to explore parameter prediction on CIFAR-10 and ImageNet. By leveraging advances in graph neural networks, we propose a hypernetwork that can predict performant parameters in a single forward pass taking a fraction of a second, even on a CPU. The proposed model achieves surprisingly good performance on unseen and diverse networks. For example, it is able to predict all 24 million parameters of a ResNet-50 achieving a 60% accuracy on CIFAR-10. On ImageNet, top-5 accuracy of some of our networks approaches 50%. Our task along with the model and results can potentially lead to a new, more computationally efficient paradigm of training networks. Our model also learns a strong representation of neural architectures enabling their analysis."

nightski 5y ago
So this is for quickly training different architectures for the same task? Which is very neat, but it makes the headline seem a lot more like click-bait (I'm not sure what I expect these days...) It's kind of like a more advanced version of hyper-parameter tuning which has been done in more primitive forms for a long time. But this approach definitely seems powerful.
- cs702 5y ago
  > So this is for quickly training different architectures for the same task? It's for instantly training/pretraining any architecture for one or more predefined tasks (e.g., you could have a GNN for visual tasks, a GNN for NLP tasks, etc.), but you could also have a multi-domain GNN which accepts two inputs: an architecture and a domain.
hwers 5y ago
Can anyone enlighten me how you'd make use of this practically though? It seems like you need like a dataset of a million models to train this beast (all of which trained on a specific dataset). Yeah it can figure out the weights for a new model but you had to train 1 million models first to be able to even do that. Am I missing something?
- cs702 5y ago
  Practical use: Once you have GNNs that predict the trained weights of headless models for vision, NLP, and other domains, you can test new architectures in all those domains much faster with fewer compute resources. Research use: Can we figure out how and why these GNNs can predict trained weights for previously unseen architectures without actually having to train the models? Can we simplify the GNNs? Can we use the knowledge gained from studying these GNNs to speed-up conventional training?
  - webmaven 5y ago
    > Practical use: Once you have GNNs that predict the trained weights of headless models for vision, NLP, and other domains, you can test new architectures in all those domains much faster with fewer compute resources. Aha. Using this as an accelerator for Neural Architecture Search would indeed be very neat.
    - hwers 5y ago
      You still need to train 1 million architectures though before you can start the search (dataset inflexible)… I suppose there’s some value in possibly finding radically different networks (though I wonder if it’d fare well in such outlier regions). Perhaps if/since the GNN model is also differentiable you could invert it to maximize some compute/quality trade off point.
      - joshuamorton 5y ago
        But like it already exists. That's a sunk cost. Presuming it actually works and gives useful weights, if it lowers training time of novel models by 10% or something, that's dollars saved in the long run.
joe_the_user 5y ago
ResNet-50 achieving a 60% accuracy on CIFAR-10. As far as I can tell, SOTA ("state of the art") is 99% on CIFAR-10 [1]. The first entry in wikipedia, for 2010, shows 21.1% error rate, so 78.9% accuracy. Which is to say a 60% accuracy rate is far from useful to put it mildly. Essentially, with that low an accuracy rate, I don't think there's reason to think the predictions are meaningful in a practical sense. But maybe there's something I'm missing in accuracy description. [1] https://en.wikipedia.org/wiki/CIFAR-10 https://en.wikipedia.org/wiki/CIFAR-10
- cs702 5y ago
  Same goes for ImageNet: These GNNs are getting only 50% top-5 accuracy, well below SOTA. But I find that figure impressive: It implies that I can concoct a newfangled architecture for visual recognition, put it through one of these GNNs, and begin training at ~50% top-5 accuracy. Also, AFAIK this is the first effort of its kind; I would expect the figures to improve over time, as usual.
- deleted 5y ago
  [deleted]
- joshuamorton 5y ago
  Consider that a random baseline is 10% accuracy, so 60% is pretty good, especially if the training time is functionally free to get to that accuracy. Granted the comparison is 2500 iterations of sgd which isn't a lot. I haven't played with cifar in a while, but that's probably not a ton of savings.
nestorD 5y ago
Very cool and sensical when you pause to think about it. While different architectures might have different components, they are also likely to reuse building blocks (ie: the first layers of convolutional neural networks are famously very similar from one problem to another). In essense this detects those building blocks and plugs the known weights in for you.

by cs702 5y ago

nightski 5y ago

So this is for quickly training different architectures for the same task? Which is very neat, but it makes the headline seem a lot more like click-bait (I'm not sure what I expect these days...) It's kind of like a more advanced version of hyper-parameter tuning which has been done in more primitive forms for a long time. But this approach definitely seems powerful.

cs702 5y ago
> So this is for quickly training different architectures for the same task? It's for instantly training/pretraining any architecture for one or more predefined tasks (e.g., you could have a GNN for visual tasks, a GNN for NLP tasks, etc.), but you could also have a multi-domain GNN which accepts two inputs: an architecture and a domain.

hwers 5y ago

Can anyone enlighten me how you'd make use of this practically though? It seems like you need like a dataset of a million models to train this beast (all of which trained on a specific dataset). Yeah it can figure out the weights for a new model but you had to train 1 million models first to be able to even do that. Am I missing something?

cs702 5y ago
Practical use: Once you have GNNs that predict the trained weights of headless models for vision, NLP, and other domains, you can test new architectures in all those domains much faster with fewer compute resources. Research use: Can we figure out how and why these GNNs can predict trained weights for previously unseen architectures without actually having to train the models? Can we simplify the GNNs? Can we use the knowledge gained from studying these GNNs to speed-up conventional training?
- webmaven 5y ago
  > Practical use: Once you have GNNs that predict the trained weights of headless models for vision, NLP, and other domains, you can test new architectures in all those domains much faster with fewer compute resources. Aha. Using this as an accelerator for Neural Architecture Search would indeed be very neat.
  - hwers 5y ago
    You still need to train 1 million architectures though before you can start the search (dataset inflexible)… I suppose there’s some value in possibly finding radically different networks (though I wonder if it’d fare well in such outlier regions). Perhaps if/since the GNN model is also differentiable you could invert it to maximize some compute/quality trade off point.
    - joshuamorton 5y ago
      But like it already exists. That's a sunk cost. Presuming it actually works and gives useful weights, if it lowers training time of novel models by 10% or something, that's dollars saved in the long run.

joe_the_user 5y ago

ResNet-50 achieving a 60% accuracy on CIFAR-10. As far as I can tell, SOTA ("state of the art") is 99% on CIFAR-10 [1]. The first entry in wikipedia, for 2010, shows 21.1% error rate, so 78.9% accuracy. Which is to say a 60% accuracy rate is far from useful to put it mildly. Essentially, with that low an accuracy rate, I don't think there's reason to think the predictions are meaningful in a practical sense. But maybe there's something I'm missing in accuracy description. [1] https://en.wikipedia.org/wiki/CIFAR-10 https://en.wikipedia.org/wiki/CIFAR-10

cs702 5y ago
Same goes for ImageNet: These GNNs are getting only 50% top-5 accuracy, well below SOTA. But I find that figure impressive: It implies that I can concoct a newfangled architecture for visual recognition, put it through one of these GNNs, and begin training at ~50% top-5 accuracy. Also, AFAIK this is the first effort of its kind; I would expect the figures to improve over time, as usual.
deleted 5y ago
[deleted]
joshuamorton 5y ago
Consider that a random baseline is 10% accuracy, so 60% is pretty good, especially if the training time is functionally free to get to that accuracy. Granted the comparison is 2500 iterations of sgd which isn't a lot. I haven't played with cifar in a while, but that's probably not a ton of savings.

nestorD 5y ago

Very cool and sensical when you pause to think about it. While different architectures might have different components, they are also likely to reuse building blocks (ie: the first layers of convolutional neural networks are famously very similar from one problem to another). In essense this detects those building blocks and plugs the known weights in for you.