4 ms·
One thing I didn't quite understand is how training sub-graphs in parallel works. If you are editing a sub-graph of a monolith type model, aren't you affecting
by eanzenberg 7y ago
One thing I didn't quite understand is how training sub-graphs in parallel works. If you are editing a sub-graph of a monolith type model, aren't you affecting other graphs that have dependencies on the one you're editing? If these are independent graphs, then what's a "sub-graph" even mean?
- punnerud 7y agoIn PyTorch you have full control on the graph and weight, everything feels like Python. So feeding some of the learning between “sub-graph” is easy. Not sure if this is possible on Tensorflow/Keras? He describes the sub-graph training in the context that they they have all the predictors in one big model, and with control of the network can feedforward and train sub-graph (read sub-parts) of the model.
- eanzenberg 7y agoThis is possible in keras, just drive new models that are functions of a monolith model and train independently. I still don’t understand the point though. If you train a “subgraph”, the other tasks dependent on the part of the graph will have to get retrained anyways, since those edits will affect the other tasks.
- paraschopra 7y agoI think their architecture might be their secret sauce. But I'm curious about this too.
- antpls 7y agoFirst time I read about "sub-network" is in this AI Google blog post : https://ai.googleblog.com/2019/09/recursive-sketches-for-modular-deep.html?m=1 https://ai.googleblog.com/2019/09/recursive-sketches-for-mod... They talk about the concept of "modular network". The article itself links to the Wikipedia page : https://en.wikipedia.org/wiki/Modular_neural_network https://en.wikipedia.org/wiki/Modular_neural_network Not sure it's exactly the same idea, but it looks similar.