3 ms·
The trick, IMO, is to manually unroll them. So what I do is something like this: func (m *model) costFn() (cost *Node, n int, err error) { var pre
by chewxy 10y ago
The trick, IMO, is to manually unroll them.
So what I do is something like this:
func (m *model) costFn() (cost *Node, n int, err error) {
var prev *recurrentOutput
for ...{
// build your recurrent graph here
}
}
func (m *model) runOne() (err error) {
cost, n err := costFn()
g := m.g.SubgraphRoots(cost)
machine := NewLispMachine(g)
if err = machine.RunAll(); err != nil{
return
}
}
The LispMachine type allows for rapid prototyping. Then once that's all done and stuff, you would probably have figured out what the size of your graph is and you can then create a TapeMachine that have the sizes you defined (though I have removed all the bits of the TapeMachine that made it turing complete, so it may be a bit hard to do that).
- PeterisP 10y agoManual unrolling fails as soon as the number of repetitions is not fixed (e.g. NLP tasks where it matches number of tokens in a sentence); so you want the looping to happen after the graph has been made, preferably on the GPU.
- chewxy 10y agoFor Gorgonia it's the other way around. You build the graph by manually unrolling it. It'll be a big goddamn graph, which is why you subgraph it and run it. source: actually have various LSTMs running. Some with attention, some with ADHD
- PeterisP 10y agoIs performance acceptable if you're required to re-build/adjust the graph for every single minibatch, because the number of unrolled items is different every time? That is what I mean by "Manual unrolling fails as soon as the number of repetitions is not fixed"; you can do manual unrolling in every framework but generally repeated graph building makes it unusably slow if you're unable to do the loop "in the system" with a runtime-chosen number of repetitions. Padding is also not really a solution, since your average sequence length is likely 5 or 10 times less than the maximum sequence length that you want to support, so just padding to a fixed size will mean 5-10 times slower processing.