4 ms·

Hi, I wrote Gorgonia: https://github.com/chewxy/gorgonia https://github.com/chewxy/gorgonia If there are any questions I'm available to answer them

• IshKebab 10y ago
  How does it handle recurrent connections. I really like Microsoft's approach with CNTK - they use a DSL called 'BrainScript' to define the network, and it's basically a computational network (hence then name), but there are special operations like x = PastValue(y) and y = FutureValue(x) for forward and backward recurrent connections. It even automatically only unrolls the recurrent part of the network. In contrast all of the other NN systems I've used have premade layers like LSTM and GRU, and to be honest their methods are so hacky I haven't really been able to work out how to do custom RNNs in any other system. How does Gorgonia handle it?
  • chewxy 10y ago
    The trick, IMO, is to manually unroll them. So what I do is something like this: func (m *model) costFn() (cost *Node, n int, err error) { var prev *recurrentOutput for ...{ // build your recurrent graph here } } func (m *model) runOne() (err error) { cost, n err := costFn() g := m.g.SubgraphRoots(cost) machine := NewLispMachine(g) if err = machine.RunAll(); err != nil{ return } } The LispMachine type allows for rapid prototyping. Then once that's all done and stuff, you would probably have figured out what the size of your graph is and you can then create a TapeMachine that have the sizes you defined (though I have removed all the bits of the TapeMachine that made it turing complete, so it may be a bit hard to do that).
    • PeterisP 10y ago
      Manual unrolling fails as soon as the number of repetitions is not fixed (e.g. NLP tasks where it matches number of tokens in a sentence); so you want the looping to happen after the graph has been made, preferably on the GPU.
      • chewxy 10y ago
        For Gorgonia it's the other way around. You build the graph by manually unrolling it. It'll be a big goddamn graph, which is why you subgraph it and run it. source: actually have various LSTMs running. Some with attention, some with ADHD
        • PeterisP 10y ago
          Is performance acceptable if you're required to re-build/adjust the graph for every single minibatch, because the number of unrolled items is different every time? That is what I mean by "Manual unrolling fails as soon as the number of repetitions is not fixed"; you can do manual unrolling in every framework but generally repeated graph building makes it unusably slow if you're unable to do the loop "in the system" with a runtime-chosen number of repetitions. Padding is also not really a solution, since your average sequence length is likely 5 or 10 times less than the maximum sequence length that you want to support, so just padding to a fixed size will mean 5-10 times slower processing.
  • chewxy 10y ago
    I just checked out BrainScript. Looks pretty cool. Hacking up a parser for that into Gorgonia should be fairly trivial. Could be a fun weekend hacking project.