3 ms·

> Understanding modern ML algebra is really "general relativity hard" if not actually harder. Spiking NNs are "quantum physics hard". The math is very much tran

> Understanding modern ML algebra is really "general relativity hard" if not actually harder. Spiking NNs are "quantum physics hard". The math is very much translatable between these domains. That's not my experience at all. What kind of ML requires something beyond basic linear algebra? Being comfortable manipulating matrices is certainly harder than plugging data into an sklearn function, but it's also significantly easier than understanding general relativity.

Retric 7y ago
The NN complexity is not a question of doing the calculations. Go on a 5x5 board would be a very straightforward game, go in a 19x19 board is not. Similarly, understanding how NN fiction at a nuts and bolts level stops being enough as you build larger networks.
- nitrogen 7y ago
  As a curious onlooker to this discussion, could you point to some references for learning the math that applies to emergent behavior in these larger networks?
- dorgo 7y ago
  > Go on a 5x5 board would be a very straightforward game I think it is not a good comparison. A small board calls for simpler algorithms: just try out all possible games and choose the most favorable one. For larger boards this simple approach doesn't work any more and you start looking for something different. But the architecture of NN networks doesn't change at all if you move from small to large scales. Only the number and size of layers increases. Edit: actually, that's the reason they call it "deep learning". it has more layers..
  - hadsed 7y ago
    The architecture doesn't change but the way it works often changes dramatically with scale.
- sweeneyrod 7y ago
  But isn't the main theme of NNs that the low level understanding is the best we have? In other words, there is no body of theory we can use to analyse architectural decisions so we just have to go off experimentation and heuristics derived from that.
  - Retric 7y ago
    I think a huge number of people publishing on the field would disagree with that assessment. Which is not to say they are right, but what looks like oversimplification is rarely accurate.

Retric 7y ago

The NN complexity is not a question of doing the calculations. Go on a 5x5 board would be a very straightforward game, go in a 19x19 board is not. Similarly, understanding how NN fiction at a nuts and bolts level stops being enough as you build larger networks.

nitrogen 7y ago
As a curious onlooker to this discussion, could you point to some references for learning the math that applies to emergent behavior in these larger networks?
dorgo 7y ago
> Go on a 5x5 board would be a very straightforward game I think it is not a good comparison. A small board calls for simpler algorithms: just try out all possible games and choose the most favorable one. For larger boards this simple approach doesn't work any more and you start looking for something different. But the architecture of NN networks doesn't change at all if you move from small to large scales. Only the number and size of layers increases. Edit: actually, that's the reason they call it "deep learning". it has more layers..
- hadsed 7y ago
  The architecture doesn't change but the way it works often changes dramatically with scale.
sweeneyrod 7y ago
But isn't the main theme of NNs that the low level understanding is the best we have? In other words, there is no body of theory we can use to analyse architectural decisions so we just have to go off experimentation and heuristics derived from that.
- Retric 7y ago
  I think a huge number of people publishing on the field would disagree with that assessment. Which is not to say they are right, but what looks like oversimplification is rarely accurate.