4 ms·

Using matrices to perform the calculations is an optimization over doing a bunch of for loops. This vectorization results in faster code within higher level lan

by davedx 9y ago

Using matrices to perform the calculations is an optimization over doing a bunch of for loops. This vectorization results in faster code within higher level languages and on certain hardware platforms (SIMD). It's nothing to do with "slicker notation", although having written gradient descent with for loops and matrix operations, the vectorized version is simpler and cleaner to read in my opinion.

return0 9y ago
This has nothing to do with understanding backpropagation (which (correct me if i m wrong) is really the core of DL). In fact in the old days backpropagation was all about "propagating the deltas" and nothing about vectorizing.
Houshalter 9y ago
He's not complaining about using vectorization in code. The problem is papers and even explanations targeted at non-experts, often use obfuscated math in place of clear explanations. I've complained about this before here: https://news.ycombinator.com/item?id=13953530 https://news.ycombinator.com/item?id=13953530 Mathematical notation is basically a programming language. A programming language with weird symbols you can't type to search for, single letter variable names for everything, and no comments. And it's written by programmers that are obsessed with fitting everything into a simple line and making it as small as possible, no matter how difficult it is to read. Any programmer understands this is incredibly bad practice. And even if parse every step and perfectly follow what the code is doing, without explanation, it's pretty difficult to figure out why.
- gnaritas 9y ago
  > Mathematical notation is basically a programming language. A very bad one that can only be executed by brains with the requisite existing historical knowledge; in fact it's more like bad pseudo-code that lacks the explicitness necessary to translate into actual instructions. It's basically condensed jargon intended for the already converted. It'd probably be vastly easier to teach math with an actual programming language than with traditional notation. Scheme would be ideal for this.
- davedx 9y ago
  OK, I see what you're saying. I think you have the same issue with "real" programming languages too. If you compare some very concise Clojure or Scala code with the equivalent in Java, it can be quite hard to understand if you're not very familiar with the language. But I wouldn't necessarily say it's "incredibly bad practice". A Scala programmer can write concise and elegant code that to another Scala programmer is actually faster to understand because of that conciseness. Whereas the same code written with for loops and class method calls and all the boilerplate in Java would take more studying to filter out the low level constructions. It's about the level of abstraction. And yeah if you don't understand the notation or syntax at the level of abstraction you're studying, it will be very hard. (FWIW I find Scala code quite hard to understand sometimes, but I also find the more I know about the language, the more comprehensible it gets).
  - Houshalter 9y ago
    It's not necessarily the conciseness that's a problem. Using foreach instead of a full for loop is one thing. What I'm complaining about is code in place of an explanation. E.g. imagine coming across some nasty piece of code like this: https://en.wikipedia.org/wiki/Fast_inverse_square_root#Overview_of_the_code https://en.wikipedia.org/wiki/Fast_inverse_square_root#Overv... It doesn't matter how familiar you are with the language. Without an explanation of what the hell is going on, just looking at the code is useless.
    - davedx 9y ago
      Now we're talking about documentation. You are correct, no code is conpletely self documenting. But that Quake code is very low level, the opposite of what I think the grandparent doesn't like (very high level abstract notation)
cr0sh 9y ago
I think OP means vector notation in the research papers, not using vector operations and representations at the code level. That said, it can be useful for the beginner to implement a basic NN library from the ground up, to understand how the vector processing works, as well as what is going on step-wise with backprop and such. Once that is fully understood, the next step of utilizing true vector processing libraries can be taken, and so on - eventually culminating in using and understanding libraries like TensorFlow. Having the background of the lower levels gives you an appreciation and even insights when you transition to higher level frameworks. That's just my opinion, though.

by davedx 9y ago

return0 9y ago

This has nothing to do with understanding backpropagation (which (correct me if i m wrong) is really the core of DL). In fact in the old days backpropagation was all about "propagating the deltas" and nothing about vectorizing.

Houshalter 9y ago

He's not complaining about using vectorization in code. The problem is papers and even explanations targeted at non-experts, often use obfuscated math in place of clear explanations. I've complained about this before here: https://news.ycombinator.com/item?id=13953530 https://news.ycombinator.com/item?id=13953530 Mathematical notation is basically a programming language. A programming language with weird symbols you can't type to search for, single letter variable names for everything, and no comments. And it's written by programmers that are obsessed with fitting everything into a simple line and making it as small as possible, no matter how difficult it is to read. Any programmer understands this is incredibly bad practice. And even if parse every step and perfectly follow what the code is doing, without explanation, it's pretty difficult to figure out why.

gnaritas 9y ago
> Mathematical notation is basically a programming language. A very bad one that can only be executed by brains with the requisite existing historical knowledge; in fact it's more like bad pseudo-code that lacks the explicitness necessary to translate into actual instructions. It's basically condensed jargon intended for the already converted. It'd probably be vastly easier to teach math with an actual programming language than with traditional notation. Scheme would be ideal for this.
davedx 9y ago
OK, I see what you're saying. I think you have the same issue with "real" programming languages too. If you compare some very concise Clojure or Scala code with the equivalent in Java, it can be quite hard to understand if you're not very familiar with the language. But I wouldn't necessarily say it's "incredibly bad practice". A Scala programmer can write concise and elegant code that to another Scala programmer is actually faster to understand because of that conciseness. Whereas the same code written with for loops and class method calls and all the boilerplate in Java would take more studying to filter out the low level constructions. It's about the level of abstraction. And yeah if you don't understand the notation or syntax at the level of abstraction you're studying, it will be very hard. (FWIW I find Scala code quite hard to understand sometimes, but I also find the more I know about the language, the more comprehensible it gets).
- Houshalter 9y ago
  It's not necessarily the conciseness that's a problem. Using foreach instead of a full for loop is one thing. What I'm complaining about is code in place of an explanation. E.g. imagine coming across some nasty piece of code like this: https://en.wikipedia.org/wiki/Fast_inverse_square_root#Overview_of_the_code https://en.wikipedia.org/wiki/Fast_inverse_square_root#Overv... It doesn't matter how familiar you are with the language. Without an explanation of what the hell is going on, just looking at the code is useless.
  - davedx 9y ago
    Now we're talking about documentation. You are correct, no code is conpletely self documenting. But that Quake code is very low level, the opposite of what I think the grandparent doesn't like (very high level abstract notation)

cr0sh 9y ago

I think OP means vector notation in the research papers, not using vector operations and representations at the code level. That said, it can be useful for the beginner to implement a basic NN library from the ground up, to understand how the vector processing works, as well as what is going on step-wise with backprop and such. Once that is fully understood, the next step of utilizing true vector processing libraries can be taken, and so on - eventually culminating in using and understanding libraries like TensorFlow. Having the background of the lower levels gives you an appreciation and even insights when you transition to higher level frameworks. That's just my opinion, though.