5 ms·

If you’ve listened to their earnings calls this is par for the course. Say what you will about his tweets, but I find these discussions fantastically informati

by mattrp 7y ago

If you’ve listened to their earnings calls this is par for the course. Say what you will about his tweets, but I find these discussions fantastically informative and transparent.

isoprophlex 7y ago
absolutely glorious, they're taking questions. first question: can you use anything else besides ReLU activations in your neural net engine? this is wonderful
- akhilcacharya 7y ago
  ...is it? I don't know the context here but it feels like they're trying to ML-splain Karpathy?
- 2bitencryption 7y ago
  what a funny question. "you say you can put bananas in your smoothie. just a thought, but perhaps, might I ask, do you have the capability to put strawberries as well, or are we not there yet?"
  - pixelpp 7y ago
    haha
  - chrisa 7y ago
    That was my first instinct to that question as well, but I think he was just asking if ReLU was required by the hardware design - or if it was possible to use other activation functions as well. If the the ReLU was part of the hardware itself somehow, then it wouldn't be possible to use tanh or sigmoid (which may be better in certain situations); so I think he was just asking if ReLU was required, or if there was flexibility allowed in the activation function.
    - 2bitencryption 7y ago
      ah, makes sense. guess I was the fool :P
- cr0sh 7y ago
  I'm not a NN expert, but based on what I have found, the point of using RELU instead of other activation functions is what is called the problem of "vanishing gradients". Basically (IIRC), during backprop the error difference gets ever smaller the further back in layers you go, ultimately getting "lost in the noise", making learning in the earlier layers more difficult to impossible. I'm not saying RELU is the only option to make this work, or that it's the only activation function that provides a "fix" for the issue; I'm sure there are other ways to deal with vanishing gradients that I don't know about. I also lack the mathematical knowledge as to why RELU helps in this manner, but I suspect something having to do with the lack of "asymptotic structure" approaching the extremes (I don't know what the proper term would be). Or maybe it allows for some form of "forgetting", in the prevention of multiplying very small numbers (such values just go to zero ultimately)? Maybe someone else here with the knowledge can explain it better, and we can both learn...?
  - Qub3d 7y ago
    ReLu is useful, but the question as asked was more about other (also essential) functions for other steps in RNNs -- in particular, sigmoid "squash" functions, as well as MaxPooling.

by mattrp 7y ago

If you’ve listened to their earnings calls this is par for the course. Say what you will about his tweets, but I find these discussions fantastically informative and transparent.

isoprophlex 7y ago

absolutely glorious, they're taking questions. first question: can you use anything else besides ReLU activations in your neural net engine? this is wonderful

akhilcacharya 7y ago
...is it? I don't know the context here but it feels like they're trying to ML-splain Karpathy?
2bitencryption 7y ago
what a funny question. "you say you can put bananas in your smoothie. just a thought, but perhaps, might I ask, do you have the capability to put strawberries as well, or are we not there yet?"
- pixelpp 7y ago
  haha
- chrisa 7y ago
  That was my first instinct to that question as well, but I think he was just asking if ReLU was required by the hardware design - or if it was possible to use other activation functions as well. If the the ReLU was part of the hardware itself somehow, then it wouldn't be possible to use tanh or sigmoid (which may be better in certain situations); so I think he was just asking if ReLU was required, or if there was flexibility allowed in the activation function.
  - 2bitencryption 7y ago
    ah, makes sense. guess I was the fool :P
cr0sh 7y ago
I'm not a NN expert, but based on what I have found, the point of using RELU instead of other activation functions is what is called the problem of "vanishing gradients". Basically (IIRC), during backprop the error difference gets ever smaller the further back in layers you go, ultimately getting "lost in the noise", making learning in the earlier layers more difficult to impossible. I'm not saying RELU is the only option to make this work, or that it's the only activation function that provides a "fix" for the issue; I'm sure there are other ways to deal with vanishing gradients that I don't know about. I also lack the mathematical knowledge as to why RELU helps in this manner, but I suspect something having to do with the lack of "asymptotic structure" approaching the extremes (I don't know what the proper term would be). Or maybe it allows for some form of "forgetting", in the prevention of multiplying very small numbers (such values just go to zero ultimately)? Maybe someone else here with the knowledge can explain it better, and we can both learn...?
- Qub3d 7y ago
  ReLu is useful, but the question as asked was more about other (also essential) functions for other steps in RNNs -- in particular, sigmoid "squash" functions, as well as MaxPooling.