ekelsen

20 ms·

211.
by ekelsen 9y ago
My comment was meant to imply that if you're using JS, then you already care about other things _more_ than you care about performance. Once you've decided that you do care about those other things, then getting the best performan
212.
by ekelsen 9y ago
If you care about performance, don't use JS for training neural networks.
213.
by ekelsen 9y ago
It also seems like the price comparison should compare with the fp16 numbers on both platforms, not the fp32 numbers.
214.
by ekelsen 9y ago
The parent said nothing about purity of vitamins. That this is the first sentence of your response only seems to confirm his fears about marketing hype. I'm disappointed to see a legitimate PhD used for pseudo scientific snake oil sal
215.
by ekelsen 9y ago
One of the authors here, happy to answer any questions.
216.
by ekelsen 9y ago
PlaidML seems entirely geared towards inference (I only see batch size 1 anywhere). Training is important.
217.
by ekelsen 9y ago
I think it's likely you were solving the Reynold's Averaged NS - https://en.wikipedia.org/wiki/Reynolds-averaged_Navier%E2%80... and dropping the turbulence term to get what you call "steady state".
218.
by ekelsen 9y ago
You didn't actually say _why_ monitors are better for the author's use case: programming. "A much nicer experience" - how? in what way? is this quantifiable in any way?
219.
by ekelsen 9y ago
If you are dealing with numbers, then you can do this is true O(n) time with radix-select. (Basically do a MSB radix sort and only do subsequent passes in the bucket where you know k must be).
220.
by ekelsen 10y ago
I've never seen anyone optimize a neural network using LARS or cyclical coordinate descent. I thought that's what this entire discussion was about - not arbitrary optimization theory.
221.
by ekelsen 10y ago
1) Neural Network Loss functions are not convex. But that isn't the issue here. 2) When you use actual numerical optimization techniques with floating point arithmetic, you don't find an exact minimum (global or local). And you d
222.
by ekelsen 10y ago
Yes, but how do you actually optimize that loss in practice? I'm not saying that a perfect solution with an L1 penalty wouldn't have weights exactly equal to 0. I'm saying that with the optimization techniques that are comm
223.
by ekelsen 10y ago
If we have a parameter x, and some cost function J(x), then with L1 regularization the cost function would be J(x) + beta * abs(x). The derivative of that loss with respect to x would be J'(x) + beta * sgn(x). So using some variant of
224.
by ekelsen 10y ago
L1 regularization does not lead to values that are _exactly_ 0 either.
225.
by ekelsen 10y ago
LeakyReLUs often have very small slopes on the negative side; this can help solve the problem of no gradients getting through a layer because the activations were all 0. But I would say it still has a sparsity effect on a trained network b
226.
by ekelsen 10y ago
Citations: - MKL (hand written assembly for most routines) - gotoBLAS (hand written assembly) - fastest gpu gemm library https://github.com/openai/openai-gemm (hand written assembly) - fastest gpu conv libraries (cudnn
227.
by ekelsen 10y ago
I'm not sure I completely agree and/or I misunderstand. Compilers are still not used for many compute intensive applications that have been around for a long time. On both CPUs and GPUs the fastest gemm, convolution kernels, md f
228.
by ekelsen 10y ago
I'm not exactly sure what you mean, but it sounds like sarcasm. 1) It isn't actually obvious that if you can train a model faster this will allow you to get better final accuracy. It obviously allows you to get to a result faster,
229.
by ekelsen 11y ago
I think you should consider all of the results: tables 13-15 have comparisons to humans across a wide range of datasets. The human WER performance varies from 3.5 to 22.2 (on clean speech) - some datasets are much harder than others. And
230.
Optimizing Recurrent Neural Network Performance(svail.github.io)
21 points by ekelsen 11y ago|0 comments