3 ms·

Can anybody who knows or understands better, how applicable is this new technique to being used in a GEMM implementation in a BLAS library in mainstream numeric

by sevagh 3y ago

Can anybody who knows or understands better, how applicable is this new technique to being used in a GEMM implementation in a BLAS library in mainstream numerical libraries?

infradig 3y ago
I know nothing but read the article. These improvements are about analytic solutions and have no practical use. At least that's what it said.
deleted 3y ago
[deleted]
klyrs 3y ago
It's been a great many years since I've touched this stuff but two rules of thumb probably haven't changed in the meantime: 1. in general, algorithms tend to slow down on small problems in exchange for the asymptotic speedup -- so you probably don't want to use this algorithm for anything that fits in memory (or perhaps the solar system) 2. in specific, asymptotically fast matrix multiplication tend to be numerically unstable. So you probably don't want to use this algorithm unless you're working with infinite precision.
- Arech 3y ago
  > 1. in general, algorithms tend to slow down on small problems in exchange for the asymptotic speedup -- so you probably don't want to use this algorithm for anything that fits in memory (or perhaps the solar system) This is not necessary true and very much depend on specifics of an algo. Some improvements could be very applicable for quite normally sized tensors. For example, Winograd algorithm is quite often a routine choice for 3x3xc convolutions. > 2. in specific, asymptotically fast matrix multiplication tend to be numerically unstable. This should be investigated in each particular case of an algorithm, but numeric stability is indeed a major concern for a practitioner. I skimmed very superficially through the article but didn't notice mentions of that.
sdenton4 3y ago
These tend to be galactic algorithms. Furthermore, I don't know that they do any work involving parallelism, or considering the narrowing gap between time needed for addition vs multiplication - typically only multiplies are counted.
- rerdavies 3y ago
  The presumption that additions are free, when, in current hardware, there's not a dramatic difference between the cost of an add and a multiply, seems particularly damning. These algorithms reduce the number of multiplies, but seem to increase the total number of floating point ops (including both adds and multiplies), unless I'm mistaken.

by sevagh 3y ago

Can anybody who knows or understands better, how applicable is this new technique to being used in a GEMM implementation in a BLAS library in mainstream numerical libraries?

infradig 3y ago

I know nothing but read the article. These improvements are about analytic solutions and have no practical use. At least that's what it said.

deleted 3y ago

[deleted]

klyrs 3y ago

It's been a great many years since I've touched this stuff but two rules of thumb probably haven't changed in the meantime: 1. in general, algorithms tend to slow down on small problems in exchange for the asymptotic speedup -- so you probably don't want to use this algorithm for anything that fits in memory (or perhaps the solar system) 2. in specific, asymptotically fast matrix multiplication tend to be numerically unstable. So you probably don't want to use this algorithm unless you're working with infinite precision.

Arech 3y ago
> 1. in general, algorithms tend to slow down on small problems in exchange for the asymptotic speedup -- so you probably don't want to use this algorithm for anything that fits in memory (or perhaps the solar system) This is not necessary true and very much depend on specifics of an algo. Some improvements could be very applicable for quite normally sized tensors. For example, Winograd algorithm is quite often a routine choice for 3x3xc convolutions. > 2. in specific, asymptotically fast matrix multiplication tend to be numerically unstable. This should be investigated in each particular case of an algorithm, but numeric stability is indeed a major concern for a practitioner. I skimmed very superficially through the article but didn't notice mentions of that.

sdenton4 3y ago

These tend to be galactic algorithms. Furthermore, I don't know that they do any work involving parallelism, or considering the narrowing gap between time needed for addition vs multiplication - typically only multiplies are counted.

rerdavies 3y ago
The presumption that additions are free, when, in current hardware, there's not a dramatic difference between the cost of an add and a multiply, seems particularly damning. These algorithms reduce the number of multiplies, but seem to increase the total number of floating point ops (including both adds and multiplies), unless I'm mistaken.