3 ms·

At least in scientific computing this applies. And note I say most decent implementations, so I assume you took it to a language without performance issues like

At least in scientific computing this applies. And note I say most decent implementations, so I assume you took it to a language without performance issues like C/Fortran/Rust/Julia/Go/etc. and got rid of temporary allocations, made sure you looped in the correct ordering, etc. After you do all of that, for mathematical and scientific problems you tend to hit a wall. [BLAS libraries tend to be within that range of each other](https://github.com/barche/julia-blas-benchmarks/blob/master/BenchmarkResults.ipynb https://github.com/barche/julia-blas-benchmarks/blob/master/...), which pretty much directly means that regression, linear classification, etc. libraries are all going to be within this amount. Different differential equations solvers which are sufficiently optimized with the same algorithm tend to be within that amount from each other. Neural network libraries tend to be within that amount from each other. General optimization algorithms over a large class of problems tend to be the same. That doesn't mean it's not a worthy enterprise to keep making the "general fallbacks" better, indeed this helps everyone. If you're looking to make your machine learning software faster, writing a new piece of fundamental GPU code to beat cuDNN could possibly be faster but probably won't get you very far. But changing your model to better match the data could make the problem orders of magnitude easier.

Veedrac 9y ago
You can't prove the effect of "temporary arrays, choice of language, and parallelism" and "pointer indirection, cache efficiency, and SIMD vectorization" don't matter unless you compare one thing that does those things to another thing that doesn't. BLAS libraries are all within the range of each other because they all store memory in basically the same way, do indirection in basically the same way, care for the cache in basically the same way and do SIMD in basically the same way. As soon as you step out of those prebuilt blocks and build your own function over squares of numbers, you're going to be losing far more than a factor of 3 until you understand how to handle these issues.
- newen 9y ago
  >You can't prove the effect of "temporary arrays, choice of language, and parallelism" and "pointer indirection, cache efficiency, and SIMD vectorization" don't matter unless you compare one thing that does those things to another thing that doesn't. I think he said the exact opposite. He's basically saying algorithmic time complexity matters (and therefore identifying your problem class matters, since you might get a more efficient algorithm out of it) since you can only get so far with the generic algorithm, however good your implementation is.
  - Veedrac 9y ago
    The argument is symmetric; once you've tuned the algorithm the only way to get >10x speedups is by working with those low-level things. Both the broad-strokes algorithm and the fine details are responsible for huge performance differences.
adrianratnapala 9y ago
Actually scientific code has huge variation in speed. The same high-level algorithm might be implemented inside an old, highly optimised FORTRAN program and then get transliterated into a use-once MATLAB script which runs 100 times slower because the vectorisation voodoo wasn't quite right. In the latter case, people might not bother fixing it because the hour it would take is longer than the run-time. Or maybe the run-time cost is weeks, but the people doing it are scientists and understanding computers isn't their job.
- ChrisRackauckas 9y ago
  I made the caveat that it's sufficiently optimized. I'm not talking about full loops on native Python arrays or something silly like that. If two people who know what they're doing try to optimize some scientific algorithm in a language without extra penalties (C, C++, Fortran, Julia, Go, Rust, etc.) the runtimes end up quite similar, at least within a few x. Of course someone can make it worse (there exist many bad coders in science...), but I'm not talking about that. I'm just saying that spending 2 years to play with some assembler code is not likely to give you some amazing speedup, while specializing algorithms to your problem is a clear and tested way to get something that is much more efficient.

Veedrac 9y ago

You can't prove the effect of "temporary arrays, choice of language, and parallelism" and "pointer indirection, cache efficiency, and SIMD vectorization" don't matter unless you compare one thing that does those things to another thing that doesn't. BLAS libraries are all within the range of each other because they all store memory in basically the same way, do indirection in basically the same way, care for the cache in basically the same way and do SIMD in basically the same way. As soon as you step out of those prebuilt blocks and build your own function over squares of numbers, you're going to be losing far more than a factor of 3 until you understand how to handle these issues.

newen 9y ago
>You can't prove the effect of "temporary arrays, choice of language, and parallelism" and "pointer indirection, cache efficiency, and SIMD vectorization" don't matter unless you compare one thing that does those things to another thing that doesn't. I think he said the exact opposite. He's basically saying algorithmic time complexity matters (and therefore identifying your problem class matters, since you might get a more efficient algorithm out of it) since you can only get so far with the generic algorithm, however good your implementation is.
- Veedrac 9y ago
  The argument is symmetric; once you've tuned the algorithm the only way to get >10x speedups is by working with those low-level things. Both the broad-strokes algorithm and the fine details are responsible for huge performance differences.

adrianratnapala 9y ago

Actually scientific code has huge variation in speed. The same high-level algorithm might be implemented inside an old, highly optimised FORTRAN program and then get transliterated into a use-once MATLAB script which runs 100 times slower because the vectorisation voodoo wasn't quite right. In the latter case, people might not bother fixing it because the hour it would take is longer than the run-time. Or maybe the run-time cost is weeks, but the people doing it are scientists and understanding computers isn't their job.

ChrisRackauckas 9y ago
I made the caveat that it's sufficiently optimized. I'm not talking about full loops on native Python arrays or something silly like that. If two people who know what they're doing try to optimize some scientific algorithm in a language without extra penalties (C, C++, Fortran, Julia, Go, Rust, etc.) the runtimes end up quite similar, at least within a few x. Of course someone can make it worse (there exist many bad coders in science...), but I'm not talking about that. I'm just saying that spending 2 years to play with some assembler code is not likely to give you some amazing speedup, while specializing algorithms to your problem is a clear and tested way to get something that is much more efficient.