4 ms·

I thought baking matrix arithmetic into the language was more about making matrix-related code look nice than making it faster. That is, most languages just us

I thought baking matrix arithmetic into the language was more about making matrix-related code look nice than making it faster. That is, most languages just use LAPACK for matrix arithmetic, even Matlab and Julia, and they are distinguished only in that they provide special syntax for it. "In Julia (as in much of scientific computation), dense linear-algebra operations are based on the LAPACK library, which in turn is built on top of basic linear-algebra building-blocks known as the BLAS" [0]. How does baking matrix arithmetic into the language itself (maybe I misunderstood what you meant by this) provide new opportunities for compilers to optimize the code? Is it that if every bit of a program was implemented in the same language (e.g., Fortran), the compiler can do whole-program optimizations it otherwise could not? [0] https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/#BLAS-functions https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/#BLAS-...

_hl_ 5y ago
I think the point they were making is that LAPACK et al are a pain to use directly (they really are!), and abstractions are limited by the capabilities of the language or heavy to compile (not sure I agree, Eigen is pretty okay).
duped 5y ago
With dedicated semantics it becomes possible for a compiler to optimize things explicitly rather than implicitly. It's sort of equivalent to how a language without generics or inheritance can't perform certain optimizations (like inlining method calls) because the compiler isn't aware of those semantics in the first place. The overhead of BLAS is significant for small matrices. However, a compiler can tell if matrices are small at compile time (or JIT time) and inline the expressions which presents enormous speed gains. It can also make judgement calls about when to use sparse or dense arrays but that's a bit trickier. Compilers can do this without knowing that matrices or linear algebra exist, but it usually takes PGO which can be difficult to use. Iirc there's a linear algebra library for Julia that can beat the LAPACK backends for small matrices, which is a pretty big deal. There are also classes of optimizations like SIMD that don't just depend on matrix size but the target platform, which may not be the platform the software was compiled on. The way to do this is with function multiversioning but not all compilers/languages support it, this is kind of a classic usecase for a JIT compiler. One of the reasons Eigen is great is because it can use template semantics to make these optimizations at compile time. It's just slightly annoying in how it harms compile times.
- adgjlsfhk1 5y ago
  yeah. It's Octavian https://raw.githubusercontent.com/JuliaLinearAlgebra/Octavian.jl/master/docs/src/assets/bench10980xe.svg https://raw.githubusercontent.com/JuliaLinearAlgebra/Octavia... It can beat MKL convincingly up to 100x100 and remains slightly ahead up to 1000x1000 (and performance is still improving).

_hl_ 5y ago

I think the point they were making is that LAPACK et al are a pain to use directly (they really are!), and abstractions are limited by the capabilities of the language or heavy to compile (not sure I agree, Eigen is pretty okay).

duped 5y ago

With dedicated semantics it becomes possible for a compiler to optimize things explicitly rather than implicitly. It's sort of equivalent to how a language without generics or inheritance can't perform certain optimizations (like inlining method calls) because the compiler isn't aware of those semantics in the first place. The overhead of BLAS is significant for small matrices. However, a compiler can tell if matrices are small at compile time (or JIT time) and inline the expressions which presents enormous speed gains. It can also make judgement calls about when to use sparse or dense arrays but that's a bit trickier. Compilers can do this without knowing that matrices or linear algebra exist, but it usually takes PGO which can be difficult to use. Iirc there's a linear algebra library for Julia that can beat the LAPACK backends for small matrices, which is a pretty big deal. There are also classes of optimizations like SIMD that don't just depend on matrix size but the target platform, which may not be the platform the software was compiled on. The way to do this is with function multiversioning but not all compilers/languages support it, this is kind of a classic usecase for a JIT compiler. One of the reasons Eigen is great is because it can use template semantics to make these optimizations at compile time. It's just slightly annoying in how it harms compile times.

adgjlsfhk1 5y ago
yeah. It's Octavian https://raw.githubusercontent.com/JuliaLinearAlgebra/Octavian.jl/master/docs/src/assets/bench10980xe.svg https://raw.githubusercontent.com/JuliaLinearAlgebra/Octavia... It can beat MKL convincingly up to 100x100 and remains slightly ahead up to 1000x1000 (and performance is still improving).