4 ms·

The GPU stuff is basically just a teaser into the future, it's still much more immature than that CPU stuff for Tullio. > Basically CPU is at this point outdat

The GPU stuff is basically just a teaser into the future, it's still much more immature than that CPU stuff for Tullio. > Basically CPU is at this point outdated technology for matrix multiplication. This is an incredibly naive statement. There are many many circumstances where you need fast matmul on the CPU even if you have a GPU available because it takes too long to send the memory to the GPU and fetch it once the kernel runs. If you have many chained matmuls, then yes the GPU is your friend. That said, regardless of how useful the CPU is in practice for matrix multiplication, BLAS kernels are some of the most overengineered, most optimized pieces of code in wide use. Tullio is incredibly flexibly and general. The fact that matrix multiplication falls out as a special case in Tullio and can outperform OpenBLAS and match MKL is absolutely stunning.

xiphias2 6y ago
Can you give concrete examples when data processing on the CPU is clearly better than on the GPU? Before I was working with time series, and there the CPU really shined, but right now I'm classifying sick patients using DNA methylation data (about 100-300 GB / dataset). I'm using a convolutional network that already gives me state of the art results, but I want to experiment with self attention based neural networks. While I'm doing this, most of the biologists are still using logistic regression that can be computed on the CPU. I see more and more domains (even simulations) where neural networks can outperform classical methods if you know how to apply them, and CPUs can never match the GPU performance. Also as I wrote, the slowest part of all these networks is the matrix multiplication (self attention is especially depending on it).
- eigenspace 6y ago
  > Can you give concrete examples when data processing on the CPU is clearly better than on the GPU? Pretty much anything where the matmul is important to performance, but doesn't dominate it and requires serial steps. A classic example off the top of my head would be solving a matrix differential equation[1]. Here's an example in Julia using DifferentialEquations.jl and CUDA.jl for the GPU part: This is solving the differential equation du/dt = A * u for the cases where u and A are arrays on the GPU versus when they are arrays on the CPU. I'm doing the solving in-place to try and minimize the amount of expensive allocations. using DifferentialEquations, CUDA function mysolve(u0, A, tspan) f!(du, u, A, t) = mul!(du, A, u) prob = ODEProblem(f!, u0, tspan, A) solve(prob) end mysolve (generic function with 1 method) julia> let n = 50, u0 = rand(Float32, n), A = randn(Float32, n, n) # Make GPU versions of u0 and A cuu0 = cu(u0) cuA = cu(A) # Do a run of the code so there's no compiler latency biasing the results mysolve( u0, A, (0.0, 1.0)) mysolve(cuu0, cuA, (0.0, 1.0)) # Now run the compiler code and time it's execution: @time mysolve( u0, A, (0.0, 1.0)) @time mysolve(cuu0, cuA, (0.0, 1.0)) end; 0.000314 seconds (621 allocations: 98.281 KiB) 0.007218 seconds (10.33 k allocations: 382.266 KiB) In this case, the CPU code was ~20x faster than the GPU code. True, I could get a better GPU (I'm using a RTX 2060), but I could also get a better CPU (Ryzen 5 2600). There's not much point in using a GPU for this example, and I think that just comes down to the fact that there's a non-trivial amount of serial work that needs to be done between the matmuls. [1] https://en.wikipedia.org/wiki/Matrix_differential_equation https://en.wikipedia.org/wiki/Matrix_differential_equation
  - xiphias2 6y ago
    I wrote big (10000x10000x10000) matrix multiplication and data processing (gigabytes of data) in the original comment, in which case the matrix size is small compared to the number of multiplications that the device has to make. Try n=10000 in your example (although I'm not sure about the differential solver part, as I don't run simulations), it's more interesting. Also differential expressions are more useful for simulation than data processing in my experience. Of course CPU is faster for small matrices, but again, if I have hundreds of gigabytes of data that you want to process, CPU is always slower.

xiphias2 6y ago

Can you give concrete examples when data processing on the CPU is clearly better than on the GPU? Before I was working with time series, and there the CPU really shined, but right now I'm classifying sick patients using DNA methylation data (about 100-300 GB / dataset). I'm using a convolutional network that already gives me state of the art results, but I want to experiment with self attention based neural networks. While I'm doing this, most of the biologists are still using logistic regression that can be computed on the CPU. I see more and more domains (even simulations) where neural networks can outperform classical methods if you know how to apply them, and CPUs can never match the GPU performance. Also as I wrote, the slowest part of all these networks is the matrix multiplication (self attention is especially depending on it).

eigenspace 6y ago
> Can you give concrete examples when data processing on the CPU is clearly better than on the GPU? Pretty much anything where the matmul is important to performance, but doesn't dominate it and requires serial steps. A classic example off the top of my head would be solving a matrix differential equation[1]. Here's an example in Julia using DifferentialEquations.jl and CUDA.jl for the GPU part: This is solving the differential equation du/dt = A * u for the cases where u and A are arrays on the GPU versus when they are arrays on the CPU. I'm doing the solving in-place to try and minimize the amount of expensive allocations. using DifferentialEquations, CUDA function mysolve(u0, A, tspan) f!(du, u, A, t) = mul!(du, A, u) prob = ODEProblem(f!, u0, tspan, A) solve(prob) end mysolve (generic function with 1 method) julia> let n = 50, u0 = rand(Float32, n), A = randn(Float32, n, n) # Make GPU versions of u0 and A cuu0 = cu(u0) cuA = cu(A) # Do a run of the code so there's no compiler latency biasing the results mysolve( u0, A, (0.0, 1.0)) mysolve(cuu0, cuA, (0.0, 1.0)) # Now run the compiler code and time it's execution: @time mysolve( u0, A, (0.0, 1.0)) @time mysolve(cuu0, cuA, (0.0, 1.0)) end; 0.000314 seconds (621 allocations: 98.281 KiB) 0.007218 seconds (10.33 k allocations: 382.266 KiB) In this case, the CPU code was ~20x faster than the GPU code. True, I could get a better GPU (I'm using a RTX 2060), but I could also get a better CPU (Ryzen 5 2600). There's not much point in using a GPU for this example, and I think that just comes down to the fact that there's a non-trivial amount of serial work that needs to be done between the matmuls. [1] https://en.wikipedia.org/wiki/Matrix_differential_equation https://en.wikipedia.org/wiki/Matrix_differential_equation
- xiphias2 6y ago
  I wrote big (10000x10000x10000) matrix multiplication and data processing (gigabytes of data) in the original comment, in which case the matrix size is small compared to the number of multiplications that the device has to make. Try n=10000 in your example (although I'm not sure about the differential solver part, as I don't run simulations), it's more interesting. Also differential expressions are more useful for simulation than data processing in my experience. Of course CPU is faster for small matrices, but again, if I have hundreds of gigabytes of data that you want to process, CPU is always slower.