4 ms·
Maybe I'm thinking about it too simply but yeah I agree. Language models in particular are very similar architectures and effectively a lot of dot products. An
by version_five 3y ago
Maybe I'm thinking about it too simply but yeah I agree.
Language models in particular are very similar architectures and effectively a lot of dot products. And running them on GPU's is arguably overkill. Look at llama.cpp for the way the industry is going. I want a fast parallel quantized dot product instruction on a CPU, and I want the memory bandwidth to keep it loaded up. Intel should be able to deliver that, with none of the horrible baggage that comes from CUDA and nvidia drivers.
- CoastalCoder 3y agoDoes Intel have a credibility problem w.r.t. ISA extensions to support deep learning? I'm thinking about the widespread confusion they caused by having different CPUs support different subsets of the AVX-512 ISA.
- refulgentis 3y agoThis reads like parody (from llama.cpp, to it being a beacon of where industry is going (!?), to GPUs are overkill for what is effectively a lot of dot products)
- NavinF 3y agoYeah using CPUs for inference or training is ridiculous. We're talking 1/20th the performance for 1/4th the energy
- pjmlp 3y agoThe reason CUDA has won is precisily because it isn't horribly stuck in a C dialect, have embraced polyglot workloads since 2010, have a great developer experience where GPUs can be debugged like regular CPUs, and the library ecosystem. Now while NVidia is making standard C++ run on CUDA, Intel is still having SYSCL and oneAPI extensions. Similarly with Python and RAPIDS framework. Intel and AMD have to up their game for the same kind of developer experience.
- fransje26 3y agoErr.. Last time I checked, CUDA was the one with the partially compliant C++ implementation, while, on the contrary, SYSCL was being base on pure C++17..
- pjmlp 3y agoTime to check again, as CUDA is C++20 for a bit now (minus modules), and NVidia is the one driving the senders/receivers work for C++26, based on their CUDA libraries. SYCL isn't pure C++, meaning writing STL code that goes into the GPU, like CUDA allows for, nor requires the hardware to follow C++ memory model.