4 ms·
If anyone's interested in non-toy usage, there's the souper project as well https://github.com/google/souper https://github.com/google/souper It's a shame it's
by viraptor 3y ago
If anyone's interested in non-toy usage, there's the souper project as well https://github.com/google/souper https://github.com/google/souper
It's a shame it's a bit hard to actually build these days. Its target is people working on compilers, not end users. But there are some odd cases where I'd really like to use it myself, like trying to get speedup in llama.cpp.
- fooker 3y agoThat's an interesting idea. Are llama.cpp (and similar targets) made into standalone C++ source files without dependencies doing the heavy lifting?
- HanClinto 3y agoYes, that's the idea of the project. One can now compile against cuBLAS and other GPU libraries to help take some of the load, but the core of the project is indeed to run on the CPU and do all of the work without dependencies.
- mhh__ 3y agoIs llama performance not basically bound by the memory hierarchy?
- viraptor 3y agoBound by: yes. Currently restricted by: possibly not. For example the performance difference between clang 11 and 15 compiled llama.cpp is ~10% (I tested a while ago, can't remember the exact number)