3 ms·
That's an interesting idea. Are llama.cpp (and similar targets) made into standalone C++ source files without dependencies doing the heavy lifting?
by fooker 3y ago
That's an interesting idea.
Are llama.cpp (and similar targets) made into standalone C++ source files without dependencies doing the heavy lifting?
- HanClinto 3y agoYes, that's the idea of the project. One can now compile against cuBLAS and other GPU libraries to help take some of the load, but the core of the project is indeed to run on the CPU and do all of the work without dependencies.
- mhh__ 3y agoIs llama performance not basically bound by the memory hierarchy?
- viraptor 3y agoBound by: yes. Currently restricted by: possibly not. For example the performance difference between clang 11 and 15 compiled llama.cpp is ~10% (I tested a while ago, can't remember the exact number)