3 ms·
Unless I'm working in an academic project I couldn't care less about elegancy. Speed (and ergo costs) trumps elegancy in industrial settings imho, specially co
by mrbungie 3y ago
Unless I'm working in an academic project I couldn't care less about elegancy.
Speed (and ergo costs) trumps elegancy in industrial settings imho, specially considering how expensive to run LLMs are. Such an OSS dependency can be improved or at least "wrapped" to isolate it from the rest of your codebase.
- brucethemoose2 3y agoAs do features. Even if llama.cpp was ~30% slower and the same size as ONNX, something like a custom grammar implementation or an extended context would be a huge deal.
- kanwisher 3y agoONNX doesn’t give you any benefits over something like llama.cpp and it’s significantly slower for zero advantage