4 ms·
What do you recommend that is faster that I can package into an app for distribution?
by sroussey 3y ago
What do you recommend that is faster that I can package into an app for distribution?
- modeless 3y agoI have packaged exllamav2 (plus a lot of other stuff) into an app for distribution here: https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7 I used pyinstaller. It was difficult because Python makes these things difficult. But it works. It does require an Nvidia GPU. MLC-LLM is another option that might be easier to package and potentially able to run on AMD.
- sroussey 3y agoOh yeah, I want to work on AMD/Intel/NVIDIA and MacOS, even iOS/Android. I've been following MLC-LLM as well. Right now I am just using JS/WASM from Huggingface, but later I will want something more performant.
- modeless 3y agoYeah if you want maximum performance on multiple platforms you'll probably have to package multiple frameworks. Llama.cpp might be a decently fast option on Apple Silicon, I'm not sure of the state of the art there.