2 ms·

There is no other way than shipping your own model, because you will want an abstracted API over the inference, and you don't know what the user has installed.

by alex7o 5mo ago

There is no other way than shipping your own model, because you will want an abstracted API over the inference, and you don't know what the user has installed. Also you can ship 9b fp4 model but it all just depends

LPisGood 5mo ago
You can know what the user has installed if the OS developer offers something.
_heimdall 5mo ago
Knowing what's installed would have to be an OS API. If LLMs provide a standard API surface to the OS, likely including metadata related to feature support.