3 ms·
i humbly request you to add mlc-llm to your quant test when/if you get around to doing it
by 0xDEADFED5 3y ago
i humbly request you to add mlc-llm to your quant test when/if you get around to doing it
- lhl 3y agoSure, I think their quant format is pretty basic, something similar to bnb q4 - my plan will to be scripting a framework for testing, so should do that as well since the omniquant implementation is in mlc-llm anyways.
- acosmism 3y agoi was trying to get this to work with mlc-llm. i'd appreciate any pointers
- acosmism 3y agomore specifically on a non-cuda gpu - mali on orangepi via opencl