You can fine tune an LLM to output a particular token for category X. This way, you get 1) fast inference (just need to generate a single token) and 2) confiden

You can fine tune an LLM to output a particular token for category X. This way, you get 1) fast inference (just need to generate a single token) and 2) confidence metrics (just use the logprobs for the token that was chosen). Gets me to 93% accuracy with a 7B model, trained with only 15 examples per category (around 10-15 categories in total, I think).