4 ms·
its so long, so much waste of compute during inference. Wondering why they couldn't finetune it through some instructions.
by riku_iki 2y ago
its so long, so much waste of compute during inference. Wondering why they couldn't finetune it through some instructions.
- tayo42 2y agohas anything been done to like turn common phrases into a single token? like "can you please" maps to 3895 instead of something like "10 245 87 941" Or does it not matter since tokenization is already a kind of compression?
- naveen99 2y agoYou can try cyp but ymmv
- hiddencost 2y agoFine-tuning is expensive and slow compared to prompt engineering, for making changes to a production system. You can develop validate and push a new prompt in hours.
- WithinReason 2y agoYou need to include the prompt in every query, which makes it very expensive
- GaggiX 2y agoThe prompt is kv-cached, it's precomputed.
- WithinReason 2y agoGood point, but it still increases the compute of all subsequent tokens
- WesolyKubeczek 2y agoI imagine the tone you set at the start affects the tone of responses, as it makes completions in that same tone more likely. I would very much like to see my assumption checked — if you are as terse as possible in your system prompt, would it turn into a drill sergeant or an introvert?
- isoprophlex 2y agoThey're most likely using prefix caching so it doesn't materially change the inference time