3 ms·

Having a tough time keeping token usage low, and context beefy enough to response to user's with useful information. One strategy I've applied is to qualify a

Having a tough time keeping token usage low, and context beefy enough to response to user's with useful information. One strategy I've applied is to qualify a question into one of several categories and then use a small hyperfocused context. Another is to remove commas and currency symbols from money values.

fnordpiglet 3y ago
I’ve found most advanced LLM are pretty resilient to abbreviations as well. “Tho” can replace “though”, and you can even be as racy as “b4” instead of before. Depending on the model it can start degenerating into abbreviating talk in response for obvious reasons, but for GPT4 it doesn’t seem to be impacted.
- sergiotapia 3y ago
  That's a great tip! Do you happen to have a list of abbreviations for common words? I guess I could ask gpt-4 lmao
  - fnordpiglet 3y ago
    Sorry I’ve been doing this a bit adhoc - there are other techniques I do as well such as choosing shorter synonyms. A structured approach might be doing a dictionary translation by inverting a thesaurus and mapping synonyms to either themselves or a shorter word. There may be some “texting culture” thesaurus in the world that you could merge with a standard one. But it’s also remarkably resilient to dropping single characters or even a few so long as the spelling is very similar. Wish I had a more rigorous approach for you though.
  - phillipcarter 3y ago
    It took us quite a bit of work to reduce the static part of our prompt from ~2300 tokens to ~500 tokens. I wish there was a good playbook for this. I used gpt-4 a bit for some ideas, but almost all of it was just getting extremely creative with terse ways to represent info and cut cruft. Additionally, we've learned that LLMs lose a lot of context in the middle of a prompt, and so it might also be that a bunch of data not near the beginning or end of a prompt could just get removed entirely, or at least much more tersely summarized.

fnordpiglet 3y ago

I’ve found most advanced LLM are pretty resilient to abbreviations as well. “Tho” can replace “though”, and you can even be as racy as “b4” instead of before. Depending on the model it can start degenerating into abbreviating talk in response for obvious reasons, but for GPT4 it doesn’t seem to be impacted.

sergiotapia 3y ago
That's a great tip! Do you happen to have a list of abbreviations for common words? I guess I could ask gpt-4 lmao
- fnordpiglet 3y ago
  Sorry I’ve been doing this a bit adhoc - there are other techniques I do as well such as choosing shorter synonyms. A structured approach might be doing a dictionary translation by inverting a thesaurus and mapping synonyms to either themselves or a shorter word. There may be some “texting culture” thesaurus in the world that you could merge with a standard one. But it’s also remarkably resilient to dropping single characters or even a few so long as the spelling is very similar. Wish I had a more rigorous approach for you though.
- phillipcarter 3y ago
  It took us quite a bit of work to reduce the static part of our prompt from ~2300 tokens to ~500 tokens. I wish there was a good playbook for this. I used gpt-4 a bit for some ideas, but almost all of it was just getting extremely creative with terse ways to represent info and cut cruft. Additionally, we've learned that LLMs lose a lot of context in the middle of a prompt, and so it might also be that a bunch of data not near the beginning or end of a prompt could just get removed entirely, or at least much more tersely summarized.