4 ms·

Why would you want to/what do you expect to get out of it? A token per second... what is that token going to say? Accurate information?

deleted 3y ago
[deleted]
pkoird 3y ago
A simple "yes / no" would probably be valuable even for being a single token.
marshray 3y ago
That's as fast or faster than most people can type.
- MuffinFlavored 3y ago
  I wasn't referring to the speed, I meant the quality/validity/value of the tokens.
- canttestthis 3y ago
  The information density of the average LLM response is much much lower than my coworkers' responses to my questions.
  - phito 3y ago
    You're not prompting it right then.
    - collyw 3y ago
      We are starting to sound like the response to agile being overrated, "you aren't doing it right". These comments never come with suggestions of how to do it right or what is being done wrong. They will never consider the possibility that the criticism is justified.
      - phito 3y ago
        You can literally just ask the model to give succinct answers. If you have a chatGPT subscription, there is a pre prompt option where you can describe exactly how the model should answer you. It works pretty well in my experience. LLMs are definitely not perfect and have limitations, but this isn't one of them imo. It's also the case that most people describe their experience with LLM from their use by
gmt2027 3y ago
One of the most powerful ways to integrate LLMs with existing systems is constrained generation. Libraries such as outlines[1] and instructor[2] allow structural specification of the expected outputs as regex patterns, simple types, jsonschema or pydantic models. Llama.cpp supports bnf grammars. These outputs often consume significantly fewer tokens than chat or text completions. [1] https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines [2] https://github.com/jxnl/instructor https://github.com/jxnl/instructor
- Der_Einzige 3y ago
  I wrote a whole paper about this idea but for syntactic or phonetic constraints: https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai-1 https://paperswithcode.com/paper/most-language-models-can-be...
wkat4242 3y ago
Because most of the online models are neutered so much they are almost useless. I get the morals lecture several times a day now.

deleted 3y ago

[deleted]

pkoird 3y ago

A simple "yes / no" would probably be valuable even for being a single token.

marshray 3y ago

That's as fast or faster than most people can type.

MuffinFlavored 3y ago
I wasn't referring to the speed, I meant the quality/validity/value of the tokens.
canttestthis 3y ago
The information density of the average LLM response is much much lower than my coworkers' responses to my questions.
- phito 3y ago
  You're not prompting it right then.
  - collyw 3y ago
    We are starting to sound like the response to agile being overrated, "you aren't doing it right". These comments never come with suggestions of how to do it right or what is being done wrong. They will never consider the possibility that the criticism is justified.
    - phito 3y ago
      You can literally just ask the model to give succinct answers. If you have a chatGPT subscription, there is a pre prompt option where you can describe exactly how the model should answer you. It works pretty well in my experience. LLMs are definitely not perfect and have limitations, but this isn't one of them imo. It's also the case that most people describe their experience with LLM from their use by

gmt2027 3y ago

One of the most powerful ways to integrate LLMs with existing systems is constrained generation. Libraries such as outlines[1] and instructor[2] allow structural specification of the expected outputs as regex patterns, simple types, jsonschema or pydantic models. Llama.cpp supports bnf grammars. These outputs often consume significantly fewer tokens than chat or text completions. [1] https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines [2] https://github.com/jxnl/instructor https://github.com/jxnl/instructor

Der_Einzige 3y ago
I wrote a whole paper about this idea but for syntactic or phonetic constraints: https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai-1 https://paperswithcode.com/paper/most-language-models-can-be...

wkat4242 3y ago

Because most of the online models are neutered so much they are almost useless. I get the morals lecture several times a day now.