4 ms·

Le sigh. Do I need to fix this sampler too? The default output should be excellent, not a drunk librarian. Rip open the code and make sure it has top_k 40, tem

Le sigh. Do I need to fix this sampler too? The default output should be excellent, not a drunk librarian. Rip open the code and make sure it has top_k 40, temperature 0.7, no top_p, and a repetition penalty of 1/0.85. Proof: https://twitter.com/theshawwn/status/1632569215348531201?s=61&t=jQbmCk1JqL7depzFWJNuPA https://twitter.com/theshawwn/status/1632569215348531201?s=6... Implementation: https://github.com/shawwn/llama https://github.com/shawwn/llama It feels basically identical to davinci. I think no one realizes it yet because Facebook released a horrible sampler that everyone copied.

Roritharr 4y ago
I can set top_k, temperature and top_p via shell already, but I don't think repetition penalty is dynamic yet. Looking into it. Edit: This fork ( https://github.com/beiller/llama.cpp/tree/feature/repeat_penalty https://github.com/beiller/llama.cpp/tree/feature/repeat_pen... ) adds repetition penalty via two values, repeat_last_n = 64, repeat_penalty = 1.300000. Playing with those now. I wish I could somehow force it to output longer texts. It just stops at arbitrary points, often in the middle of sentences.
- eigenvalue 4y ago
  Thanks for flagging this. Using that repo, I'm getting pretty good results with the following: ./main -m ./models/13B/ggml-model-q4_0.bin -t 7 -b 10 --top_p 0.0 -n 5000 --temp 0.7 --repeat_last_n 64 --repeat_penalty 1.3 --prompt "You are a question answering bot that is able to answer questions about the world. You are extremely smart, knowledgeable, capable, and helpful. You always give complete, accurate, and very detailed responses to questions, and never stop a response in mid-sentence or mid-thought. You answer questions in the following format: Question: What’s the history of bullfighting in Spain? Answer: Bullfighting, also known as "tauromachia," has a long and storied history in Spain, with roots that can be traced back to ancient civilizations. The sport is believed to have originated in 7th-century BCE Iberian Peninsula as a form of animal worship, and it evolved over time to become a sport and form of entertainment. Bullfighting as it is known today became popular in Spain in the 17th and 18th centuries. During this time, the sport was heavily influenced by the traditions of medieval jousts and was performed by nobles and other members of the upper classes. Over time, bullfighting became more democratized and was performed by people from all walks of life. Bullfighting reached the height of its popularity in the 19th and early 20th centuries and was considered a national symbol of Spain. However, in recent decades, bullfighting has faced increasing opposition from animal rights activists, and its popularity has declined. Some regions of Spain have banned bullfighting, while others continue to hold bullfights as a cherished tradition. Despite its declining popularity, bullfighting remains an important part of Spanish culture and history, and it continues to be performed in many parts of the country to this day. Now complete the following questions: Question: What happened to the field of cybernetics in the 1970s? Answer: "
  - Roritharr 4y ago
    Interesting, that one segfaults for me. Gonna try with fresh eyes tomorrow, thanks!
  - Retr0id 4y ago
    > and never stop a response in mid-sentence or mid-thought I'm curious if this part actually does anything to improve the output, my intuition says it doesn't help much. I don't have it running locally to test yet, though.
    - aardvarkr 4y ago
      These prompts become “key pillars” of the LLM’s “identity”. If you are curious you should at look about how much personality MS Sydney acquired through it’s own startup prompt.
  - dr_dshiv 4y ago
    Lol, was that blank an accurate answer about cybernetics?
    - delusional 4y ago
      That's the prompt, it's where the inserts the stuff it generates. The AI generated content has not been posted.
      - eigenvalue 4y ago
        Yes I believe the parent was making a joke :)
  - alfonsodev 4y ago
    I think accidentally the bot is kind of correct, you are asking to "complete the following questions" and that question seems complete, maybe the bot could have asked for clarification, I've never had that situation yet, would be something like .. "That question seems very well formulated, do you want me to answer it ? or to complete the question? we could complete the question by being more specific about location, for example what happened to the filed of cybernetics in EEUU in the 1970s ?"
- nsxwolf 4y ago
  What kind of effort does it take to have any idea what you guys are talking about? Is there a For Dummies book?
  - enjoylife 4y ago
    It’s essentially a question of did you take a look the code? There is comments in the codebase directly where the config values in question are being used. Eg. > # repetition penalty from CTRL paper (https://arxiv.org/abs/1909.05858 https://arxiv.org/abs/1909.05858) Now I’m assuming a base rate of knowledge for this to help, but in general I find diving into the code paths of open source models, usually is a good use of time.
  - pksebben 4y ago
    there's no for dummies, this stuff is too new. there's a bunch of white papers to read; _attention is all you need_ is a good place to start, kinda. if you troll HN for AI related stuff and just read read read, progressively googling the terms you don't understand, eventually the haze will lift and things will start to make some sense. source: in the middle of doing that right now
    - ben_w 4y ago
      > if you troll HN for AI related stuff and just read read read Do you mean "troll" or "trawl"? (I'm thinking of an old saying that the best way to get an answer isn't to ask a question, but to say the thing is impossible. Can't remember it well enough to google for the exact form, however).
      - Qworg 4y ago
        IIRC, both forms work. Trolling is done by hand with a fishing pole, sometimes on a dock. Trawling is from the back of a boat, generally with a net.
        ben_w 4y ago
        Interesting. Until now I only knew the verb "troll" as in "being mean on the internet".
jerpint 4y ago
Having it be on par with DaVinci is a big claim, will definitely be putting that to the test
eigenvalue 4y ago
What does no top_p mean exactly? Set top_p to 0.0? or to 1.0?
- toxik 4y ago
  1.0
Der_Einzige 4y ago
BTW - Now that discussions about sampling techniques within NLP are going mainstream, I have to shamelessly self-plug my own research work in this direction, specifically in the context of Constrained Text Generation. Simply filtering tokens according to constraints before generating text is wonderful for fixing stupid failure cases in LLMs like ChatGPT - such as asking it to not write with a particular letter, asking it to respect syllable counts, or asking it to rhyme properly (look carefully, ChatGPT struggles with exact rhymes in poetry!) https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai https://paperswithcode.com/paper/most-language-models-can-be... My work got cited by Gwern in this article! https://gwern.net/gpt-3 https://gwern.net/gpt-3
- JonathanFly 4y ago
  I love this. I first tried it with Grover in 2019 and ran into similar token issues: https://twitter.com/jonathanfly/status/1143773270451396608?lang=en https://twitter.com/jonathanfly/status/1143773270451396608?l... Looks like you don't have a 'most common/simple words' vocab restriction? That's a pretty fun one too. https://twitter.com/jonathanfly/status/1144345934857474048 https://twitter.com/jonathanfly/status/1144345934857474048
  - Der_Einzige 4y ago
    I feel bad for not being active on Twitter, specifically because I missed out on stuff like this (which predates my paper). I should have cited this thread since it's evidence that someone else had utilized this technique in my paper! I think this vocabulary filtering technique will pick up steam again now that good quality open source LLMs are proliferating. I've always wanted to try these techniques with ChatGPT, and the inability to do so has caused me to somewhat publicly complain about their lack of access to ChatGPT's vocabulary probability distribution (they don't give more than the top 10 most likely continuations).
    - kastnerkyle 4y ago
      You might be interested in this thread from 2019 here on HN as well, I linked to a line of work on constrained sequence generation from Sony CSL that is pretty nifty https://news.ycombinator.com/item?id=21334639 https://news.ycombinator.com/item?id=21334639 As an aside - do you have any academic references for masking / constraining based next step transitions? It's a trick I've been messing with for years, but I probably picked this up from some older poetry generation work. Haven't been able to find any direct reference for simple 1 step masking, let alone more involved things like constraining based on the unrolled, modified markov sequence or things bigger than 1 step.
taf2 4y ago
Thanks even the 13B model that runs way faster with top_p disabled works better!
ComplexSystems 4y ago
What does "1/0.85" mean here? The reciprocal of 0.85 is ~1.176, so is that what you are recommending?

Roritharr 4y ago

I can set top_k, temperature and top_p via shell already, but I don't think repetition penalty is dynamic yet. Looking into it. Edit: This fork ( https://github.com/beiller/llama.cpp/tree/feature/repeat_penalty https://github.com/beiller/llama.cpp/tree/feature/repeat_pen... ) adds repetition penalty via two values, repeat_last_n = 64, repeat_penalty = 1.300000. Playing with those now. I wish I could somehow force it to output longer texts. It just stops at arbitrary points, often in the middle of sentences.

eigenvalue 4y ago
Thanks for flagging this. Using that repo, I'm getting pretty good results with the following: ./main -m ./models/13B/ggml-model-q4_0.bin -t 7 -b 10 --top_p 0.0 -n 5000 --temp 0.7 --repeat_last_n 64 --repeat_penalty 1.3 --prompt "You are a question answering bot that is able to answer questions about the world. You are extremely smart, knowledgeable, capable, and helpful. You always give complete, accurate, and very detailed responses to questions, and never stop a response in mid-sentence or mid-thought. You answer questions in the following format: Question: What’s the history of bullfighting in Spain? Answer: Bullfighting, also known as "tauromachia," has a long and storied history in Spain, with roots that can be traced back to ancient civilizations. The sport is believed to have originated in 7th-century BCE Iberian Peninsula as a form of animal worship, and it evolved over time to become a sport and form of entertainment. Bullfighting as it is known today became popular in Spain in the 17th and 18th centuries. During this time, the sport was heavily influenced by the traditions of medieval jousts and was performed by nobles and other members of the upper classes. Over time, bullfighting became more democratized and was performed by people from all walks of life. Bullfighting reached the height of its popularity in the 19th and early 20th centuries and was considered a national symbol of Spain. However, in recent decades, bullfighting has faced increasing opposition from animal rights activists, and its popularity has declined. Some regions of Spain have banned bullfighting, while others continue to hold bullfights as a cherished tradition. Despite its declining popularity, bullfighting remains an important part of Spanish culture and history, and it continues to be performed in many parts of the country to this day. Now complete the following questions: Question: What happened to the field of cybernetics in the 1970s? Answer: "
- Roritharr 4y ago
  Interesting, that one segfaults for me. Gonna try with fresh eyes tomorrow, thanks!
- Retr0id 4y ago
  > and never stop a response in mid-sentence or mid-thought I'm curious if this part actually does anything to improve the output, my intuition says it doesn't help much. I don't have it running locally to test yet, though.
  - aardvarkr 4y ago
    These prompts become “key pillars” of the LLM’s “identity”. If you are curious you should at look about how much personality MS Sydney acquired through it’s own startup prompt.
- dr_dshiv 4y ago
  Lol, was that blank an accurate answer about cybernetics?
  - delusional 4y ago
    That's the prompt, it's where the inserts the stuff it generates. The AI generated content has not been posted.
    - eigenvalue 4y ago
      Yes I believe the parent was making a joke :)
- alfonsodev 4y ago
  I think accidentally the bot is kind of correct, you are asking to "complete the following questions" and that question seems complete, maybe the bot could have asked for clarification, I've never had that situation yet, would be something like .. "That question seems very well formulated, do you want me to answer it ? or to complete the question? we could complete the question by being more specific about location, for example what happened to the filed of cybernetics in EEUU in the 1970s ?"
nsxwolf 4y ago
What kind of effort does it take to have any idea what you guys are talking about? Is there a For Dummies book?
- enjoylife 4y ago
  It’s essentially a question of did you take a look the code? There is comments in the codebase directly where the config values in question are being used. Eg. > # repetition penalty from CTRL paper (https://arxiv.org/abs/1909.05858 https://arxiv.org/abs/1909.05858) Now I’m assuming a base rate of knowledge for this to help, but in general I find diving into the code paths of open source models, usually is a good use of time.
- pksebben 4y ago
  there's no for dummies, this stuff is too new. there's a bunch of white papers to read; _attention is all you need_ is a good place to start, kinda. if you troll HN for AI related stuff and just read read read, progressively googling the terms you don't understand, eventually the haze will lift and things will start to make some sense. source: in the middle of doing that right now
  - ben_w 4y ago
    > if you troll HN for AI related stuff and just read read read Do you mean "troll" or "trawl"? (I'm thinking of an old saying that the best way to get an answer isn't to ask a question, but to say the thing is impossible. Can't remember it well enough to google for the exact form, however).
    - Qworg 4y ago
      IIRC, both forms work. Trolling is done by hand with a fishing pole, sometimes on a dock. Trawling is from the back of a boat, generally with a net.
      - ben_w 4y ago
        Interesting. Until now I only knew the verb "troll" as in "being mean on the internet".

jerpint 4y ago

Having it be on par with DaVinci is a big claim, will definitely be putting that to the test

eigenvalue 4y ago

What does no top_p mean exactly? Set top_p to 0.0? or to 1.0?

toxik 4y ago
1.0

Der_Einzige 4y ago

BTW - Now that discussions about sampling techniques within NLP are going mainstream, I have to shamelessly self-plug my own research work in this direction, specifically in the context of Constrained Text Generation. Simply filtering tokens according to constraints before generating text is wonderful for fixing stupid failure cases in LLMs like ChatGPT - such as asking it to not write with a particular letter, asking it to respect syllable counts, or asking it to rhyme properly (look carefully, ChatGPT struggles with exact rhymes in poetry!) https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai https://paperswithcode.com/paper/most-language-models-can-be... My work got cited by Gwern in this article! https://gwern.net/gpt-3 https://gwern.net/gpt-3

JonathanFly 4y ago
I love this. I first tried it with Grover in 2019 and ran into similar token issues: https://twitter.com/jonathanfly/status/1143773270451396608?lang=en https://twitter.com/jonathanfly/status/1143773270451396608?l... Looks like you don't have a 'most common/simple words' vocab restriction? That's a pretty fun one too. https://twitter.com/jonathanfly/status/1144345934857474048 https://twitter.com/jonathanfly/status/1144345934857474048
- Der_Einzige 4y ago
  I feel bad for not being active on Twitter, specifically because I missed out on stuff like this (which predates my paper). I should have cited this thread since it's evidence that someone else had utilized this technique in my paper! I think this vocabulary filtering technique will pick up steam again now that good quality open source LLMs are proliferating. I've always wanted to try these techniques with ChatGPT, and the inability to do so has caused me to somewhat publicly complain about their lack of access to ChatGPT's vocabulary probability distribution (they don't give more than the top 10 most likely continuations).
  - kastnerkyle 4y ago
    You might be interested in this thread from 2019 here on HN as well, I linked to a line of work on constrained sequence generation from Sony CSL that is pretty nifty https://news.ycombinator.com/item?id=21334639 https://news.ycombinator.com/item?id=21334639 As an aside - do you have any academic references for masking / constraining based next step transitions? It's a trick I've been messing with for years, but I probably picked this up from some older poetry generation work. Haven't been able to find any direct reference for simple 1 step masking, let alone more involved things like constraining based on the unrolled, modified markov sequence or things bigger than 1 step.

taf2 4y ago

Thanks even the 13B model that runs way faster with top_p disabled works better!

ComplexSystems 4y ago

What does "1/0.85" mean here? The reciprocal of 0.85 is ~1.176, so is that what you are recommending?