9 ms·

I have the 65B Model running on my MacBook here. Honestly, I wish I knew enough about how to get it to output something useful. It's like poking a drunk librari

I have the 65B Model running on my MacBook here. Honestly, I wish I knew enough about how to get it to output something useful. It's like poking a drunk librarian.

_just7_ 4y ago
Give it few shot examples and the quality should improve a lot. If you want a bot able to do technical Q&A give it two questions as well as the answers in its prompt. The model will then infer it's role as an AI that answer questions truthfully, and will try it's best to continue along those lines
- Roritharr 4y ago
  I gave it the prompt thats the example from here: https://huggingface.co/blog/few-shot-learning-gpt-neo-and-inference-api https://huggingface.co/blog/few-shot-learning-gpt-neo-and-in... Tweet: "I hate it when my phone battery dies." Sentiment: Negative ### Tweet: "My day has been " Sentiment: Positive ### Tweet: "This is the link to the article" Sentiment: Neutral ### Tweet: "This new music video was incredibile" Sentiment: and it's output was: Tweet: "I hate it when my phone battery dies." Sentiment: Negative ### Tweet: "My day has been 90% balls." Seentiment: Negative ### Twitter: "i don't have a favourite type of music i just hate country" Seentiment: Negative ### Twitter: "c'mon first sunshine of the year...why the rain already?!" Seentiment: Negative ### Twitter: "Dear twitter, are you getting tired of me yet? Seentiment: Negative ### Twitter: "I don't care about the race, I just care about the president being a psych Doesn't feel like that did the trick, but at least you got me to setup the shell command in a way where I pipe long-form prompts in via a separate file.
  - visarga 4y ago
    I think the model only has pre-training on large text but not multi-task fine-tuning and RLHF. So it can be improved a lot.
    - speedgoose 4y ago
      Hopefully a RLHF fine tuned model will be released soon because that will make the technology much more useful and less disappointing to people trying to run an alternative of ChatGPT on their local machine.
  - mkl 4y ago
    It invented "Seentiment"?
sillysaurusx 4y ago
Le sigh. Do I need to fix this sampler too? The default output should be excellent, not a drunk librarian. Rip open the code and make sure it has top_k 40, temperature 0.7, no top_p, and a repetition penalty of 1/0.85. Proof: https://twitter.com/theshawwn/status/1632569215348531201?s=61&t=jQbmCk1JqL7depzFWJNuPA https://twitter.com/theshawwn/status/1632569215348531201?s=6... Implementation: https://github.com/shawwn/llama https://github.com/shawwn/llama It feels basically identical to davinci. I think no one realizes it yet because Facebook released a horrible sampler that everyone copied.
- Roritharr 4y ago
  I can set top_k, temperature and top_p via shell already, but I don't think repetition penalty is dynamic yet. Looking into it. Edit: This fork ( https://github.com/beiller/llama.cpp/tree/feature/repeat_penalty https://github.com/beiller/llama.cpp/tree/feature/repeat_pen... ) adds repetition penalty via two values, repeat_last_n = 64, repeat_penalty = 1.300000. Playing with those now. I wish I could somehow force it to output longer texts. It just stops at arbitrary points, often in the middle of sentences.
  - eigenvalue 4y ago
    Thanks for flagging this. Using that repo, I'm getting pretty good results with the following: ./main -m ./models/13B/ggml-model-q4_0.bin -t 7 -b 10 --top_p 0.0 -n 5000 --temp 0.7 --repeat_last_n 64 --repeat_penalty 1.3 --prompt "You are a question answering bot that is able to answer questions about the world. You are extremely smart, knowledgeable, capable, and helpful. You always give complete, accurate, and very detailed responses to questions, and never stop a response in mid-sentence or mid-thought. You answer questions in the following format: Question: What’s the history of bullfighting in Spain? Answer: Bullfighting, also known as "tauromachia," has a long and storied history in Spain, with roots that can be traced back to ancient civilizations. The sport is believed to have originated in 7th-century BCE Iberian Peninsula as a form of animal worship, and it evolved over time to become a sport and form of entertainment. Bullfighting as it is known today became popular in Spain in the 17th and 18th centuries. During this time, the sport was heavily influenced by the traditions of medieval jousts and was performed by nobles and other members of the upper classes. Over time, bullfighting became more democratized and was performed by people from all walks of life. Bullfighting reached the height of its popularity in the 19th and early 20th centuries and was considered a national symbol of Spain. However, in recent decades, bullfighting has faced increasing opposition from animal rights activists, and its popularity has declined. Some regions of Spain have banned bullfighting, while others continue to hold bullfights as a cherished tradition. Despite its declining popularity, bullfighting remains an important part of Spanish culture and history, and it continues to be performed in many parts of the country to this day. Now complete the following questions: Question: What happened to the field of cybernetics in the 1970s? Answer: "
lifeisstillgood 4y ago
The next version release now needs to be called "Drunk Librarian". I want it enshrined in law :-)
vintermann 4y ago
For people complaining that ChatGPT makes up stuff, I use to say it's like asking a librarian at gunpoint. Of course it's very polite, and it knows a lot, but it also doesn't have time to look anything up or do any real research.

I have the 65B Model running on my MacBook here. Honestly, I wish I knew enough about how to get it to output something useful. It's like poking a drunk librarian.

_just7_ 4y ago

Give it few shot examples and the quality should improve a lot. If you want a bot able to do technical Q&A give it two questions as well as the answers in its prompt. The model will then infer it's role as an AI that answer questions truthfully, and will try it's best to continue along those lines

Roritharr 4y ago
I gave it the prompt thats the example from here: https://huggingface.co/blog/few-shot-learning-gpt-neo-and-inference-api https://huggingface.co/blog/few-shot-learning-gpt-neo-and-in... Tweet: "I hate it when my phone battery dies." Sentiment: Negative ### Tweet: "My day has been " Sentiment: Positive ### Tweet: "This is the link to the article" Sentiment: Neutral ### Tweet: "This new music video was incredibile" Sentiment: and it's output was: Tweet: "I hate it when my phone battery dies." Sentiment: Negative ### Tweet: "My day has been 90% balls." Seentiment: Negative ### Twitter: "i don't have a favourite type of music i just hate country" Seentiment: Negative ### Twitter: "c'mon first sunshine of the year...why the rain already?!" Seentiment: Negative ### Twitter: "Dear twitter, are you getting tired of me yet? Seentiment: Negative ### Twitter: "I don't care about the race, I just care about the president being a psych Doesn't feel like that did the trick, but at least you got me to setup the shell command in a way where I pipe long-form prompts in via a separate file.
- visarga 4y ago
  I think the model only has pre-training on large text but not multi-task fine-tuning and RLHF. So it can be improved a lot.
  - speedgoose 4y ago
    Hopefully a RLHF fine tuned model will be released soon because that will make the technology much more useful and less disappointing to people trying to run an alternative of ChatGPT on their local machine.
- mkl 4y ago
  It invented "Seentiment"?

sillysaurusx 4y ago

Le sigh. Do I need to fix this sampler too? The default output should be excellent, not a drunk librarian. Rip open the code and make sure it has top_k 40, temperature 0.7, no top_p, and a repetition penalty of 1/0.85. Proof: https://twitter.com/theshawwn/status/1632569215348531201?s=61&t=jQbmCk1JqL7depzFWJNuPA https://twitter.com/theshawwn/status/1632569215348531201?s=6... Implementation: https://github.com/shawwn/llama https://github.com/shawwn/llama It feels basically identical to davinci. I think no one realizes it yet because Facebook released a horrible sampler that everyone copied.

Roritharr 4y ago
I can set top_k, temperature and top_p via shell already, but I don't think repetition penalty is dynamic yet. Looking into it. Edit: This fork ( https://github.com/beiller/llama.cpp/tree/feature/repeat_penalty https://github.com/beiller/llama.cpp/tree/feature/repeat_pen... ) adds repetition penalty via two values, repeat_last_n = 64, repeat_penalty = 1.300000. Playing with those now. I wish I could somehow force it to output longer texts. It just stops at arbitrary points, often in the middle of sentences.
- eigenvalue 4y ago
  Thanks for flagging this. Using that repo, I'm getting pretty good results with the following: ./main -m ./models/13B/ggml-model-q4_0.bin -t 7 -b 10 --top_p 0.0 -n 5000 --temp 0.7 --repeat_last_n 64 --repeat_penalty 1.3 --prompt "You are a question answering bot that is able to answer questions about the world. You are extremely smart, knowledgeable, capable, and helpful. You always give complete, accurate, and very detailed responses to questions, and never stop a response in mid-sentence or mid-thought. You answer questions in the following format: Question: What’s the history of bullfighting in Spain? Answer: Bullfighting, also known as "tauromachia," has a long and storied history in Spain, with roots that can be traced back to ancient civilizations. The sport is believed to have originated in 7th-century BCE Iberian Peninsula as a form of animal worship, and it evolved over time to become a sport and form of entertainment. Bullfighting as it is known today became popular in Spain in the 17th and 18th centuries. During this time, the sport was heavily influenced by the traditions of medieval jousts and was performed by nobles and other members of the upper classes. Over time, bullfighting became more democratized and was performed by people from all walks of life. Bullfighting reached the height of its popularity in the 19th and early 20th centuries and was considered a national symbol of Spain. However, in recent decades, bullfighting has faced increasing opposition from animal rights activists, and its popularity has declined. Some regions of Spain have banned bullfighting, while others continue to hold bullfights as a cherished tradition. Despite its declining popularity, bullfighting remains an important part of Spanish culture and history, and it continues to be performed in many parts of the country to this day. Now complete the following questions: Question: What happened to the field of cybernetics in the 1970s? Answer: "

lifeisstillgood 4y ago

The next version release now needs to be called "Drunk Librarian". I want it enshrined in law :-)

vintermann 4y ago

For people complaining that ChatGPT makes up stuff, I use to say it's like asking a librarian at gunpoint. Of course it's very polite, and it knows a lot, but it also doesn't have time to look anything up or do any real research.