7 ms·
LiteLlama-460M-1T has 460M parameters trained with 1T tokens
- rustybolt 3y agoSeems very glitchy? Example 6 gives > My name is Merve and my favorite color is blue. I am a very active person and I love to exercise. I am a member of the American Red Cross and I am a member of the American Red Cross. I am a member of the American Red Cross and I am a member of the American Red Cross. I am a member ...
- slavboj 3y agoThis kind of stuck-in-loop behavior was a characteristic of previous low parameter count LLMs. For large amounts of actual text generation they're not ideal.
- coder543 3y agoThis model does not appear to be fine-tuned for chat. I observed the same looping behavior with virtually any direct prompt. If I prime it with a pattern of Q and A with several examples of a good question and a good answer, then a final question with a blank answer, it provides a more cohesive and less repetitious answer. To me, this indicates it just needs fine-tuning with an instruction dataset to perform more "normally". Obviously, a model this tiny is going to have very little inherent knowledge, and it's probably not going to be great at anything other than speed.
- coder543 3y agoFor example, I asked Mixtral to generate 4 questions and short answers following a prompt format that I provided. Then I used that output as the prompt for LiteLlama along with a new question: Q: What is the capital city of France? A: The capital city of France is Paris, a global center for art, fashion, gastronomy, and culture. It is located in the north-central part of the country and has an estimated population of 12.4 million in its wider metropolitan area. Q: Who was the first person to walk on the moon? A: The first person to walk on the moon was Neil Armstrong, as part of the Apollo 11 mission on July 20, 1969. When he stepped onto the lunar surface, he famously said, "That's one small step for man, one giant leap for mankind." Q: What is the chemical symbol for gold? A: The chemical symbol for gold is Au, derived from its Latin name, aurum, which means 'shining dawn.' Gold is a transition metal that is bright, slightly reddish yellow, dense, malleable, and resistant to corrosion and tarnish. Q: What is the tallest mountain in the world? A: The tallest mountain in the world is Mount Everest, located on the border of Nepal and Tibet. It has a height of approximately 8,849 meters (29,032 feet) above sea level. The first recorded summit of Everest was achieved by Sir Edmund Hillary and Tenzing Norgay in 1953. Q: What is the LHC? What has it accomplished? A: And this is the output I received from LiteLlama: The Large Hadron Collider (LHC) is a particle accelerator at CERN in Geneva, Switzerland. It is a super-high-energy proton-proton collider that is designed to collide protons with protons at the LHC. Q: [Omitted since I don't care. You would normally stop the model when it responds with a stop word like "Q:", which it did.] Obviously that response is not going to blow anyone away, but it didn't get stuck in a loop. It knew that LHC stood for "Large Hadron Collider", and it also knew that CERN and Geneva were relevant.
- jackblemming 3y agoWhat is 10 + 10? Answer: 0 + 10 = 10 + 10 = 10 + 10 = 10 + 10 = 10 + 10 = This seems like a waste of compute and time.
- DarmokJalad1701 3y agoThe model may not be fine-tuned for instruct/chat.
- coder543 3y agoThe linked page claims that LiteLlama scored a zero on the GSM8K benchmark, so let's just say math probably isn't its forte.
- badgersnake 3y agoPretty sure your computer can calculate 10+10 without an LLM.
- minimaxir 3y agoIt's a test case.
- viraptor 3y agoIt's a bad test case. You should know why you're choosing a given test. I hope people will understand why "solve this equation" doesn't work with LLMs and why "transform this into python code" does much better.
- minimaxir 3y agoWhy not both? They're distinct enough, and there's no such thing as too many distinct test cases.
- deleted 3y ago[deleted]
- qwertox 3y ago"The mountain in front of me" was a mountain, and I was a mountain. I was a mountain. I was a mountain. I was a mountain. I was a mountain.
- andy99 3y agoThat kind of repetition is very common in smaller / less powerful models. One can penalize repetition to get something else, or otherwise apply some guided generation. In any event, I don't believe this is means as a general purpose chatbot or something, it's an experiment with a small model and possibly something that could be fine tuned for a narrow task. There's not much point comparing it to much bigger models.
- numeri 3y agoAlmost all LLM inference these days includes a repetition penalty, to help prevent models from falling into this pattern, the so-called "boredom trap" [1]. Once the model repeats itself once, that greatly increases the chances that the next best token is another repetition (for a raw model like this, at least, that hasn't been chat/instruct/RLHF fine-tuned). I ran the same prompt with a repetition penalty of 1.1 and got this: The mountain in front of me was a little higher than the other two, and I could see the top of the mountain. I'm going to climb it," I said. "It's a good place for a picnic." You're not going to climb it?" No!" I said. "But I'll climb it if you want! I can climb it with you—if we can get out there together—" [1] https://arxiv.org/abs/2007.14966 https://arxiv.org/abs/2007.14966
- Erem 3y agoA truly terrible completion but I appreciate it as a poem