3 ms·

Parakeet: A Tiny LLM

31 points by razodactyl 3y ago

Hi all, https://colab.research.google.com/drive/1gI8CM9Bz9ov0-E6aL2jF808rE56UtZyF?usp=sharing This post is more for gauging interest, I plan to release the entire end to end code including: - Dataset curation (including citations). - Model checkpoints. - Inference code. - Synthetic data generation. - etc. Parakeet is the name of a small language model I've designed from the ground up for the purpose of research. The challenge was to see how far I could push the limits of LLM tech. given a massively constrained environment. It was trained on a 3080 Ti and has a considerable amount more training to do but here are the results so far. Specs: - 18 layers / 18 heads - 8K context. - 1152 embedding dimension. - cl100k tokenizer (TikToken) - ALiBi (max I can train is 1200 tokens so this was crucial) - KV caching for improved inference. - Grouped Query Attention (2 layers per group / speeds up inference) - `min_p`: Cut-off low quality tokens. - Softmax1: https://github.com/kyegomez/AttentionIsOFFByOne - Not sure if this really made much of a difference / it's hard to train comparable models as compute resources are limited. - Sub 400M parameters (378M from memory) Edit: - Things I forgot to mention: NO RLHF / DPO, it's entirely dataset driven. - The model seems mostly harmless due to being trained only with synthetic data. - A side-effect of only being trained on synthetic data is that the model learns quite fast. - There's probably less than 2 weeks of actual training time in the model so far. - You don't need to start from scratch when altering model parameters. Weights can be copied/merged in and out of smaller/larger models. Why? - Curiosity got to me - I wanted to know what would happen if a model with a considerably small amount of parameters was bombarded with data. - There were many results showing these language models with room for more training but instead many are scaled up. - I wanted to see what happens if you just keep training them. References: - "./datasets/wikipedia-20220301.en.jsonl"), - "./datasets/euclaise_littletown.jsonl"), # https://huggingface.co/datasets/euclaise/LittleTown?row=99 - "./datasets/squad-v2.0-processed.jsonl"), - "./datasets/huggingface_ultrachat200k.jsonl"), # https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k - "./datasets/wizardlm_evol_instruct_v2_196k.jsonl"), # https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_V2_196k - "./datasets/crumb_clean_instruct_440k.jsonl"), # https://huggingface.co/datasets/crumb/Clean-Instruct-440k - Generate a story starting with the sentence "It was already late when they stepped out of the house". - "./datasets/openorca_4m.jsonl"), # https://huggingface.co/datasets/Open-Orca/OpenOrca - "./datasets/databricks_dolly15k.jsonl"), # https://huggingface.co/datasets/databricks/databricks-dolly-15k - Common-sense reasoning. - "./datasets/teven_code_contests4m.jsonl"), # https://huggingface.co/datasets/teven/code_contests - ['PYTHON', 'PYTHON3', 'JAVA', 'CPP'] - "./datasets/squad-v2.0-summaries.jsonl"), - "./datasets/google-boolq.jsonl"), - "./datasets/stingning_ultrachat.jsonl"), # https://huggingface.co/datasets/stingning/ultrachat - "./datasets/wikimovies-train.jsonl"), - "./datasets/kunishou-databricks-dolly-15k-ja.jsonl"), - "./datasets/wizardlm_evol_instruct_70k.jsonl"), # https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_70k - "./datasets/map_codefeedback.jsonl"), Sorry for bad formatting! ...continues in reply due to 4000 char. limit.

razodactyl 3y ago
Learnings: There's A LOT of data on the internet, good luck getting through all of it. These models show interesting recency bias. - The model seems to be aware of its training and how it has changed over time? - Due to model size it seems to easily forget information gathered during training so every sample provided is picked in a cycle from the datasets above. - The `InterleavedDataset` when `shuffle=True` messes with the above constraint so needs to be fixed. - JSONL became necessary due to datasets taking many GBs on disk, currently it's using a primitive method of simply streaming lines from disk during training (offsets are calculated when the datasets are loaded) - It displays interesting abilities to reason even at this size. - It performs best at summarisation when the information is provided. - It hallucinates considerably. - Story-telling is very "we went here, then we went there" but there's no "depth" - it's just a sequence of events. Future work: I'm going to continue hacking away at this project as I want to see where it leads but I'm very keen to get it out into the wild when time permits. (it's exhausting trying to keep up with this field) Some examples below: >>> Human: Can you help me with a puzzle? I bought sushi for lunch but I don't remember where I left it. Here are the steps I took before it went missing: I came home, tossed my keys on the table and brought my lunch to the kitchen. I took my lunch out of its bag and threw the bag in the bin. I put my lunch on a plate then went to my bedroom. My phone rang so I went back to the kitchen. --- Where did I put my sushi? Please think step-by-step.<|endofprompt|> Assistant: ->Based on the steps I took before it went missing, I came home and placed my keys on the table. I then placed my lunch on a plate then went to my bedroom. I put my lunch on a plate then went to my bedroom. Took: 0.7614264488220215s >>> Human: Read this sentence: "The Parakeet hopped out of its cage and landed on the table." Pretend you're the Parakeet. What do you see around the room after you hop out of your cage? Please be descriptive.<|endofprompt|> Assistant: ->As I look out the window, I see a small room with a large table surrounded by chairs. The walls are lined with various items that make for an immersive experience. The furniture is made of high-quality wood and has a sleek design that adds to the overall ambiance. The table is adorned with colorful paintings and sculptures, giving the space a cozy and inviting feel. The furniture is made of high-quality wood, which adds to its charm and sophistication. The chairs are made from high-quality materials that add to the overall elegance and sophistication of the space. The tables are arranged in a way that creates an elegant ambiance. The chairs are arranged in a way that allows for easy movement around the room, making it easy to find what you're looking for. Overall, I am sure you'll find something that will leave you feeling refreshed and rejuvenated after your workout. Took: 5.08751106262207s >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Human: Please create a website about a new Robot Cafe. Show the entire HTML script.<|endofprompt|> Assistant: -><!DOCTYPE html> <html> <head> <title> Robot Cafe</title> <style> body { font-family: Arial, sans-serif; font-size: 16px; line-height: 1.5; } </style> </head> <body> <div class="container"> <h1> Robot Cafe</h1> <p>The Robot Cafe is a new Robot Cafe that offers a variety of products, from electronics to medical devices. The products range from basic medical devices to more advanced medical devices.</p> <h2>The Robot Cafe is a new Robot Cafe that offers a variety of products, from medical devices to advanced medical devices.</p> <h2>The Robot Cafe is a new Robot Cafe that offers a variety of products, from medical devices to advanced medical devices.</p> <h2>The Robot Cafe is a new Robot Cafe that specializes in using natural and sustainable materials. The products range from basic medical devices to more advanced medical devices.</p> </h2> <h2>The Robot Cafe is another new robot cafe that specializes in using eco-friendly materials and has a wide range of customizable options. The products range from basic medical devices to more advanced medical devices.</p> <h3>The Robot Cafe is another new robot cafe that specializes in using eco-friendly materials and has an extensive range of customizable options. The products range from basic medical devices to more advanced medical devices.</p> </h2> </div> </body> </html> ``` Took: 7.056133508682251s

31 points by razodactyl 3y ago

razodactyl 3y ago

Learnings: There's A LOT of data on the internet, good luck getting through all of it. These models show interesting recency bias. - The model seems to be aware of its training and how it has changed over time? - Due to model size it seems to easily forget information gathered during training so every sample provided is picked in a cycle from the datasets above. - The `InterleavedDataset` when `shuffle=True` messes with the above constraint so needs to be fixed. - JSONL became necessary due to datasets taking many GBs on disk, currently it's using a primitive method of simply streaming lines from disk during training (offsets are calculated when the datasets are loaded) - It displays interesting abilities to reason even at this size. - It performs best at summarisation when the information is provided. - It hallucinates considerably. - Story-telling is very "we went here, then we went there" but there's no "depth" - it's just a sequence of events. Future work: I'm going to continue hacking away at this project as I want to see where it leads but I'm very keen to get it out into the wild when time permits. (it's exhausting trying to keep up with this field) Some examples below: >>> Human: Can you help me with a puzzle? I bought sushi for lunch but I don't remember where I left it. Here are the steps I took before it went missing: I came home, tossed my keys on the table and brought my lunch to the kitchen. I took my lunch out of its bag and threw the bag in the bin. I put my lunch on a plate then went to my bedroom. My phone rang so I went back to the kitchen. --- Where did I put my sushi? Please think step-by-step.<|endofprompt|> Assistant: ->Based on the steps I took before it went missing, I came home and placed my keys on the table. I then placed my lunch on a plate then went to my bedroom. I put my lunch on a plate then went to my bedroom. Took: 0.7614264488220215s >>> Human: Read this sentence: "The Parakeet hopped out of its cage and landed on the table." Pretend you're the Parakeet. What do you see around the room after you hop out of your cage? Please be descriptive.<|endofprompt|> Assistant: ->As I look out the window, I see a small room with a large table surrounded by chairs. The walls are lined with various items that make for an immersive experience. The furniture is made of high-quality wood and has a sleek design that adds to the overall ambiance. The table is adorned with colorful paintings and sculptures, giving the space a cozy and inviting feel. The furniture is made of high-quality wood, which adds to its charm and sophistication. The chairs are made from high-quality materials that add to the overall elegance and sophistication of the space. The tables are arranged in a way that creates an elegant ambiance. The chairs are arranged in a way that allows for easy movement around the room, making it easy to find what you're looking for. Overall, I am sure you'll find something that will leave you feeling refreshed and rejuvenated after your workout. Took: 5.08751106262207s >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Human: Please create a website about a new Robot Cafe. Show the entire HTML script.<|endofprompt|> Assistant: -><!DOCTYPE html> <html> <head> <title> Robot Cafe</title> <style> body { font-family: Arial, sans-serif; font-size: 16px; line-height: 1.5; } </style> </head> <body> <div class="container"> <h1> Robot Cafe</h1> <p>The Robot Cafe is a new Robot Cafe that offers a variety of products, from electronics to medical devices. The products range from basic medical devices to more advanced medical devices.</p> <h2>The Robot Cafe is a new Robot Cafe that offers a variety of products, from medical devices to advanced medical devices.</p> <h2>The Robot Cafe is a new Robot Cafe that offers a variety of products, from medical devices to advanced medical devices.</p> <h2>The Robot Cafe is a new Robot Cafe that specializes in using natural and sustainable materials. The products range from basic medical devices to more advanced medical devices.</p> </h2> <h2>The Robot Cafe is another new robot cafe that specializes in using eco-friendly materials and has a wide range of customizable options. The products range from basic medical devices to more advanced medical devices.</p> <h3>The Robot Cafe is another new robot cafe that specializes in using eco-friendly materials and has an extensive range of customizable options. The products range from basic medical devices to more advanced medical devices.</p> </h2> </div> </body> </html> ``` Took: 7.056133508682251s