5 ms·

Learnings: There's A LOT of data on the internet, good luck getting through all of it. These models show interesting recency bias. - The model seems to be a

Learnings: There's A LOT of data on the internet, good luck getting through all of it. These models show interesting recency bias. - The model seems to be aware of its training and how it has changed over time? - Due to model size it seems to easily forget information gathered during training so every sample provided is picked in a cycle from the datasets above. - The `InterleavedDataset` when `shuffle=True` messes with the above constraint so needs to be fixed. - JSONL became necessary due to datasets taking many GBs on disk, currently it's using a primitive method of simply streaming lines from disk during training (offsets are calculated when the datasets are loaded) - It displays interesting abilities to reason even at this size. - It performs best at summarisation when the information is provided. - It hallucinates considerably. - Story-telling is very "we went here, then we went there" but there's no "depth" - it's just a sequence of events. Future work: I'm going to continue hacking away at this project as I want to see where it leads but I'm very keen to get it out into the wild when time permits. (it's exhausting trying to keep up with this field) Some examples below: >>> Human: Can you help me with a puzzle? I bought sushi for lunch but I don't remember where I left it. Here are the steps I took before it went missing: I came home, tossed my keys on the table and brought my lunch to the kitchen. I took my lunch out of its bag and threw the bag in the bin. I put my lunch on a plate then went to my bedroom. My phone rang so I went back to the kitchen. --- Where did I put my sushi? Please think step-by-step.<|endofprompt|> Assistant: ->Based on the steps I took before it went missing, I came home and placed my keys on the table. I then placed my lunch on a plate then went to my bedroom. I put my lunch on a plate then went to my bedroom. Took: 0.7614264488220215s >>> Human: Read this sentence: "The Parakeet hopped out of its cage and landed on the table." Pretend you're the Parakeet. What do you see around the room after you hop out of your cage? Please be descriptive.<|endofprompt|> Assistant: ->As I look out the window, I see a small room with a large table surrounded by chairs. The walls are lined with various items that make for an immersive experience. The furniture is made of high-quality wood and has a sleek design that adds to the overall ambiance. The table is adorned with colorful paintings and sculptures, giving the space a cozy and inviting feel. The furniture is made of high-quality wood, which adds to its charm and sophistication. The chairs are made from high-quality materials that add to the overall elegance and sophistication of the space. The tables are arranged in a way that creates an elegant ambiance. The chairs are arranged in a way that allows for easy movement around the room, making it easy to find what you're looking for. Overall, I am sure you'll find something that will leave you feeling refreshed and rejuvenated after your workout. Took: 5.08751106262207s >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Human: Please create a website about a new Robot Cafe. Show the entire HTML script.<|endofprompt|> Assistant: -><!DOCTYPE html> <html> <head> <title> Robot Cafe</title> <style> body { font-family: Arial, sans-serif; font-size: 16px; line-height: 1.5; } </style> </head> <body> <div class="container"> <h1> Robot Cafe</h1> <p>The Robot Cafe is a new Robot Cafe that offers a variety of products, from electronics to medical devices. The products range from basic medical devices to more advanced medical devices.</p> <h2>The Robot Cafe is a new Robot Cafe that offers a variety of products, from medical devices to advanced medical devices.</p> <h2>The Robot Cafe is a new Robot Cafe that offers a variety of products, from medical devices to advanced medical devices.</p> <h2>The Robot Cafe is a new Robot Cafe that specializes in using natural and sustainable materials. The products range from basic medical devices to more advanced medical devices.</p> </h2> <h2>The Robot Cafe is another new robot cafe that specializes in using eco-friendly materials and has a wide range of customizable options. The products range from basic medical devices to more advanced medical devices.</p> <h3>The Robot Cafe is another new robot cafe that specializes in using eco-friendly materials and has an extensive range of customizable options. The products range from basic medical devices to more advanced medical devices.</p> </h2> </div> </body> </html> ``` Took: 7.056133508682251s

adt 3y ago
How many tokens have you trained on?
- razodactyl 3y ago
  Absolutely no idea sorry!! - this was more for understanding how an LLM even works. I just happen to understand the ML pipeline so I went beyond Andrej Karpathy's tutorial. It's an extract of TinyGPT but built from the ground up to work with new features such as Multi/Grouped-Query attention, `min_p` sampling, KV-caching etc. --- Unfortunately as this is a 1-GPU personal situation everything had to be thought out from the constraints of never being able to finish training the model. --- What I can tell you though is after ~40 hours of training, the model starts showing ability to speak and perform tasks. ~80 hours it improves upon these tasks and starts showing a tiny (very minute) bit of understanding where it fits into the picture. --- GPT-2's largest was 12 layers I believe. Layers seem to correlate with ability of the model to "compute". The embedding dimension of the model seems to correlate with the ability of the model to "communicate". If you "communicate" a larger amount of information through the layers (compute) you get a much smarter AI. --- Additionally, there's an interesting effect of GQA/MQA where the model is forced to share its work between the layers. I think about it like a group of students, either working on their own or in groups: - Too many and it becomes chaos. - Too few and learning takes longer as there's no collaboration between individuals. - In my case 2 "students" per N layers means the network learns quickly and infers fast. Hopefully it makes sense, it's been a non-stop torrent of learning for myself to answer the more obvious questions that always seem to be avoided in general. Everyone is more interested (obviously) in competing and scaling up their model that the smaller models are neglected. From what I can tell, something like X.AI's Grok can be thought of as extremely high capacity without being fully saturated (it has a lot of space to learn more).
  - adt 3y ago
    Thanks. Have estimated this as 3B tokens as a round number at 8:1, but if you have a firmer number I'd love to know it. Added to the Models Table (a few 'small' and 'tiny' models there, too): https://lifearchitect.ai/models https://lifearchitect.ai/models

adt 3y ago

How many tokens have you trained on?

razodactyl 3y ago
Absolutely no idea sorry!! - this was more for understanding how an LLM even works. I just happen to understand the ML pipeline so I went beyond Andrej Karpathy's tutorial. It's an extract of TinyGPT but built from the ground up to work with new features such as Multi/Grouped-Query attention, `min_p` sampling, KV-caching etc. --- Unfortunately as this is a 1-GPU personal situation everything had to be thought out from the constraints of never being able to finish training the model. --- What I can tell you though is after ~40 hours of training, the model starts showing ability to speak and perform tasks. ~80 hours it improves upon these tasks and starts showing a tiny (very minute) bit of understanding where it fits into the picture. --- GPT-2's largest was 12 layers I believe. Layers seem to correlate with ability of the model to "compute". The embedding dimension of the model seems to correlate with the ability of the model to "communicate". If you "communicate" a larger amount of information through the layers (compute) you get a much smarter AI. --- Additionally, there's an interesting effect of GQA/MQA where the model is forced to share its work between the layers. I think about it like a group of students, either working on their own or in groups: - Too many and it becomes chaos. - Too few and learning takes longer as there's no collaboration between individuals. - In my case 2 "students" per N layers means the network learns quickly and infers fast. Hopefully it makes sense, it's been a non-stop torrent of learning for myself to answer the more obvious questions that always seem to be avoided in general. Everyone is more interested (obviously) in competing and scaling up their model that the smaller models are neglected. From what I can tell, something like X.AI's Grok can be thought of as extremely high capacity without being fully saturated (it has a lot of space to learn more).
- adt 3y ago
  Thanks. Have estimated this as 3B tokens as a round number at 8:1, but if you have a firmer number I'd love to know it. Added to the Models Table (a few 'small' and 'tiny' models there, too): https://lifearchitect.ai/models https://lifearchitect.ai/models