3 ms·
This is awesome! Since some years ago we don't see NLP things that we can run in normal computers we have at home (no, a 24GB GPU still isn't normal). I have so
by estebarb 3y ago
This is awesome! Since some years ago we don't see NLP things that we can run in normal computers we have at home (no, a 24GB GPU still isn't normal). I have some questions:
How much time do you take for training?
Which are the specs of your system?
In total, how much is the training dataset?
- razodactyl 3y ago3080Ti (12GB VRAM), DDR4 64GB AMD Ryzen 7 8TB drive-space (I built it with the idea that I would be hoarding a lot of data for potential AI research) --- The model has probably spent ~2 weeks straight of training to get it to this level (it learns VERY fast I suspect from GQA+ALiBi + the type of training I've given it) It's why I think there's a considerable amount more training it can do.
- nickpsecurity 3y agoYou can also use Project Gutenberg if you want a huge, legal dataset. I’ve been collecting papers on small models, training with small data, using small ones to jump start big ones, alternatives to back propagation, etc. I wanted to do a sub-500M with those techniques eventually which could be reused in other projects. I may or may not get around to it. Email me and I’ll send you some of those links for your research.
- razodactyl 3y agoRE: How much is the training dataset. It's probably over 50GB so it's very unlikely I will be able to overfit the model, the best part is this means the model is always learning new things and showing new abilities. I noticed that straight after giving it ONLY a coding dataset it became a bit better at logical puzzles so I think there's a side-effect of training an LLM on sequential information like code. The LLM seems more inclined to learn structure for example: Structure of language Structure of poems Structure of music lyrics etc etc.