3 ms·
3080Ti (12GB VRAM), DDR4 64GB AMD Ryzen 7 8TB drive-space (I built it with the idea that I would be hoarding a lot of data for potential AI research) --- The
by razodactyl 3y ago
3080Ti (12GB VRAM), DDR4 64GB
AMD Ryzen 7
8TB drive-space (I built it with the idea that I would be hoarding a lot of data for potential AI research)
---
The model has probably spent ~2 weeks straight of training to get it to this level (it learns VERY fast I suspect from GQA+ALiBi + the type of training I've given it)
It's why I think there's a considerable amount more training it can do.
- nickpsecurity 3y agoYou can also use Project Gutenberg if you want a huge, legal dataset. I’ve been collecting papers on small models, training with small data, using small ones to jump start big ones, alternatives to back propagation, etc. I wanted to do a sub-500M with those techniques eventually which could be reused in other projects. I may or may not get around to it. Email me and I’ll send you some of those links for your research.