You can also use Project Gutenberg if you want a huge, legal dataset. I’ve been collecting papers on small models, training with small data, using small ones t

You can also use Project Gutenberg if you want a huge, legal dataset. I’ve been collecting papers on small models, training with small data, using small ones to jump start big ones, alternatives to back propagation, etc. I wanted to do a sub-500M with those techniques eventually which could be reused in other projects. I may or may not get around to it. Email me and I’ll send you some of those links for your research.