35 ms·
WyGPT: Minimal mature GPT model in C++
- helloplanets 3y agoThe readme page is cryptic. What does 'mature' mean in this context? What is the sample text a continuation of? Having a gif the thing in use would be great, similar to the gpt4all readme page. (https://github.com/nomic-ai/gpt4all https://github.com/nomic-ai/gpt4all)
- bt1a 3y agoIf you think the readme is cryptic, you're in for a treat
- nemothekid 3y agoWas pleasantly surprised at how accurate this comment is.
- deleted 3y ago[deleted]
- ie21 3y agoWent to see that code repo...guy threw a party there
- dhoe 3y agosexy is part of Neanderthal. Hope this helps
- LoganDark 3y agotried reading the cuda source and I can see why it took 2.5 years to get working properly
- loxias 3y agoGross. Why in the world do I want a gif for a tiny code release? Just read the source. This looks extremely interesting, considering the author I'm inclined to give it the benefit of the doubt. The source looks fugly, but who cares, I can reformat it. Compilers don't care.
- helloplanets 3y agoTrue, maybe a gif is a bit of an inane ask for a relase of this sort.
- deleted 3y ago[deleted]
- totorovirus 3y agosexy self is a cool naming convention that will possibly replace foobar https://github.com/wangyi-fudan/wyGPT/blob/main/cpu.cpp#L140 https://github.com/wangyi-fudan/wyGPT/blob/main/cpu.cpp#L140
- kgeist 3y agoThis project comes from the author of wyhash, which is now the default hashing algorithm in Go, Zig etc. What's lacking is more information about the project. I was able to compile it and just started training on GPU, but I have no idea what's going on, when it will end, and what the characteristics are in general. Currently the only output I get is: ./train book-war-and-peace.txt book-war-and-peace.txt 3.05397 1 para 1679360 32 1.3654 206.1985 I guess I need to dive into the source code in the meantime :)
- hskalin 3y ago2.5 years of effort but the author certainly didn't spend even 2.5 mins writing code for more descriptive outputs. Meanwhile I'm also training in order to find how well it works but I only have a small GPU. I would appreciate if someone could provide a trained model.
- kgeist 3y agoSo it's a character-based model (makes sense, because there's no tokenizer to be found). The model file is 3.3 MB Output after 3 hours of training (loss = 0.4160): hello, the sounds of the rain assistants. In provations which had been banitz had taken step in, the spirit of the crowd, and the right winot to scrap on , Valuex, one of marks othhhanled me in a most powerful power. The other Augun had been in commands among that guests, the regiments were placed by imists, and science of irritation, but or thoughts that he would never cease to point before meant in the losses were discussing, and at once she had lived in the church. From the looks of it, it's similar to MinGPT/nanoGPT (I found a Reddit thread [0] where the author compares it to MinGPT) [0] https://www.reddit.com/r/MachineLearning/comments/o2u2cm/pwygpt_improved_small_gpt_model_in_c_from_scratch/ https://www.reddit.com/r/MachineLearning/comments/o2u2cm/pwy...
- tt567x 3y agoVery neat.
- ashvardanian 3y agoI love every ambitious and tiny codebase, but the formatting looks very uncommon and for me personally - hard to read. Is there a tool that produces it?
- _lvbh 3y agoThe code is wizardry. It looks more confusing than obfuscated JavaScript