3 ms·
I've posted a few "fake XYZ written by AI" on reddit over the years, and the model that got the best reactions was GPT-2. Markov chains aren't good enough to ma
by hatthew 2y ago
I've posted a few "fake XYZ written by AI" on reddit over the years, and the model that got the best reactions was GPT-2. Markov chains aren't good enough to make something that is interesting for more than a sentence or two, and GPT-3 onwards is too clean and boring. GPT-2 is the perfect middle ground that can get grammar largely correct and maintain a cohesive idea, but doesn't yet know enough of the specifics of various topics to make things that make sense in the context of that topic.
- metadat 2y agoCan you share any of the best examples?
- PawgerZ 2y agohttps://www.reddit.com/r/SubSimulatorGPT2/top/?t=all https://www.reddit.com/r/SubSimulatorGPT2/top/?t=all This is all GPT2 generations trained on reddit data. https://www.reddit.com/r/SubSimulatorGPT2/comments/btfhks/what_is_rsubsimulatorgpt2/ https://www.reddit.com/r/SubSimulatorGPT2/comments/btfhks/wh... Here's the subreddit explained
- Sohcahtoa82 2y agoI used my 15+ years of IRC logs to fine-tune a GPT-2 model to try to imitate me. My plan was to deploy a bot in my regular IRC channel and see how long it took for people to notice it was a bot. Any time someone would send a message, it would send the last 10 messages to the LLM, and if the result started with <Sohcahtoa>, then I would send that message to the channel. Unfortunately, I quickly found GPT-2 isn't nearly good enough. It would generate slightly-coherent yet on-topic nonsense. Once I overhaul my system, I'll try fine-tuning a 7B model.
- loa_in_ 2y agoI'm pretty sure you could make it work if you expanded the context to hundred or so messages.
- Sohcahtoa82 2y agoWith the 7B parameter models, sure. Not with GPT-2 though. The context window is only 1024 tokens. Even with only 10 messages, if they're long messages, it will exceed the context window.