3 ms·
That's really interesting! I've wanted to do something similar but assumed that the token count required to ingrain a meaningful amount of information would mak
by gsuuon 3y ago
That's really interesting! I've wanted to do something similar but assumed that the token count required to ingrain a meaningful amount of information would make it very difficult. How many tokens do you typically need to feed into these base models to get consistent content out? Does generated text ever end up leaving the specialized domain? (like League of Legends characters mixed into a Minecraft story)
- fnordpiglet 3y agoI’ve not done anything particularly scientific to measure but my observation is if symbols are fairly rare it doesn’t require a huge amount. Herobrine for instance as a string isn’t very common, so it seems to be pretty good at keeping in the semantic region. Steve and Alex are popular Minecraft characters but are not very unique so you see more drift. But the more you mix in specific semantics, like creepers, or other tokens that are pretty specific to the genre, things improve. You see similar behavior with stable diffusion Lora. I’ve never seen it wander between genres, but it can tell pretty generic stories particularly if you’re pretty generic in your promoting.