3 ms·
> 1 could be fixed by using a tokenizer that only splits at syllable boundaries Maybe? Wouldn't that require the tokenizer to understand the word's usage in co
by brookst 4y ago
> 1 could be fixed by using a tokenizer that only splits at syllable boundaries
Maybe? Wouldn't that require the tokenizer to understand the word's usage in context of the surrounding text, or even worse, dialects?
What do you do with "fire" or "laboratory" or, god help us, "nuclear"?
- sp332 4y agoYes, you will need to write the tokenizer to behave in the dialect you are trying to write poetry in. An alternate approach is to filter on the output side by writing a sampler that only allows acceptable outputs. I think it would be easier to handle the ambiguous ones that way, because instead of encoding the ambiguity directly in the input or output stream, the sampler can just allow either pronunciation. https://www.youtube.com/watch?v=92hyQJRcFpM https://www.youtube.com/watch?v=92hyQJRcFpM https://github.com/Hellisotherpeople/Constrained-Text-Generation-Studio https://github.com/Hellisotherpeople/Constrained-Text-Genera...