3 ms·

I found that asking it to write a lipogrammatic text omitting the letter "E" was impossible. It gets very confused by which words contain the letter E, and when

I found that asking it to write a lipogrammatic text omitting the letter "E" was impossible. It gets very confused by which words contain the letter E, and when you ask it to omit it, it simply rewrites words like "forever" as "forvr", which is honestly hilarious.

Der_Einzige 4y ago
Ugh, I have to post this every damn place You can write lipograms, and poetry easily with language models. You just need a trick. See the example huggingface space in this paper. Lipograms are the first example. https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai https://paperswithcode.com/paper/most-language-models-can-be...
- jaggederest 4y ago
  I don't think you get what I was saying. Specifically ChatGPT, the current interface, is hilariously bad at lipograms. Nothing about other models, obviously you can build a model to model just about anything you like. I appreciate the link though, it will be fun to play with the huggingface example. Edit: also that example isn't exactly sophisticated. It literally just retries until it finds an example that doesn't contain the banned letters, bit of a brute force method. Would be more interesting to build a new model from scratch that just completely omits the letters.
  - Der_Einzige 4y ago
    Retries what exactly? It filters before the decoding step. I think you misread the code and the paper. I wouldn't know though, I only wrote it and all that...
    - jaggederest 4y ago
      Lol living up to the name "hellisotherpeople" I guess. https://huggingface.co/spaces/Hellisotherpeople/Gadsby/blob/main/Text-Generation.py https://huggingface.co/spaces/Hellisotherpeople/Gadsby/blob/... next_token_candidates = torch.multinomial(probs, num_samples=int(number_of_tokens_to_sample)) ## 10000 random samples word_list = [] for candidate_string in next_token_candidates: for candidate in candidate_string: resulting_string = tokenizer.decode(candidate) #skip_special_tokens=True, clean_up_tokenization_spaces=True) ###Constrained text generation starts HERE ##Lipogram - No naughty strings used if mode == "lipogram" or mode == "e-prime": if all(nauty_string not in resulting_string for nauty_string in naughty_strings): ## This returns at the first naughty strings return resulting_string That, to me, reads as "grab tokens as long as they don't contain the bad string(s)", and the code that uses it just repeats that. It doesn't seem, to me, to eliminate tokens at the model level, but I could be wrong, I'm no expert. I'd expect the model itself not to generate candidates that contained the "bad" string. Merry Christmas.

Der_Einzige 4y ago

Ugh, I have to post this every damn place You can write lipograms, and poetry easily with language models. You just need a trick. See the example huggingface space in this paper. Lipograms are the first example. https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai https://paperswithcode.com/paper/most-language-models-can-be...

jaggederest 4y ago
I don't think you get what I was saying. Specifically ChatGPT, the current interface, is hilariously bad at lipograms. Nothing about other models, obviously you can build a model to model just about anything you like. I appreciate the link though, it will be fun to play with the huggingface example. Edit: also that example isn't exactly sophisticated. It literally just retries until it finds an example that doesn't contain the banned letters, bit of a brute force method. Would be more interesting to build a new model from scratch that just completely omits the letters.
- Der_Einzige 4y ago
  Retries what exactly? It filters before the decoding step. I think you misread the code and the paper. I wouldn't know though, I only wrote it and all that...
  - jaggederest 4y ago
    Lol living up to the name "hellisotherpeople" I guess. https://huggingface.co/spaces/Hellisotherpeople/Gadsby/blob/main/Text-Generation.py https://huggingface.co/spaces/Hellisotherpeople/Gadsby/blob/... next_token_candidates = torch.multinomial(probs, num_samples=int(number_of_tokens_to_sample)) ## 10000 random samples word_list = [] for candidate_string in next_token_candidates: for candidate in candidate_string: resulting_string = tokenizer.decode(candidate) #skip_special_tokens=True, clean_up_tokenization_spaces=True) ###Constrained text generation starts HERE ##Lipogram - No naughty strings used if mode == "lipogram" or mode == "e-prime": if all(nauty_string not in resulting_string for nauty_string in naughty_strings): ## This returns at the first naughty strings return resulting_string That, to me, reads as "grab tokens as long as they don't contain the bad string(s)", and the code that uses it just repeats that. It doesn't seem, to me, to eliminate tokens at the model level, but I could be wrong, I'm no expert. I'd expect the model itself not to generate candidates that contained the "bad" string. Merry Christmas.