It's mentioned somewhere in here (https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027) that the language filtering that was used on the source mat

It's mentioned somewhere in here (https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027) that the language filtering that was used on the source material wasn't perfect, so apparently enough non-English material slipped through the cracks for GPT-J to pick it up. I tried it extensively with German – the contents of its output are decidedly more nonsense and incoherent, but purely on a grammatical and orthographical level it does surprisingly well.