Very cool! If I understand correctly, this approach can generate steganographic output that is virtually indistinguishable from the output generated by the lan

Very cool! If I understand correctly, this approach can generate steganographic output that is virtually indistinguishable from the output generated by the language model used. Quoting from section 4.3 of the paper: "Most striking, arithmetic coding with the unmodulated language model induces a q distribution with a KL of 4e-8 nats. This indicates that, consistent with theory (Sallee, 2004), arithmetic coding enables generative steganography matching the exact distribution of the language model used."[a] Regardless of theory, I can't help but think, holy cow, the K-L divergence between the distribution of tokens in the steganographic output versus the output of a GPT-2 model is a minuscule 0.00000004 nats! Did I understand correctly? Given that language models are getting even larger and the distribution of their output is getting harder and harder to distinguish from that of human output[b], I find this incredibly powerful. Anyway, congratulations and thank you for sharing on HN! [a] https://arxiv.org/abs/1909.01496 https://arxiv.org/abs/1909.01496 [b] https://nv-adlr.github.io/MegatronLM https://nv-adlr.github.io/MegatronLM