>Model capacity. Our findings may also be of independent interest to researchers who otherwise do not find privacy mo- tivating. In order for GPT-Neo 6B to be a

>Model capacity. Our findings may also be of independent interest to researchers who otherwise do not find privacy mo- tivating. In order for GPT-Neo 6B to be able to emit nearly a gigabyte of training data, this information must be stored somewhere in the model weights. And because this model can be compressed to just a few GB on disk without loss of utility, this means that approximately 10% of the entire model capacity is “wasted” on verbatim memorized training data. Would models perform better or worse if this data was not memorized