If the probability distribution is enough to reproduce a copyrighted work to the level of substantial similarity, then yes, a copy has legally been made. Howev

If the probability distribution is enough to reproduce a copyrighted work to the level of substantial similarity, then yes, a copy has legally been made. However, that's not the only question involved in a copyright lawsuit[0]. So far most of the evidence of copying has been circumstantial: a regurgitated Quake lighting function here, a Getty Images watermark there, but everything else has looked like wholly original output. We know from how these models are trained that copyrighted work is involved somewhere, but a court could just say it's Fair Use to scrape data and train a model on it. However, that defense is way harder to make if we can actually open up a model and show "ok, this is where and how it's storing copied training set data". At a minimum, it takes the "how much was used" Fair Use factor from "a few watermarks" to "your honor, the entire fucking Internet". [0] As usual we will assume jurisdiction in US court