Sure, but you've missed a step: the act of an LLM/AI spitting out a block of copyright-encumbered code to you is itself copyright infringement, for which OpenAI

Sure, but you've missed a step: the act of an LLM/AI spitting out a block of copyright-encumbered code to you is itself copyright infringement, for which OpenAI (et al.) should be liable for. You can then commit further copyright infringement by copying that code into your project an distributing it. It's similar for Stack Overflow: they require contributors to only post code that they have a legal right to post, but nothing actually stops them from including copyright-encumbered code in an answer that they don't have the rights to. The copyright holder would be within their rights to take legal action against the person who posted it, and/or send a DMCA takedown notice to Stack Overflow. And, again, you can commit further copyright infringement by copying that code into your project an distributing it.