I bet a lot of the issue is just the CLIP encoding. It's doing a ton of work getting text prompts into vector form, and it's incredible it does so well as is, b

I bet a lot of the issue is just the CLIP encoding. It's doing a ton of work getting text prompts into vector form, and it's incredible it does so well as is, but past a point you're squeezing blood from a stone. It's just not a full language model! Bump up the size, or figure something out about the space it's working in, I bet you'd see huge improvements.