That works, but I found a simpler / cheaper alternative to fix this particular Cloudfront problem: Create a custom "Cache policy" that only includes the header

That works, but I found a simpler / cheaper alternative to fix this particular Cloudfront problem: Create a custom "Cache policy" that only includes the header "origin" and then the answer from Cloudfront will always include "Vary: origin", even if it wasn't in the request (That behaviour is not documented anywhere I could find, stumbled upon it by accident)