The expiration date+HMAC is still an option. Plus, the number of users blocking referrer headers is probably small enough that a site owner doing this wouldn't

The expiration date+HMAC is still an option. Plus, the number of users blocking referrer headers is probably small enough that a site owner doing this wouldn't care. And browsers dropping the header from HTTPS -> HTTP isn't an issue if you only want to allow access from your own domain. (If you want to authorize third-party sites that use HTTPS, you'd just have to use HTTPS yourself.)