I understand validating the `Host` header but my understanding that the referer field is optional and theres quite a few users who either block referer headers

I understand validating the `Host` header but my understanding that the referer field is optional and theres quite a few users who either block referer headers for privacy/anti-tracking purposes. Would this stop those users from accessing the site also?