Been annoyed about this for a long time. Google still knows about my home address as I have that label set in my Google Maps settings, it just won't let me use

Been annoyed about this for a long time. Google still knows about my home address as I have that label set in my Google Maps settings, it just won't let me use it unless I allow them to track all my search history. Wtf. Evil design pattern for sure.