I had my home address set since couple years without turning history on. This week I wanted to change address, but Google wouldn't let me unless I give access

I had my home address set since couple years without turning history on. This week I wanted to change address, but Google wouldn't let me unless I give access to all location, web and app activity.