If your phone is rooted, you can try installing XPrivacy to manage permissions for Apps. I used it for a while when I had to have Skype on my phone. It worked g

If your phone is rooted, you can try installing XPrivacy to manage permissions for Apps. I used it for a while when I had to have Skype on my phone. It worked great to lock Skype out of GPS, Contacts, etc. There was a noticable performance hit, but my phone is 3+ years old. It likely would not be as much on a newer phone.