It's not. There's a lot of metadata there that's not dependent on your contact list. They still have the phone numbers of the people you're in touch with, they

It's not. There's a lot of metadata there that's not dependent on your contact list. They still have the phone numbers of the people you're in touch with, they have the groups you're part of and all the phone numbers there. To get (most of) your contact list, they just need to cross that information with everybody else's contacts list. Your not sharing is not worth much when everybody else is sharing. On top of that, they also know times you're online, potentially location, who you're messaging with, who you're making audio/video calls with and for how long, etc.