I don't understand why a scammer wouldn't just add the number to their address book before calling. Also as an individual this would be kinda scary. Not to men

I don't understand why a scammer wouldn't just add the number to their address book before calling. Also as an individual this would be kinda scary. Not to mention the fact I have to share my private address book, but also if my friend forgets to add me I can randomly have to pay?