I use Fastmail, and everything is great (for me) except one very annoying aspect with Contact sync. I’m unable to sync only manually created contacts via cardd

I use Fastmail, and everything is great (for me) except one very annoying aspect with Contact sync. I’m unable to sync only manually created contacts via carddav. If I sync contacts to my phone, I get every automatically created contact also showing in my contact list, which is basically anyone I’ve ever sent an email reply to. Somehow Gmail handles this - I think they simply exclude the auto created group from being exported. I asked Fastmail support about this and there is nothing in the works to allow it. I hope someone from Fastmail has a fix coming for this, or some other suggestion.