Valid question, not sure why the downvotes. You're right, the server can modify the contact list (and this is used extensively within organizations, and in Snik

Valid question, not sure why the downvotes. You're right, the server can modify the contact list (and this is used extensively within organizations, and in Snikket to ensure family members can see each other by default). But this isn't the cause for concern here - if the server is malicious it can broadcast this info without the need to modify your contact list. This again is independent of protocol - if someone wants to log or share the times and durations you are actively connected to their server, they can do so quite easily. It doesn't require XMPP.