Stick to the existing protocols. TLS for session control. ESP for bulk traffic. IKE - for the ESP key exchange or piggy-back on TLS instead. Multiplex as needed

Stick to the existing protocols. TLS for session control. ESP for bulk traffic. IKE - for the ESP key exchange or piggy-back on TLS instead. Multiplex as needed. Which libraries to use is secondary.