Yes, although it might be unnecessary to do all of that. I would think that calling recv with MSG_PEEK could be used to read the first byte; if it is 0x16 then

Yes, although it might be unnecessary to do all of that. I would think that calling recv with MSG_PEEK could be used to read the first byte; if it is 0x16 then it can continue with TLS (using OpenSSL or other libraries to wrap the connection), and if it is anything else then it will continue without TLS. (This should work with any protocol that expects the client to send data first before the server does, and that 0x16 is not a valid first byte of the request.) However, I have not actually tried this (since I have not set up any servers with TLS yet).