It is a custom protocol that follows IRC very closely, they just have a lot of extensions on it to add support for the things on twitch. to start a connection y

It is a custom protocol that follows IRC very closely, they just have a lot of extensions on it to add support for the things on twitch. to start a connection you send PASS, NICK, USER and then JOIN, over the websocket so I would not really call it a custom protocol. The url it uses is wss://irc-ws.chat.twitch.tv/ as well.