20 ms·

How Discord Handles Two and Half Million Concurrent Voice Users Using WebRTC

649 points by jhgg 8y ago

sdegutis 8y ago
I made several websocket-based apps that I want to turn into Electron desktop apps where people can use them only with their friends, but WebRTC has been really confusing and hard to know how to get started with. Especially considering ideally I don't want to have to host a server, but all the P2P JavaScript libraries that I found seem to assume you'll at least have a server for hosting the lobby to look for peers.
- jrcii 8y ago
  Look into the DHT protocol
- ty_a 8y ago
  Iirc there are some free servers you can use provided by Google and/or twilio. Edit: STUN is free, Turn is not. Been a while since I worked with those.
- chuckhays 8y ago
  WebRTC requires some sort of external communication channel to exchange information about how to set up the connection. (Often referred to as signaling messages). Disclaimer: Google employee, work on the WebRTC team.
  - marknadal 8y ago
    Grandparent: After getting frustrated with all the other WebRTC libraries over the last 4 years, I finally wrote our own ( https://github.com/amark/gun/blob/master/lib/webrtc.js https://github.com/amark/gun/blob/master/lib/webrtc.js ) which is capable of using a set of decentralized DHT relay-peers in GUN for signaling - and once peers are already on WebRTC, they can signal (daisy-chain "DAM" as we call it) to other WebRTC peers via WebRTC! Meaning, you don't/won't have to run any servers!!! Ping us on our chatroom if you want the list of DHT peers. Parent: Chuck! Sounds like you'd be a very useful person to know. Me, Feross, etc., plenty others have been requesting additional API/protocol access over the last 4 years (some of which FireFox is adding in libdweb extension!). Any chance we could connect and chat? Ping me at mark@gun.eco ?
    - anderspitman 8y ago
      Would it be a problem to keep a list of DHT peers in the git repo?
      - marknadal 8y ago
        Good idea. I need to get approval they are OK with their peers being publicly listed (easy DDoS potential, we don't have code yet to [but working on it] mitigate it). I'll start one with just mine, and then others can PR to opt-in. You want to join?
        anderspitman 8y ago
        Awesome, thanks. I'm not actually using WebRTC (or GUN) yet, but I hope to for a project relatively soon and the possibility of not requiring a STUN/ICE server is very enticing to me.
    - Orphis 8y ago
      If you need to request more APIs, you should try to talk about your usecase to the WebRTC working group using the users mailing list ( https://groups.google.com/forum/#!forum/discuss-webrtc https://groups.google.com/forum/#!forum/discuss-webrtc ). Depending on the question, you may also ask on the spec Github project ( https://github.com/w3c/webrtc-pc https://github.com/w3c/webrtc-pc ). Source: Google employee, in the WebRTC WG.
- eboyjr 8y ago
  You will need some sort of signal transport for sending ICE negotiation data in JSON. Because the two peers need to know how to coordinate how to connect to each other before actually connecting. It can be a websocket server, REST API, email, QR codes, carrier pigeon, etc. Also in general, I highly recommend webrtc-adapter.
- Klathmon 8y ago
  The fact is that you can't easily do the last part without a central server. There might be some public or semi-public servers for this kind of thing available somewhere, or the alternative that I played with a few years ago was to compress and base64 encode the connection information into a string, and allow users to share the link with friends via whatever method they want, and that can then be expanded client side and used to establish the connection. (Or something like qr codes I've also seen used) Sadly I never finished that project, so I don't really have any code to show you, but in theory it should work okay.
  - sdegutis 8y ago
    I mean if everyone already has a way to contact their friends (discord) and say "hey let's play this game together", sharing that link does away with the need for a coordination server. But there's still the need for at least a server, even if it's just one of the clients that knows how to take charge (like Minecraft).
    - Klathmon 8y ago
      Yeah, I skipped over a lot of the details, but you would need someone to choose to be a "host" (at least in terms of a "host" for the connection signaling stuff), but you can also implicitly make it the person who created the link safely.
      - derefr 8y ago
        If everybody is behind NATs (without uPnP privileges), nobody can be the "host" for connection-signalling purposes.
        Klathmon 8y ago
        There are public ICE/STUN servers that can facilitate NAT-punching, google even hosts one. But that's not the aspect that I thought the root comment was talking about, but more the aspect of "A is hosting a room called 'stuff', and B and C want to browse the list of rooms and connect to it".
- stcredzero 8y ago
  For my game, I wrote a simple UDP to WebRTC datachannel proxy in NodeJS, suing node-electron, webrtc-electron and SimplePeer. My proxy only works downstream, with the upstream signalling using websockets. I'm going to move over to an all WebRTC architecture, however. https://www.emergencevector.com https://www.emergencevector.com
  - kayoone 8y ago
    Interesting, i am working on a realtime action game and currently implemented multiplayer via websockets including authorative server, client side predicition and server reconciliation which works failry well but based on the nature of TCP, i still sometimes get some visible stutter in the movement. I guess with WebRTC you can handle things more like traditional UDP? What are your experiences vs Websockets? Did you write how you did it somewhere? Would be very interesting
    - stcredzero 8y ago
      I got really big stutter with head of line blocking on TCP. You can make things better by conspiring to send only 1 packet messages of about 512 bytes or less, and not too often. You can register for an account on my server. WebRTC downstream seems to be working at the moment. Be sure to read the instructions, however. https://www.emergencevector.com https://www.emergencevector.com I am thinking of turning my game server into a serverless PaaS offering. I'm open to collaborating on this.
      - kayoone 8y ago
        i emailed you regarding this ;)
- nine_k 8y ago
  There's a number of decentralized networks that could send the connection info without a single central server, e.g. torrents or Dat. OTOH the time required to connect to them and pass the info would be quite noticeable. A single connection-initiation server could be the easiest solution. It won't need to withstand a heavy load.
- jessaustin 8y ago
  There are probably some problems with this idea, but why not just use a popular social network or similar as the lobby? For example, if there is a twitter feed "@MyWebRTCLobby" that everyone follows, clients could just @ a tweet at it with current contact details.
  - rcraft 8y ago
    If random Twitter users send @mentions to "@mywebrtclobby" then no other users will likely see these tweets unless they already follow each of the various Twitter users sending the @mentions. However if you connected @mywebRTClobby with https://GroupTweet.com https://GroupTweet.com you could configure things so that any @mentions (from authorized users or anyone) would be converted into actual tweets from the @mywebRTClobby account so that all followers would actually see those tweets/contact details.
    - jessaustin 8y ago
      Haha I don't use Twitter so I'm not surprised I got something wrong... thanks for fixing that bug.
      - jsjohnst 8y ago
        It used to be the case you could see all @mentions someone received, but alas not anymore.
- matthewaveryusa 8y ago
  You need a STUN/TURN server, that's just how the cookie crumbles unfortunately. Blame NAT :) I used this dockerized coturn server successfully: https://github.com/strukturag/docker-webrtc-turnserver https://github.com/strukturag/docker-webrtc-turnserver
jrockway 8y ago
I'd be interested to see how the system works during high-load events. It sounds like if a server gets overloaded, it shuts down and moves the clients to the next non-overloaded server. If the health-check used by the load assignment server is not perfectly tuned, then that failover overloads that server, shutting it down. Eventually you have no servers left. I suppose that a perfect health check will prevent this, since the failover will assign failover traffic at exactly the level that the new server can successfully handle. But if it's wrong on the other side (rejects connections when capacity actually exists), then compute resources are wasted "just to be safe". I imagine that estimating capacity is even more difficult since people can join and leave at any time, and the client doesn't send any packets when there is silence. So the load changes based on how talkative people are being (which means your server always crashes during the best parts of whatever you're discussing). Anyway, I'm wondering how this all compares to the naive strategy of "pick a random server for this channel, if it crashes, bad luck".
- qmarchi 8y ago
  > ... and the client doesn't send any packets when there is silence. Yes and no, there's no packets being passed over the WebRTC connection, but the server maintains a WebSocket connection for state changes. There's two, one to the guild service, which handles assignment of the voice service. Want to count the number of clients connected to a voice server? Count the number of connections. All Discord clients are connected to Guild services and discord publishes events for voice channel updates to all clients in the guild. (Ex. when a person joins a voice channel.) Also, for larger discord guilds, there are exclusive voice server pools for them to consume.[1] These servers are configured with more resources and are usually pretty close to exclusive to the single guild. Most of the voice servers are for the millions of other guilds though. [1] https://discordapp.com/partners https://discordapp.com/partners
- ivanfon 8y ago
  I remember in the past the Discord devs have moved particularly overloaded servers to their own dedicated hardware to improve performance.
- jhgg 8y ago
jorams 8y ago
Slightly off-topic, but: > For clarity, we will use the term “guild” to represent a collection of users and channels — they are called “servers” in the client. The term “server” will instead be used here to describe our backend infrastructure. I still don't understand why you ever chose the name "server" to refer to something that is many things, but not a server. Guild is a way better name, and doesn't need disclaimers about your self-chosen terminology being confusing.
- jczhang 8y ago
  I believe this is a tradition that has been grandfathered into voice chat apps. Previous ones like Ventrilo have you actually connect to a private server (that someone pays for / hosts).
  - elefanten 8y ago
    I second this explanation. Additionally, from a UX perspective, if a user wanted to join your server you had to give them IP/port/pw and they'd type it in manually. So even the users were accustomed to addressing servers. "Server" entered the community lexicon as "the place we chat".
  - ObsoleteNerd 8y ago
    I assumed it was based on IRC servers, because each Discord server can have multiple "channels" just like IRC.
  - androsyang 8y ago
    This is most likely the case. Discord's introduction was mainly to pull in the gaming audience using Skype while providing the quality behind Mumble and Ventrillo
- savanaly 8y ago
  I'm a programmer who well knows what a server is to a programmer, but I'm also a gamer and Discord calling its servers servers never seemed weird to me. From the perspective of the user, the term makes sense. The servers have users, those users have permissions, they host files, they host chat, you can install plugins to them.
hfourm 8y ago
This comment isn't really related. I obviously love Discord and have been using pretty much since it became an option, but I wish there was a better UX for people who are in TONS of channels/servers. I find it hard to navigate all my current servers with just the icon -- especially when server owners are changing their avatars frequently.
- anafh83 8y ago
  I think their UX is awful, and their client is very slow in channels with thousands of users. They should open up their protocol so alternative clients can be created, or at least some IRC gateway. I haven't made many calls but the sound was always crisp.
  - scrollaway 8y ago
    I agree on opening the protocol but if what you want is a gateway, it's already open enough for that. https://github.com/42wim/matterbridge https://github.com/42wim/matterbridge
  - marksomnian 8y ago
    > open up their protocol Both the Rest API (that official clients use internally too), as well as the real-time WebSocket protocol are described here: https://discordapp.com/developers/docs/reference https://discordapp.com/developers/docs/reference https://discordapp.com/developers/docs/topics/gateway https://discordapp.com/developers/docs/topics/gateway
    - Ycros 8y ago
      Unfortunately you can't use these APIs to create third party clients because that violates their ToS, they are meant only for writing bots. If they catch you using these APIs with your full account credentials they'll ban your account.
      - tenryuu 8y ago
        They can't catch you using these apis since these are the same apis the first party client runs off of
Karrot_Kream 8y ago
I really wish there were non-browser implementations of WebRTC. So far, it seems like the standard as-it-is is defined by browser code and browsers contain the canonical implementations.
- brian-armstrong 8y ago
  Does https://webrtc.googlesource.com/src https://webrtc.googlesource.com/src not count?
  - Karrot_Kream 8y ago
    Right, so it's a chromium C++ repo that you need to create bindings for to interact with. I have no doubt that an organization like Discord has the engineering resources to either write their own bindings to WebRTC or to roll their own implementation of the components, but if I'm a an individual developer that wants to interact with WebRTC without a browser, then it's pretty difficult. One common desired use case is using WebRTC for p2p torrent communications. Right now the best way to do this is in browser, or to use an Electron app that can bridge the WebRTC clients with the standard desktop torrent clients.
    - brian-armstrong 8y ago
      Ah that makes sense. I have seen forks that de-chromium it some that look pretty reasonable. Most of it doesn't depend on anything chromium, just happens to use its build system (for better or worse). But I believe the fork switches it to CMake
    - jgrowl 8y ago
      There's nothing tied specifically to the browser, but it is written in C++. It seems like the discord folks just built a layer on top of it. It's been a while since I played around with it, but you just have to build the peerconnection library. This guide seems pretty close to what I remember having to do: https://webrtc.org/native-code/development/ https://webrtc.org/native-code/development/ It still leaves you writing in c++, but it lets you build a self contained server without any browser stuff. There also were java bindings which is what I used when I was experimenting with it. You just have to build your .so and .jar files. EDIT: Here's some scala code that I wrote that uses the native bindings. (Please forgive the messy test project that I have abandoned). https://github.com/jgrowl/livehq/blob/master/media/src/main/scala/tv/camfire/media/webrtc/WebRtcHelper.scala https://github.com/jgrowl/livehq/blob/master/media/src/main/...
ReidZB 8y ago
I really wish Discord would implement an audio compressor (or if they use one, make it much more aggressive) for voice chat. For example, Mumble doesn't require manually adjusting anyone's volume - instead, it uses a compressor to automatically normalize volumes for everyone. In contrast, Discord has a volume slider that I end up having to adjust for most people. That requires manual tweaking, often while I'm playing a game, and isn't future-proof; just the other day, one of my friends changed his microphone setup and was extremely loud (I had him at 200% when in the new setup he should've been at about 75%). This is really my only complaint with Discord's VOIP functionality. It is a major step backward from Mumble. (Also, for some reason the Discord community mostly uses voice activated mic instead of push-to-talk, but I don't think that's Discord's fault.)
- iMerNibor 8y ago
  > voice activated mic Which is a very sane default for most people The only reason you'd want to use push to talk is either you have a lot of people in the channel, you have a loud background or you don't want people hearing stuff not designated for the discord voice (if you're streaming, have irl people in the same room - that kinda stuff)
  - r_smart 8y ago
    Personally, I hate the sound of people chewing and smacking their lips directly in my ear as they work their way through their 'fresh that afternoon' gas station nachos. But that's just me :)
    - Zekio 8y ago
      if you are in control of the Discord server you can force push to talk in voice channels
    - LinuxBender 8y ago
      It is not just you. I know of several gaming sites that will block or ban people for leaving their mic open or making breathing sounds into it.
Reedx 8y ago
> we have seen 1000 people taking turns speaking Wow. Has anyone here been in a channel like that? I'd imagine it to be complete chaos. But I suppose it's possible with good moderation and/or bots to ensure people take turns? Is that what they do?
- wwwigham 8y ago
  Yeah. Typically it starts with a priority speaker giving a speech, ends with a Q&A moderated via text chat.
- Operyl 8y ago
  There's usually some sort of moderator, and a bot to facilitate turn taking, at least in the channels that I had these in :).
- crystalPalace 8y ago
  I've been in a couple channels with several hundred people. The admins handled it by shouting until everyone was quiet, enforcing one person speaking at a time, and swiftly muting, moving, or banning anyone who disobeyed. I'm not sure if Discord's API is flexible enough for a bot like that.
  - Operyl 8y ago
    The API is flexible enough for a bot like this, I'm contemplating opensourcing mine at some point! :)
    - zrobotics 8y ago
      Please do, it would be an actual act of charity! I've left a lot of interest i 8 servers due to the voice chat being clogged.
- Kagerjay 8y ago
  1000 people taking their turn speaking? It must be some political rally or reddit discord channel for a large national outbreak news story All I know is that official fortnite (discord's largest gaming server last year AFAIK) had serious amounts of disconnect issues, only on that server though Fortnite didn't have more than 100 battlerooms (4 players max) at any given point in time. There's a few battle royale scrimming servers with 70-100 players in one room for tournaments. But the rule is everyone needs to be mute & deafened though
merb 8y ago
OT: actually what I always wonder, is how Discord will make money. I mean they offer this huge free service for everybody (2.6 million concurrent voice users) over 1000 servers that must be a huge GCloud invoice
- meowface 8y ago
  I imagine they make most of it through Nitro: https://discordapp.com/nitro https://discordapp.com/nitro
- vertex-four 8y ago
  They’ve just started a game store. Encouraging integration of their social features into games will allow them to take a healthy bite out of Steam’s pie.
  - Godel_unicode 8y ago
    Related: steam has built in voice chat now. https://www.engadget.com/2018/06/12/steam-finally-adds-voice-chat-flexible-friend-lists-new-beta/ https://www.engadget.com/2018/06/12/steam-finally-adds-voice...
    - maemilius 8y ago
      Steam has had voice chat (group and 1-on-1) for a much longer period of time than that. I've used it on a regular basis for the last 5 years or so, well before the current updates they've made. What they added was voice chat to static groups[0], whereas before you had to invite everyone to a custom group chat every time. Definitely a QoL improvement, but not exactly "new". 0: https://steamcommunity.com/search/groups https://steamcommunity.com/search/groups
      - fro0116 8y ago
        The new chat is a huge improvement over the old one in terms of UX, but lack of persistent history is pretty pathetic considering the amount of resources Valve has available compared to a startup like Discord.
Siecje 8y ago
Does anyone have a solution to prevent notifications when a server creates a new channel and then I am notified for every message until I mute the channel.
- ihuman 8y ago
  You can mute the server, or change the default notification setting to @mentions only for the server
Operyl 8y ago
If you skip the use of ICE, why is it that 25% of the time I get a message relating to ICE and it gets stuck there? "CONNECTION_STATUS_ICE_CHECKING" ICE Checking.
GranPC 8y ago
> Instead of DTLS/SRTP, we decided to use the faster Salsa20 encryption. In addition, we avoid sending audio data during periods of silence — a frequent occurrence especially with larger groups. I wonder whether this would help an attacker infer voice data, with a method similar to the one from the paper "Uncovering Spoken Phrases in Encrypted Voice over IP Conversations" [1] [1]: http://www.cs.unc.edu/~fabian/papers/tissec2010.pdf http://www.cs.unc.edu/~fabian/papers/tissec2010.pdf
- EvangelicalPig 8y ago
  I've never tried doing pcaps with Discord voice chat yet but it would be interesting "estimating" what was said best on the amount of voice traffic transferred. Might be harder if it still looks like standard Discord TLS "control traffic" blending in though, no?
  - wahern 8y ago
    Keystroke timing analysis has some solid research. See, e.g., Timing Analysis of Keystrokes and Timing Attacks on SSH (https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf), Don’t Skype & Type! Acoustic Eavesdropping in Voice-Over-IP (https://arxiv.org/pdf/1609.09359.pdf https://arxiv.org/pdf/1609.09359.pdf). Deciphering content from latencies in packetized speech is likely much more difficult, but I wouldn't put much stock in it being too difficult. Which is to say, if you're transferring high-value information assets over VoIP you should probably assume it's decipherable. That doesn't mean you should change what you're doing. You could simple say, "M'eh, I'll worry about it when it becomes a thing." But I wouldn't assume it's confidential to someone willing to invest the time to target and capture the conversations. And I might leave a few choice comments in the source code and documentation so nobody could excuse imprudent reliance on confidentiality with, "But nobody warned me".
- cvwright 8y ago
  First author of the linked paper here. It's an interesting question. Actually the voice activation (on/off periods of sending vs silence) was the first thing we looked at in that project. There's definitely some information leakage there, but it was really hard to learn anything meaningful from it. The problem seems to be that long strings of words all get lumped into a single activation. It's really hard to discern anything about what those words are, or even what language they come from. We got some very weak results on language detection from VAD. You can learn something about the language, but it's not very precise. For example, maybe you could tell that a given conversation is definitely not language A, B, or C, but it might still be language X, Y, or Z.
anderspitman 8y ago
For those interested in open source/decentralized alternatives to Slack/Discord/etc, I've had a pretty great experience with both the voice and video integrated into Riot.im[0] (uses Matrix protocol[1] for chat, and Jitsi[2] for voice/video). I've said before that I can't really justify nonfree software for something as simple as text chat. That said, rooms with very large numbers of people trying to communicate with voice/video is one use case where maybe it makes more sense for a commercial product to solve it. [0] http://riot.im/ http://riot.im/ [1] https://matrix.org/blog/home/ https://matrix.org/blog/home/ [2] https://jitsi.org/ https://jitsi.org/
- singularity2001 8y ago
  Why is the riot "TRY now" link so damn hidden? Two clicks, one long search here https://riot.im/app/#/home https://riot.im/app/#/home No private room? Not working without google? Very bad first impression :(
  - anderspitman 8y ago
    Not sure what you mean by not working without google?
    - hegz 8y ago
      Iirc there is a captcha on sign up. Same as discord.
  - Arathorn 8y ago
    isn’t the “try now” link the big flashing button labelled “launch now!” on http://riot.im http://riot.im?
    - anderspitman 8y ago
      That's how I always start it. Unfortunately that link is on a carousel for some reason, which automatically advances. Definitely not ideal UX in my opinion.
orliesaurus 8y ago
Can anyone who works on the discord app explain to me: How am I impacted for having (being part of) let's say 100 server (guilds) on my app versus having 1? Should I quit the discords that I don't use? I feel like they stay in sync in the background and if I am in 100 servers and they all post say tons of text I will get massive lags. Does that make sense? Is there a way to prioritize only the server I have active/I am talking in and freeze the rest while im full-screen/in-game/focused?
- highesttide 8y ago
  I'm pretty sure there isn't a ton of background fetching asides from marking servers as unread, it can take a bit to actually load the messages when you click an inactive server, and it won't have any new messages if you don't have a connection (even if it had a connection when it was sent)
- b1naryth1ef 8y ago
  We've put a lot of work into our clients and our backend to make sure the impact of being on 1 vs 100 guilds is negligible. It helps that most of the folks building Discord are power users in a bunch of servers (so we feel the pain of poorly optimized paths early). Generally if you don't look at a server often it shouldn't effect the performance of the app / bandwidth usage. I think we have some more blog posts in the works regarding some of these topics so look forward to those!
  - ehsankia 8y ago
    If a guild is completely muted (including @everyone), and I don't open it, do you have a rough estimate of how much resources it uses? In theory, it should be zero, but in practice, does is the server sending me updates about messages even if I'm muted?
    - jhgg 8y ago
      It depends on the activity of the server. But generally, you're only receiving message create events and updates to the server. But you aren't receiving things like member list updates, presence or typing events from the server until you focus it initially. Additionally, we unsubscribe you from these events if you unfocus the server for a given amount of time.
fulafel 8y ago
"Instead of DTLS/SRTP, we decided to use the faster Salsa20 encryption." This is just confused and sounds quite worrying :I
- fulafel 8y ago
  Judging by the votes, this may need spelling out: Salsa20 is a stream cipher. DTLS and SRTP are higher level security protocols that use ciphers (among other things) as building blocks, to ensure things like replay protection, integrity protection, mutual authentication, secure session key agreement and forward secrecy in addition to confidentiality. If you replace an engineered security protocol with a raw cipher and key, you create many vulnerabilities and make a much less secure system. VOIP applications are especially vulnerable to replay attacks, for instance. A better approach would be to continue using DTLS and SRTP, and use Salsa20 as the SRTP cipher. Non-crypto-experts saying things like "we replaced <estabilished security protocol> with <my own idea> and it's much faster" is a well known bad sign - It often indicates the person has fumbled things without realizing it.
  - Orphis 8y ago
    They say they have done it for performance. I am wondering if they wanted to optimize for the client performance or the SFU's.
bg0 8y ago
My buddies and I use Discord as an easy conference call group chat while playing games (no push to talk, mic is always on). One thing I've found on the app is when a person/voice is far away from the mic it often glitches and doesn't send all of the audio data over like, for a example, a normal FaceTime Audio call would. Not sure why this happens, but just thought I'd put that out in the ether.
- gizmo385 8y ago
  Sounds like you just need to adjust your microphone sensitivity settings? If you have the mic on voice activation and the person is far from their mic than it might be that they're just not loud enough (from the perspective of their microphone) to actually trigger Discord. That's my guess at least.
  - bg0 8y ago
    Ah, never thought of it like that. I always perceived it as a "always on" scenario. Just found it in the settings and I'll give it a try, thank you!
quackerhacker 8y ago
So I have an affection for Discord...so I appreciate posts like this. >"Using the WebRTC native library allows us to use a lower level API from WebRTC (webrtc::Call) to create both send stream and receive stream." So I'm gathering that discord's voice servers receive multiple persistent connections, then compress the audio streams for delivery to each end user. THIS part is where I can't imagine the on-the-fly cpu usage. Each client's receiving compression needs to also negate their own audio to prevent an echo effect (no point to hear your own voice), but it also means separate compression streams per user. >" All the voice channels within a guild are assigned to the same Discord Voice server." I imagine this helps significantly with I/O in converting live streams into 1 stream per end user. I've dealt with video compression (only in ffmpeg) and live syncing time stampings, and I can say from experience that, this is no easy feature. I understand this is audio streams (so lower overhead), but still the persistent voice server needs to handle the incoming connections, web socket heartbeats (negligible), compression (high I/O), and deliver the streams (high memory usage too). I'm impressed, but would love to hear the specs on the media servers and their DL/UL speeds. My old setup to deliver live video (in sync and compressed) was 6 mini-itx's, 4GB of ram per board, and i3's...my bottleneck was my isp, which I solved with multiple docsis modems and an internal switch (each board had 2 ethernet ports).
- jhgg 8y ago
  We don't transcode audio/video on the server. Each stream is processed and muxed by the client. The server is merely relaying rtp packets and tagging them with a given ssrc per peer. The client does the rest of the work. The bulk of the user-space time on the SFU is spent doing encryption (xalsa/dtls). We also avoid memory allocations in the hot paths, using fixed-size ring buffers as much as possible. Additionally, we coalesce sends using sendmmsg, to reduce syscalls in the write path: (http://man7.org/linux/man-pages/man2/sendmmsg.2.html http://man7.org/linux/man-pages/man2/sendmmsg.2.html) I posted some about the specs here: https://news.ycombinator.com/item?id=17954163 https://news.ycombinator.com/item?id=17954163
  - quackerhacker 8y ago
    If Discord is basically proxying the raw packets from one client to the others, isn't that wasted bandwidth (for discord, not the clients). I understand from the post that the goal would be to mask the ip of the users, to shoulder user privacy and the ddos vector. Kudos on silence detection to save overhead. So video w/audio broadcasting has to be compressed client side, then proxied through Discord's media servers, to the end user's. That's pretty smart...I just wished that I could send my raw stream to a LAN host so I could offload the compression, and allow my LAN host to provide delivery (I'm a nitro user).
winterismute 8y ago
I love discord but am I the only one that thinks that the name is hindering wider adoption? It's hard to suggest to some people that an with that name (and to a lesser extent visual theme) is the skype+slack+others replacement they are actually looking for...
- Kaveren 8y ago
  I suppose you could say the same for Slack. I don't think that the connotations of the names for either particularly matter to users. What's hindering adoption outside of the gaming community is the branding and niche focus.
- Vishnevskiy 8y ago
  We found the name is not an issue as growth continues at a rapid pace. We are already multiple times larger than Slack :)
dep_b 8y ago
I've been reading the article hoping to find out a bit more about the actual voice server but the SFU is actually not very deeply explained at all.
- jsjohnst 8y ago
  That’s because there isn’t a lot to it from the sounds of it in this post: https://news.ycombinator.com/item?id=17955309 https://news.ycombinator.com/item?id=17955309
  - dep_b 8y ago
    I would implement a similar dump pipe as well for a very different application. I mean the application still needs to set up the WebSockets with the clients, which isn't trivial to code yourself.
tylerplz 8y ago
> Since it’s the only service directly accessible from the public Internet, we will focus on Discord Voice server failovers. Why is the Gateway not directly accessible from the public Internet?
- jhgg 8y ago
  It's a clustered service that is behind an SSL Load Balancer[1] that is behind Cloudflare. So not really a viable DDoS target. [1]: https://cloud.google.com/load-balancing/docs/ssl/ https://cloud.google.com/load-balancing/docs/ssl/

649 points by jhgg 8y ago

I made several websocket-based apps that I want to turn into Electron desktop apps where people can use them only with their friends, but WebRTC has been really confusing and hard to know how to get started with. Especially considering ideally I don't want to have to host a server, but all the P2P JavaScript libraries that I found seem to assume you'll at least have a server for hosting the lobby to look for peers.

jrcii 8y ago
Look into the DHT protocol
ty_a 8y ago
Iirc there are some free servers you can use provided by Google and/or twilio. Edit: STUN is free, Turn is not. Been a while since I worked with those.
chuckhays 8y ago
WebRTC requires some sort of external communication channel to exchange information about how to set up the connection. (Often referred to as signaling messages). Disclaimer: Google employee, work on the WebRTC team.
- marknadal 8y ago
  Grandparent: After getting frustrated with all the other WebRTC libraries over the last 4 years, I finally wrote our own ( https://github.com/amark/gun/blob/master/lib/webrtc.js https://github.com/amark/gun/blob/master/lib/webrtc.js ) which is capable of using a set of decentralized DHT relay-peers in GUN for signaling - and once peers are already on WebRTC, they can signal (daisy-chain "DAM" as we call it) to other WebRTC peers via WebRTC! Meaning, you don't/won't have to run any servers!!! Ping us on our chatroom if you want the list of DHT peers. Parent: Chuck! Sounds like you'd be a very useful person to know. Me, Feross, etc., plenty others have been requesting additional API/protocol access over the last 4 years (some of which FireFox is adding in libdweb extension!). Any chance we could connect and chat? Ping me at mark@gun.eco ?
  - anderspitman 8y ago
    Would it be a problem to keep a list of DHT peers in the git repo?
    - marknadal 8y ago
      Good idea. I need to get approval they are OK with their peers being publicly listed (easy DDoS potential, we don't have code yet to [but working on it] mitigate it). I'll start one with just mine, and then others can PR to opt-in. You want to join?
      - anderspitman 8y ago
        Awesome, thanks. I'm not actually using WebRTC (or GUN) yet, but I hope to for a project relatively soon and the possibility of not requiring a STUN/ICE server is very enticing to me.
  - Orphis 8y ago
    If you need to request more APIs, you should try to talk about your usecase to the WebRTC working group using the users mailing list ( https://groups.google.com/forum/#!forum/discuss-webrtc https://groups.google.com/forum/#!forum/discuss-webrtc ). Depending on the question, you may also ask on the spec Github project ( https://github.com/w3c/webrtc-pc https://github.com/w3c/webrtc-pc ). Source: Google employee, in the WebRTC WG.
eboyjr 8y ago
You will need some sort of signal transport for sending ICE negotiation data in JSON. Because the two peers need to know how to coordinate how to connect to each other before actually connecting. It can be a websocket server, REST API, email, QR codes, carrier pigeon, etc. Also in general, I highly recommend webrtc-adapter.
Klathmon 8y ago
The fact is that you can't easily do the last part without a central server. There might be some public or semi-public servers for this kind of thing available somewhere, or the alternative that I played with a few years ago was to compress and base64 encode the connection information into a string, and allow users to share the link with friends via whatever method they want, and that can then be expanded client side and used to establish the connection. (Or something like qr codes I've also seen used) Sadly I never finished that project, so I don't really have any code to show you, but in theory it should work okay.
- sdegutis 8y ago
  I mean if everyone already has a way to contact their friends (discord) and say "hey let's play this game together", sharing that link does away with the need for a coordination server. But there's still the need for at least a server, even if it's just one of the clients that knows how to take charge (like Minecraft).
  - Klathmon 8y ago
    Yeah, I skipped over a lot of the details, but you would need someone to choose to be a "host" (at least in terms of a "host" for the connection signaling stuff), but you can also implicitly make it the person who created the link safely.
    - derefr 8y ago
      If everybody is behind NATs (without uPnP privileges), nobody can be the "host" for connection-signalling purposes.
      - Klathmon 8y ago
        There are public ICE/STUN servers that can facilitate NAT-punching, google even hosts one. But that's not the aspect that I thought the root comment was talking about, but more the aspect of "A is hosting a room called 'stuff', and B and C want to browse the list of rooms and connect to it".
stcredzero 8y ago
For my game, I wrote a simple UDP to WebRTC datachannel proxy in NodeJS, suing node-electron, webrtc-electron and SimplePeer. My proxy only works downstream, with the upstream signalling using websockets. I'm going to move over to an all WebRTC architecture, however. https://www.emergencevector.com https://www.emergencevector.com
- kayoone 8y ago
  Interesting, i am working on a realtime action game and currently implemented multiplayer via websockets including authorative server, client side predicition and server reconciliation which works failry well but based on the nature of TCP, i still sometimes get some visible stutter in the movement. I guess with WebRTC you can handle things more like traditional UDP? What are your experiences vs Websockets? Did you write how you did it somewhere? Would be very interesting
  - stcredzero 8y ago
    I got really big stutter with head of line blocking on TCP. You can make things better by conspiring to send only 1 packet messages of about 512 bytes or less, and not too often. You can register for an account on my server. WebRTC downstream seems to be working at the moment. Be sure to read the instructions, however. https://www.emergencevector.com https://www.emergencevector.com I am thinking of turning my game server into a serverless PaaS offering. I'm open to collaborating on this.
    - kayoone 8y ago
      i emailed you regarding this ;)
nine_k 8y ago
There's a number of decentralized networks that could send the connection info without a single central server, e.g. torrents or Dat. OTOH the time required to connect to them and pass the info would be quite noticeable. A single connection-initiation server could be the easiest solution. It won't need to withstand a heavy load.
jessaustin 8y ago
There are probably some problems with this idea, but why not just use a popular social network or similar as the lobby? For example, if there is a twitter feed "@MyWebRTCLobby" that everyone follows, clients could just @ a tweet at it with current contact details.
- rcraft 8y ago
  If random Twitter users send @mentions to "@mywebrtclobby" then no other users will likely see these tweets unless they already follow each of the various Twitter users sending the @mentions. However if you connected @mywebRTClobby with https://GroupTweet.com https://GroupTweet.com you could configure things so that any @mentions (from authorized users or anyone) would be converted into actual tweets from the @mywebRTClobby account so that all followers would actually see those tweets/contact details.
  - jessaustin 8y ago
    Haha I don't use Twitter so I'm not surprised I got something wrong... thanks for fixing that bug.
    - jsjohnst 8y ago
      It used to be the case you could see all @mentions someone received, but alas not anymore.
matthewaveryusa 8y ago
You need a STUN/TURN server, that's just how the cookie crumbles unfortunately. Blame NAT :) I used this dockerized coturn server successfully: https://github.com/strukturag/docker-webrtc-turnserver https://github.com/strukturag/docker-webrtc-turnserver

jrockway 8y ago

I'd be interested to see how the system works during high-load events. It sounds like if a server gets overloaded, it shuts down and moves the clients to the next non-overloaded server. If the health-check used by the load assignment server is not perfectly tuned, then that failover overloads that server, shutting it down. Eventually you have no servers left. I suppose that a perfect health check will prevent this, since the failover will assign failover traffic at exactly the level that the new server can successfully handle. But if it's wrong on the other side (rejects connections when capacity actually exists), then compute resources are wasted "just to be safe". I imagine that estimating capacity is even more difficult since people can join and leave at any time, and the client doesn't send any packets when there is silence. So the load changes based on how talkative people are being (which means your server always crashes during the best parts of whatever you're discussing). Anyway, I'm wondering how this all compares to the naive strategy of "pick a random server for this channel, if it crashes, bad luck".

qmarchi 8y ago
> ... and the client doesn't send any packets when there is silence. Yes and no, there's no packets being passed over the WebRTC connection, but the server maintains a WebSocket connection for state changes. There's two, one to the guild service, which handles assignment of the voice service. Want to count the number of clients connected to a voice server? Count the number of connections. All Discord clients are connected to Guild services and discord publishes events for voice channel updates to all clients in the guild. (Ex. when a person joins a voice channel.) Also, for larger discord guilds, there are exclusive voice server pools for them to consume.[1] These servers are configured with more resources and are usually pretty close to exclusive to the single guild. Most of the voice servers are for the millions of other guilds though. [1] https://discordapp.com/partners https://discordapp.com/partners
ivanfon 8y ago
I remember in the past the Discord devs have moved particularly overloaded servers to their own dedicated hardware to improve performance.
jhgg 8y ago

jorams 8y ago

Slightly off-topic, but: > For clarity, we will use the term “guild” to represent a collection of users and channels — they are called “servers” in the client. The term “server” will instead be used here to describe our backend infrastructure. I still don't understand why you ever chose the name "server" to refer to something that is many things, but not a server. Guild is a way better name, and doesn't need disclaimers about your self-chosen terminology being confusing.

jczhang 8y ago
I believe this is a tradition that has been grandfathered into voice chat apps. Previous ones like Ventrilo have you actually connect to a private server (that someone pays for / hosts).
- elefanten 8y ago
  I second this explanation. Additionally, from a UX perspective, if a user wanted to join your server you had to give them IP/port/pw and they'd type it in manually. So even the users were accustomed to addressing servers. "Server" entered the community lexicon as "the place we chat".
- ObsoleteNerd 8y ago
  I assumed it was based on IRC servers, because each Discord server can have multiple "channels" just like IRC.
- androsyang 8y ago
  This is most likely the case. Discord's introduction was mainly to pull in the gaming audience using Skype while providing the quality behind Mumble and Ventrillo
savanaly 8y ago
I'm a programmer who well knows what a server is to a programmer, but I'm also a gamer and Discord calling its servers servers never seemed weird to me. From the perspective of the user, the term makes sense. The servers have users, those users have permissions, they host files, they host chat, you can install plugins to them.

hfourm 8y ago

This comment isn't really related. I obviously love Discord and have been using pretty much since it became an option, but I wish there was a better UX for people who are in TONS of channels/servers. I find it hard to navigate all my current servers with just the icon -- especially when server owners are changing their avatars frequently.

anafh83 8y ago
I think their UX is awful, and their client is very slow in channels with thousands of users. They should open up their protocol so alternative clients can be created, or at least some IRC gateway. I haven't made many calls but the sound was always crisp.
- scrollaway 8y ago
  I agree on opening the protocol but if what you want is a gateway, it's already open enough for that. https://github.com/42wim/matterbridge https://github.com/42wim/matterbridge
- marksomnian 8y ago
  > open up their protocol Both the Rest API (that official clients use internally too), as well as the real-time WebSocket protocol are described here: https://discordapp.com/developers/docs/reference https://discordapp.com/developers/docs/reference https://discordapp.com/developers/docs/topics/gateway https://discordapp.com/developers/docs/topics/gateway
  - Ycros 8y ago
    Unfortunately you can't use these APIs to create third party clients because that violates their ToS, they are meant only for writing bots. If they catch you using these APIs with your full account credentials they'll ban your account.
    - tenryuu 8y ago
      They can't catch you using these apis since these are the same apis the first party client runs off of

Karrot_Kream 8y ago

I really wish there were non-browser implementations of WebRTC. So far, it seems like the standard as-it-is is defined by browser code and browsers contain the canonical implementations.

brian-armstrong 8y ago
Does https://webrtc.googlesource.com/src https://webrtc.googlesource.com/src not count?
- Karrot_Kream 8y ago
  Right, so it's a chromium C++ repo that you need to create bindings for to interact with. I have no doubt that an organization like Discord has the engineering resources to either write their own bindings to WebRTC or to roll their own implementation of the components, but if I'm a an individual developer that wants to interact with WebRTC without a browser, then it's pretty difficult. One common desired use case is using WebRTC for p2p torrent communications. Right now the best way to do this is in browser, or to use an Electron app that can bridge the WebRTC clients with the standard desktop torrent clients.
  - brian-armstrong 8y ago
    Ah that makes sense. I have seen forks that de-chromium it some that look pretty reasonable. Most of it doesn't depend on anything chromium, just happens to use its build system (for better or worse). But I believe the fork switches it to CMake
  - jgrowl 8y ago
    There's nothing tied specifically to the browser, but it is written in C++. It seems like the discord folks just built a layer on top of it. It's been a while since I played around with it, but you just have to build the peerconnection library. This guide seems pretty close to what I remember having to do: https://webrtc.org/native-code/development/ https://webrtc.org/native-code/development/ It still leaves you writing in c++, but it lets you build a self contained server without any browser stuff. There also were java bindings which is what I used when I was experimenting with it. You just have to build your .so and .jar files. EDIT: Here's some scala code that I wrote that uses the native bindings. (Please forgive the messy test project that I have abandoned). https://github.com/jgrowl/livehq/blob/master/media/src/main/scala/tv/camfire/media/webrtc/WebRtcHelper.scala https://github.com/jgrowl/livehq/blob/master/media/src/main/...

ReidZB 8y ago

I really wish Discord would implement an audio compressor (or if they use one, make it much more aggressive) for voice chat. For example, Mumble doesn't require manually adjusting anyone's volume - instead, it uses a compressor to automatically normalize volumes for everyone. In contrast, Discord has a volume slider that I end up having to adjust for most people. That requires manual tweaking, often while I'm playing a game, and isn't future-proof; just the other day, one of my friends changed his microphone setup and was extremely loud (I had him at 200% when in the new setup he should've been at about 75%). This is really my only complaint with Discord's VOIP functionality. It is a major step backward from Mumble. (Also, for some reason the Discord community mostly uses voice activated mic instead of push-to-talk, but I don't think that's Discord's fault.)

iMerNibor 8y ago
> voice activated mic Which is a very sane default for most people The only reason you'd want to use push to talk is either you have a lot of people in the channel, you have a loud background or you don't want people hearing stuff not designated for the discord voice (if you're streaming, have irl people in the same room - that kinda stuff)
- r_smart 8y ago
  Personally, I hate the sound of people chewing and smacking their lips directly in my ear as they work their way through their 'fresh that afternoon' gas station nachos. But that's just me :)
  - Zekio 8y ago
    if you are in control of the Discord server you can force push to talk in voice channels
  - LinuxBender 8y ago
    It is not just you. I know of several gaming sites that will block or ban people for leaving their mic open or making breathing sounds into it.

Reedx 8y ago

> we have seen 1000 people taking turns speaking Wow. Has anyone here been in a channel like that? I'd imagine it to be complete chaos. But I suppose it's possible with good moderation and/or bots to ensure people take turns? Is that what they do?

wwwigham 8y ago
Yeah. Typically it starts with a priority speaker giving a speech, ends with a Q&A moderated via text chat.
Operyl 8y ago
There's usually some sort of moderator, and a bot to facilitate turn taking, at least in the channels that I had these in :).
crystalPalace 8y ago
I've been in a couple channels with several hundred people. The admins handled it by shouting until everyone was quiet, enforcing one person speaking at a time, and swiftly muting, moving, or banning anyone who disobeyed. I'm not sure if Discord's API is flexible enough for a bot like that.
- Operyl 8y ago
  The API is flexible enough for a bot like this, I'm contemplating opensourcing mine at some point! :)
  - zrobotics 8y ago
    Please do, it would be an actual act of charity! I've left a lot of interest i 8 servers due to the voice chat being clogged.
Kagerjay 8y ago
1000 people taking their turn speaking? It must be some political rally or reddit discord channel for a large national outbreak news story All I know is that official fortnite (discord's largest gaming server last year AFAIK) had serious amounts of disconnect issues, only on that server though Fortnite didn't have more than 100 battlerooms (4 players max) at any given point in time. There's a few battle royale scrimming servers with 70-100 players in one room for tournaments. But the rule is everyone needs to be mute & deafened though

merb 8y ago

OT: actually what I always wonder, is how Discord will make money. I mean they offer this huge free service for everybody (2.6 million concurrent voice users) over 1000 servers that must be a huge GCloud invoice

meowface 8y ago
I imagine they make most of it through Nitro: https://discordapp.com/nitro https://discordapp.com/nitro
vertex-four 8y ago
They’ve just started a game store. Encouraging integration of their social features into games will allow them to take a healthy bite out of Steam’s pie.
- Godel_unicode 8y ago
  Related: steam has built in voice chat now. https://www.engadget.com/2018/06/12/steam-finally-adds-voice-chat-flexible-friend-lists-new-beta/ https://www.engadget.com/2018/06/12/steam-finally-adds-voice...
  - maemilius 8y ago
    Steam has had voice chat (group and 1-on-1) for a much longer period of time than that. I've used it on a regular basis for the last 5 years or so, well before the current updates they've made. What they added was voice chat to static groups[0], whereas before you had to invite everyone to a custom group chat every time. Definitely a QoL improvement, but not exactly "new". 0: https://steamcommunity.com/search/groups https://steamcommunity.com/search/groups
    - fro0116 8y ago
      The new chat is a huge improvement over the old one in terms of UX, but lack of persistent history is pretty pathetic considering the amount of resources Valve has available compared to a startup like Discord.

Siecje 8y ago

Does anyone have a solution to prevent notifications when a server creates a new channel and then I am notified for every message until I mute the channel.

ihuman 8y ago
You can mute the server, or change the default notification setting to @mentions only for the server

Operyl 8y ago

If you skip the use of ICE, why is it that 25% of the time I get a message relating to ICE and it gets stuck there? "CONNECTION_STATUS_ICE_CHECKING" ICE Checking.

GranPC 8y ago

> Instead of DTLS/SRTP, we decided to use the faster Salsa20 encryption. In addition, we avoid sending audio data during periods of silence — a frequent occurrence especially with larger groups. I wonder whether this would help an attacker infer voice data, with a method similar to the one from the paper "Uncovering Spoken Phrases in Encrypted Voice over IP Conversations" [1] [1]: http://www.cs.unc.edu/~fabian/papers/tissec2010.pdf http://www.cs.unc.edu/~fabian/papers/tissec2010.pdf

EvangelicalPig 8y ago
I've never tried doing pcaps with Discord voice chat yet but it would be interesting "estimating" what was said best on the amount of voice traffic transferred. Might be harder if it still looks like standard Discord TLS "control traffic" blending in though, no?
- wahern 8y ago
  Keystroke timing analysis has some solid research. See, e.g., Timing Analysis of Keystrokes and Timing Attacks on SSH (https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf), Don’t Skype & Type! Acoustic Eavesdropping in Voice-Over-IP (https://arxiv.org/pdf/1609.09359.pdf https://arxiv.org/pdf/1609.09359.pdf). Deciphering content from latencies in packetized speech is likely much more difficult, but I wouldn't put much stock in it being too difficult. Which is to say, if you're transferring high-value information assets over VoIP you should probably assume it's decipherable. That doesn't mean you should change what you're doing. You could simple say, "M'eh, I'll worry about it when it becomes a thing." But I wouldn't assume it's confidential to someone willing to invest the time to target and capture the conversations. And I might leave a few choice comments in the source code and documentation so nobody could excuse imprudent reliance on confidentiality with, "But nobody warned me".
cvwright 8y ago
First author of the linked paper here. It's an interesting question. Actually the voice activation (on/off periods of sending vs silence) was the first thing we looked at in that project. There's definitely some information leakage there, but it was really hard to learn anything meaningful from it. The problem seems to be that long strings of words all get lumped into a single activation. It's really hard to discern anything about what those words are, or even what language they come from. We got some very weak results on language detection from VAD. You can learn something about the language, but it's not very precise. For example, maybe you could tell that a given conversation is definitely not language A, B, or C, but it might still be language X, Y, or Z.

anderspitman 8y ago

For those interested in open source/decentralized alternatives to Slack/Discord/etc, I've had a pretty great experience with both the voice and video integrated into Riot.im[0] (uses Matrix protocol[1] for chat, and Jitsi[2] for voice/video). I've said before that I can't really justify nonfree software for something as simple as text chat. That said, rooms with very large numbers of people trying to communicate with voice/video is one use case where maybe it makes more sense for a commercial product to solve it. [0] http://riot.im/ http://riot.im/ [1] https://matrix.org/blog/home/ https://matrix.org/blog/home/ [2] https://jitsi.org/ https://jitsi.org/

singularity2001 8y ago
Why is the riot "TRY now" link so damn hidden? Two clicks, one long search here https://riot.im/app/#/home https://riot.im/app/#/home No private room? Not working without google? Very bad first impression :(
- anderspitman 8y ago
  Not sure what you mean by not working without google?
  - hegz 8y ago
    Iirc there is a captcha on sign up. Same as discord.
- Arathorn 8y ago
  isn’t the “try now” link the big flashing button labelled “launch now!” on http://riot.im http://riot.im?
  - anderspitman 8y ago
    That's how I always start it. Unfortunately that link is on a carousel for some reason, which automatically advances. Definitely not ideal UX in my opinion.

orliesaurus 8y ago

Can anyone who works on the discord app explain to me: How am I impacted for having (being part of) let's say 100 server (guilds) on my app versus having 1? Should I quit the discords that I don't use? I feel like they stay in sync in the background and if I am in 100 servers and they all post say tons of text I will get massive lags. Does that make sense? Is there a way to prioritize only the server I have active/I am talking in and freeze the rest while im full-screen/in-game/focused?

highesttide 8y ago
I'm pretty sure there isn't a ton of background fetching asides from marking servers as unread, it can take a bit to actually load the messages when you click an inactive server, and it won't have any new messages if you don't have a connection (even if it had a connection when it was sent)
b1naryth1ef 8y ago
We've put a lot of work into our clients and our backend to make sure the impact of being on 1 vs 100 guilds is negligible. It helps that most of the folks building Discord are power users in a bunch of servers (so we feel the pain of poorly optimized paths early). Generally if you don't look at a server often it shouldn't effect the performance of the app / bandwidth usage. I think we have some more blog posts in the works regarding some of these topics so look forward to those!
- ehsankia 8y ago
  If a guild is completely muted (including @everyone), and I don't open it, do you have a rough estimate of how much resources it uses? In theory, it should be zero, but in practice, does is the server sending me updates about messages even if I'm muted?
  - jhgg 8y ago
    It depends on the activity of the server. But generally, you're only receiving message create events and updates to the server. But you aren't receiving things like member list updates, presence or typing events from the server until you focus it initially. Additionally, we unsubscribe you from these events if you unfocus the server for a given amount of time.

fulafel 8y ago

"Instead of DTLS/SRTP, we decided to use the faster Salsa20 encryption." This is just confused and sounds quite worrying :I

fulafel 8y ago
Judging by the votes, this may need spelling out: Salsa20 is a stream cipher. DTLS and SRTP are higher level security protocols that use ciphers (among other things) as building blocks, to ensure things like replay protection, integrity protection, mutual authentication, secure session key agreement and forward secrecy in addition to confidentiality. If you replace an engineered security protocol with a raw cipher and key, you create many vulnerabilities and make a much less secure system. VOIP applications are especially vulnerable to replay attacks, for instance. A better approach would be to continue using DTLS and SRTP, and use Salsa20 as the SRTP cipher. Non-crypto-experts saying things like "we replaced <estabilished security protocol> with <my own idea> and it's much faster" is a well known bad sign - It often indicates the person has fumbled things without realizing it.
- Orphis 8y ago
  They say they have done it for performance. I am wondering if they wanted to optimize for the client performance or the SFU's.

bg0 8y ago

My buddies and I use Discord as an easy conference call group chat while playing games (no push to talk, mic is always on). One thing I've found on the app is when a person/voice is far away from the mic it often glitches and doesn't send all of the audio data over like, for a example, a normal FaceTime Audio call would. Not sure why this happens, but just thought I'd put that out in the ether.

gizmo385 8y ago
Sounds like you just need to adjust your microphone sensitivity settings? If you have the mic on voice activation and the person is far from their mic than it might be that they're just not loud enough (from the perspective of their microphone) to actually trigger Discord. That's my guess at least.
- bg0 8y ago
  Ah, never thought of it like that. I always perceived it as a "always on" scenario. Just found it in the settings and I'll give it a try, thank you!

quackerhacker 8y ago

So I have an affection for Discord...so I appreciate posts like this. >"Using the WebRTC native library allows us to use a lower level API from WebRTC (webrtc::Call) to create both send stream and receive stream." So I'm gathering that discord's voice servers receive multiple persistent connections, then compress the audio streams for delivery to each end user. THIS part is where I can't imagine the on-the-fly cpu usage. Each client's receiving compression needs to also negate their own audio to prevent an echo effect (no point to hear your own voice), but it also means separate compression streams per user. >" All the voice channels within a guild are assigned to the same Discord Voice server." I imagine this helps significantly with I/O in converting live streams into 1 stream per end user. I've dealt with video compression (only in ffmpeg) and live syncing time stampings, and I can say from experience that, this is no easy feature. I understand this is audio streams (so lower overhead), but still the persistent voice server needs to handle the incoming connections, web socket heartbeats (negligible), compression (high I/O), and deliver the streams (high memory usage too). I'm impressed, but would love to hear the specs on the media servers and their DL/UL speeds. My old setup to deliver live video (in sync and compressed) was 6 mini-itx's, 4GB of ram per board, and i3's...my bottleneck was my isp, which I solved with multiple docsis modems and an internal switch (each board had 2 ethernet ports).

jhgg 8y ago
We don't transcode audio/video on the server. Each stream is processed and muxed by the client. The server is merely relaying rtp packets and tagging them with a given ssrc per peer. The client does the rest of the work. The bulk of the user-space time on the SFU is spent doing encryption (xalsa/dtls). We also avoid memory allocations in the hot paths, using fixed-size ring buffers as much as possible. Additionally, we coalesce sends using sendmmsg, to reduce syscalls in the write path: (http://man7.org/linux/man-pages/man2/sendmmsg.2.html http://man7.org/linux/man-pages/man2/sendmmsg.2.html) I posted some about the specs here: https://news.ycombinator.com/item?id=17954163 https://news.ycombinator.com/item?id=17954163
- quackerhacker 8y ago
  If Discord is basically proxying the raw packets from one client to the others, isn't that wasted bandwidth (for discord, not the clients). I understand from the post that the goal would be to mask the ip of the users, to shoulder user privacy and the ddos vector. Kudos on silence detection to save overhead. So video w/audio broadcasting has to be compressed client side, then proxied through Discord's media servers, to the end user's. That's pretty smart...I just wished that I could send my raw stream to a LAN host so I could offload the compression, and allow my LAN host to provide delivery (I'm a nitro user).

winterismute 8y ago

I love discord but am I the only one that thinks that the name is hindering wider adoption? It's hard to suggest to some people that an with that name (and to a lesser extent visual theme) is the skype+slack+others replacement they are actually looking for...

Kaveren 8y ago
I suppose you could say the same for Slack. I don't think that the connotations of the names for either particularly matter to users. What's hindering adoption outside of the gaming community is the branding and niche focus.
Vishnevskiy 8y ago
We found the name is not an issue as growth continues at a rapid pace. We are already multiple times larger than Slack :)

dep_b 8y ago

I've been reading the article hoping to find out a bit more about the actual voice server but the SFU is actually not very deeply explained at all.

jsjohnst 8y ago
That’s because there isn’t a lot to it from the sounds of it in this post: https://news.ycombinator.com/item?id=17955309 https://news.ycombinator.com/item?id=17955309
- dep_b 8y ago
  I would implement a similar dump pipe as well for a very different application. I mean the application still needs to set up the WebSockets with the clients, which isn't trivial to code yourself.

tylerplz 8y ago

> Since it’s the only service directly accessible from the public Internet, we will focus on Discord Voice server failovers. Why is the Gateway not directly accessible from the public Internet?

jhgg 8y ago
It's a clustered service that is behind an SSL Load Balancer[1] that is behind Cloudflare. So not really a viable DDoS target. [1]: https://cloud.google.com/load-balancing/docs/ssl/ https://cloud.google.com/load-balancing/docs/ssl/