6 ms·

I tried this myself too and when I try p2p with 4 people, out of 10 tests about 50% of the time I won't be able to see all 4 people or someone wouldn't be able

I tried this myself too and when I try p2p with 4 people, out of 10 tests about 50% of the time I won't be able to see all 4 people or someone wouldn't be able to see all 4 people. It was really hard to make p2p work and debugging the ice connections was even harder.

moron4hire 5y ago
That's probably Network Address Translation (NAT), which requires TURN (a fancy name for a central relay for all media) to "punch through". TURN literally stands for "Traversal Using Relay around NAT". And it's just a traditional, centralized. non-p2p fallback for people on paternalistic networks that don't allow them to create UDP connections or TCP connections on any ports other than 80 or 443. Which, as it turns out, is a lot of users. I've seen estimates in the range of 10 to 20% of users. Which means, for a random selection of 7 users, you pretty much have a 50/50 chance of not being able to peer everyone using just STUN.
- kwindla 5y ago
  I think it is likely also bandwidth and cpu issues with mesh peer-to-peer. Unless you're capping the video bitrate, the browser will try to use whatever the browser's default target is, for each connection. On Chrome that's 3mb/s, which is a lot of network bandwidth, and turns out to be a lot of cpu as well just shuffling those packets through the encoding->sending->bandwidth-estimation and receiving->decoding->rendering pipelines. Capping the video bitrate is more complicated and confusing than it should be. It's better now that the browser implementations are all more or less closing in on "WebRTC 1.0" compliance. But you still need to reach into either the raw SDP you are exchanging during signaling, or the RTCPeerConnection objects, and set the encoding bitrate target. The SaaS platforms that offer WebRTC APIs and infrastructure all do a lot of work under the covers to set bitrate caps, track constraints (resolution, for example), and other bits and pieces of WebRTC config that work well on a wide variety of networks, devices, and browsers.
- throw14082020 5y ago
  I always assumed everyone is behind NAT, you're saying on 10 to 20% of people are, and therefore only they need TURN. I'd love to see where you got that number. If I were to guess, the problem GP is facing is bandwidth, a mesh network uses exponentially more bandwidth. For each user, the bandwidth is linear, N more people requires N more bandwidth. This is fine for downloads, but uploading N more can be much more challenging for certain networks.
  - cma 5y ago
    He's mistaken that NAT always requires TURN. Consumer NAT typically still allows incoming UDP, using STUN/punch-through, or TCP with uPNP support. He maybe meant to talk about only about more restrictive NAT situations or campus/corporate/ISP/nation-state/scientology-compound firewalls.
    - throw14082020 5y ago
      Im not sure about UDP hole punching and how it relates to WebRTC, i don't see it being talked about much. In general hole punching is a rare thing to hear, I cannot find much resources about it. And uPNP (Universal Plug and Play) sounds like its for device discovery in the same local network, so again, it doesn't sound related to webRTC, we can connect directly with each other on the same local network anyway.
      - mypalmike 5y ago
        Hole punching in the context of webrtc is usually referred to as ICE. UPNP has a number of functions, including forwarding of WAN packets to a specific LAN device.
  - deleted 5y ago
    [deleted]
  - mypalmike 5y ago
    The 10-20 percent would reflect people with symmetric NAT, which is rather common with mobile networks, corporate NAT, etc. Symmetric NAT requires TURN relays. Typical home router configurations are not symmetric NAT, and usually work with just STUN.
- Sean-Der 5y ago
  The only numbers I have ever seen published are Whereby's[0] they saw 17% used TURN. There is a little more nuance then just paternalistic networks though. In same cases like NAT Mapping exhaustion you just can't give an individual user multiple long lived mappings. Address Dependendent filtering/mapping also makes sense in some cases. It makes P2P harder, but does give you the ability to provide your users more sessions at least! https://medium.com/the-making-of-whereby/what-kind-of-turn-server-is-being-used-d67dbfc2ff5d https://medium.com/the-making-of-whereby/what-kind-of-turn-s...
  - kwindla 5y ago
    We see about the same numbers Whereby does at Daily, globally across our whole user base. Bounces around a little but is usually just under 20%. Way more for customers that are mostly serving corporate users, of course (firewalls). And more for mobile-heavy user populations. Actually, that's a good reminder that it would be nice to understand the mobile data networks breakdown in more detail. Most of the US mobile data networks require TURN, as far as I remember when I last looked at this. But I don't know if that's true everywhere in the world.
    - Sean-Der 5y ago
      That is awesome, thanks for sharing :) Lots of little details and they all effect each other. I really enjoy networking because of this. I wonder if we come back to this in 10 years what this number will be. Linux on the desktop and IPv6 is just around the corner...
      - mekkkkkk 5y ago
        Thinking about it, v4 addresses and oil have a lot in common. The exhaustion/depletion is coming, but we keep finding ways to circumvent it. For oil you got fracking and sand. For v4 you see wider adoption of wide/carrier-grade NAT. In both cases it temporarily solves the supply problem really effectively. However it also ruins the environment.
deneb150 5y ago
Debugging this is indeed quite hard. I have only used it for 1 on 1 p2p calls.
Sean-Der 5y ago
WebRTC + networking is frustrating. IMO it is a leaky abstraction. There was a hope that ICE+TURN would work everywhere and users would never need to worry. That isn't true so we need to do a better job educating developers about what/why things went wrong. I am working on a Open Source book that includes a WebRTC networking chapter[0]. Would love your opinions/feedback if this would have actually been helpful when learning this stuff! [0] https://webrtcforthecurious.com/docs/03-connecting/ https://webrtcforthecurious.com/docs/03-connecting/
- parhamn 5y ago
  Wonderful and very well written, thanks! I too experimented with a p2p golang webchat setup. All the jargon was confusing and very hard to look up. This post has already given me much more clarity!!
VWWHFSfQ 5y ago
iOS and Safari is riddled with WebRTC bugs like this. Sounds similar to my experience. Everything consistently works great in Chrome and Firefox and then only kinda works Safari. worst browser on the planet
- nobleach 5y ago
  I've jokingly referred to Safari as SafarIE for the last 5 years. It does tick all these boxes: 1. Backed by an OS manufacturer that doesn't care about the web 2. Spends more time working on features that suit itself than meeting standards agreed upon by a body of which they're a part. 3. The only sanctioned/allowed browser on their platform (MS didn't even achieve this holy grail) 4. Lagging behind most other popular browsers by years in some cases But due to it being the ONLY browser that'll run on iOS, I have no choice but to dumb down user experience for it. This year's lovely issue has been MediaRecorder - but supposedly that's made it into the most recent release.
  - mguerville 5y ago
    I use firefox on iOS, it’s even set as my default browser. Curious what you mean by “safari is the only browser that works in iOS”? For in browser video chat? Genuinely curious, what are the limitations?
    - danielheath 5y ago
      Firefox on iOS uses the safari js runtime and the safari layout engine.
      - mguerville 5y ago
        Ah I see, thanks I had no idea (I assume most average users are in the same situation). I assume IpadOS is the same, and only MacOS allows a true non safari browser.
    - VWWHFSfQ 5y ago
      Firefox and chrome on iOS are just re-skins of safari. Apple doesn't allow other browser backends
yawnxyz 5y ago
I thought I was just dumb for not being able to figure this out, but turns out it's just a hard problem for everyone haha
- 6510 5y ago
  Yeah, it made me feel like I had to read the manual but there wasn't any. Seeing such a bad one reminded me how spoiled we are with beautiful abstractions.
meheleventyone 5y ago
The worst one for me so far is mDNS not working on my local network so the one circumstance you should basically be able to guarantee an easy P2P connection doesn’t work.

moron4hire 5y ago

That's probably Network Address Translation (NAT), which requires TURN (a fancy name for a central relay for all media) to "punch through". TURN literally stands for "Traversal Using Relay around NAT". And it's just a traditional, centralized. non-p2p fallback for people on paternalistic networks that don't allow them to create UDP connections or TCP connections on any ports other than 80 or 443. Which, as it turns out, is a lot of users. I've seen estimates in the range of 10 to 20% of users. Which means, for a random selection of 7 users, you pretty much have a 50/50 chance of not being able to peer everyone using just STUN.

kwindla 5y ago
I think it is likely also bandwidth and cpu issues with mesh peer-to-peer. Unless you're capping the video bitrate, the browser will try to use whatever the browser's default target is, for each connection. On Chrome that's 3mb/s, which is a lot of network bandwidth, and turns out to be a lot of cpu as well just shuffling those packets through the encoding->sending->bandwidth-estimation and receiving->decoding->rendering pipelines. Capping the video bitrate is more complicated and confusing than it should be. It's better now that the browser implementations are all more or less closing in on "WebRTC 1.0" compliance. But you still need to reach into either the raw SDP you are exchanging during signaling, or the RTCPeerConnection objects, and set the encoding bitrate target. The SaaS platforms that offer WebRTC APIs and infrastructure all do a lot of work under the covers to set bitrate caps, track constraints (resolution, for example), and other bits and pieces of WebRTC config that work well on a wide variety of networks, devices, and browsers.
throw14082020 5y ago
I always assumed everyone is behind NAT, you're saying on 10 to 20% of people are, and therefore only they need TURN. I'd love to see where you got that number. If I were to guess, the problem GP is facing is bandwidth, a mesh network uses exponentially more bandwidth. For each user, the bandwidth is linear, N more people requires N more bandwidth. This is fine for downloads, but uploading N more can be much more challenging for certain networks.
- cma 5y ago
  He's mistaken that NAT always requires TURN. Consumer NAT typically still allows incoming UDP, using STUN/punch-through, or TCP with uPNP support. He maybe meant to talk about only about more restrictive NAT situations or campus/corporate/ISP/nation-state/scientology-compound firewalls.
  - throw14082020 5y ago
    Im not sure about UDP hole punching and how it relates to WebRTC, i don't see it being talked about much. In general hole punching is a rare thing to hear, I cannot find much resources about it. And uPNP (Universal Plug and Play) sounds like its for device discovery in the same local network, so again, it doesn't sound related to webRTC, we can connect directly with each other on the same local network anyway.
    - mypalmike 5y ago
      Hole punching in the context of webrtc is usually referred to as ICE. UPNP has a number of functions, including forwarding of WAN packets to a specific LAN device.
- deleted 5y ago
  [deleted]
- mypalmike 5y ago
  The 10-20 percent would reflect people with symmetric NAT, which is rather common with mobile networks, corporate NAT, etc. Symmetric NAT requires TURN relays. Typical home router configurations are not symmetric NAT, and usually work with just STUN.
Sean-Der 5y ago
The only numbers I have ever seen published are Whereby's[0] they saw 17% used TURN. There is a little more nuance then just paternalistic networks though. In same cases like NAT Mapping exhaustion you just can't give an individual user multiple long lived mappings. Address Dependendent filtering/mapping also makes sense in some cases. It makes P2P harder, but does give you the ability to provide your users more sessions at least! https://medium.com/the-making-of-whereby/what-kind-of-turn-server-is-being-used-d67dbfc2ff5d https://medium.com/the-making-of-whereby/what-kind-of-turn-s...
- kwindla 5y ago
  We see about the same numbers Whereby does at Daily, globally across our whole user base. Bounces around a little but is usually just under 20%. Way more for customers that are mostly serving corporate users, of course (firewalls). And more for mobile-heavy user populations. Actually, that's a good reminder that it would be nice to understand the mobile data networks breakdown in more detail. Most of the US mobile data networks require TURN, as far as I remember when I last looked at this. But I don't know if that's true everywhere in the world.
  - Sean-Der 5y ago
    That is awesome, thanks for sharing :) Lots of little details and they all effect each other. I really enjoy networking because of this. I wonder if we come back to this in 10 years what this number will be. Linux on the desktop and IPv6 is just around the corner...
    - mekkkkkk 5y ago
      Thinking about it, v4 addresses and oil have a lot in common. The exhaustion/depletion is coming, but we keep finding ways to circumvent it. For oil you got fracking and sand. For v4 you see wider adoption of wide/carrier-grade NAT. In both cases it temporarily solves the supply problem really effectively. However it also ruins the environment.

deneb150 5y ago

Debugging this is indeed quite hard. I have only used it for 1 on 1 p2p calls.

Sean-Der 5y ago

WebRTC + networking is frustrating. IMO it is a leaky abstraction. There was a hope that ICE+TURN would work everywhere and users would never need to worry. That isn't true so we need to do a better job educating developers about what/why things went wrong. I am working on a Open Source book that includes a WebRTC networking chapter[0]. Would love your opinions/feedback if this would have actually been helpful when learning this stuff! [0] https://webrtcforthecurious.com/docs/03-connecting/ https://webrtcforthecurious.com/docs/03-connecting/

parhamn 5y ago
Wonderful and very well written, thanks! I too experimented with a p2p golang webchat setup. All the jargon was confusing and very hard to look up. This post has already given me much more clarity!!

VWWHFSfQ 5y ago

iOS and Safari is riddled with WebRTC bugs like this. Sounds similar to my experience. Everything consistently works great in Chrome and Firefox and then only kinda works Safari. worst browser on the planet

nobleach 5y ago
I've jokingly referred to Safari as SafarIE for the last 5 years. It does tick all these boxes: 1. Backed by an OS manufacturer that doesn't care about the web 2. Spends more time working on features that suit itself than meeting standards agreed upon by a body of which they're a part. 3. The only sanctioned/allowed browser on their platform (MS didn't even achieve this holy grail) 4. Lagging behind most other popular browsers by years in some cases But due to it being the ONLY browser that'll run on iOS, I have no choice but to dumb down user experience for it. This year's lovely issue has been MediaRecorder - but supposedly that's made it into the most recent release.
- mguerville 5y ago
  I use firefox on iOS, it’s even set as my default browser. Curious what you mean by “safari is the only browser that works in iOS”? For in browser video chat? Genuinely curious, what are the limitations?
  - danielheath 5y ago
    Firefox on iOS uses the safari js runtime and the safari layout engine.
    - mguerville 5y ago
      Ah I see, thanks I had no idea (I assume most average users are in the same situation). I assume IpadOS is the same, and only MacOS allows a true non safari browser.
  - VWWHFSfQ 5y ago
    Firefox and chrome on iOS are just re-skins of safari. Apple doesn't allow other browser backends

yawnxyz 5y ago

I thought I was just dumb for not being able to figure this out, but turns out it's just a hard problem for everyone haha

6510 5y ago
Yeah, it made me feel like I had to read the manual but there wasn't any. Seeing such a bad one reminded me how spoiled we are with beautiful abstractions.

meheleventyone 5y ago

The worst one for me so far is mDNS not working on my local network so the one circumstance you should basically be able to guarantee an easy P2P connection doesn’t work.