3 ms·

I'm not familiar with the nitty gritty of video streaming infra but I'm curious, with the crazy array of services on AWS/most other cloud providers, what in par

I'm not familiar with the nitty gritty of video streaming infra but I'm curious, with the crazy array of services on AWS/most other cloud providers, what in particular is missing or cannot scale down cost efficiently?

everforward 4y ago
I would guess a big part is switching/routing specifically geared towards low-latency audio/video. Just as an example, they probably use UDP for latency, which has no ordering as part of the protocol. However, for streaming stuff, you really want to drop out of order packets and prioritize whatever the latest packet is. Zoom probably has some kind of custom routing that handles that. If they get congested or clients are unstable, it prioritizes serving the latest audio/video rather than trying to push things out in the order they came in, only for the Zoom client to discard the out of order packets anyways. Amazon can't really do that, because they don't know which bits of data are the ordering. On the pricing side, people are probably the component that scales the least. You're probably going to need a dedicated person to monitor/fix/upgrade it, and deal with calls from people who have issues. Between salary and benefits, that's at least a 6 figure expense. Zoom's most expensive tier is $240/license/year, so $100,000 would buy you 416 licenses. And $100,000 is a lowball here; that's roundabouts a $60k salary, and the other $40k is paid out in benefits/space/etc. This also doesn't count the bandwidth charges, nor the hardware to run the system on. Imo, it's also just generally a bad idea to DIY anything business-critical that isn't a core competency of the company. It tends to cause issues. What do you do when the person that built this bespoke Zoom alternative quits? Can you risk it breaking and not working until you can hire someone new and they figure out how it works? You could hire 2 of them, but Zoom is definitely cheaper then. What do you do if the open source project gets abandoned? It also distracts from things that actually make you money. It makes more sense to me to focus on your core line of business, and use the additional proceeds to just pay for Zoom.
jech 4y ago
There's nothing particularly difficult on the server side — a quality SFU should be capable to handle on the order of 400 video flows per core, and there are quite a few high-quality free software SFUs available (Janus, Jitsi, ion-sfu, livekit, Galene). To give some perspective: we're using Galene for lectures, and our single-CPU server uses around 40% CPU usage in a room with 120 students (who keep their cameras switched off during the lecture, of course, and only occasionally switch them on to ask questions). As the grandparent mentioned, the problem is the client side. Since there is no standard videoconferencing protocol, every free software project needs to develop their own clients. And it's difficult for a free software project to have the manpower and expertise to develop quality clients for the web, Android and iOS, so in effect what we currently have are mostly half-baked web clients. There is some hope, though. The IETF have been working on standard protocols for ingress (https://datatracker.ietf.org/wg/wish/ https://datatracker.ietf.org/wg/wish/), and if their protocols get deployed, you'll be able to use the same streaming software (think OBS) or IP camera with multiple distinct videoconferencing servers. An interoperable interactive videoconferencing protocol is nowhere near, but as more people understand videoconferencing technology, there is some hope that people will get together and start working on multi-protocol clients (remember Pidgin?). Full disclosure: I'm the author of Galene (https://galene.org https://galene.org), and I've been actively participating in the Pion community (https://github.com/pion/webrtc https://github.com/pion/webrtc) and collaborating with the authors of ion-sfu (https://github.com/pion/ion-sfu https://github.com/pion/ion-sfu) and LiveKit (https://github.com/livekit https://github.com/livekit).

everforward 4y ago

I would guess a big part is switching/routing specifically geared towards low-latency audio/video. Just as an example, they probably use UDP for latency, which has no ordering as part of the protocol. However, for streaming stuff, you really want to drop out of order packets and prioritize whatever the latest packet is. Zoom probably has some kind of custom routing that handles that. If they get congested or clients are unstable, it prioritizes serving the latest audio/video rather than trying to push things out in the order they came in, only for the Zoom client to discard the out of order packets anyways. Amazon can't really do that, because they don't know which bits of data are the ordering. On the pricing side, people are probably the component that scales the least. You're probably going to need a dedicated person to monitor/fix/upgrade it, and deal with calls from people who have issues. Between salary and benefits, that's at least a 6 figure expense. Zoom's most expensive tier is $240/license/year, so $100,000 would buy you 416 licenses. And $100,000 is a lowball here; that's roundabouts a $60k salary, and the other $40k is paid out in benefits/space/etc. This also doesn't count the bandwidth charges, nor the hardware to run the system on. Imo, it's also just generally a bad idea to DIY anything business-critical that isn't a core competency of the company. It tends to cause issues. What do you do when the person that built this bespoke Zoom alternative quits? Can you risk it breaking and not working until you can hire someone new and they figure out how it works? You could hire 2 of them, but Zoom is definitely cheaper then. What do you do if the open source project gets abandoned? It also distracts from things that actually make you money. It makes more sense to me to focus on your core line of business, and use the additional proceeds to just pay for Zoom.

jech 4y ago

There's nothing particularly difficult on the server side — a quality SFU should be capable to handle on the order of 400 video flows per core, and there are quite a few high-quality free software SFUs available (Janus, Jitsi, ion-sfu, livekit, Galene). To give some perspective: we're using Galene for lectures, and our single-CPU server uses around 40% CPU usage in a room with 120 students (who keep their cameras switched off during the lecture, of course, and only occasionally switch them on to ask questions). As the grandparent mentioned, the problem is the client side. Since there is no standard videoconferencing protocol, every free software project needs to develop their own clients. And it's difficult for a free software project to have the manpower and expertise to develop quality clients for the web, Android and iOS, so in effect what we currently have are mostly half-baked web clients. There is some hope, though. The IETF have been working on standard protocols for ingress (https://datatracker.ietf.org/wg/wish/ https://datatracker.ietf.org/wg/wish/), and if their protocols get deployed, you'll be able to use the same streaming software (think OBS) or IP camera with multiple distinct videoconferencing servers. An interoperable interactive videoconferencing protocol is nowhere near, but as more people understand videoconferencing technology, there is some hope that people will get together and start working on multi-protocol clients (remember Pidgin?). Full disclosure: I'm the author of Galene (https://galene.org https://galene.org), and I've been actively participating in the Pion community (https://github.com/pion/webrtc https://github.com/pion/webrtc) and collaborating with the authors of ion-sfu (https://github.com/pion/ion-sfu https://github.com/pion/ion-sfu) and LiveKit (https://github.com/livekit https://github.com/livekit).