5 ms·

I have some experience with large-scale CDN I've cofounded some years ago. The short answer is - the perceived benefits you'd receive doesn't match the decrease

I have some experience with large-scale CDN I've cofounded some years ago. The short answer is - the perceived benefits you'd receive doesn't match the decrease in quality of service. So the economics doesn't work. Content distribution is widely skewed towards 1% of content makes 90% of traffic nodes so the cases that p2p may work for very popular content but the long tail is not easily cached/served. Try to watch/download a torrent file with small number of seeders and you'll get the experience you will be providing for the end user. Also internet networking is tricky and connectivity in different parts of the world can differ widely. South America or Asia have almost none of the peering hubs infrastructure that is available in Europe. (I've cofounded a CDN company and have some knowledge of the market)

jdrc 5y ago
That sounds like a perfect match for internet content which is often viral and temporary peaks. Ideally in a P2P world creators would care to have at least 1 node running which would serve the long tail.
sitkack 5y ago
So your origin needs to be able to handle the tail and P2P can handle the popular stuff no? It isn't all or nothing. P2P could absolutely be used to peak shave. CDNs also have cold start issues with long tail. Time to first byte is pretty good, https://webtorrent.io/ https://webtorrent.io/ To the OP, you should research some other options, are you sure that you are asking the right question?
- zinekeller 5y ago
  > So your origin needs to be able to handle the tail and P2P can handle the popular stuff no? It isn't all or nothing. P2P could absolutely be used to peak shave. CDNs also have cold start issues with long tail. No, it'll be genuinely be worse, as you can't really have efficient P2P today. Let's start with the points; I'm not talking about Europe or US today. P2P or CDN is a coin flip there, and I'll agree that P2P is as reliable as a CDN. I'll also agree that theoretically P2P and CDN will be a coin flip (performance-wise, I'm leaving monetary considerations in this discussion). Those points break down quickly in Latin America, Asia and Africa. I'll talk about Asia as it is where I'm more familiar of, but the points apply also in Africa and Latin America. Outside of Jio and actual hosters/colos (meaning nearly all of residential and mobile connections), IPv6 in Asia is actually more unreliable than you think (it should not be, but that's peering fights for you) and IPv4 is CGNATTed heavily so P2P is no dice. Even if somehow there's a unique IPv4 for every device, internet connections there is very top-down, unlike with Europe's mesh-like connectivity. How bad? It'll be better to route Telkom Indonesia's connections to America than to another hoster located in Indonesia or Singapore (nearest regional hub), unless you're Akamai which have agreed to buy colocation space inside Telin just to have good connectivity. And before you comment, OVH in Singapore has the precise issues we're discussing (https://lowendtalk.com/discussion/172659/ovh-routing-issues-singapore-indonesia-singapore-philippines https://lowendtalk.com/discussion/172659/ovh-routing-issues-...). Thus, it is miles better to use your time to build good CDNs with good connections (or spend money to Akamai) than bothering with the issues you'll face with P2P. Moreover, you can concentrate your money into buying a better data link (either by renting/IRU a dedicated wavelength or even fiber pairs) than relying on a best-effort service. Funnily, even bittorent downloads in practice seems to be concentrated to a few seedboxens - even for purely legal downloads like GIMP. Postscript: technically P2P and CDN would be the best for reliability, but you're spending the time to do it (instead dealing with a single thing) and deal with the possible fallout (like Windows Updates before Microsoft decided to limit P2P to local networks due to bandwidth and privacy concerns). Speaking of bandwidth concerns, I promised myself to limit monetary concerns just to point out that's it's not good in actual deployment, but I forgot data caps (not just a problem in Africa but in the US too)! It'll be a disaster when P2P is not voluntary because you just wasted their precious money!
  - conradev 5y ago
    So, your origin needs to handle the parts of the world with bad internet infrastructure, but can use P2P in Europe and the US, then?
    - zinekeller 5y ago
      US is out because data caps. You can in Europe, but it is worth the dev time? It doesn't matter when monetary costs are second to principles, but it be wrong to think that most companies are spending their time fancying P2P.
    - giantrobot 5y ago
      Besides data caps on residential broadband, a non-trivial number of clients will be on mobile and have effectively zero ability to reseed content. To enable streaming from mobile devices requires a lot of server infrastructure to support hole poking and reflection. Even with residential broadband without worries of data caps, there's still issues of firewalls and terrible upload speeds. Hole poking in residential firewalls like on mobile requires third party infrastructure. Most end users are not readily able to forward ports on their home routers. It's also not uncommon to see 10:1 downstream to upstream ratios. There's plenty of connections that couldn't manage to reliably upload a single FHD stream to a single client. It would take a hundred seeders with Comcastic upload speeds of 10Mbps to equal a single cheap VPS on a 1Gbps connection.
- giantrobot 5y ago
  Streaming video is not like static files from a torrent. The term "popular content" is a bit of a misnomer. Nominally with streaming a single uploaded video will be encoded into a number of different versions at different frame sizes and bitrates packaged into individual stream segments. So while a particular video URL might be popular, the actual video segments served to viewers will depend heavily on the configuration/environment of all the viewing devices. So a FHD stream for a video could be super popular while a 480p version might not have any viewers. When some 480p viewers load the stream they don't get any benefit from the P2P/cache popularity of the FHD stream. This is compounded by the fact that for video time to first byte doesn't matter very much. A decoder can't do anything with a single byte or even a partial stream segment. It's the time to locally buffer a decodable stream segment and a steady reception of segments as the play head reaches their time stamp. If a segment gets held up (as happens with P2P) it can't be decoded so you either pause playback or drop frames (if you've got later segments available). You want the media segments to cover at minimum a single GOP (Group of Pictures) which is a relatively large I-frame and the following smaller P and B frames. If you were to tune the video segmentation such that each segment was some tiny time slice it would balloon the bitrate since you need way more I-frames. Super long GOPs don't help either because segment drops end up ruining playback until a new I-frame is received. Streaming video is hard enough with a reliable CDN, adding P2P makes much harder and orders of magnitude less reliable.
qwerty456127 5y ago
> Try to watch/download a torrent file with small number of seeders and you'll get the experience you will be providing for the end user. I always wondered why not combine the ways the media is served into a hybrid of P2P with the classic way by just using P2P (e.g. BitTorrent) and also seeding yourself (as the hoster). Just run a seedbox of the capacity you want/can and let the audience help you by handling a share of the load they can. Wouldn't this save you some bandwidth while keeping the same service quality?
- aasasd 5y ago
  Archive.org seems to do something like that with its downloads via torrent: it appears as a web-seed for the torrents, and I usually snatch the files from those seeds faster than DHT seeds usually show up (which kinda undermines the idea, though). Frankly, I never saw any other seeds on Archive's torrents—but one of those I snatched has a ratio of 3 now, so presumably it does work as intended.
- matheusmoreira 5y ago
  > let the audience help you by handling a share of the load So why should we help these for-profit corporations? We're supposed to just donate our bandwidth to them? Are we getting a discount in exchange or are they just gonna pocket their bandwidth cost savings at our expense? Seeding internet archive data is one thing. Letting corporations invisibly take advantage of resources we pay for is a completely different matter.
  - Retric 5y ago
    Pass some of the savings in to the customer via lower prices and it’s a win win. Alternatively, enabling P2P gives faster downloads. That’s rarely noticeable with streaming but people often want to download stuff for offline viewing.
    - matheusmoreira 5y ago
      > Pass some of the savings in to the customer via lower prices Does that ever happen? They'll probably keep prices exactly the same.
iib 5y ago
I wonder if an asymmetric scheme would work best, even though it may be unbalanced? For example, for any one popular video you watch, while you watch it, you also have to download and seed a small part of an unpopular video, even if you don't see it? Maybe bandwidth and storage caps would work better. You seed X MB, even if you don't need the full X yourself.
samwillis 5y ago
This is more or less exactly why Spotify dropped P2P from its platform back in 2014. Good summary here: https://techcrunch.com/2014/04/17/spotify-removes-peer-to-peer-technology-from-its-desktop-client/ https://techcrunch.com/2014/04/17/spotify-removes-peer-to-pe... "This is when the company took advantage of cached songs. Before today’s change, if you streamed a popular song for the first time, the client would download the song from other users, using peer-to-peer. All of this happened in the background, but it greatly contributed to making the overall user experience snappier." "Yet, now that the company has many servers, using peer-to-peer in addition to direct downloads actually adds a bit of overhead. Moreover, the company has to maintain the peer-to-peer code base, and update it with each new version."
- black_puppydog 5y ago
  > the company has to maintain the peer-to-peer code base, and update it with each new version. Really, that's (part of) their excuse? A bit lame, that. :|
  - dymk 5y ago
    What, you don’t think code has maintenance costs associated with it, just because it’s cool?
  - powerslacker 5y ago
    I might have agreed with you a few years ago but these days I see Spotify's perspective more clearly. Today I see code as a liability. Like the parts in my car it is eventually going to break in some strange way and cause me a headache. The only real solution is to never drive a car, but that's almost impossible if you want to get around. Same with code, it's a vehicle that gets a business to a destination, which is nice: bit it can also be quite expensive when things go wrong. Efficient engineering teams balance this inherent risk against the needs of the business.
    - bradknowles 5y ago
      Every line of code has a cost. The question you have to ask yourself is whether that cost is worth the return on your investment? No code is less expensive than no code.
dehrmann 5y ago
The other part is that CDNs result in less total network utilization. They limit most bulky traffic to the last miles.
- freefaler 5y ago
  It's not always so straightforward. You decrease your traffic costs, but you'll incur capex (servers, switches, connectivity) or operation costs (renting servers). Additionally outside EU network contracts can be a hassle to sign because of market consolidation or outright oligopoly. So you'd need to commit capital to get bandwidth. It's an interesting optimisation problem and sometimes you can route your traffic differently based on SNMP data, price per Mbps and route performance. The higher you pay, the better connectivity you get. But off-peak times you can route traffic via cheaper provider.
pmayrgundter 5y ago
İ also have experience building a CDN, tho not for video, and this sound right. I wonder if this might be improved by an appointment system, to stream popular titles at pre-planned times so that people could benefit from others watching at the same time. It would be like watching a broadcast television program during prime time. I guess this has been tried before but also never heard of it
- freefaler 5y ago
  The clients that do plan in advance are more sensitive about the quality of service rather than price. There are QoS SLA contractually required and if they can't be met clients will leave.
eternityforest 5y ago
P2P with fallback to centralized would solve a lot of it. Unfortunately P2P projects always try to be replacements for centralized rather than supplements, and almost none of them aside from BitTorrent are what I'd call great.

jdrc 5y ago

That sounds like a perfect match for internet content which is often viral and temporary peaks. Ideally in a P2P world creators would care to have at least 1 node running which would serve the long tail.

sitkack 5y ago

So your origin needs to be able to handle the tail and P2P can handle the popular stuff no? It isn't all or nothing. P2P could absolutely be used to peak shave. CDNs also have cold start issues with long tail. Time to first byte is pretty good, https://webtorrent.io/ https://webtorrent.io/ To the OP, you should research some other options, are you sure that you are asking the right question?

zinekeller 5y ago
> So your origin needs to be able to handle the tail and P2P can handle the popular stuff no? It isn't all or nothing. P2P could absolutely be used to peak shave. CDNs also have cold start issues with long tail. No, it'll be genuinely be worse, as you can't really have efficient P2P today. Let's start with the points; I'm not talking about Europe or US today. P2P or CDN is a coin flip there, and I'll agree that P2P is as reliable as a CDN. I'll also agree that theoretically P2P and CDN will be a coin flip (performance-wise, I'm leaving monetary considerations in this discussion). Those points break down quickly in Latin America, Asia and Africa. I'll talk about Asia as it is where I'm more familiar of, but the points apply also in Africa and Latin America. Outside of Jio and actual hosters/colos (meaning nearly all of residential and mobile connections), IPv6 in Asia is actually more unreliable than you think (it should not be, but that's peering fights for you) and IPv4 is CGNATTed heavily so P2P is no dice. Even if somehow there's a unique IPv4 for every device, internet connections there is very top-down, unlike with Europe's mesh-like connectivity. How bad? It'll be better to route Telkom Indonesia's connections to America than to another hoster located in Indonesia or Singapore (nearest regional hub), unless you're Akamai which have agreed to buy colocation space inside Telin just to have good connectivity. And before you comment, OVH in Singapore has the precise issues we're discussing (https://lowendtalk.com/discussion/172659/ovh-routing-issues-singapore-indonesia-singapore-philippines https://lowendtalk.com/discussion/172659/ovh-routing-issues-...). Thus, it is miles better to use your time to build good CDNs with good connections (or spend money to Akamai) than bothering with the issues you'll face with P2P. Moreover, you can concentrate your money into buying a better data link (either by renting/IRU a dedicated wavelength or even fiber pairs) than relying on a best-effort service. Funnily, even bittorent downloads in practice seems to be concentrated to a few seedboxens - even for purely legal downloads like GIMP. Postscript: technically P2P and CDN would be the best for reliability, but you're spending the time to do it (instead dealing with a single thing) and deal with the possible fallout (like Windows Updates before Microsoft decided to limit P2P to local networks due to bandwidth and privacy concerns). Speaking of bandwidth concerns, I promised myself to limit monetary concerns just to point out that's it's not good in actual deployment, but I forgot data caps (not just a problem in Africa but in the US too)! It'll be a disaster when P2P is not voluntary because you just wasted their precious money!
- conradev 5y ago
  So, your origin needs to handle the parts of the world with bad internet infrastructure, but can use P2P in Europe and the US, then?
  - zinekeller 5y ago
    US is out because data caps. You can in Europe, but it is worth the dev time? It doesn't matter when monetary costs are second to principles, but it be wrong to think that most companies are spending their time fancying P2P.
  - giantrobot 5y ago
    Besides data caps on residential broadband, a non-trivial number of clients will be on mobile and have effectively zero ability to reseed content. To enable streaming from mobile devices requires a lot of server infrastructure to support hole poking and reflection. Even with residential broadband without worries of data caps, there's still issues of firewalls and terrible upload speeds. Hole poking in residential firewalls like on mobile requires third party infrastructure. Most end users are not readily able to forward ports on their home routers. It's also not uncommon to see 10:1 downstream to upstream ratios. There's plenty of connections that couldn't manage to reliably upload a single FHD stream to a single client. It would take a hundred seeders with Comcastic upload speeds of 10Mbps to equal a single cheap VPS on a 1Gbps connection.
giantrobot 5y ago
Streaming video is not like static files from a torrent. The term "popular content" is a bit of a misnomer. Nominally with streaming a single uploaded video will be encoded into a number of different versions at different frame sizes and bitrates packaged into individual stream segments. So while a particular video URL might be popular, the actual video segments served to viewers will depend heavily on the configuration/environment of all the viewing devices. So a FHD stream for a video could be super popular while a 480p version might not have any viewers. When some 480p viewers load the stream they don't get any benefit from the P2P/cache popularity of the FHD stream. This is compounded by the fact that for video time to first byte doesn't matter very much. A decoder can't do anything with a single byte or even a partial stream segment. It's the time to locally buffer a decodable stream segment and a steady reception of segments as the play head reaches their time stamp. If a segment gets held up (as happens with P2P) it can't be decoded so you either pause playback or drop frames (if you've got later segments available). You want the media segments to cover at minimum a single GOP (Group of Pictures) which is a relatively large I-frame and the following smaller P and B frames. If you were to tune the video segmentation such that each segment was some tiny time slice it would balloon the bitrate since you need way more I-frames. Super long GOPs don't help either because segment drops end up ruining playback until a new I-frame is received. Streaming video is hard enough with a reliable CDN, adding P2P makes much harder and orders of magnitude less reliable.

qwerty456127 5y ago

> Try to watch/download a torrent file with small number of seeders and you'll get the experience you will be providing for the end user. I always wondered why not combine the ways the media is served into a hybrid of P2P with the classic way by just using P2P (e.g. BitTorrent) and also seeding yourself (as the hoster). Just run a seedbox of the capacity you want/can and let the audience help you by handling a share of the load they can. Wouldn't this save you some bandwidth while keeping the same service quality?

aasasd 5y ago
Archive.org seems to do something like that with its downloads via torrent: it appears as a web-seed for the torrents, and I usually snatch the files from those seeds faster than DHT seeds usually show up (which kinda undermines the idea, though). Frankly, I never saw any other seeds on Archive's torrents—but one of those I snatched has a ratio of 3 now, so presumably it does work as intended.
matheusmoreira 5y ago
> let the audience help you by handling a share of the load So why should we help these for-profit corporations? We're supposed to just donate our bandwidth to them? Are we getting a discount in exchange or are they just gonna pocket their bandwidth cost savings at our expense? Seeding internet archive data is one thing. Letting corporations invisibly take advantage of resources we pay for is a completely different matter.
- Retric 5y ago
  Pass some of the savings in to the customer via lower prices and it’s a win win. Alternatively, enabling P2P gives faster downloads. That’s rarely noticeable with streaming but people often want to download stuff for offline viewing.
  - matheusmoreira 5y ago
    > Pass some of the savings in to the customer via lower prices Does that ever happen? They'll probably keep prices exactly the same.

iib 5y ago

I wonder if an asymmetric scheme would work best, even though it may be unbalanced? For example, for any one popular video you watch, while you watch it, you also have to download and seed a small part of an unpopular video, even if you don't see it? Maybe bandwidth and storage caps would work better. You seed X MB, even if you don't need the full X yourself.

samwillis 5y ago

This is more or less exactly why Spotify dropped P2P from its platform back in 2014. Good summary here: https://techcrunch.com/2014/04/17/spotify-removes-peer-to-peer-technology-from-its-desktop-client/ https://techcrunch.com/2014/04/17/spotify-removes-peer-to-pe... "This is when the company took advantage of cached songs. Before today’s change, if you streamed a popular song for the first time, the client would download the song from other users, using peer-to-peer. All of this happened in the background, but it greatly contributed to making the overall user experience snappier." "Yet, now that the company has many servers, using peer-to-peer in addition to direct downloads actually adds a bit of overhead. Moreover, the company has to maintain the peer-to-peer code base, and update it with each new version."

black_puppydog 5y ago
> the company has to maintain the peer-to-peer code base, and update it with each new version. Really, that's (part of) their excuse? A bit lame, that. :|
- dymk 5y ago
  What, you don’t think code has maintenance costs associated with it, just because it’s cool?
- powerslacker 5y ago
  I might have agreed with you a few years ago but these days I see Spotify's perspective more clearly. Today I see code as a liability. Like the parts in my car it is eventually going to break in some strange way and cause me a headache. The only real solution is to never drive a car, but that's almost impossible if you want to get around. Same with code, it's a vehicle that gets a business to a destination, which is nice: bit it can also be quite expensive when things go wrong. Efficient engineering teams balance this inherent risk against the needs of the business.
  - bradknowles 5y ago
    Every line of code has a cost. The question you have to ask yourself is whether that cost is worth the return on your investment? No code is less expensive than no code.

dehrmann 5y ago

The other part is that CDNs result in less total network utilization. They limit most bulky traffic to the last miles.

freefaler 5y ago
It's not always so straightforward. You decrease your traffic costs, but you'll incur capex (servers, switches, connectivity) or operation costs (renting servers). Additionally outside EU network contracts can be a hassle to sign because of market consolidation or outright oligopoly. So you'd need to commit capital to get bandwidth. It's an interesting optimisation problem and sometimes you can route your traffic differently based on SNMP data, price per Mbps and route performance. The higher you pay, the better connectivity you get. But off-peak times you can route traffic via cheaper provider.

pmayrgundter 5y ago

İ also have experience building a CDN, tho not for video, and this sound right. I wonder if this might be improved by an appointment system, to stream popular titles at pre-planned times so that people could benefit from others watching at the same time. It would be like watching a broadcast television program during prime time. I guess this has been tried before but also never heard of it

freefaler 5y ago
The clients that do plan in advance are more sensitive about the quality of service rather than price. There are QoS SLA contractually required and if they can't be met clients will leave.

eternityforest 5y ago

P2P with fallback to centralized would solve a lot of it. Unfortunately P2P projects always try to be replacements for centralized rather than supplements, and almost none of them aside from BitTorrent are what I'd call great.