6 ms·

Idea: a legal botnet CDN

16 points by marcamillion 17y ago

icefox 17y ago
In a similar vain to double click v.s. adsense in the 90's. Back in the day they would count each ad install, done via an ad team and they have several thousand or something. Comes along google with a little javascript you insert into any page with little to no pre-screening and they get millions of installs. Rather then setting up and maintaining a box at sites just put up a vmware image for people to run :)
- marcamillion 17y ago
  Exactly...so I would love to see someone do it =)
  - milkshakes 17y ago
    why don't you?
    - marcamillion 17y ago
      Very good question. The truth is that the timing wasn't right for me when I originally thought about it, but depending on how the discussion goes here I might re-consider.
brazzy 17y ago
ISPs could not be profitable if end users started reselling bandwidth while paying flatrate fees. They'd be forced to enforce volume caps.
- marcamillion 17y ago
  That's interesting brazzy, but I think we would be surprised with the results of such a project. For example, if I were a user and this was available to me, I would have multiple ISP accounts and boxes. I imagine many others would do the same. I also think, an interesting side-effect would be a reduction in piracy. The way to cure piracy is not to sue everybody that pirates, the way to do it is to create an explicit opportunity cost for your time/bandwidth/storage. If I know that for every GB I pirate, I lose $1/month (or whatever the figure turns out to be) that changes my decision making process. So I figure that yes, along the limit (i.e. approaching infinity), assuming no one pirates and all customers are a CDN peer, the ISPs might not be profitable. But, I think we would see some very interesting social behavior changes before then, that the ISPs might be able to benefit from.
  - brazzy 17y ago
    ISPs would lose money on those accounts at current rates, so they'd have to make them so expensive that the user pays more money for bandwidth than he gets from the CDN. Would you have multiple ISP accounts paying perhaps $20 from the CDN when they cost you $400 each? Because that's how much they'd have to cost based on what little percentage of the bandwidth price the user gets in that spreadsheet. As for the anti-piracy argument, Have you actually looked at the numbers? It assumes $0.02 per GB bandwidth fee to the user - I doubt many people pirate movies they consider worth less thant 2 cents. And no, jacking up the bandwidth fee would not change the picture much. The point is that CDNs are already paying ISPs for bandwidth. There is no money to be made paying users, who get their bandwidth from ISPs, period. And storage fees would have to be relative to available bandwidth because 2TB storage on a 1MBit line is worth much less to the CDN than on a 100Mbit line.
- milkshakes 17y ago
  i think the point was to reduce transit traffic between providers. "local" mirrors accomplish that. regardless, users still have underutilized bandwidth, capped or not. why not resell it?
  - brazzy 17y ago
    Even if you could have a very smart network where only users within the same ISP exchange data (this not incurring peering fees), all that data still needs to be routed and transported between the ISP's access points, a cost that scales up with usage. If all users started using all of their bandwidth (or a larger portion of it than they do now), costs for the ISP would go up, and they would have to pass that on to the users. Earning $5 per month from reselling bandwidth is not so attractive when it means your ISP bill goes up $20. The business plan does not account for this. It assumes bandwidth to cost the users nothing, which only is true now because of flat fees, which are only viable when users use a very limited portion of the total available bandwidth.
    - milkshakes 17y ago
      all that data still needs to be routed and transported between the ISP's access points, a cost that scales up with usage. isn't this happening anyway?
- jerf 17y ago
  As you say: second-order effects. If user home broadband connections are being sold cheaply enough to be able to be resold for a profit, the ISPs will either raise the price to recoup this (the value of the connection has gone up to you and therefore you will be willing to pay more), or cut out the middleman and directly provide these services with their bandwidth (forbidding end users from participating), pocketing the money. This entire plan hinges on the ISPs doing nothing in response. And the responses the ISPs would make aren't even "evil" or anything, it would just be rational market behavior on their part. Furthermore, there would be major quality issues. Clever programming and design could make this work with enough redundancy in the system that even as people log in and out the content wouldn't disappear, but this does not come for free. You must pay a price somewhere else: increased latency (one of the most likely outcomes, many P2P systems make this tradeoff), decreased bandwidth, grotesque requirements for content replication that goes a long ways towards mitigating the capacity of the system. You can often trade one kind of quality for another, but you can't increase the total quality available, and a home consumer broadband connection is a low-quality connection for this purpose. And the quality of home connections aggregates distinctly sublinearly, especially since so many of them are badly asymmetric. This is a terrible idea and anybody going into it will lose everything they invest, excepting the IMHO faint possibility that some interesting technology could emerge that could eventually be repurposed for something sane. (I call this a faint possibility because that would then be facing competition from technologies directly purposed for whatever this later domain is and given the likelihood of a large difference between the initial design constraints and the final design constraints you're likely to have a very inferior solution.)
  - milkshakes 17y ago
    I still don't understand why this is a terrible idea. Is there absolutely no way for ISPs to react to this in such a way that they save money rather than lose it? If decreasing traffic between providers doesn't save money, then wouldn't there be some savings at least from decreasing traffic between subnets?
    - brazzy 17y ago
      CDNs don't scale down that well. The likelihood that there is demand for one particular piece of data within the same subnet (and thus the value of having it there) decreases dramatically with the size of the subnet. As the article itself points out, some CDNs already do this with large ISPs, directly. This kind of thing may increase and get more sophisticated, but it will never include the users themselves as middle men.
- grogers 17y ago
  Maybe, but have they yet? Until the concept becomes massively popular, there is still a business opportunity. Keep in mind also, that it only matters that the end users get more money from their bandwidth than it costs them to buy it from their ISP. As long as operating a distributed CDN node is more valuable than just the bandwidth to drive it, then there is still a business model. That added value is from the spare disk/cpu the end user would be contributing. In that case ISPs might jump into it themselves, but I still see that happening only after the idea became massively popular.
- eli 17y ago
  Also note that most home users have asynchronous connections -- uploads are often an order of magnitude slower than downloads
  - nkohari 17y ago
    Nitpick: it's "asymmetric". Good point though.
    - eli 17y ago
      Doh. It's been a long day.
marcamillion 17y ago
I would love to see PG add this as a rfs. Just a thought.
gojomo 17y ago
There were a gaggle of companies trying similar things with bandwidth and/or computer cycles in the 1999-2001 era. RedSwoosh was was eventually acquired by Akamai. BitTorrent still plugs its 'BitTorrentDNA' offering as a P2P CDN alternative. Kontiki was bought by VeriSign... but later spun back out. There was some speculation Adobe was adding P2P CDN capabilities to Flash Player in 2007, based on some combination of Kontiki technology and that of another P2P VOIP company they'd acquired, Amicima. (See <http://gigaom.com/2007/01/24/adobe-and-its-p2p-ambitions/>. http://gigaom.com/2007/01/24/adobe-and-its-p2p-ambitions/&#6...) But, none of these have taken off in a giant way. There may yet be a magic formula, but many variants have already been tried.
- kierank 17y ago
  Mainly because all the software developed for P2P CDNs so far were so cumbersome unlike file-sharing counterparts. Redswoosh in my opinion was the least cumbersome but having used it I never actually saw it use P2P, instead relying on its main server. Kontiki (used by BBC, Sky and Channel 4) was a joke too. It was difficult to remove; users couldn't control bandwidth easily and it forcibly started upon boot. When you removed it, it put itself back in the startup list again.
- marcamillion 17y ago
  I am fully aware of all the attempts, but like file-sharing had been attempted many times before Dropbox, I think there is still room for it to be done right. Not saying I am the one to do it right, but I definitely wouldn't use history as an accurate predictor of future attempts.
- wmf 17y ago
  More recently, Octoshape has gotten some high-profile adoption doing P2P streaming of live video events.
wallflower 17y ago
In case you missed it, HighScalability recently had a long, well-written thought-provoking essay on Planet-scale distributed systems: "Notice how global, how plentiful, and how fast the flashes flicker. It looks as if the whole world is afire, every ember burning with computation. That's a planet full of compute power. With millions of available nodes botnets wield more collective processing power than the world's top 10 supercomputers. Now imagine if all this power was available to your application. Take a moment...welcome back." http://highscalability.com/blog/2009/12/16/building-super-scalable-systems-blade-runner-meets-autonomic.html http://highscalability.com/blog/2009/12/16/building-super-sc...
Pahalial 17y ago
Given the global pushback against ISPs from the content industry and assertions they're knowingly profiting at the expense of content - including various threats of legal or legislative action - do you really believe any of them would let you profit at their expense? Such a system would be the very first thing throttled into oblivion by any packet shaping. The ISPs would naturally (and rightly) demand a profit-sharing agreement in exchange for unblocking this app. Their share would have to cover the costs of their bandwidth - and at that point, why not just do as every CDN worthy of the name has already done and put servers in their network? While this is a nice thought experiment, I frankly don't see it being any more successful this time around than previous attempts at the same. Perhaps less so, as network administrators everywhere now have more experience disrupting P2P and ISPs are more vigilant.
m_eiman 17y ago
The article is a bit hazy on some rather important details, such as "are you going to use P2P for downloads" and "how will you route traffic to the nodes". Using P2P means custom client software, which means that it can only be used to very specific things like application updates. I'm assuming that there will need to be some central point of contact that knows what content is available where - how will this be distributed and scaled? There's also the question of how to handle nodes dropping out of the network.
- marcamillion 17y ago
  This article wasn't meant to spell out everything, but just meant to prime the discussion pump in an effort to flesh out the idea from a 50,000 foot view. But to clear up any misconceptions, I am not purporting a new P2P application that people use to download stuff. I am essentially purporting a new protocol that replaces rmtp (the flash media streaming protocol) and other media streaming/distribution protocols. Routing the traffic is all a software issue. Same thing with nodes dropping out of the network. Bit Torrent (the protocol) has solved many - if not all - of those issues.
  - m_eiman 17y ago
    I am essentially purporting a new protocol that replaces rmtp (the flash media streaming protocol) and other media streaming/distribution protocols. Ah. So you just need to either convince Adobe to add that to Flash, or have the users run an application that can act as a proxy between it and RTMP. It's not a question of if it's possible or not, the problem is that it's hard to sell the concept to a website after you mention the "oh, and every one of your users will need to install this piece of software to be able to access the site" detail. It'll probably be a reasonable trade-off for some sites with devoted users, but for sites that have a lot of random users that only go there once or very seldom, it'll probably make more sense to pay the higher price of a normal CDN. Or maybe you intend to run the P2P/RTMP proxy on the nodes instead? Either way, the problem with the node dropping is to handle it in a graceful way that doesn't rely on custom software on the user side. Since your nodes are going to be pretty unreliable, your users will need to be able to switch between nodes without interrupting a large download. For smaller files they'll just need to be able to restart the transfer, which is a bit easier. I'm assuming that you want to use the CDN for serving various web content, for which HTTP should do just fine for just about everything. You probably don't even need RTMP, progressive downloading with a hack to allow non-linear access works just as well for most uses.
dnsworks 17y ago
One of the big problems with this is that DOCSIS is a shared bandwidth protocol. It's actually better for a lot of data to stream into a DOCSIS network from the outside, then for data to originate from that network, even if it's just going to other peers within that network. I have a friend at Charter who says that the biggest network engineering problem they face is the saturation of the "uplink frequencies" for users running bittorrent.

16 points by marcamillion 17y ago

In a similar vain to double click v.s. adsense in the 90's. Back in the day they would count each ad install, done via an ad team and they have several thousand or something. Comes along google with a little javascript you insert into any page with little to no pre-screening and they get millions of installs. Rather then setting up and maintaining a box at sites just put up a vmware image for people to run :)

marcamillion 17y ago
Exactly...so I would love to see someone do it =)
- milkshakes 17y ago
  why don't you?
  - marcamillion 17y ago
    Very good question. The truth is that the timing wasn't right for me when I originally thought about it, but depending on how the discussion goes here I might re-consider.

brazzy 17y ago

ISPs could not be profitable if end users started reselling bandwidth while paying flatrate fees. They'd be forced to enforce volume caps.

marcamillion 17y ago
That's interesting brazzy, but I think we would be surprised with the results of such a project. For example, if I were a user and this was available to me, I would have multiple ISP accounts and boxes. I imagine many others would do the same. I also think, an interesting side-effect would be a reduction in piracy. The way to cure piracy is not to sue everybody that pirates, the way to do it is to create an explicit opportunity cost for your time/bandwidth/storage. If I know that for every GB I pirate, I lose $1/month (or whatever the figure turns out to be) that changes my decision making process. So I figure that yes, along the limit (i.e. approaching infinity), assuming no one pirates and all customers are a CDN peer, the ISPs might not be profitable. But, I think we would see some very interesting social behavior changes before then, that the ISPs might be able to benefit from.
- brazzy 17y ago
  ISPs would lose money on those accounts at current rates, so they'd have to make them so expensive that the user pays more money for bandwidth than he gets from the CDN. Would you have multiple ISP accounts paying perhaps $20 from the CDN when they cost you $400 each? Because that's how much they'd have to cost based on what little percentage of the bandwidth price the user gets in that spreadsheet. As for the anti-piracy argument, Have you actually looked at the numbers? It assumes $0.02 per GB bandwidth fee to the user - I doubt many people pirate movies they consider worth less thant 2 cents. And no, jacking up the bandwidth fee would not change the picture much. The point is that CDNs are already paying ISPs for bandwidth. There is no money to be made paying users, who get their bandwidth from ISPs, period. And storage fees would have to be relative to available bandwidth because 2TB storage on a 1MBit line is worth much less to the CDN than on a 100Mbit line.
milkshakes 17y ago
i think the point was to reduce transit traffic between providers. "local" mirrors accomplish that. regardless, users still have underutilized bandwidth, capped or not. why not resell it?
- brazzy 17y ago
  Even if you could have a very smart network where only users within the same ISP exchange data (this not incurring peering fees), all that data still needs to be routed and transported between the ISP's access points, a cost that scales up with usage. If all users started using all of their bandwidth (or a larger portion of it than they do now), costs for the ISP would go up, and they would have to pass that on to the users. Earning $5 per month from reselling bandwidth is not so attractive when it means your ISP bill goes up $20. The business plan does not account for this. It assumes bandwidth to cost the users nothing, which only is true now because of flat fees, which are only viable when users use a very limited portion of the total available bandwidth.
  - milkshakes 17y ago
    all that data still needs to be routed and transported between the ISP's access points, a cost that scales up with usage. isn't this happening anyway?
jerf 17y ago
As you say: second-order effects. If user home broadband connections are being sold cheaply enough to be able to be resold for a profit, the ISPs will either raise the price to recoup this (the value of the connection has gone up to you and therefore you will be willing to pay more), or cut out the middleman and directly provide these services with their bandwidth (forbidding end users from participating), pocketing the money. This entire plan hinges on the ISPs doing nothing in response. And the responses the ISPs would make aren't even "evil" or anything, it would just be rational market behavior on their part. Furthermore, there would be major quality issues. Clever programming and design could make this work with enough redundancy in the system that even as people log in and out the content wouldn't disappear, but this does not come for free. You must pay a price somewhere else: increased latency (one of the most likely outcomes, many P2P systems make this tradeoff), decreased bandwidth, grotesque requirements for content replication that goes a long ways towards mitigating the capacity of the system. You can often trade one kind of quality for another, but you can't increase the total quality available, and a home consumer broadband connection is a low-quality connection for this purpose. And the quality of home connections aggregates distinctly sublinearly, especially since so many of them are badly asymmetric. This is a terrible idea and anybody going into it will lose everything they invest, excepting the IMHO faint possibility that some interesting technology could emerge that could eventually be repurposed for something sane. (I call this a faint possibility because that would then be facing competition from technologies directly purposed for whatever this later domain is and given the likelihood of a large difference between the initial design constraints and the final design constraints you're likely to have a very inferior solution.)
- milkshakes 17y ago
  I still don't understand why this is a terrible idea. Is there absolutely no way for ISPs to react to this in such a way that they save money rather than lose it? If decreasing traffic between providers doesn't save money, then wouldn't there be some savings at least from decreasing traffic between subnets?
  - brazzy 17y ago
    CDNs don't scale down that well. The likelihood that there is demand for one particular piece of data within the same subnet (and thus the value of having it there) decreases dramatically with the size of the subnet. As the article itself points out, some CDNs already do this with large ISPs, directly. This kind of thing may increase and get more sophisticated, but it will never include the users themselves as middle men.
grogers 17y ago
Maybe, but have they yet? Until the concept becomes massively popular, there is still a business opportunity. Keep in mind also, that it only matters that the end users get more money from their bandwidth than it costs them to buy it from their ISP. As long as operating a distributed CDN node is more valuable than just the bandwidth to drive it, then there is still a business model. That added value is from the spare disk/cpu the end user would be contributing. In that case ISPs might jump into it themselves, but I still see that happening only after the idea became massively popular.
eli 17y ago
Also note that most home users have asynchronous connections -- uploads are often an order of magnitude slower than downloads
- nkohari 17y ago
  Nitpick: it's "asymmetric". Good point though.
  - eli 17y ago
    Doh. It's been a long day.

marcamillion 17y ago

I would love to see PG add this as a rfs. Just a thought.

gojomo 17y ago

There were a gaggle of companies trying similar things with bandwidth and/or computer cycles in the 1999-2001 era. RedSwoosh was was eventually acquired by Akamai. BitTorrent still plugs its 'BitTorrentDNA' offering as a P2P CDN alternative. Kontiki was bought by VeriSign... but later spun back out. There was some speculation Adobe was adding P2P CDN capabilities to Flash Player in 2007, based on some combination of Kontiki technology and that of another P2P VOIP company they'd acquired, Amicima. (See <http://gigaom.com/2007/01/24/adobe-and-its-p2p-ambitions/>. http://gigaom.com/2007/01/24/adobe-and-its-p2p-ambitions/&#6...) But, none of these have taken off in a giant way. There may yet be a magic formula, but many variants have already been tried.

kierank 17y ago
Mainly because all the software developed for P2P CDNs so far were so cumbersome unlike file-sharing counterparts. Redswoosh in my opinion was the least cumbersome but having used it I never actually saw it use P2P, instead relying on its main server. Kontiki (used by BBC, Sky and Channel 4) was a joke too. It was difficult to remove; users couldn't control bandwidth easily and it forcibly started upon boot. When you removed it, it put itself back in the startup list again.
marcamillion 17y ago
I am fully aware of all the attempts, but like file-sharing had been attempted many times before Dropbox, I think there is still room for it to be done right. Not saying I am the one to do it right, but I definitely wouldn't use history as an accurate predictor of future attempts.
wmf 17y ago
More recently, Octoshape has gotten some high-profile adoption doing P2P streaming of live video events.

wallflower 17y ago

In case you missed it, HighScalability recently had a long, well-written thought-provoking essay on Planet-scale distributed systems: "Notice how global, how plentiful, and how fast the flashes flicker. It looks as if the whole world is afire, every ember burning with computation. That's a planet full of compute power. With millions of available nodes botnets wield more collective processing power than the world's top 10 supercomputers. Now imagine if all this power was available to your application. Take a moment...welcome back." http://highscalability.com/blog/2009/12/16/building-super-scalable-systems-blade-runner-meets-autonomic.html http://highscalability.com/blog/2009/12/16/building-super-sc...

Pahalial 17y ago

Given the global pushback against ISPs from the content industry and assertions they're knowingly profiting at the expense of content - including various threats of legal or legislative action - do you really believe any of them would let you profit at their expense? Such a system would be the very first thing throttled into oblivion by any packet shaping. The ISPs would naturally (and rightly) demand a profit-sharing agreement in exchange for unblocking this app. Their share would have to cover the costs of their bandwidth - and at that point, why not just do as every CDN worthy of the name has already done and put servers in their network? While this is a nice thought experiment, I frankly don't see it being any more successful this time around than previous attempts at the same. Perhaps less so, as network administrators everywhere now have more experience disrupting P2P and ISPs are more vigilant.

m_eiman 17y ago

The article is a bit hazy on some rather important details, such as "are you going to use P2P for downloads" and "how will you route traffic to the nodes". Using P2P means custom client software, which means that it can only be used to very specific things like application updates. I'm assuming that there will need to be some central point of contact that knows what content is available where - how will this be distributed and scaled? There's also the question of how to handle nodes dropping out of the network.

marcamillion 17y ago
This article wasn't meant to spell out everything, but just meant to prime the discussion pump in an effort to flesh out the idea from a 50,000 foot view. But to clear up any misconceptions, I am not purporting a new P2P application that people use to download stuff. I am essentially purporting a new protocol that replaces rmtp (the flash media streaming protocol) and other media streaming/distribution protocols. Routing the traffic is all a software issue. Same thing with nodes dropping out of the network. Bit Torrent (the protocol) has solved many - if not all - of those issues.
- m_eiman 17y ago
  I am essentially purporting a new protocol that replaces rmtp (the flash media streaming protocol) and other media streaming/distribution protocols. Ah. So you just need to either convince Adobe to add that to Flash, or have the users run an application that can act as a proxy between it and RTMP. It's not a question of if it's possible or not, the problem is that it's hard to sell the concept to a website after you mention the "oh, and every one of your users will need to install this piece of software to be able to access the site" detail. It'll probably be a reasonable trade-off for some sites with devoted users, but for sites that have a lot of random users that only go there once or very seldom, it'll probably make more sense to pay the higher price of a normal CDN. Or maybe you intend to run the P2P/RTMP proxy on the nodes instead? Either way, the problem with the node dropping is to handle it in a graceful way that doesn't rely on custom software on the user side. Since your nodes are going to be pretty unreliable, your users will need to be able to switch between nodes without interrupting a large download. For smaller files they'll just need to be able to restart the transfer, which is a bit easier. I'm assuming that you want to use the CDN for serving various web content, for which HTTP should do just fine for just about everything. You probably don't even need RTMP, progressive downloading with a hack to allow non-linear access works just as well for most uses.

dnsworks 17y ago

One of the big problems with this is that DOCSIS is a shared bandwidth protocol. It's actually better for a lot of data to stream into a DOCSIS network from the outside, then for data to originate from that network, even if it's just going to other peers within that network. I have a friend at Charter who says that the biggest network engineering problem they face is the saturation of the "uplink frequencies" for users running bittorrent.