8 ms·

Peer-to-peer overlay networks are a bad idea on a DSL-based internet.

56 points by laughinghan 15y ago

PaulHoule 15y ago
I've met so many people who've got this silly fantasy that there's something 'democratic' about hosting a web server at home. It's fine for a toy web site, but if you want your site to be fast and reliable, it's much more sensible to keep your server in a data center... in most cases, cheaper too.
tptacek 15y ago
This is a sensible post but note that there are interesting and important applications for overlay networks that are not "peer to peer" (ie, that do not use last-mile bandwidth for transit).
- sqrt17 15y ago
  The alternative to using last-mile transit is for everyone to have a virtual server somewhere. Even this seems to be a bit off since 90% of all people only ever consume content and can't be bothered with anything like creating their own content.
  - DanWaterworth 15y ago
    Even if just a small minority of people had VPSs that were on some P2P network it would be better than the current system of monopolies.
  - kragen 15y ago
    I think that's a good idea. It might not even have to be virtual, given servers the size of the Raspberry Pi: http://www.geek.com/articles/chips/raspberry-pi-25-pc-goes-into-alpha-production-20110728/ http://www.geek.com/articles/chips/raspberry-pi-25-pc-goes-i...
    - DanWaterworth 15y ago
      It would be cheaper to buy big servers with enormous amounts of RAM and use visualization than to have thousands of raspberry pis. Unless you mean each person has a raspberry pi on there home network, in which case you're still using expensive last-mile bandwidth.
      - kragen 15y ago
        It may or may not be cheaper. I was envisioning ten thousand Raspberry Pis in your local telephone central office.
      - lsc 15y ago
        I have been following this math /very closely/ - and I am fairly certain that if someone wanted to build a board like this in quantity, with more ram, of course, and if I could get my hands on a suitable power supply so I would have one psu for every twenty of these servers, it would likely come out pretty close to the cost of a big box. The problem is that all boards I can get for a reasonable price in the quantities I can afford have a bunch of crap on them I don't need; video and the like, which pushes the cost up higher than I'd like. I've been talking with a friend of mine who does ARM stuff about maybe seeing if we could get venture funding, as I think we could get into reasonable quantities in the low-six figure range. But, like I said, as far as I can tell, right now the cost difference is 'close' - even if I could get a custom board with lots of ram and no video crap, for most things it wouldn't be dramatically better than virtualizing a massive supermicro box, and considering that design time would mean that the thing would be kinda old by the time we were done, there probably isn't a lot of profit to be had. But, if trends continue, this may change; and I can tell you right now, customers are willing to pay a premium for dedicated hardware, and for good reason. (Yes, you have the management problem of thousands of servers, but most of the things that don't have PXE have some way to boot from serial, and I can deal with that. Having managed thousands of virtuals and tens of thousands of hardware nodes, I'm not convinced that virtuals are all that much easier to manage, once you have a good system in place.)
        DanWaterworth 15y ago
  - illume 15y ago
    Most people create content. See facebook, and email as proof if you are skeptical.
- wmf 15y ago
  I like the term server-to-server (S2S) from Jabber; basically P2P where the peers are located in datacenters.
- kragen 15y ago
  Thanks.
schiptsov 15y ago
Internet is already GSM-based which is much much worse.. ^_^
- schiptsov 15y ago
  It is amazing to see that current HN's audience is unable to notice that consumers are switching from PC+ADSL scheme to laptop+HSDPA USB stick or some Android/Apple device and that global GSM carriers were already won the market with cheap data-only plans. But it is OK that typical US citizen hardly knew that other countries exists. ^_^ btw, I'm currently in Sweden, where HSDPA coverage even in rural areas is a defacto standard. Same trends going on everywhere in the world (Asia and Europe), except, of course, US. ^_^
losethos 15y ago
I'm not sure, but somebody might be in denial. 10 i = i + 1 15 IF i > 99999 THEN i = 0 20 IF LEN(INKEY$) = 0 THEN PRINT ".";: GOTO 10 30 PRINT "King James Bible, Line:", i house of Israel, and in the midst of them stood Jaazaniah the son of Shaphan, with every man his censer in his hand; and a thick cloud of incense went up. 8:12 Then said he unto me, Son of man, hast thou seen what the ancients of the house of Israel do in the dark, every man in the chambers of his imagery? for they say, the LORD seeth us not; the LORD hath forsaken the earth. 8:13 He said also unto me, Turn thee yet again, and thou shalt see greater abominations that they do. 8:14 Then he brought me to the door of the gate of the LORD's house which was toward the north; and, behold, there sat women weeping for Tammuz.
banderson623 15y ago
It would be a much better read without the attitudes declared through statements like "The horrifying existence of abominations like Hulu and the iTunes Music Store..." I get the technical details, but the attitude is obnoxious at best and diluting the point at worst.
- rmrm 15y ago
  People seem to like Hulu, Netflix, Amazon, eBay, Youtube, etc. They haven't stopped the internet from the 1990s from existing, its still there and available. Most people just seem to like this other corporate internet a lot more. It seems to me that in many arguments, it isn't about whether it's possible for people to have their non-corporate, idealized internet (because it is), it's about not allowing others to also have the internet they want (Hulu, iTunes, Facebook, etc), because ultimately people aren't visiting the idealized, non corporate internet, and aren't going to -- because the majority of people prefer the other one. Ultimately, it seems to be a disappointment rooted in the behaviors and wants of the majority, tho its rarely ever framed as such.
  - kragen 15y ago
    People don't always choose the things that make them happy, especially collectively. If we step back a bit from the internet, this point is obvious: the people of the US, for example, collectively chose to have the Civil War and then abolish slavery, when abolishing slavery without the war would have been a much better choice; and alcoholics who die of exposure in the street would probably have preferred, in retrospect, to never start drinking. Some of them will tell you that explicitly even before dying. So it's not really disappointment. It's advocacy: there is a better way, a brighter future. And I have a selfish interest in telling people about it, because despite what you say, it's not a choice I can make entirely on my own. > They haven't stopped the internet from the 1990s from existing; it's still there and available. In the 1990s, I could email my mother from the mail server in my house; I could walk to the local bookstore and browse the books on the shelves; I could chat online with my friends without giving Microsoft, AOL, Yahoo, Google, and Facebook minute-by-minute updates on when I'm at my computer; I could read and post to Usenet, which had useful discussions on it, without broadcasting my reading habits to whoever was wiretapping (because my ISP had a local news server); and I could run years-old software on my computer without becoming a victim of the latest worm. So the two internets you're talking about do not exist independently of each other. They interact in a lot of ways. Sometimes the interference is constructive (internet access is available in a lot more places now, and bandwidth is cheaper), and sometimes it's destructive (Usenet is dead.) I don't think the problem is really even the wants of the majority. I think the problem is that the majority of people are getting tricked into choosing things they don't want, both because they aren't aware of the implications of their choices and because there are prisoner's-dilemma games going on. (Recruit all your friends to Farmville and your farm will be bigger!)
kleiba 15y ago
My hobby: reposting links from inside other HN front page articles.
- kragen 15y ago
  Not everything on kragen-tol started out as a comment on an HN article. Most of the linked post, for example, did not.
MaysonL 15y ago
Please somebody fix the title to read "an ADSL" instead of "a DSL".
- kragen 15y ago
  The middle half or so of my post is devoted to explaining why SDSL is a bad idea, which I think justifies the more general title. I take it you disagree? Why?
  - DanWaterworth 15y ago
    A better title would have been, "Why it's a bad idea for P2P overlay networks to use last-mile bandwidth".
    - kragen 15y ago
      If the last-mile bandwidth is provided by something that flexibly distributes bandwidth between "upstream" and "downstream" (thus inverting the meaning of the terms on a millisecond's notice!), or by fiber optics, or by shared media like a neighborhood Ethernet or cable modems, or by hop-by-hop forwarding in something other than a star topology, the reasoning doesn't apply. It specifically applies to DSL.
      - DanWaterworth 15y ago
        With physical connections, one high bandwidth connection will always be cheaper than many low bandwidth connections. So long as your p2p network is working over the internet, the reasoning still holds. Using expensive, last-mile bandwidth any more than necessary is not a good idea.
        kragen 15y ago
        If you have a shared neighborhood Ethernet or geographically-based hop-by-hop forwarding, it's cheaper to get a file from your neighbor than from a data center, because it avoids using expensive last-mile bandwidth. If you have an inherently-symmetric medium like fiber, traffic in the less congested direction is free, and data-center space isn't. So peer-to-peer works better than warehouse-scale computing in those cases. The dynamic-allocation case is a little more complicated, and I'm less certain of my reasoning there. It's true that peer-to-peer architectures will still use more last-mile bandwidth overall in that case than data-center architectures, as much as twice as much, and so they'll still be less efficient. The difference is that you can make that tradeoff on a moment-by-moment basis, instead of choosing to permanently cut your bandwidth in half when you sign up for the service, before you know anything about the internet. That's a much lower bar.
tuomasb 15y ago
I think the decision by Google to not buy Skype is great. Skype uses supernodes/decentralized address book and other trickery to keep the most of the load off their servers/NAT proxies. Whatever advantage Skype got from decentralization when it started in 2003 is now long gone. Bandwidth is ridiculously cheap and voice calls don't really take more than 16kbps with modern codecs. If two Skype clients can't connect to each other directly due to NAT, they proxy the connection via supernode. Any Skype client with a public IP address can be a supernode, so they might use your bandwidth and computer to route calls. Skype brand and userbase alone are not worth $8,5B. Their P2P decentralization technology is worth ~$0.

56 points by laughinghan 15y ago

I've met so many people who've got this silly fantasy that there's something 'democratic' about hosting a web server at home. It's fine for a toy web site, but if you want your site to be fast and reliable, it's much more sensible to keep your server in a data center... in most cases, cheaper too.

tptacek 15y ago

This is a sensible post but note that there are interesting and important applications for overlay networks that are not "peer to peer" (ie, that do not use last-mile bandwidth for transit).

sqrt17 15y ago
The alternative to using last-mile transit is for everyone to have a virtual server somewhere. Even this seems to be a bit off since 90% of all people only ever consume content and can't be bothered with anything like creating their own content.
- DanWaterworth 15y ago
  Even if just a small minority of people had VPSs that were on some P2P network it would be better than the current system of monopolies.
- kragen 15y ago
  I think that's a good idea. It might not even have to be virtual, given servers the size of the Raspberry Pi: http://www.geek.com/articles/chips/raspberry-pi-25-pc-goes-into-alpha-production-20110728/ http://www.geek.com/articles/chips/raspberry-pi-25-pc-goes-i...
  - DanWaterworth 15y ago
    It would be cheaper to buy big servers with enormous amounts of RAM and use visualization than to have thousands of raspberry pis. Unless you mean each person has a raspberry pi on there home network, in which case you're still using expensive last-mile bandwidth.
    - kragen 15y ago
      It may or may not be cheaper. I was envisioning ten thousand Raspberry Pis in your local telephone central office.
    - lsc 15y ago
      I have been following this math /very closely/ - and I am fairly certain that if someone wanted to build a board like this in quantity, with more ram, of course, and if I could get my hands on a suitable power supply so I would have one psu for every twenty of these servers, it would likely come out pretty close to the cost of a big box. The problem is that all boards I can get for a reasonable price in the quantities I can afford have a bunch of crap on them I don't need; video and the like, which pushes the cost up higher than I'd like. I've been talking with a friend of mine who does ARM stuff about maybe seeing if we could get venture funding, as I think we could get into reasonable quantities in the low-six figure range. But, like I said, as far as I can tell, right now the cost difference is 'close' - even if I could get a custom board with lots of ram and no video crap, for most things it wouldn't be dramatically better than virtualizing a massive supermicro box, and considering that design time would mean that the thing would be kinda old by the time we were done, there probably isn't a lot of profit to be had. But, if trends continue, this may change; and I can tell you right now, customers are willing to pay a premium for dedicated hardware, and for good reason. (Yes, you have the management problem of thousands of servers, but most of the things that don't have PXE have some way to boot from serial, and I can deal with that. Having managed thousands of virtuals and tens of thousands of hardware nodes, I'm not convinced that virtuals are all that much easier to manage, once you have a good system in place.)
      - DanWaterworth 15y ago
- illume 15y ago
  Most people create content. See facebook, and email as proof if you are skeptical.
wmf 15y ago
I like the term server-to-server (S2S) from Jabber; basically P2P where the peers are located in datacenters.
kragen 15y ago
Thanks.

schiptsov 15y ago

Internet is already GSM-based which is much much worse.. ^_^

schiptsov 15y ago
It is amazing to see that current HN's audience is unable to notice that consumers are switching from PC+ADSL scheme to laptop+HSDPA USB stick or some Android/Apple device and that global GSM carriers were already won the market with cheap data-only plans. But it is OK that typical US citizen hardly knew that other countries exists. ^_^ btw, I'm currently in Sweden, where HSDPA coverage even in rural areas is a defacto standard. Same trends going on everywhere in the world (Asia and Europe), except, of course, US. ^_^

losethos 15y ago

I'm not sure, but somebody might be in denial. 10 i = i + 1 15 IF i > 99999 THEN i = 0 20 IF LEN(INKEY$) = 0 THEN PRINT ".";: GOTO 10 30 PRINT "King James Bible, Line:", i house of Israel, and in the midst of them stood Jaazaniah the son of Shaphan, with every man his censer in his hand; and a thick cloud of incense went up. 8:12 Then said he unto me, Son of man, hast thou seen what the ancients of the house of Israel do in the dark, every man in the chambers of his imagery? for they say, the LORD seeth us not; the LORD hath forsaken the earth. 8:13 He said also unto me, Turn thee yet again, and thou shalt see greater abominations that they do. 8:14 Then he brought me to the door of the gate of the LORD's house which was toward the north; and, behold, there sat women weeping for Tammuz.

banderson623 15y ago

It would be a much better read without the attitudes declared through statements like "The horrifying existence of abominations like Hulu and the iTunes Music Store..." I get the technical details, but the attitude is obnoxious at best and diluting the point at worst.

rmrm 15y ago
People seem to like Hulu, Netflix, Amazon, eBay, Youtube, etc. They haven't stopped the internet from the 1990s from existing, its still there and available. Most people just seem to like this other corporate internet a lot more. It seems to me that in many arguments, it isn't about whether it's possible for people to have their non-corporate, idealized internet (because it is), it's about not allowing others to also have the internet they want (Hulu, iTunes, Facebook, etc), because ultimately people aren't visiting the idealized, non corporate internet, and aren't going to -- because the majority of people prefer the other one. Ultimately, it seems to be a disappointment rooted in the behaviors and wants of the majority, tho its rarely ever framed as such.
- kragen 15y ago
  People don't always choose the things that make them happy, especially collectively. If we step back a bit from the internet, this point is obvious: the people of the US, for example, collectively chose to have the Civil War and then abolish slavery, when abolishing slavery without the war would have been a much better choice; and alcoholics who die of exposure in the street would probably have preferred, in retrospect, to never start drinking. Some of them will tell you that explicitly even before dying. So it's not really disappointment. It's advocacy: there is a better way, a brighter future. And I have a selfish interest in telling people about it, because despite what you say, it's not a choice I can make entirely on my own. > They haven't stopped the internet from the 1990s from existing; it's still there and available. In the 1990s, I could email my mother from the mail server in my house; I could walk to the local bookstore and browse the books on the shelves; I could chat online with my friends without giving Microsoft, AOL, Yahoo, Google, and Facebook minute-by-minute updates on when I'm at my computer; I could read and post to Usenet, which had useful discussions on it, without broadcasting my reading habits to whoever was wiretapping (because my ISP had a local news server); and I could run years-old software on my computer without becoming a victim of the latest worm. So the two internets you're talking about do not exist independently of each other. They interact in a lot of ways. Sometimes the interference is constructive (internet access is available in a lot more places now, and bandwidth is cheaper), and sometimes it's destructive (Usenet is dead.) I don't think the problem is really even the wants of the majority. I think the problem is that the majority of people are getting tricked into choosing things they don't want, both because they aren't aware of the implications of their choices and because there are prisoner's-dilemma games going on. (Recruit all your friends to Farmville and your farm will be bigger!)

kleiba 15y ago

My hobby: reposting links from inside other HN front page articles.

kragen 15y ago
Not everything on kragen-tol started out as a comment on an HN article. Most of the linked post, for example, did not.

MaysonL 15y ago

Please somebody fix the title to read "an ADSL" instead of "a DSL".

kragen 15y ago
The middle half or so of my post is devoted to explaining why SDSL is a bad idea, which I think justifies the more general title. I take it you disagree? Why?
- DanWaterworth 15y ago
  A better title would have been, "Why it's a bad idea for P2P overlay networks to use last-mile bandwidth".
  - kragen 15y ago
    If the last-mile bandwidth is provided by something that flexibly distributes bandwidth between "upstream" and "downstream" (thus inverting the meaning of the terms on a millisecond's notice!), or by fiber optics, or by shared media like a neighborhood Ethernet or cable modems, or by hop-by-hop forwarding in something other than a star topology, the reasoning doesn't apply. It specifically applies to DSL.
    - DanWaterworth 15y ago
      With physical connections, one high bandwidth connection will always be cheaper than many low bandwidth connections. So long as your p2p network is working over the internet, the reasoning still holds. Using expensive, last-mile bandwidth any more than necessary is not a good idea.
      - kragen 15y ago
        If you have a shared neighborhood Ethernet or geographically-based hop-by-hop forwarding, it's cheaper to get a file from your neighbor than from a data center, because it avoids using expensive last-mile bandwidth. If you have an inherently-symmetric medium like fiber, traffic in the less congested direction is free, and data-center space isn't. So peer-to-peer works better than warehouse-scale computing in those cases. The dynamic-allocation case is a little more complicated, and I'm less certain of my reasoning there. It's true that peer-to-peer architectures will still use more last-mile bandwidth overall in that case than data-center architectures, as much as twice as much, and so they'll still be less efficient. The difference is that you can make that tradeoff on a moment-by-moment basis, instead of choosing to permanently cut your bandwidth in half when you sign up for the service, before you know anything about the internet. That's a much lower bar.

tuomasb 15y ago

I think the decision by Google to not buy Skype is great. Skype uses supernodes/decentralized address book and other trickery to keep the most of the load off their servers/NAT proxies. Whatever advantage Skype got from decentralization when it started in 2003 is now long gone. Bandwidth is ridiculously cheap and voice calls don't really take more than 16kbps with modern codecs. If two Skype clients can't connect to each other directly due to NAT, they proxy the connection via supernode. Any Skype client with a public IP address can be a supernode, so they might use your bandwidth and computer to route calls. Skype brand and userbase alone are not worth $8,5B. Their P2P decentralization technology is worth ~$0.