6 ms·

My guess is it's a combination of development complexity and (probably more importantly) firewall/router issues. It's very hard — impossible, really — to deplo

My guess is it's a combination of development complexity and (probably more importantly) firewall/router issues. It's very hard — impossible, really — to deploy P2P technologies on a mass scale without thousands of users encountering problems with their routers & firewalls. For mass market products, you can't get away with asking people to whitelist your app, make sure port 28777 is open for UDP, etc. Many P2P systems have freeloader issues, which can usually be resolved/avoided/ignored on desktop systems, but when you add mobile into the equation — with its paltry bandwidth limits and sky-high overage charges — the potential for it to become a problem is much greater.

api 12y ago
Hmm... so... (1) Developer ease of use... must be almost as easy as simply opening a 'dang socket. (2) Must be able to fall back on non-p2p easily and transparently if p2p is not possible for a given customer. (3) Must be able to set a bandwidth quota on mobile devices, or possibly fall back to non-p2p easily and transparently when on a capped cellular connection. (4) (via another reply) Must not do constant keep alives all the time at least on smaller mobile devices like phones, which will eat battery life-- must support some kind of sleep mode with instant wake. This being HN, I have ulterior market research motives with this ask post. I have thoughts about building something, and want to know if it's worth doing. :-) But I think it's an interesting question in the abstract too.
- ColinCera 12y ago
  I have a hunch that WebRTC is going to make P2P a much more mainstream technology. Once most people are running browsers with WebRTC support, and there are good high-level libraries available for developers, I think we'll start to see it used for all kinds of things — not just real-time video conferencing, but all kinds of information sharing and dissemination, both real-time and who-cares-about-real-time, just because WebRTC will be there, will (mostly) work, and will be (relatively) easy. You mentioned something about saving bandwidth costs, and I admit I've had a few ideas for bootstrap startups where I've thought, "Well, I could do this, but the second it became semi-popular I'd go broke and have to shut it down," and then, "If only there was a way to have the apps distribute the data P2P, so I didn't have to pay for every single user downloading from the server."
  - api 12y ago
    Check the thread and see my other reply.
- Mandatum 12y ago
  If you could get that going on both mobile and desktop apps, I would definitely implement it across several software implementations I work on. It'd likely be a year from now before it was implemented, but the amount we'd save would arguably outweigh the cons to this. Of course we'd implement user settings to disable, only transfer of public data, make sure the client is robust and secure, etc. but peer-to-peer in this day-and-age seems to be the way to go.
  - api 12y ago
    I'm considering doing just that. I already have most of it done in fact... It's the engine behind this: https://www.zerotier.com/ https://www.zerotier.com/ It does most of the things I listed. The remainder would be a matter of proper packaging and integration with mobile OSes for the mobile version. The idea would be to package it with an ultralight IP stack so that each app appeared on a virtual network as an ip endpoint. Servers could join the same virtual networks using the already released software I just linked, and services could just talk standard ip as if they were talking to any other network. No rewrites on the server end needed, and the client may just need to link in a library and call some init functions. Respecting battery life might be hard, but I see no reason integration with Apple or Google push notification systems couldn't be used to wake the endpoint app when it needs to do something. Idea is that desktop nodes could run in a heavier mode than mobile nodes, with the latter going dormant after a few minutes of inactivity and waking on coarse grained push. Drop me an email at contact@zerotier.com if you're really interested in this kind of thing. I'm doing some research to see if this is worth building.
- imron 12y ago
  It's not so much 'Developer ease of use' but 'End user ease of use'. If end users can't use your product because it's blocked by their firewall then that's going to cause you problems (either in extra support costs, or lost customers).
- superuser2 12y ago
  > (1) Developer ease of use You can't fix it with developer tools. You'd need to completely rewrite the way many hundreds of thousands of diverse, entrenched, autonomous entities with a vested interest in your failure think about networking. Most users are behind NAT. Several thousand employees in an office or students in a dorm will have the same public IP address and will never, under any circumstances, administer the router where NATing takes place. For users who do not pay their own ISP bills, it's impossible. By design. You are not going to convince the entire IT management/security establishment that they should allow arbitrary connections to their users' machines. With unlimited resources and political power, you might be able to convince all the SOHO router manufacturers to include some kind of standardized port-forwarding API. It will be several years before all the old devices are replaced, though. But you might get to a point where there are peer-to-peer systems for nontechnical users who own their routers. You still need client-server for when people are on an institutional connection/mobile. There are no peer-to-peer connections between users behind firewalls or NAT. This is true even in BitTorrent - if you're behind a firewall, your only peers are people who are not behind firewalls (seedboxes in datacenters, geeks who went into their router config, and people who are wasting IPv4 space by handing out fully routable IPs to individual clients.)
  - api 12y ago
    Most of this is incorrect. Look into hole punching as a starting point. BitTorrent, SIP VoIP phones, desktop Skype, etc. all do this. It's sort of a hack that became a semi de facto standard, but so is NAT. There are some percentage who cannot do p2p even with smart traversal techniques. In my system I have measured this to be <5% of total users, few enough that free relaying is basically free to provide. The hardest problems surround mobile battery life, quotas, and coarse grained and limited multitasking. These are also likely solvable. They may require a rethink at the encapsulated layer 3 level, like TCP proxy ACK or linear coding. Or maybe push notification. Still researching this.
    - superuser2 12y ago
      >BitTorrent http://www.bittorrent.com/help/manual/chapter0203 http://www.bittorrent.com/help/manual/chapter0203 BitTorrent users who do not have port forwarding enabled do not communicate with each other. BitTorrent has a highly technical user base and chances are good that someone with the file you want actually has enabled port forwarding or is running a server in a datacenter (seedbox). That wouldn't work for mainstream messaging. >desktop Skype Firewalled Skype users do not talk directly to each other, they talk to Supernodes (servers) running on nodes that do have port forwarding / no firewall. Conscripting random PCs as servers drew public outrage and Microsoft eventually (citing scalability and reliability) made all the Supernodes server-class hardware sitting in Microsoft datacenters. >SIP VoIP phones If you are in an office and can receive calls on a SIP phone, it's because your organization runs (or pays someone to run) a central server called a gateway, usually but not always as part of a PBX. The most common open-source PBX is called Asterisk. People who call you from the outside are connecting to your phone, but to your organization's gateway. Your gateway then sits on a (V)LAN with your phone, or your phone maintains an outgoing connection to it. If my friend has her router configured but I do not, then (absent a gateway) I could call her but she couldn't call me. So you use central gateways Many VoIP phones come with VPN clients so that they can behave like peers to phones within the organization if deployed at employees' homes or in certain branch office configurations where bridging is not already in place. In this case, you still have centralization at the level of the VPN server.

api 12y ago

Hmm... so... (1) Developer ease of use... must be almost as easy as simply opening a 'dang socket. (2) Must be able to fall back on non-p2p easily and transparently if p2p is not possible for a given customer. (3) Must be able to set a bandwidth quota on mobile devices, or possibly fall back to non-p2p easily and transparently when on a capped cellular connection. (4) (via another reply) Must not do constant keep alives all the time at least on smaller mobile devices like phones, which will eat battery life-- must support some kind of sleep mode with instant wake. This being HN, I have ulterior market research motives with this ask post. I have thoughts about building something, and want to know if it's worth doing. :-) But I think it's an interesting question in the abstract too.

ColinCera 12y ago
I have a hunch that WebRTC is going to make P2P a much more mainstream technology. Once most people are running browsers with WebRTC support, and there are good high-level libraries available for developers, I think we'll start to see it used for all kinds of things — not just real-time video conferencing, but all kinds of information sharing and dissemination, both real-time and who-cares-about-real-time, just because WebRTC will be there, will (mostly) work, and will be (relatively) easy. You mentioned something about saving bandwidth costs, and I admit I've had a few ideas for bootstrap startups where I've thought, "Well, I could do this, but the second it became semi-popular I'd go broke and have to shut it down," and then, "If only there was a way to have the apps distribute the data P2P, so I didn't have to pay for every single user downloading from the server."
- api 12y ago
  Check the thread and see my other reply.
Mandatum 12y ago
If you could get that going on both mobile and desktop apps, I would definitely implement it across several software implementations I work on. It'd likely be a year from now before it was implemented, but the amount we'd save would arguably outweigh the cons to this. Of course we'd implement user settings to disable, only transfer of public data, make sure the client is robust and secure, etc. but peer-to-peer in this day-and-age seems to be the way to go.
- api 12y ago
  I'm considering doing just that. I already have most of it done in fact... It's the engine behind this: https://www.zerotier.com/ https://www.zerotier.com/ It does most of the things I listed. The remainder would be a matter of proper packaging and integration with mobile OSes for the mobile version. The idea would be to package it with an ultralight IP stack so that each app appeared on a virtual network as an ip endpoint. Servers could join the same virtual networks using the already released software I just linked, and services could just talk standard ip as if they were talking to any other network. No rewrites on the server end needed, and the client may just need to link in a library and call some init functions. Respecting battery life might be hard, but I see no reason integration with Apple or Google push notification systems couldn't be used to wake the endpoint app when it needs to do something. Idea is that desktop nodes could run in a heavier mode than mobile nodes, with the latter going dormant after a few minutes of inactivity and waking on coarse grained push. Drop me an email at contact@zerotier.com if you're really interested in this kind of thing. I'm doing some research to see if this is worth building.
imron 12y ago
It's not so much 'Developer ease of use' but 'End user ease of use'. If end users can't use your product because it's blocked by their firewall then that's going to cause you problems (either in extra support costs, or lost customers).
superuser2 12y ago
> (1) Developer ease of use You can't fix it with developer tools. You'd need to completely rewrite the way many hundreds of thousands of diverse, entrenched, autonomous entities with a vested interest in your failure think about networking. Most users are behind NAT. Several thousand employees in an office or students in a dorm will have the same public IP address and will never, under any circumstances, administer the router where NATing takes place. For users who do not pay their own ISP bills, it's impossible. By design. You are not going to convince the entire IT management/security establishment that they should allow arbitrary connections to their users' machines. With unlimited resources and political power, you might be able to convince all the SOHO router manufacturers to include some kind of standardized port-forwarding API. It will be several years before all the old devices are replaced, though. But you might get to a point where there are peer-to-peer systems for nontechnical users who own their routers. You still need client-server for when people are on an institutional connection/mobile. There are no peer-to-peer connections between users behind firewalls or NAT. This is true even in BitTorrent - if you're behind a firewall, your only peers are people who are not behind firewalls (seedboxes in datacenters, geeks who went into their router config, and people who are wasting IPv4 space by handing out fully routable IPs to individual clients.)
- api 12y ago
  Most of this is incorrect. Look into hole punching as a starting point. BitTorrent, SIP VoIP phones, desktop Skype, etc. all do this. It's sort of a hack that became a semi de facto standard, but so is NAT. There are some percentage who cannot do p2p even with smart traversal techniques. In my system I have measured this to be <5% of total users, few enough that free relaying is basically free to provide. The hardest problems surround mobile battery life, quotas, and coarse grained and limited multitasking. These are also likely solvable. They may require a rethink at the encapsulated layer 3 level, like TCP proxy ACK or linear coding. Or maybe push notification. Still researching this.
  - superuser2 12y ago
    >BitTorrent http://www.bittorrent.com/help/manual/chapter0203 http://www.bittorrent.com/help/manual/chapter0203 BitTorrent users who do not have port forwarding enabled do not communicate with each other. BitTorrent has a highly technical user base and chances are good that someone with the file you want actually has enabled port forwarding or is running a server in a datacenter (seedbox). That wouldn't work for mainstream messaging. >desktop Skype Firewalled Skype users do not talk directly to each other, they talk to Supernodes (servers) running on nodes that do have port forwarding / no firewall. Conscripting random PCs as servers drew public outrage and Microsoft eventually (citing scalability and reliability) made all the Supernodes server-class hardware sitting in Microsoft datacenters. >SIP VoIP phones If you are in an office and can receive calls on a SIP phone, it's because your organization runs (or pays someone to run) a central server called a gateway, usually but not always as part of a PBX. The most common open-source PBX is called Asterisk. People who call you from the outside are connecting to your phone, but to your organization's gateway. Your gateway then sits on a (V)LAN with your phone, or your phone maintains an outgoing connection to it. If my friend has her router configured but I do not, then (absent a gateway) I could call her but she couldn't call me. So you use central gateways Many VoIP phones come with VPN clients so that they can behave like peers to phones within the organization if deployed at employees' homes or in certain branch office configurations where bridging is not already in place. In this case, you still have centralization at the level of the VPN server.