4 ms·

I like the P2P nature of the internet, but how do you get connected to a peer on good terms? Comcast, Verizon and AT&T limit not only who but what devices you

I like the P2P nature of the internet, but how do you get connected to a peer on good terms? Comcast, Verizon and AT&T limit not only who but what devices you can route packets for (e.g. hotspot data). Any ideas on finding good peers to work with? I'm hoping for few usage restrictions first and then quality links characteristics next (e.g. bandwidth, latency, cost, etc...).

johngalt 7y ago
It's somewhat ad hoc. Generally you'll want to find who owns the actual infrastructure, and get as close to an agreement with them as possible. I.E. There will be some huge Zayo fiber that is resold by a bunch of different carriers. They are B2B only and don't sell directly to end users and have fairly high minimum commitments. Find the closest point that you can connect to it, and do wireless from that point outwards. Or you can do an interconnect/backhaul type agreement (L3 used to be great at these before they were acquired by Clink). This is where you are only buying a link back to a colocation. From there you can do direct peering or pick from several transit providers at pricing you can only get in that sort of environment. Fundamentally the problem is: If the fiber is there, then likely you'll have 50 carriers all reselling it and you are just number 51. If the fiber isn't there, then you'll have a difficult time providing suitable bandwidth.
jcrawfordor 7y ago
As a new ISP, you're not going to get any peering arrangements. Really, as an existing consumer ISP not in the top 10 or 20, you're not going to get any peering arrangements. Consumer ISPs generate very asymmetric traffic (more down than up) which tampers with the core "equality" assumption of traditional peering arrangements. You'll need to purchase transit from an existing provider, such as CenturyLink's commercial arm or Verizon Business, there are plenty more. This won't be free but it also probably won't be as expensive as you think. In a city you can get a Gigabit dedicated to you for $2k a month, maybe less. Shop around. Most of these providers will put you on the phone with a network engineer as part of the quoting process and they can be helpful in recommending what service level and equipment you get. IPv4 allocations are hard to come by unless you want to bid on them at auction which can be expensive. You probably want to get a block from your provider, but they'll be hard to talk into giving you more than say a /29 and they might charge you more for it. Often they don't, though, they just put you in a queue sorted by customer size and you'll never get to the top of it. As a small ISP, CGNAT is probably going to be your only choice for IPv4. This also has the downside of being non-portable since the addresses will always belong to your provider. Maybe save up money to bid on your very own. Once you get one you'll want to start participating in BGP which is complicated anyway, so get to that later, once you have customers.
- rahimnathwani 7y ago
  'Consumer ISPs generate very asymmetric traffic (more down than up) which tampers with the core "equality" assumption of traditional peering arrangements.' Shouldn't this be in favour of the ISP? If the traffic isn't balanced, then doesn't the peer that's sending more (e.g. YouTube, Netflix) usually pay? I have no experience of setting up an ISP, so this is just based on what I've read. I'd love to understand it better. "IPv4 allocations are hard to come by unless you want to bid on them at auction which can be expensive." Looking online, it looks like the prices are $15 to $30 per IP address, depending on the size of the block. So I guess this cost swallows the first month of revenue from each new customer.
  - Scoundreller 7y ago
    > 'Consumer ISPs generate very asymmetric traffic (more down than up) which tampers with the core "equality" assumption of traditional peering arrangements.' It's a good situation for the ISP to start a DC business with unmetered ports to help balance things!
    - jcrawfordor 7y ago
      This is a lot of how the major ISPs smooth out the peering situation, they also offer colocation and carrier hotel services to large customers. Unfortunately, the popularity of Netflix and other streaming services has largely upset this balance and it's very hard for any ISP to keep traffic equality.
      - Scoundreller 7y ago
        At least with Netflix, you can get their Appliances and cut your inbound Netflix bandwidth by 80%+ for the cost of power and 4U.
        iptrans 7y ago
        Yeah, if you have at least 5 Gbps of Netflix traffic on your network... No small or startup ISP is going to qualify for a Netflix appliance. Also to cut your Netflix bandwidth by 80+%, you need more than 6 Gbps of Netflix traffic, as Netflix requires 1.2 Gbps of inbound bandwidth just to fill the cache. Netflix share of Internet traffic is about 15-20%, so in practice Netflix appliances are used by ISPs with 25-35 Gbps of bandwidth and up. This means ISPs with tens of thousands of subscribers or more.
Scoundreller 7y ago
It's kinda a strange thing in Canada. The incumbent telecoms generally refuse to peer domestically with independent ISPs and smaller transit providers that cheaper DCs use. One ISP reported that Bell would only connect to them in Equinix in Chicago, even though Equinix has a PoP in Toronto. https://www.peeringdb.com/net/1550 https://www.peeringdb.com/net/1550 This means if you subscribe to internet with the local telephone monopoly (Bell) in Toronto, your traffic to toronto[1-8].voip.ms goes through NYC or Chicago. Have seen some routes that add Washington DC to the mix too. A 5km trip made into several thousand km. The incumbents even don't peer much at our big IX: http://torix.ca http://torix.ca (they buy ports, but don't peer freely). In a lot of ways, because the independents will peer with everyone and love IXs, your traffic often stays a lot more local in Canada.
- dylz 7y ago
  The US is the same. All major ISPs refuse to peer with any independents at all; refuse to be on routeservers; and demand some minimum of like 50-100 Gbps for a peering session, with insane requirements. I've seen numerous ones also demand things like RPKI/ROA and good filtering - while themselves having everything ROA and IRR invalid, announcing customer space with no IRR records whatsoever, etc.
  - Scoundreller 7y ago
    So how do independents interconnect their routes? I think the Canadian issue is that incumbents aren't even peering with transit providers.
    - dylz 7y ago
      For this context, when I say independents, I mean smaller providers - content providers, independent ISPs that aren't large enough yet. They will obviously peer with Google, Akamai, etc. Independents more or less just buy transit/blend and deal with it. Here's an example of a large incumbent's peering policies: https://www.spectrum.com/content/dam/spectrum/residential/en/pdfs/policies/160525_Attachment_1_Charter_Communications_IP_Interconnect_Offer_and_Requirements.pdf https://www.spectrum.com/content/dam/spectrum/residential/en... The first requirement completely removes the possibility of ever getting a peering session with a small ISP or independent provider. So does the fourth. Another one: https://www.xfinity.com/peering https://www.xfinity.com/peering > minimum of four mutually agreeable geographically diverse points in the US. Interconnection points must include at least one city on the US east coast, one in the central region, and one on the US west coast This makes "launch an ISP from garage" bordering on "never" because they do not want regional providers getting settlement free peering - they just want them to buy IP transit from them. If you look at virtually every major US/CA ISP on peeringdb, they will be Restrictive / Selective or Private peering only, most likely.
lapinot 7y ago
As others have said, as a small isp you're gonna need to start off by buying transit. But the core philosophy is to peer with your peers, might sound stupid, but you should look towards other isp that look like you. There are already a lot of diy internet communities, most doing one kind or another of wireless mesh to connect to end-users. Some have as numbers and can peer at regional colo centers. * guifi.net (spain, mostly in catalunya, as49835) * ffdn.org (france) (see also: gitoyen.net, as20766) * freifunk.net (germany) (see also: community-ix.de) * ninux.org (italy, as197835) * riseup.net (as16652) Most networks can be found on peeringdb.com. You can also start poking at local NOG mailing lists (https://en.wikipedia.org/wiki/Internet_network_operators'_group https://en.wikipedia.org/wiki/Internet_network_operators'_gr...)

johngalt 7y ago

It's somewhat ad hoc. Generally you'll want to find who owns the actual infrastructure, and get as close to an agreement with them as possible. I.E. There will be some huge Zayo fiber that is resold by a bunch of different carriers. They are B2B only and don't sell directly to end users and have fairly high minimum commitments. Find the closest point that you can connect to it, and do wireless from that point outwards. Or you can do an interconnect/backhaul type agreement (L3 used to be great at these before they were acquired by Clink). This is where you are only buying a link back to a colocation. From there you can do direct peering or pick from several transit providers at pricing you can only get in that sort of environment. Fundamentally the problem is: If the fiber is there, then likely you'll have 50 carriers all reselling it and you are just number 51. If the fiber isn't there, then you'll have a difficult time providing suitable bandwidth.

jcrawfordor 7y ago

As a new ISP, you're not going to get any peering arrangements. Really, as an existing consumer ISP not in the top 10 or 20, you're not going to get any peering arrangements. Consumer ISPs generate very asymmetric traffic (more down than up) which tampers with the core "equality" assumption of traditional peering arrangements. You'll need to purchase transit from an existing provider, such as CenturyLink's commercial arm or Verizon Business, there are plenty more. This won't be free but it also probably won't be as expensive as you think. In a city you can get a Gigabit dedicated to you for $2k a month, maybe less. Shop around. Most of these providers will put you on the phone with a network engineer as part of the quoting process and they can be helpful in recommending what service level and equipment you get. IPv4 allocations are hard to come by unless you want to bid on them at auction which can be expensive. You probably want to get a block from your provider, but they'll be hard to talk into giving you more than say a /29 and they might charge you more for it. Often they don't, though, they just put you in a queue sorted by customer size and you'll never get to the top of it. As a small ISP, CGNAT is probably going to be your only choice for IPv4. This also has the downside of being non-portable since the addresses will always belong to your provider. Maybe save up money to bid on your very own. Once you get one you'll want to start participating in BGP which is complicated anyway, so get to that later, once you have customers.

rahimnathwani 7y ago
'Consumer ISPs generate very asymmetric traffic (more down than up) which tampers with the core "equality" assumption of traditional peering arrangements.' Shouldn't this be in favour of the ISP? If the traffic isn't balanced, then doesn't the peer that's sending more (e.g. YouTube, Netflix) usually pay? I have no experience of setting up an ISP, so this is just based on what I've read. I'd love to understand it better. "IPv4 allocations are hard to come by unless you want to bid on them at auction which can be expensive." Looking online, it looks like the prices are $15 to $30 per IP address, depending on the size of the block. So I guess this cost swallows the first month of revenue from each new customer.
- Scoundreller 7y ago
  > 'Consumer ISPs generate very asymmetric traffic (more down than up) which tampers with the core "equality" assumption of traditional peering arrangements.' It's a good situation for the ISP to start a DC business with unmetered ports to help balance things!
  - jcrawfordor 7y ago
    This is a lot of how the major ISPs smooth out the peering situation, they also offer colocation and carrier hotel services to large customers. Unfortunately, the popularity of Netflix and other streaming services has largely upset this balance and it's very hard for any ISP to keep traffic equality.
    - Scoundreller 7y ago
      At least with Netflix, you can get their Appliances and cut your inbound Netflix bandwidth by 80%+ for the cost of power and 4U.
      - iptrans 7y ago
        Yeah, if you have at least 5 Gbps of Netflix traffic on your network... No small or startup ISP is going to qualify for a Netflix appliance. Also to cut your Netflix bandwidth by 80+%, you need more than 6 Gbps of Netflix traffic, as Netflix requires 1.2 Gbps of inbound bandwidth just to fill the cache. Netflix share of Internet traffic is about 15-20%, so in practice Netflix appliances are used by ISPs with 25-35 Gbps of bandwidth and up. This means ISPs with tens of thousands of subscribers or more.

Scoundreller 7y ago

It's kinda a strange thing in Canada. The incumbent telecoms generally refuse to peer domestically with independent ISPs and smaller transit providers that cheaper DCs use. One ISP reported that Bell would only connect to them in Equinix in Chicago, even though Equinix has a PoP in Toronto. https://www.peeringdb.com/net/1550 https://www.peeringdb.com/net/1550 This means if you subscribe to internet with the local telephone monopoly (Bell) in Toronto, your traffic to toronto[1-8].voip.ms goes through NYC or Chicago. Have seen some routes that add Washington DC to the mix too. A 5km trip made into several thousand km. The incumbents even don't peer much at our big IX: http://torix.ca http://torix.ca (they buy ports, but don't peer freely). In a lot of ways, because the independents will peer with everyone and love IXs, your traffic often stays a lot more local in Canada.

dylz 7y ago
The US is the same. All major ISPs refuse to peer with any independents at all; refuse to be on routeservers; and demand some minimum of like 50-100 Gbps for a peering session, with insane requirements. I've seen numerous ones also demand things like RPKI/ROA and good filtering - while themselves having everything ROA and IRR invalid, announcing customer space with no IRR records whatsoever, etc.
- Scoundreller 7y ago
  So how do independents interconnect their routes? I think the Canadian issue is that incumbents aren't even peering with transit providers.
  - dylz 7y ago
    For this context, when I say independents, I mean smaller providers - content providers, independent ISPs that aren't large enough yet. They will obviously peer with Google, Akamai, etc. Independents more or less just buy transit/blend and deal with it. Here's an example of a large incumbent's peering policies: https://www.spectrum.com/content/dam/spectrum/residential/en/pdfs/policies/160525_Attachment_1_Charter_Communications_IP_Interconnect_Offer_and_Requirements.pdf https://www.spectrum.com/content/dam/spectrum/residential/en... The first requirement completely removes the possibility of ever getting a peering session with a small ISP or independent provider. So does the fourth. Another one: https://www.xfinity.com/peering https://www.xfinity.com/peering > minimum of four mutually agreeable geographically diverse points in the US. Interconnection points must include at least one city on the US east coast, one in the central region, and one on the US west coast This makes "launch an ISP from garage" bordering on "never" because they do not want regional providers getting settlement free peering - they just want them to buy IP transit from them. If you look at virtually every major US/CA ISP on peeringdb, they will be Restrictive / Selective or Private peering only, most likely.

lapinot 7y ago

As others have said, as a small isp you're gonna need to start off by buying transit. But the core philosophy is to peer with your peers, might sound stupid, but you should look towards other isp that look like you. There are already a lot of diy internet communities, most doing one kind or another of wireless mesh to connect to end-users. Some have as numbers and can peer at regional colo centers. * guifi.net (spain, mostly in catalunya, as49835) * ffdn.org (france) (see also: gitoyen.net, as20766) * freifunk.net (germany) (see also: community-ix.de) * ninux.org (italy, as197835) * riseup.net (as16652) Most networks can be found on peeringdb.com. You can also start poking at local NOG mailing lists (https://en.wikipedia.org/wiki/Internet_network_operators'_group https://en.wikipedia.org/wiki/Internet_network_operators'_gr...)