4 ms·

Sounds like you would be better picking up a decent transit locally to you with a good blend of upstreams.

by josephb 14y ago

lsc 14y ago
that's what they say... but the first rule of business is that you will get fucked if it's hard for you to switch providers. I'm already in that position with datacentere space. I've been in that position with bandwidth, too, and it's super painful. Also, nobody is going to give a shit about your three customers way out in the middle of "nobody cares" that can't reach your stuff 'cause cogent and at&t refuse to upgrade a peering link that is also in the middle of "nobody cares" - If I'm controlling it, I at least have a chance of doing something about it. Also, the cost is 3x to 10x what you get direct from the super-low end big guys. Everyone fucks it up, and with two up-streams? It's my fault when it fucks up (and I can do something about it.) I guess having that preference is kinda stupid, but it is what it is; I am the network Engineer I've got... not the network Engineer I wish I had. (that, and in my experience? competency of tech support does not scale with price paid. Neither does quality of routes.)
- jnw2 14y ago
  If the problem turns out to be that AT&T and Cogent's peering links are overloaded, and the links between AT&T and Hurricane Electric don't have similar problems, then your border routers should be setting localpref to something low for any route with AT&T's ASN learned on the Cogent BGP session, which should send your outbound traffic to AT&T via Hurricane Electric except when your he.net link is down. The opposite direction looks more problematic. Some ISPs will let you tag your routes with BGP communities that say ``when advertising to AT&T, prepend some extra copies of some ASN so that AT&T will be unlikely to send me traffic via this path.'' http://www.onesc.net/communities/as174/ http://www.onesc.net/communities/as174/ does not list any communities which would affect how Cogent advertises routes to AT&T without affecting how Cogent advertises to all of their other peers. On the other hand, while 174:3001 is an extremely coarse tool, your traffic ratios may be sufficiently outbound only that sending most of your inbound traffic via Hurricane Electric might turn out OK, even with Hurricane Electric being only 1G and Cogent 10G (although this change might make your network less robust in the face of a DDOS attack, and you might want to drop 174:3001 when dealing with such an attack). If the overloaded links are only overloaded in one direction, then you may only need to do one of these things, not both.
  - lsc 14y ago
    I tried switching my outbound traffic to the customers in question over to he.net (with a static route, which I know is the dumb way to do it.) with no improvement. Hm. I need to learn more about bgp, specifically tags and communities, can you recommend me a book?
    - jnw2 14y ago
      I ended up learning most of what I know about BGP not via a book, which makes that hard. But basically, back in the days of 16 bit ASNs, communities were just 32 bit values that were attached to routes being advertised by BGP, usually expressed as the two 16 bit halves separated by a colon, generally with the ASN that defines the meaning of the community being on the left of the colon, and that ASN defining arbitrary values on the right side of the colon for flags that they want to recognize. Then, there are route maps or whatever the router vendor wants to call them that use the data from the communities. Typically when the router is deciding what route to use, it looks for the route with the highest localpref, and if there's a tie, then it looks at how many ASNs are listed in the path, and if there's still a tie, then the router may use MED. http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431.shtml http://www.cisco.com/en/US/tech/tk365/technologies_tech_note... explains how this works in the Cisco BGP implementation with all of the additional factors that are considered; Linux and Juniper BGP implementations may consider a slightly different set of factors. IIRC, Level 3 generally sets localpref to 86 on routes heard on non-customer peer sessions, and 100 on routes heard on customer BGP sessions, but if customers want localpref to be 90 or 80 or 70 instead, they can send community 3356:90 or 3356:80 or 3356:70. If you had two links to Level 3 and wanted to use one as backup only, you could send community 3356:90 on your routes on your backup link, and potentially no communities on your primary link. If you had a really slow link to Level 3 that was backup only and some other ASN was your primary, then you'd want to send Level 3 3356:80 or 3356:70. I doubt you have any desire to set your upstream's localpref values in your configuration, but this may shed some light on why they offer it to their other customers. Also, if you ever have customers speaking BGP who want to use you as backup only, you need to offer some way for routes they advertise to you to have localpref less than what you set on routes you learn from your upstreams; typically this seems to be done by letting your customers send you a community so the customer controls this, but you could just have a special route-map for customers who are backup only that would just set all of their routes to low localpref regardless of the BGP communities, and just apply the right route-map to those BGP sessions. http://www.onesc.net/communities/as3356/ http://www.onesc.net/communities/as3356/ mentions that Level 3 supports 65004:[ASN] for four prepends when advertising to ASN. If you were a Level 3 customer, and traffic coming to you via one of their peering links was overloaded, sending a community of that form would be a good way to discourage traffic from taking that link by making the path have a large number of ASNs and thus look undesirable. I say this mostly to illustrate the sort of thing that it looks like you might find useful for Cogent to support, though it appears that Cogent doesn't support that. It's likely that Hurricane Electric and Cogent will recognize some communities that you could send them that will influence their routing, but as far as I know, their peers won't necessarily end up accepting any communities from you, so the key is to either find documentation on the web of what communties Hurricane Electric and Cogent recognize, or ask them for it. You can also sometimes get geographic data about which major city is closest to a network block by recieving community data from an upstrem, but it's not obvious to me why you'd want that. It may sometimes be used when hot potato routing is not desired, or when a CDN is trying to identify which geographically close server to use.
- jnw2 14y ago
  Do you have traceroutes that demonstrate that the path goes through Cogent and AT&T in both directions? Asymmetric routing is extremely common for things like this.
  - lsc 14y ago
    not now. Next time I get a complaint like this, though, I will attempt to document the debug process in more detail, in public.

Sounds like you would be better picking up a decent transit locally to you with a good blend of upstreams.

by josephb 14y ago

Sounds like you would be better picking up a decent transit locally to you with a good blend of upstreams.

lsc 14y ago

that's what they say... but the first rule of business is that you will get fucked if it's hard for you to switch providers. I'm already in that position with datacentere space. I've been in that position with bandwidth, too, and it's super painful. Also, nobody is going to give a shit about your three customers way out in the middle of "nobody cares" that can't reach your stuff 'cause cogent and at&t refuse to upgrade a peering link that is also in the middle of "nobody cares" - If I'm controlling it, I at least have a chance of doing something about it. Also, the cost is 3x to 10x what you get direct from the super-low end big guys. Everyone fucks it up, and with two up-streams? It's my fault when it fucks up (and I can do something about it.) I guess having that preference is kinda stupid, but it is what it is; I am the network Engineer I've got... not the network Engineer I wish I had. (that, and in my experience? competency of tech support does not scale with price paid. Neither does quality of routes.)

jnw2 14y ago
If the problem turns out to be that AT&T and Cogent's peering links are overloaded, and the links between AT&T and Hurricane Electric don't have similar problems, then your border routers should be setting localpref to something low for any route with AT&T's ASN learned on the Cogent BGP session, which should send your outbound traffic to AT&T via Hurricane Electric except when your he.net link is down. The opposite direction looks more problematic. Some ISPs will let you tag your routes with BGP communities that say ``when advertising to AT&T, prepend some extra copies of some ASN so that AT&T will be unlikely to send me traffic via this path.'' http://www.onesc.net/communities/as174/ http://www.onesc.net/communities/as174/ does not list any communities which would affect how Cogent advertises routes to AT&T without affecting how Cogent advertises to all of their other peers. On the other hand, while 174:3001 is an extremely coarse tool, your traffic ratios may be sufficiently outbound only that sending most of your inbound traffic via Hurricane Electric might turn out OK, even with Hurricane Electric being only 1G and Cogent 10G (although this change might make your network less robust in the face of a DDOS attack, and you might want to drop 174:3001 when dealing with such an attack). If the overloaded links are only overloaded in one direction, then you may only need to do one of these things, not both.
- lsc 14y ago
  I tried switching my outbound traffic to the customers in question over to he.net (with a static route, which I know is the dumb way to do it.) with no improvement. Hm. I need to learn more about bgp, specifically tags and communities, can you recommend me a book?
  - jnw2 14y ago
    I ended up learning most of what I know about BGP not via a book, which makes that hard. But basically, back in the days of 16 bit ASNs, communities were just 32 bit values that were attached to routes being advertised by BGP, usually expressed as the two 16 bit halves separated by a colon, generally with the ASN that defines the meaning of the community being on the left of the colon, and that ASN defining arbitrary values on the right side of the colon for flags that they want to recognize. Then, there are route maps or whatever the router vendor wants to call them that use the data from the communities. Typically when the router is deciding what route to use, it looks for the route with the highest localpref, and if there's a tie, then it looks at how many ASNs are listed in the path, and if there's still a tie, then the router may use MED. http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431.shtml http://www.cisco.com/en/US/tech/tk365/technologies_tech_note... explains how this works in the Cisco BGP implementation with all of the additional factors that are considered; Linux and Juniper BGP implementations may consider a slightly different set of factors. IIRC, Level 3 generally sets localpref to 86 on routes heard on non-customer peer sessions, and 100 on routes heard on customer BGP sessions, but if customers want localpref to be 90 or 80 or 70 instead, they can send community 3356:90 or 3356:80 or 3356:70. If you had two links to Level 3 and wanted to use one as backup only, you could send community 3356:90 on your routes on your backup link, and potentially no communities on your primary link. If you had a really slow link to Level 3 that was backup only and some other ASN was your primary, then you'd want to send Level 3 3356:80 or 3356:70. I doubt you have any desire to set your upstream's localpref values in your configuration, but this may shed some light on why they offer it to their other customers. Also, if you ever have customers speaking BGP who want to use you as backup only, you need to offer some way for routes they advertise to you to have localpref less than what you set on routes you learn from your upstreams; typically this seems to be done by letting your customers send you a community so the customer controls this, but you could just have a special route-map for customers who are backup only that would just set all of their routes to low localpref regardless of the BGP communities, and just apply the right route-map to those BGP sessions. http://www.onesc.net/communities/as3356/ http://www.onesc.net/communities/as3356/ mentions that Level 3 supports 65004:[ASN] for four prepends when advertising to ASN. If you were a Level 3 customer, and traffic coming to you via one of their peering links was overloaded, sending a community of that form would be a good way to discourage traffic from taking that link by making the path have a large number of ASNs and thus look undesirable. I say this mostly to illustrate the sort of thing that it looks like you might find useful for Cogent to support, though it appears that Cogent doesn't support that. It's likely that Hurricane Electric and Cogent will recognize some communities that you could send them that will influence their routing, but as far as I know, their peers won't necessarily end up accepting any communities from you, so the key is to either find documentation on the web of what communties Hurricane Electric and Cogent recognize, or ask them for it. You can also sometimes get geographic data about which major city is closest to a network block by recieving community data from an upstrem, but it's not obvious to me why you'd want that. It may sometimes be used when hot potato routing is not desired, or when a CDN is trying to identify which geographically close server to use.
jnw2 14y ago
Do you have traceroutes that demonstrate that the path goes through Cogent and AT&T in both directions? Asymmetric routing is extremely common for things like this.
- lsc 14y ago
  not now. Next time I get a complaint like this, though, I will attempt to document the debug process in more detail, in public.