3 ms·

I rent a single ~600 meter fiber that connects my home to the network infrastructure of the company I run. The rent is $75/mo or so. I have passive CWDM muxes o

I rent a single ~600 meter fiber that connects my home to the network infrastructure of the company I run. The rent is $75/mo or so. I have passive CWDM muxes on the line to run two 10G connections to two separate edge routers. My home router is a MikroTik CRS309—it’s about $250, fanless, has 8 SFP+ ports and advertises my home network blocks over BGP to the two routers (for HA). The setup works great. The best part is how it’s small and fanless and fits inside the very small wall box the fiber terminates in.

techsupporter 4y ago
> My home router is a MikroTik CRS309 ... advertises my home network blocks over BGP How do you fit two full BGP tables into 512MB of RAM? I've looked into MikroTik boxes before and maybe they're doing something I'm not understanding. On the routers I have manage, two IPv4 feeds take up about 1.1GB and three IPv6 adds another 450MB.
- no_carrier 4y ago
  If by 'full BGP tables' you mean the entire internet routing table, you don't need that to advertise a network on the internet. You can receive just a default route if you wish and still be able to advertise to any carriers you have a connection with. You won't lose out on any functionality in dual homing or anything like that.
  - oarsinsync 4y ago
    > You can receive just a default route … You won't lose out on any functionality in dual homing or anything like that. Except you do lose out on best path routing / any other outbound TE, and you’re now restricted to rudimentary load balancing methodologies / manual prefix-specific hackery.
    - iptrans 4y ago
      Only for outbound traffic, tho. For eyeball networks and other stub networks this is mostly fine.
    - jpgvm 4y ago
      If you aren't multi-homed that doesn't matter at all though. For a home network I'm guessing you are pretty unlikely to do the multi-homing from the house, more likely you will either just have a single upstream or if you are connecting it into your own collocated infrastructure you will do iBGP and let your actual edge BGP routers handle the multi-homed upstreams and sync the full route table etc. I was tempted to do this once before when I was running my own hosting company but it was prohibitive cost wise to get the circuit I wanted. :(
      - oarsinsync 4y ago
        > If you aren't multi-homed that doesn't matter at all though. The parent I was responding to was explicitly claiming no downside to being default-only while multihomed.
    - martyvis 4y ago
      And that's why SD-WAN (software defined wide area network) technology now exists. In a lot of ways it is even better than BGP in that you direct traffic based on performance measurements - either actively through SLA tests or passively watching traffic flows and measuring latency. Using BGP routing based on hop counts and AS paths is akin to following road signs rather than getting live routing that knows traffic for instance from Google Maps
      - oarsinsync 4y ago
        SD-WAN technology also creates a number of new ways for unthinking people to shoot themselves in the foot. Noction is a wonderful example of this. It has sane defaults, but insane customers, who think that it’s a good idea for them to originate more specifics for other outside networks, because they’ll never leak them outside of their own AS (narrator: of course, the prefixes leaked). I think the most notable example of this recently was Verizon (the insane customer) using Noction (the SD-WAN technology) and doing exactly that, causing mass traffic disruption as they announced more specifics for other peoples prefixes, drawing all that traffic to their own network instead.
    - throw0101c 4y ago
      > Except you do lose out on best path routing / any other outbound TE, and you’re now restricted to rudimentary load balancing methodologies / manual prefix-specific hackery. After 3-4 hops you're probably hitting a Tier 1 network, after which point you can basically think of the Internet like cloud icon you see in many diagrams, because your route choices are no longer really determining reachability, rather the choices of other people/companies are: * https://en.wikipedia.org/wiki/Tier_1_network https://en.wikipedia.org/wiki/Tier_1_network If you're talking about reachability of a network on another continent or the other side of the planet, your local decisions aren't going to much to determine the path. The main thing to have locally for routing decisions is the ASNs/networks of the other customers of your ISPs: if Service A is also a customer of ISP #1, you want to send traffic for them through that service instead of ISP #2. The other nice thing to have is the reachability to the closest IXP, as quite often many CDNs have connections to those. Beyond knowing IXP reachability and other-customers reachability, I don't think there are many other advantages for a smaller entity on the Internet, so a full Internet-wide BGP is not needed.
- markonen 4y ago
  I’m just getting the default route from both upstream routers. They’re not diverse in terms of the internet routes they have, this is just for HA.
  - throw0101c 4y ago
    > I’m just getting the default route from both upstream routers. If they offer the option, instead of a "default-only" feed from each ISP, you may wish to see if they have "default-plus-our-customers" feed: if Service A is also a customer of ISP #1, then why bother sending the packets to ISP #2 in the first place? * https://support.allstream.com/knowledge-base/bgp-request-information/ https://support.allstream.com/knowledge-base/bgp-request-inf... In general, at some point you'll hit a Tier 1 network, after which it won't matter, but until that point getting the connectivity to other customers of the ISPs could be useful. The other reachability destination to pay attention to would be of IXPs, where CDNs often connect to. * https://bgp.he.net/ https://bgp.he.net/ * https://bgpview.io/ https://bgpview.io/
    - markonen 4y ago
      Both of the upstream routers are mine, there’s no difference to the routes they have available. (They’re connected both to Tier 1 transit as well as the major local IXP)
- ilyt 4y ago
  Use BIRD: bird> show memory BIRD memory usage Routing tables: 262 MB Route attributes: 120 MB ROA tables: 192 B Protocols: 171 kB Total: 382 MB bird> show route count table r1 906312 of 906312 routes for 906312 networks bird> show route count table r2 903532 of 903532 routes for 903532 networks bird> routes in kernel will take less as you're only getting best one exported to kernel and no route attributes to hold
- tiernano 4y ago
  I have their RB5009 with 1Gb RAM and I got 4 full feeds and have around 200Mb free. I do need to tweak stuff to get IPv6 working (again... only been sorting this out in the last couple weeks)

techsupporter 4y ago

> My home router is a MikroTik CRS309 ... advertises my home network blocks over BGP How do you fit two full BGP tables into 512MB of RAM? I've looked into MikroTik boxes before and maybe they're doing something I'm not understanding. On the routers I have manage, two IPv4 feeds take up about 1.1GB and three IPv6 adds another 450MB.

no_carrier 4y ago
If by 'full BGP tables' you mean the entire internet routing table, you don't need that to advertise a network on the internet. You can receive just a default route if you wish and still be able to advertise to any carriers you have a connection with. You won't lose out on any functionality in dual homing or anything like that.
- oarsinsync 4y ago
  > You can receive just a default route … You won't lose out on any functionality in dual homing or anything like that. Except you do lose out on best path routing / any other outbound TE, and you’re now restricted to rudimentary load balancing methodologies / manual prefix-specific hackery.
  - iptrans 4y ago
    Only for outbound traffic, tho. For eyeball networks and other stub networks this is mostly fine.
  - jpgvm 4y ago
    If you aren't multi-homed that doesn't matter at all though. For a home network I'm guessing you are pretty unlikely to do the multi-homing from the house, more likely you will either just have a single upstream or if you are connecting it into your own collocated infrastructure you will do iBGP and let your actual edge BGP routers handle the multi-homed upstreams and sync the full route table etc. I was tempted to do this once before when I was running my own hosting company but it was prohibitive cost wise to get the circuit I wanted. :(
    - oarsinsync 4y ago
      > If you aren't multi-homed that doesn't matter at all though. The parent I was responding to was explicitly claiming no downside to being default-only while multihomed.
  - martyvis 4y ago
    And that's why SD-WAN (software defined wide area network) technology now exists. In a lot of ways it is even better than BGP in that you direct traffic based on performance measurements - either actively through SLA tests or passively watching traffic flows and measuring latency. Using BGP routing based on hop counts and AS paths is akin to following road signs rather than getting live routing that knows traffic for instance from Google Maps
    - oarsinsync 4y ago
      SD-WAN technology also creates a number of new ways for unthinking people to shoot themselves in the foot. Noction is a wonderful example of this. It has sane defaults, but insane customers, who think that it’s a good idea for them to originate more specifics for other outside networks, because they’ll never leak them outside of their own AS (narrator: of course, the prefixes leaked). I think the most notable example of this recently was Verizon (the insane customer) using Noction (the SD-WAN technology) and doing exactly that, causing mass traffic disruption as they announced more specifics for other peoples prefixes, drawing all that traffic to their own network instead.
  - throw0101c 4y ago
    > Except you do lose out on best path routing / any other outbound TE, and you’re now restricted to rudimentary load balancing methodologies / manual prefix-specific hackery. After 3-4 hops you're probably hitting a Tier 1 network, after which point you can basically think of the Internet like cloud icon you see in many diagrams, because your route choices are no longer really determining reachability, rather the choices of other people/companies are: * https://en.wikipedia.org/wiki/Tier_1_network https://en.wikipedia.org/wiki/Tier_1_network If you're talking about reachability of a network on another continent or the other side of the planet, your local decisions aren't going to much to determine the path. The main thing to have locally for routing decisions is the ASNs/networks of the other customers of your ISPs: if Service A is also a customer of ISP #1, you want to send traffic for them through that service instead of ISP #2. The other nice thing to have is the reachability to the closest IXP, as quite often many CDNs have connections to those. Beyond knowing IXP reachability and other-customers reachability, I don't think there are many other advantages for a smaller entity on the Internet, so a full Internet-wide BGP is not needed.
markonen 4y ago
I’m just getting the default route from both upstream routers. They’re not diverse in terms of the internet routes they have, this is just for HA.
- throw0101c 4y ago
  > I’m just getting the default route from both upstream routers. If they offer the option, instead of a "default-only" feed from each ISP, you may wish to see if they have "default-plus-our-customers" feed: if Service A is also a customer of ISP #1, then why bother sending the packets to ISP #2 in the first place? * https://support.allstream.com/knowledge-base/bgp-request-information/ https://support.allstream.com/knowledge-base/bgp-request-inf... In general, at some point you'll hit a Tier 1 network, after which it won't matter, but until that point getting the connectivity to other customers of the ISPs could be useful. The other reachability destination to pay attention to would be of IXPs, where CDNs often connect to. * https://bgp.he.net/ https://bgp.he.net/ * https://bgpview.io/ https://bgpview.io/
  - markonen 4y ago
    Both of the upstream routers are mine, there’s no difference to the routes they have available. (They’re connected both to Tier 1 transit as well as the major local IXP)
ilyt 4y ago
Use BIRD: bird> show memory BIRD memory usage Routing tables: 262 MB Route attributes: 120 MB ROA tables: 192 B Protocols: 171 kB Total: 382 MB bird> show route count table r1 906312 of 906312 routes for 906312 networks bird> show route count table r2 903532 of 903532 routes for 903532 networks bird> routes in kernel will take less as you're only getting best one exported to kernel and no route attributes to hold
tiernano 4y ago
I have their RB5009 with 1Gb RAM and I got 4 full feeds and have around 200Mb free. I do need to tweak stuff to get IPv6 working (again... only been sorting this out in the last couple weeks)