4 ms·

Good stuff. However - only one Linux router (VM) which means that you can't upgrade it and reboot without loss of service. The way around that is two VMs and

Good stuff. However - only one Linux router (VM) which means that you can't upgrade it and reboot without loss of service. The way around that is two VMs and VRRP or similar and a lot of very complicated NAT and firewall rules. Out of the box, pfSense can do multi WAN and CARP (similar to VRRP) clustering. At the office I have two older servers with lots of NICs and five WANs. Inbound redundancy is provided by dynamic DNS and SRV records etc. Note that to do CARP/VRRP, you do need at least a /29 IPv4 allocation. You need an address per box plus the virtual one that is actually used by services. PPPoA/E is harder to deal with than cable/leased line etc but it turns out that low cost Billion 8800NLR2 can do external IPv4 pass through as well as do the PPPoA/E. They will need an address as well from your range. You need something like them in this case because only one device can be the PPPoA/E dial up system at a time. Unless you have some very fancy secret sauce, your clustered routers' pppd or whatever are going to get confused as to who does what. I notice you have a cloud key. Unifi on an Ubuntu VM is easy, and much easier to backup and snapshot before upgrades, so is safer. You can also front it with HA Proxy for simple URLs and perhaps Lets Encrypt. pfSense has a HA Proxy package with a GUI and I believe it is CARP friendly as well ...

chrissnell 8y ago
Unfortunately, OP is using Centurylink fiber. It's been a few years since I lived in Tacoma, WA and used this service but it's something like PPPoE over VLAN. There was a FreeBSD bug a few years back where PPPoE was ridiculously slow when running on top of a VLAN interface. OpenBSD did not have this problem, which is why I ran that for a firewall instead of my preferred pfSense.
- gerdesj 8y ago
  I have four FTTC (PPPoE/A) WANs and a BT (UK) leased line at work. The FTTCs are 80/20Mbs-1 and the leased line is symmetric 100Mbs-1. I've put all five WANs down a separate 802.1q VLAN. Each of my routers has one physical NIC (Intel 1Gb) dedicated to WANs. The other nine NICs, each, are for internal VLANs. I use Draytek 120 or 130s modems for single ADSL or FTTC connections but for CARP clusters, I use Billion Bipac 8800NLR2, so I am not doing the PPPoA/E on the pfSense boxes. The Billions are able pass through bits of a /29 and do the PPPoA/E themselves - the only cheap router (~£60) I've found to do this. I've been running this thing for about four years now. PPPox is a complex beast and there are a few things to look out for such as MTU. PPPoE imposes an eight byte overhead (hence 1492) and back in the day some ill advised auth mechanism required setting a 1458 byte MTU. Apparently, some BT kit supports mini Jumbo frames of 1508 bytes which means that you could set your MTU to 1500 instead of 1492 - good luck with that as a rule of thumb. $DEITY only knows what an ISP in WA has arbitrarily decided to mandate. Here in the UK we have a near monopoly for the infrastructure but lots of providers that use it and so it should be simple. To be fair, I bet you don't get docs like this: https://www.btplc.com/SINet/SINs/index.htm https://www.btplc.com/SINet/SINs/index.htm (498 is FTTC) Anyway, if you are happy maintaining your firewall rule set manually then crack on but nowadays it is hard to do that. pfSense has a lot of quite vociferous users who kick the tyres on a regular basis. It even looks quite pretty these days - all bootstrapped up and stuff, the red thing is long gone.
  - jlgaddis 8y ago
    > The FTTCs are 80/20Mbs-1 and the leased line is symmetric 100Mbs-1. What's the "-1" in "80/20Mbs-1" and "100Mbs-1" signify? I've never seen this "syntax" or formst used before but maybe it's an EU/UK thing (I'm in .us, FWIW)?
    - F_r_k 8y ago
      OP has a scientific background. -1 is meant as "to the power of -1". Thus, s-1 becomes 1/s, and the entire thing Mb/s Never seen that either
      - gerdesj 8y ago
        OP has a scientific background. - LOL - I have an HND (technician) in Civil Engineering and I am now the MD of an IT consultancy (obvs). I picked up the habit of using s-1 etc when studying Physics 'A' level (UK) many, many moons ago. Not too sure why I persist with it these days but I dimly remember liking the fact that you can use basic arithmetic on superscripts. To be fair I should put s^-1 but s-1 is reasonably obvious.
    - gerdesj 8y ago
      Sorry, "s-1" means "per second"

chrissnell 8y ago

Unfortunately, OP is using Centurylink fiber. It's been a few years since I lived in Tacoma, WA and used this service but it's something like PPPoE over VLAN. There was a FreeBSD bug a few years back where PPPoE was ridiculously slow when running on top of a VLAN interface. OpenBSD did not have this problem, which is why I ran that for a firewall instead of my preferred pfSense.

gerdesj 8y ago
I have four FTTC (PPPoE/A) WANs and a BT (UK) leased line at work. The FTTCs are 80/20Mbs-1 and the leased line is symmetric 100Mbs-1. I've put all five WANs down a separate 802.1q VLAN. Each of my routers has one physical NIC (Intel 1Gb) dedicated to WANs. The other nine NICs, each, are for internal VLANs. I use Draytek 120 or 130s modems for single ADSL or FTTC connections but for CARP clusters, I use Billion Bipac 8800NLR2, so I am not doing the PPPoA/E on the pfSense boxes. The Billions are able pass through bits of a /29 and do the PPPoA/E themselves - the only cheap router (~£60) I've found to do this. I've been running this thing for about four years now. PPPox is a complex beast and there are a few things to look out for such as MTU. PPPoE imposes an eight byte overhead (hence 1492) and back in the day some ill advised auth mechanism required setting a 1458 byte MTU. Apparently, some BT kit supports mini Jumbo frames of 1508 bytes which means that you could set your MTU to 1500 instead of 1492 - good luck with that as a rule of thumb. $DEITY only knows what an ISP in WA has arbitrarily decided to mandate. Here in the UK we have a near monopoly for the infrastructure but lots of providers that use it and so it should be simple. To be fair, I bet you don't get docs like this: https://www.btplc.com/SINet/SINs/index.htm https://www.btplc.com/SINet/SINs/index.htm (498 is FTTC) Anyway, if you are happy maintaining your firewall rule set manually then crack on but nowadays it is hard to do that. pfSense has a lot of quite vociferous users who kick the tyres on a regular basis. It even looks quite pretty these days - all bootstrapped up and stuff, the red thing is long gone.
- jlgaddis 8y ago
  > The FTTCs are 80/20Mbs-1 and the leased line is symmetric 100Mbs-1. What's the "-1" in "80/20Mbs-1" and "100Mbs-1" signify? I've never seen this "syntax" or formst used before but maybe it's an EU/UK thing (I'm in .us, FWIW)?
  - F_r_k 8y ago
    OP has a scientific background. -1 is meant as "to the power of -1". Thus, s-1 becomes 1/s, and the entire thing Mb/s Never seen that either
    - gerdesj 8y ago
      OP has a scientific background. - LOL - I have an HND (technician) in Civil Engineering and I am now the MD of an IT consultancy (obvs). I picked up the habit of using s-1 etc when studying Physics 'A' level (UK) many, many moons ago. Not too sure why I persist with it these days but I dimly remember liking the fact that you can use basic arithmetic on superscripts. To be fair I should put s^-1 but s-1 is reasonably obvious.
  - gerdesj 8y ago
    Sorry, "s-1" means "per second"