8 ms·

VyOS From Scratch – Edition 1 (set up your own router)

121 points by elisaado 3y ago

jstanley 3y ago
Might be worth adding a paragraph about what VyOS is, maybe not for the target audience but certainly for HN. https://vyos.io/ https://vyos.io/ The VyOS website says: > Democratizing how we access networks through a universal Router and Open source software. > Our vision at VyOS is to dramatically change how we access networks so that we can all build the solutions we always dreamed of, without restrictions, limitations, or prohibitive costs. But I'm still kind of none the wiser. Does this thing use Linux or something else?
- minimaul 3y ago
  Yes - it's a Debian Linux based router distro that is at least partially modelled after Juniper's configuration style. edit: used to be a maintainer for a short while :)
- rnhmjoj 3y ago
  It's a fork vyatta[1], the same system used by Ubiquiti for EdgeOS. Yes, it's Debian with a declarative configuration system. It works more or less similarly to NixOS, if you know about that: basically it uses a bunch of perl scripts to install and set up software from a single unified configuration file, hiding all the implementation details. [1]: https://en.wikipedia.org/wiki/Vyatta https://en.wikipedia.org/wiki/Vyatta
  - sebazzz 3y ago
    And it works great because your router starts from a read-only image, then executes the scripts and applies the configuration. You can easily roll back and forward configuration and firmware versions.
  - LeBit 3y ago
    I don't think they use Perl anymore. They have been moving to Python.
  - nunez 3y ago
    Had no idea vyatta is what Ubiquiti uses. Very cool.
    - paradox460 3y ago
      Thats only on EdgeOS though. UbiquitiOS uses a different system, built around Podman and Debian
- yrro 3y ago
  It's a Network Operating System for configuring routers & switches (implemented on top of Debian). This means: * You can SSH in and configure it like you would a managed switch or router. There's a single object that models all of the device's configuration, when you commit it a bunch of scripts activate and actaully apply the configuration to the running system. * Deployments are image based, you can roll back to older images etc. * You don't need to look at any of the system's underlying configuration files or use any of the normal Linux commands to examine and manipulate the state of the system (the commands are still there for convenience of course). You don't even need to be aware that you're really using a bunch of custom bash functions to examine and manipulate the state of the system.
- Arnavion 3y ago
  It's a Debian 8 base with a lot of its own custom packages. The homepage has an FAQ that tries to reassure you that being based on Debian 8 is not a problem. Whether that convinces you or not is up to you, of course. I personally use regular Debian 12 on my router without problems. It also has "declarative config" since all the configuration, firewall rules, etc are a bunch of config files that I can scp / ansible over any time.
PreInternet01 3y ago
The question 'what is this thing' is probably best answered by the Github project page: https://github.com/vyos https://github.com/vyos It's a decent-ish option if you need advanced routing functionality; one thing to keep in mind, though, is that unless you're OK with running unstable 'nightly' code, you'll be spending USD 8K+ on an annual basis.
- awesomeMilou 3y ago
  https://support.vyos.io/en/support/solutions/103000152091 https://support.vyos.io/en/support/solutions/103000152091 They have an LTS release, no? They seem to follow the RedHat strategy though, only subscribers can download prebuilt images, but you can build the LTS ones yourself: https://blog.vyos.io/vyos-1.3.2-lts-release https://blog.vyos.io/vyos-1.3.2-lts-release
- blinkingled 3y ago
  > If you are an individual, you can get the generic ISO by donating on Open Collective. And if you are contributing to VyOS, whether you are writing code, improving the docs, or promoting VyOS publicly, we are happy to share pre-built images with you through contributor subscriptions. Finally, you can always build your own images — just follow these instructions. Sounds fair to me. Truth is there's no good alternative other than pfSense but if you want Linux (hw support etc) I don't know if you can do better than vyos for routers.
  - laurowyn 3y ago
    If you want a web GUI, then pfSense or OPNSense are the general go tos. However, if you're comfortable with CLI and modifying configs in /etc/ then just running a bare metal Alpine Linux box is perfectly doable on a tiny box. iptables/nftables for firewall/NAT, dnsmasq/bind9 for dns, dnsmasq/isc-dhcp for DHCP. I've got a handful of these boxes all interlinked via wireguard, sharing routes via BGP using bird. Sure, you miss the config verification that VyOS provides, but does mean you learn the underlying tools themselves and that knowledge is portable to any other box running those systems. Personally, I don't quite understand why VyOS is a standalone distro when it could just be a config generator/checker package. Could even support multiple different underlying tools so if you want to use dnsmasq over bind9, or vice versa, it can provide a unified config interface for them.
awesomeMilou 3y ago
VyOS is unfortunately completely useless for larger applications, since it's difficult to impossible to automate due to it's unique way of applying configurations. Don't get me wrong, for manual administration it's great, there's a lot of missed automation potential given that it's just Linux underneath. As an example, the Ansible modules for VyOS are basicially just variations of an adapted ansible.builtin.shell, instead of offering to manage state in a more first class manner (via attributes and values): https://docs.ansible.com/ansible/latest/collections/vyos/vyos/vyos_config_module.html#ansible-collections-vyos-vyos-vyos-config-module https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
- LeBit 3y ago
  You can ssh into the router, copy a new config (state) and load that config. It is not very elegant though. Do you know of an open source router that does what you are looking for?
  - chomp 3y ago
    That doesn’t sound that far off from what unifi does, it uses a daemon to do those things. (Receive configuration and then do a local apply)
    - synergy20 3y ago
      unifi runs vyos underneath
      - zrail 3y ago
        UniFi doesn't use VyOS anywhere afaik. Some Ubiquiti products run EdgeOS which is a Vyatta fork (EdgeRouter, the original security gateways). More recent UniFi products (UDM, UDM-SE, UNVR, etc) run UniFi OS which is their own Debian-based OS that runs applications inside Podman containers.
- jon-wood 3y ago
  From what I've seen of VyOS using a configuration file that is then used to generate the actual system configuration I'm not really sure its so hard to automate. Take your target state, generate a configuration file in the right format, then send it over and apply. To be honest this feels more like a limitation in Ansible, which has always felt like a bit of a hacky config management system to me in that the way it functions is generally to run a bunch of commands that gradually mutate the system's state, rather than atomically applying the target state, but then I've been spoiled by NixOS on my personal infrastructure recently.
amar0c 3y ago
Unfortunately they went full "Vyatta way" or "RedHat Way" by basically giving rolling release for free only. I remember times when Vyatta went behind paywalls Vyos was completely free. Not sure who would want something rolling on device like router. Nowdays everyone wanting something good and free go OpnSense way.
- rnhmjoj 3y ago
  The biggest problem I have is that they only support (or at least that's what they release publicly) x86_64. I am forced to use openwrt because the vast majority of consumer low-power hardware is using ARM or exotic architectures. VyOS interface is vastly superior, though.
  - sp0ck 3y ago
    VyOS is not targeted for such usage scenarios. It is ISP who are paying for support. I have VyOS systems with BGP, MPLS, PPPoE termination, etc that are pushing gigabits of traffic. This is VyOS usecase. Not 5W router handling home or small office.
    - rnhmjoj 3y ago
      I've used EdgeOS in the past and it's perfect for SOHO routers too, but unfortunately is half full of proprietary extensions and semi-abandoned. I think VyOS could take its place nicely, openwrt feels like a toy compared.
- thepill 3y ago
  You can build your own LTS image relatively easy and free of charge
  - LeBit 3y ago
    The process is not straightforward, especially if you want to customize the image. And I have been unable to build a cloud-init image. But the ISO is buildable from the LTS branch and it works well.
pshirshov 3y ago
From my experience, Nix makes lot more reliable and easier to configure router. ymmv
- LeBit 3y ago
  You are comparing bananas and windows.
- rnhmjoj 3y ago
  I just finished building a router based on NixOS and I must disagree. NixOS modules mostly targets desktop and servers, but the network specific configuration is still very lacking. A few examples: - up until a couple of weeks ago the hostapd module was basically a toy: could only manage a single SSID, no way to configure the radios, hardcoded to WPA2-PSK; - the NixOS firewall is still based on iptables and conflicts with nftables, so you must disable and manually write rules; - the `networking.nat` module (NAT44) doesn't do NAT reflection; - I had to write a module for Jool (NAT64, SIIT); - I had to write a module for libreswan (IPsec); - I had to write a module for automatic rollbacks, otherwise you can loose access if you make a mistake. Vyatta and VyOS also provide a much higher level abstraction over the software that is being configured (e.g. you don't have to deal with a specific IPsec implementation). Finally, once you do `nixos-rebuild switch` you're on your own, while with vyatta you have a clean command line interface to inspect the state of the router and manage it.
  - gbraad 3y ago
    does vyatta/vyos store the configuration in a single file or easy to back up config? I do not remember anymore how this was done, but I believe that is the appeal what 'nix' was mentioned.
    - rnhmjoj 3y ago
      Yes, it does.
  - solarkraft 3y ago
    Thanks for the warning; sounds like my mileage would have varied quite a bit. Do you think that, had your use case been simpler, you'd have enjoyed the experience?
lifeisstillgood 3y ago
A definite tangent: About three house moves ago I had OpenWRT on a cable router and knew what was going on. But with family and work and house moves I am now just staring at a flashing BT Home Hub and wondering how to tackle the inevitable "of course that router firmware wont allow that and your ISP wont give out its password and ..." Is there a uptodate reliable guide (possibly including how to persuade your wife it's a good idea to drill holes in the living room ceiling to run cat6)
- trustingtrust 3y ago
  What I find easy(ier?) is to run (x)sense on a dedicated firewall and either a mesh with cheap openwrt routers or get something like Deco mesh and run it in AP mode if you don’t have cat6 at home. I think this combination can be under 300$ for a 3-pack of mesh Deco x20 + an intel card on a refurb dell optiplex.
  - dgroshev 3y ago
    IME (x)sense is quite problematic when you start doing anything other than the bare minimum. I spent multiple days trying to figure out why it silently stopped accepting IPv6 delegation, or why does it spike latency for no visible reason under load. The underlying reality is that FreeBSD's network stack is much more conservative and has less resources than Linux's, which shows up in articles like this one [1]. On the same Celeron J4125/i226 box VyOS was absolutely perfect, not a single issue, significantly low (and always low) latency with higher throughput. On the hardware side, I think the /r/homelab hivemind doesn't get challenged enough. Dell optiplexes cost very similarly to Aliexpress Protectli alternatives (such as [2]), while being larger, having a fan, and being overall more hassle. TP Link/Ubiquiti WiFi APs seem to be overall inferior to Aruba Instant On, which is exactly the same hardware HP sells in their Aruba line, but for the same SOHO price. [1] https://teklager.se/en/knowledge-base/opnsense-performance-optimization/ https://teklager.se/en/knowledge-base/opnsense-performance-o... [2] https://www.aliexpress.com/item/1005004272231167.html https://www.aliexpress.com/item/1005004272231167.html
    - trustingtrust 3y ago
gbraad 3y ago
I was wondering if this was Vyatta... it seems it is the community continuation after Broccade ceased development. Used to run this over a decade ago on a 'router' (ThinkCentre Tiny), but eventually went to a Fedora installation a few years back. Might have a look again.
- gbraad 3y ago
  ... have to say that the offering is confusing; it is a subscription or a rolling release. Wondering how much can be automated of the installation/state; would it be possible to use version control? If not, I can see the appeal to suggest Nix over this.
  - kmbfjr 3y ago
    Both. Pay for a subscription or use the RR. You can also build your own images with docker using the annotated releases.
dgroshev 3y ago
I was pretty confused what it is too and then I loved it. It's debian plus some shell trickery and CLI tools that let you configure debian and debian packages as a router from one large config tree using neat CLI tools (that support commit/rollback). Normally you'd need iptables, a separate DNS package, DHCP server, etc etc to set up a router, with VyOS you just change VyOS config and it configures normal debian packages for you. Plus everything is exhaustively tested and configs are reverse compatible, hiding all breaking changes underneath. It's super neat and it works perfectly on a £100 fanless Celeron J4125 box from Aliexpress as a home router, routing and shaping 1gbit without breaking a sweat and with deeply sub-ms delay.
- solarkraft 3y ago
  This is probably the best explanation I've seen. Do you have an idea why the CLI tools aren't distributed independently? Why shouldn't I be able to run it on a Debian system I already have (and understand)? Running an entire new distro just seems like overkill for what it actually does over a normal Linux system. It's just a configuration manager!
  - dgroshev 3y ago
    I don't know, but I imagine it would make testing substantially more difficult (right now they have fully tested images available), and it would be impossible to implement their distro update thing (you can update the entire .iso backwards and forwards, the config will be reapplied).
solarkraft 3y ago
I like the idea of VyOS. Networking software tends to have so much hidden, hard to control state for reasons I don't understand. As a programmer this melts my brain. VyOS finally lets you have all your configuration in one, easily controlled place. Nice! For some reason I ended not actually trying it out too actively. I think I was weirded out by the distribution model and concerned by the small community.
solarkraft 3y ago
The theory of what VyOS does is (per my understanding) really simple: Configure all the networking components of a Linux system from a single place. Why isn't doing this much more popular? All the systems are already there, after all! Why aren't there (that I know of) dozens of projects to accomplish this relatively easy, but relatively useful task? I think it's a pretty big deal to be able to configure that stuff from a single place. Commercial router manufacturers all do it. Why does (as far as I know) only VyOS do it on the open source side of things?
- sofixa 3y ago
  IMO, Open Source routers are a niche thing, and open source declarative CLI routers even more of a niche thing. Most enterprises prefer buying something with a support contract from a known name vendor (Cisco, Juniper, etc.). Most home users just use what their ISP provides them with, and of those that want something more, they either pick a SOHO vendor like Ubiquiti/Mikrotik, or if DIYing the hardware, choose pfSense / OPNSense / DD-WRT for the clickOps options, because networking really isn't trivial. For those for whom networking is trivial, Debian is fine router OS if you know your way around iptables and friends. That leaves all those who want to use DIY hardware, and a enterprise-like declarative CLI. That's really not a whole lot of people in the end.
- whalesalad 3y ago
  OPNsense and pfSense exist, plus OpenWrt.
nunez 3y ago
I thought this would be about building VyOS from source, but fun article nonetheless. I use VyOS whenever I need layer 3 routing in vSphere for test environments. NSX-T is (way) faster by dint of being deeply integrated into VMkernel, but VyOS is pretty performant for what it is and is easier to install to boot.
rhn_mk1 3y ago
The idea of VyOS is awesome: have your configuration all in one place, apply it atomically. After using it for several years, the implementation is clearly lacking. It seems that the maintainers are overloaded, because contributions to fix minor issues, or add config options get ignored in my experience. As a result, the configuration is missing some nice options in the IPv6 space (so-called tethering), and it's still using iptables for packet filtering. It's also rather hard to roll your own, with your own modules: the module system is rather hard to use. Not to mention the lack of interest to roll out a version for ARM. My next router will be based on NixOS, and will attempt to recreate the awesome UI of VyOS.
js2 3y ago
There's incorrect comments in this thread re: VyOS, Vyatta, EdgeOS. Vyatta is the original OS, based on Debian, dating back to 2005. Its history is detailed here: https://en.wikipedia.org/wiki/Vyatta https://en.wikipedia.org/wiki/Vyatta In 2011, Ubiquiti launched their EdgeMax products with EdgeOS which was a fork of Vyatta Core 6.3 ported from x86 to Cavium. In 2012, Vyatta was acquired by Brocade. IN 2013, Vyatta Core 6.6 was forked as VyOS. That's the rough origin of these three OSes. I used Vyatta Core on a PC at a startup from 2009-2013 as our office router. I haven't paid attention to it or VyOS since then. I've been running various EdgeOS routers at my home since 2014 or so, first an EdgeRouter Lite and today an EdgeRouter 4. EdgeOS has been updated quite a bit over the years from its Vyatta Core origins, but the original developers are no longer with Ubiquiti. EdgeOS hasn't seen updates in quite some time now. Also, not all Ubiquiti run EdgeOS. Only the EdgeRouters do. The rest of their products run a completely different OS, generally either UbiquitiOS or UnifiOS. Sources besides my own memory: https://blog.vyos.io/versions-mystery-revealed https://blog.vyos.io/versions-mystery-revealed https://old.reddit.com/r/Ubiquiti/comments/scqlg3/what_happened_to_edgeos/hucrwxx/ https://old.reddit.com/r/Ubiquiti/comments/scqlg3/what_happe...
- whalesalad 3y ago
  My ER4 has been a very solid piece of kit.
deleted 3y ago
[deleted]
k_roy 3y ago
hey... I know this post.... I need to write a follow-up so bad. VyOS has evolved a BUNCH since I wrote this, but the same basic ideas apply. Mostly some configuration nodes have moved around.
- grawlinson 3y ago
  Hey! Thanks for writing this, this really helped me setup VyOS as my main router/firewall for my homelab. Do you have any plans to expand into IPv6 functionality?
- elisaado 3y ago
  Thank you for writing the blog, the way it is written was very engaging :)

121 points by elisaado 3y ago

Might be worth adding a paragraph about what VyOS is, maybe not for the target audience but certainly for HN. https://vyos.io/ https://vyos.io/ The VyOS website says: > Democratizing how we access networks through a universal Router and Open source software. > Our vision at VyOS is to dramatically change how we access networks so that we can all build the solutions we always dreamed of, without restrictions, limitations, or prohibitive costs. But I'm still kind of none the wiser. Does this thing use Linux or something else?

minimaul 3y ago
Yes - it's a Debian Linux based router distro that is at least partially modelled after Juniper's configuration style. edit: used to be a maintainer for a short while :)
rnhmjoj 3y ago
It's a fork vyatta[1], the same system used by Ubiquiti for EdgeOS. Yes, it's Debian with a declarative configuration system. It works more or less similarly to NixOS, if you know about that: basically it uses a bunch of perl scripts to install and set up software from a single unified configuration file, hiding all the implementation details. [1]: https://en.wikipedia.org/wiki/Vyatta https://en.wikipedia.org/wiki/Vyatta
- sebazzz 3y ago
  And it works great because your router starts from a read-only image, then executes the scripts and applies the configuration. You can easily roll back and forward configuration and firmware versions.
- LeBit 3y ago
  I don't think they use Perl anymore. They have been moving to Python.
- nunez 3y ago
  Had no idea vyatta is what Ubiquiti uses. Very cool.
  - paradox460 3y ago
    Thats only on EdgeOS though. UbiquitiOS uses a different system, built around Podman and Debian
yrro 3y ago
It's a Network Operating System for configuring routers & switches (implemented on top of Debian). This means: * You can SSH in and configure it like you would a managed switch or router. There's a single object that models all of the device's configuration, when you commit it a bunch of scripts activate and actaully apply the configuration to the running system. * Deployments are image based, you can roll back to older images etc. * You don't need to look at any of the system's underlying configuration files or use any of the normal Linux commands to examine and manipulate the state of the system (the commands are still there for convenience of course). You don't even need to be aware that you're really using a bunch of custom bash functions to examine and manipulate the state of the system.
Arnavion 3y ago
It's a Debian 8 base with a lot of its own custom packages. The homepage has an FAQ that tries to reassure you that being based on Debian 8 is not a problem. Whether that convinces you or not is up to you, of course. I personally use regular Debian 12 on my router without problems. It also has "declarative config" since all the configuration, firewall rules, etc are a bunch of config files that I can scp / ansible over any time.

PreInternet01 3y ago

The question 'what is this thing' is probably best answered by the Github project page: https://github.com/vyos https://github.com/vyos It's a decent-ish option if you need advanced routing functionality; one thing to keep in mind, though, is that unless you're OK with running unstable 'nightly' code, you'll be spending USD 8K+ on an annual basis.

awesomeMilou 3y ago
https://support.vyos.io/en/support/solutions/103000152091 https://support.vyos.io/en/support/solutions/103000152091 They have an LTS release, no? They seem to follow the RedHat strategy though, only subscribers can download prebuilt images, but you can build the LTS ones yourself: https://blog.vyos.io/vyos-1.3.2-lts-release https://blog.vyos.io/vyos-1.3.2-lts-release
blinkingled 3y ago
> If you are an individual, you can get the generic ISO by donating on Open Collective. And if you are contributing to VyOS, whether you are writing code, improving the docs, or promoting VyOS publicly, we are happy to share pre-built images with you through contributor subscriptions. Finally, you can always build your own images — just follow these instructions. Sounds fair to me. Truth is there's no good alternative other than pfSense but if you want Linux (hw support etc) I don't know if you can do better than vyos for routers.
- laurowyn 3y ago
  If you want a web GUI, then pfSense or OPNSense are the general go tos. However, if you're comfortable with CLI and modifying configs in /etc/ then just running a bare metal Alpine Linux box is perfectly doable on a tiny box. iptables/nftables for firewall/NAT, dnsmasq/bind9 for dns, dnsmasq/isc-dhcp for DHCP. I've got a handful of these boxes all interlinked via wireguard, sharing routes via BGP using bird. Sure, you miss the config verification that VyOS provides, but does mean you learn the underlying tools themselves and that knowledge is portable to any other box running those systems. Personally, I don't quite understand why VyOS is a standalone distro when it could just be a config generator/checker package. Could even support multiple different underlying tools so if you want to use dnsmasq over bind9, or vice versa, it can provide a unified config interface for them.

awesomeMilou 3y ago

VyOS is unfortunately completely useless for larger applications, since it's difficult to impossible to automate due to it's unique way of applying configurations. Don't get me wrong, for manual administration it's great, there's a lot of missed automation potential given that it's just Linux underneath. As an example, the Ansible modules for VyOS are basicially just variations of an adapted ansible.builtin.shell, instead of offering to manage state in a more first class manner (via attributes and values): https://docs.ansible.com/ansible/latest/collections/vyos/vyos/vyos_config_module.html#ansible-collections-vyos-vyos-vyos-config-module https://docs.ansible.com/ansible/latest/collections/vyos/vyo...

LeBit 3y ago
You can ssh into the router, copy a new config (state) and load that config. It is not very elegant though. Do you know of an open source router that does what you are looking for?
- chomp 3y ago
  That doesn’t sound that far off from what unifi does, it uses a daemon to do those things. (Receive configuration and then do a local apply)
  - synergy20 3y ago
    unifi runs vyos underneath
    - zrail 3y ago
      UniFi doesn't use VyOS anywhere afaik. Some Ubiquiti products run EdgeOS which is a Vyatta fork (EdgeRouter, the original security gateways). More recent UniFi products (UDM, UDM-SE, UNVR, etc) run UniFi OS which is their own Debian-based OS that runs applications inside Podman containers.
jon-wood 3y ago
From what I've seen of VyOS using a configuration file that is then used to generate the actual system configuration I'm not really sure its so hard to automate. Take your target state, generate a configuration file in the right format, then send it over and apply. To be honest this feels more like a limitation in Ansible, which has always felt like a bit of a hacky config management system to me in that the way it functions is generally to run a bunch of commands that gradually mutate the system's state, rather than atomically applying the target state, but then I've been spoiled by NixOS on my personal infrastructure recently.

amar0c 3y ago

Unfortunately they went full "Vyatta way" or "RedHat Way" by basically giving rolling release for free only. I remember times when Vyatta went behind paywalls Vyos was completely free. Not sure who would want something rolling on device like router. Nowdays everyone wanting something good and free go OpnSense way.

rnhmjoj 3y ago
The biggest problem I have is that they only support (or at least that's what they release publicly) x86_64. I am forced to use openwrt because the vast majority of consumer low-power hardware is using ARM or exotic architectures. VyOS interface is vastly superior, though.
- sp0ck 3y ago
  VyOS is not targeted for such usage scenarios. It is ISP who are paying for support. I have VyOS systems with BGP, MPLS, PPPoE termination, etc that are pushing gigabits of traffic. This is VyOS usecase. Not 5W router handling home or small office.
  - rnhmjoj 3y ago
    I've used EdgeOS in the past and it's perfect for SOHO routers too, but unfortunately is half full of proprietary extensions and semi-abandoned. I think VyOS could take its place nicely, openwrt feels like a toy compared.
thepill 3y ago
You can build your own LTS image relatively easy and free of charge
- LeBit 3y ago
  The process is not straightforward, especially if you want to customize the image. And I have been unable to build a cloud-init image. But the ISO is buildable from the LTS branch and it works well.

pshirshov 3y ago

From my experience, Nix makes lot more reliable and easier to configure router. ymmv

LeBit 3y ago
You are comparing bananas and windows.
rnhmjoj 3y ago
I just finished building a router based on NixOS and I must disagree. NixOS modules mostly targets desktop and servers, but the network specific configuration is still very lacking. A few examples: - up until a couple of weeks ago the hostapd module was basically a toy: could only manage a single SSID, no way to configure the radios, hardcoded to WPA2-PSK; - the NixOS firewall is still based on iptables and conflicts with nftables, so you must disable and manually write rules; - the `networking.nat` module (NAT44) doesn't do NAT reflection; - I had to write a module for Jool (NAT64, SIIT); - I had to write a module for libreswan (IPsec); - I had to write a module for automatic rollbacks, otherwise you can loose access if you make a mistake. Vyatta and VyOS also provide a much higher level abstraction over the software that is being configured (e.g. you don't have to deal with a specific IPsec implementation). Finally, once you do `nixos-rebuild switch` you're on your own, while with vyatta you have a clean command line interface to inspect the state of the router and manage it.
- gbraad 3y ago
  does vyatta/vyos store the configuration in a single file or easy to back up config? I do not remember anymore how this was done, but I believe that is the appeal what 'nix' was mentioned.
  - rnhmjoj 3y ago
    Yes, it does.
- solarkraft 3y ago
  Thanks for the warning; sounds like my mileage would have varied quite a bit. Do you think that, had your use case been simpler, you'd have enjoyed the experience?

lifeisstillgood 3y ago

A definite tangent: About three house moves ago I had OpenWRT on a cable router and knew what was going on. But with family and work and house moves I am now just staring at a flashing BT Home Hub and wondering how to tackle the inevitable "of course that router firmware wont allow that and your ISP wont give out its password and ..." Is there a uptodate reliable guide (possibly including how to persuade your wife it's a good idea to drill holes in the living room ceiling to run cat6)

trustingtrust 3y ago
What I find easy(ier?) is to run (x)sense on a dedicated firewall and either a mesh with cheap openwrt routers or get something like Deco mesh and run it in AP mode if you don’t have cat6 at home. I think this combination can be under 300$ for a 3-pack of mesh Deco x20 + an intel card on a refurb dell optiplex.
- dgroshev 3y ago
  IME (x)sense is quite problematic when you start doing anything other than the bare minimum. I spent multiple days trying to figure out why it silently stopped accepting IPv6 delegation, or why does it spike latency for no visible reason under load. The underlying reality is that FreeBSD's network stack is much more conservative and has less resources than Linux's, which shows up in articles like this one [1]. On the same Celeron J4125/i226 box VyOS was absolutely perfect, not a single issue, significantly low (and always low) latency with higher throughput. On the hardware side, I think the /r/homelab hivemind doesn't get challenged enough. Dell optiplexes cost very similarly to Aliexpress Protectli alternatives (such as [2]), while being larger, having a fan, and being overall more hassle. TP Link/Ubiquiti WiFi APs seem to be overall inferior to Aruba Instant On, which is exactly the same hardware HP sells in their Aruba line, but for the same SOHO price. [1] https://teklager.se/en/knowledge-base/opnsense-performance-optimization/ https://teklager.se/en/knowledge-base/opnsense-performance-o... [2] https://www.aliexpress.com/item/1005004272231167.html https://www.aliexpress.com/item/1005004272231167.html
  - trustingtrust 3y ago

gbraad 3y ago

I was wondering if this was Vyatta... it seems it is the community continuation after Broccade ceased development. Used to run this over a decade ago on a 'router' (ThinkCentre Tiny), but eventually went to a Fedora installation a few years back. Might have a look again.

gbraad 3y ago
... have to say that the offering is confusing; it is a subscription or a rolling release. Wondering how much can be automated of the installation/state; would it be possible to use version control? If not, I can see the appeal to suggest Nix over this.
- kmbfjr 3y ago
  Both. Pay for a subscription or use the RR. You can also build your own images with docker using the annotated releases.

dgroshev 3y ago

I was pretty confused what it is too and then I loved it. It's debian plus some shell trickery and CLI tools that let you configure debian and debian packages as a router from one large config tree using neat CLI tools (that support commit/rollback). Normally you'd need iptables, a separate DNS package, DHCP server, etc etc to set up a router, with VyOS you just change VyOS config and it configures normal debian packages for you. Plus everything is exhaustively tested and configs are reverse compatible, hiding all breaking changes underneath. It's super neat and it works perfectly on a £100 fanless Celeron J4125 box from Aliexpress as a home router, routing and shaping 1gbit without breaking a sweat and with deeply sub-ms delay.

solarkraft 3y ago
This is probably the best explanation I've seen. Do you have an idea why the CLI tools aren't distributed independently? Why shouldn't I be able to run it on a Debian system I already have (and understand)? Running an entire new distro just seems like overkill for what it actually does over a normal Linux system. It's just a configuration manager!
- dgroshev 3y ago
  I don't know, but I imagine it would make testing substantially more difficult (right now they have fully tested images available), and it would be impossible to implement their distro update thing (you can update the entire .iso backwards and forwards, the config will be reapplied).

solarkraft 3y ago

I like the idea of VyOS. Networking software tends to have so much hidden, hard to control state for reasons I don't understand. As a programmer this melts my brain. VyOS finally lets you have all your configuration in one, easily controlled place. Nice! For some reason I ended not actually trying it out too actively. I think I was weirded out by the distribution model and concerned by the small community.

solarkraft 3y ago

The theory of what VyOS does is (per my understanding) really simple: Configure all the networking components of a Linux system from a single place. Why isn't doing this much more popular? All the systems are already there, after all! Why aren't there (that I know of) dozens of projects to accomplish this relatively easy, but relatively useful task? I think it's a pretty big deal to be able to configure that stuff from a single place. Commercial router manufacturers all do it. Why does (as far as I know) only VyOS do it on the open source side of things?

sofixa 3y ago
IMO, Open Source routers are a niche thing, and open source declarative CLI routers even more of a niche thing. Most enterprises prefer buying something with a support contract from a known name vendor (Cisco, Juniper, etc.). Most home users just use what their ISP provides them with, and of those that want something more, they either pick a SOHO vendor like Ubiquiti/Mikrotik, or if DIYing the hardware, choose pfSense / OPNSense / DD-WRT for the clickOps options, because networking really isn't trivial. For those for whom networking is trivial, Debian is fine router OS if you know your way around iptables and friends. That leaves all those who want to use DIY hardware, and a enterprise-like declarative CLI. That's really not a whole lot of people in the end.
whalesalad 3y ago
OPNsense and pfSense exist, plus OpenWrt.

nunez 3y ago

I thought this would be about building VyOS from source, but fun article nonetheless. I use VyOS whenever I need layer 3 routing in vSphere for test environments. NSX-T is (way) faster by dint of being deeply integrated into VMkernel, but VyOS is pretty performant for what it is and is easier to install to boot.

rhn_mk1 3y ago

The idea of VyOS is awesome: have your configuration all in one place, apply it atomically. After using it for several years, the implementation is clearly lacking. It seems that the maintainers are overloaded, because contributions to fix minor issues, or add config options get ignored in my experience. As a result, the configuration is missing some nice options in the IPv6 space (so-called tethering), and it's still using iptables for packet filtering. It's also rather hard to roll your own, with your own modules: the module system is rather hard to use. Not to mention the lack of interest to roll out a version for ARM. My next router will be based on NixOS, and will attempt to recreate the awesome UI of VyOS.

js2 3y ago

There's incorrect comments in this thread re: VyOS, Vyatta, EdgeOS. Vyatta is the original OS, based on Debian, dating back to 2005. Its history is detailed here: https://en.wikipedia.org/wiki/Vyatta https://en.wikipedia.org/wiki/Vyatta In 2011, Ubiquiti launched their EdgeMax products with EdgeOS which was a fork of Vyatta Core 6.3 ported from x86 to Cavium. In 2012, Vyatta was acquired by Brocade. IN 2013, Vyatta Core 6.6 was forked as VyOS. That's the rough origin of these three OSes. I used Vyatta Core on a PC at a startup from 2009-2013 as our office router. I haven't paid attention to it or VyOS since then. I've been running various EdgeOS routers at my home since 2014 or so, first an EdgeRouter Lite and today an EdgeRouter 4. EdgeOS has been updated quite a bit over the years from its Vyatta Core origins, but the original developers are no longer with Ubiquiti. EdgeOS hasn't seen updates in quite some time now. Also, not all Ubiquiti run EdgeOS. Only the EdgeRouters do. The rest of their products run a completely different OS, generally either UbiquitiOS or UnifiOS. Sources besides my own memory: https://blog.vyos.io/versions-mystery-revealed https://blog.vyos.io/versions-mystery-revealed https://old.reddit.com/r/Ubiquiti/comments/scqlg3/what_happened_to_edgeos/hucrwxx/ https://old.reddit.com/r/Ubiquiti/comments/scqlg3/what_happe...

whalesalad 3y ago
My ER4 has been a very solid piece of kit.

deleted 3y ago

[deleted]

k_roy 3y ago

hey... I know this post.... I need to write a follow-up so bad. VyOS has evolved a BUNCH since I wrote this, but the same basic ideas apply. Mostly some configuration nodes have moved around.

grawlinson 3y ago
Hey! Thanks for writing this, this really helped me setup VyOS as my main router/firewall for my homelab. Do you have any plans to expand into IPv6 functionality?
elisaado 3y ago
Thank you for writing the blog, the way it is written was very engaging :)