5 ms·

Almost everyone I know that's ever used JunOS from a command line for 'serious' ISP things finds RouterOS painful and cumbersome. The way things are laid out i

Almost everyone I know that's ever used JunOS from a command line for 'serious' ISP things finds RouterOS painful and cumbersome. The way things are laid out in a hierarchy in a full system "/export" from a Mikrotik is so weird and annoying compared to a hierarchical junos configuration from a "show configuration" on a juniper router. If people want to make a real router of an x86-64 system rather than putting a mikrotik pci-e card into it (wtf, why?) I'd recommend they go with vyatta or VyOS instead, or install something like a barebones centos or debian and then add FRR to it.

techsupporter 5y ago
> make a real router of an x86-64 system rather than putting a mikrotik pci-e card into it (wtf, why?) I'd recommend they go with vyatta or VyOS instead One thing I've been looking for is a hardware box that can replicate what Ubiquiti's EdgeRouter Infinity does: a handful of 10Gbps SFP+ ports (sorry, I know that the term is "cages" but I just can't) and a couple of copper 1Gbps ports. So far I haven't found anything but I feel like my search will get motivated in the next couple of years since it feels like Ubiquiti has forgotten that EdgeRouter exists. Do you have any rack form factor x86-type systems you like for VyOS?
- walrus01 5y ago
  When space permits I prefer full-size 1U systems that have dual/hotswap power supplies and room for three low profile pci-e slots, such as a Dell R630/R640 or similar. With Intel chipset 4-port 10GbE SFP+ NICs this would max out at twelve ports plus whatever is on the motherboard daughtercard for network interfaces (2 x 10GbE + 2 x 1GbE copper, or whatever). for smaller or shallow stuff, supermicro, msi, tyan, asus
- logifail 5y ago
  > a hardware box [with] a handful of 10Gbps SFP+ [..] and a couple of copper 1Gbps ports I have a couple of (fanless!) CRS305-1G-4S+IN[0] at home, one in my study and one in the utility room. They each connect with 10GbE fibre (or DAC) to ConnectX-3 cards in my PCs and servers. [0] https://mikrotik.com/product/crs305_1g_4s_in https://mikrotik.com/product/crs305_1g_4s_in
  - techsupporter 5y ago
    I appreciate the recommendation but that's kind of a gap from the EdgeRouter Infinity (ER-8-XG). The Infinity has 8x10Gbps SFP+ ports, a single copper 1Gbps port, 16GB of RAM, and a multi-core processor because it's designed as an inexpensive core router for a mid-sized network. Where I work, we use one of them as our main router with multiple peering sessions and two transit uplinks. According to Cacti, right now we're pushing about 30Gbps through the router. That's what I'm looking to eventually replace, if Ubiquiti doesn't start up with software updates to the EdgeRouter line again. But I think that's the problem: the EdgeRouter line is so amazingly inexpensive for all of the power you get, there's no financial incentive for Ubiquiti to invest in it and all of the players with the "proper" routers--the Junipers and Ciscos and the like--start at three times the price of an ER-8-XG.
    - logifail 5y ago
      > that's kind of a gap from the EdgeRouter Infinity (ER-8-XG) Indeed, not least on price. How much was your ER-8-XG? My CRS305-1G-4S+IN were about USD180 each. EDIT: If there were a silent version of the CRS326-24S+2Q+RM[0][1] I'd have bought one already... "The MikroTik CRS326-24S+2Q+RM is an insane switch. Its specs are relatively mundane by modern standards. It has 24x SFP+ 10GbE ports and 2x QSFP+ 40GbE ports making it not even as powerful as mainstream previous-generation switches like the QCT QuantaMesh T3048-LY8 that we installed in our lab years ago. Instead what makes the switch insane is that it offers all of that performance at $475" [0] https://mikrotik.com/product/crs326_24s_2q_rm https://mikrotik.com/product/crs326_24s_2q_rm [1] https://www.servethehome.com/mikrotik-crs326-24s2qrm-review-insane-switch/ https://www.servethehome.com/mikrotik-crs326-24s2qrm-review-...
      - Alupis 5y ago
        For what it's worth - there is a healthy "modding" community for some of these Mikrotik switches. People convert them into fanless/silent units pretty regularly, or swap the fans for higher flow / lower rpm fans, etc.
      - walrus01 5y ago
barbazoo 5y ago
> putting a mikrotik pci-e card into it (wtf, why?) It's in the first sentence of the post: > Save space in your server room
- walrus01 5y ago
  if you want a mikrotik, buy a mikrotik hardware 1U router, despite the many issues with them the one thing they do have going for them are low power consumption and small space use. an actual ccr2004 1U box is not that large and can be mounted almost anywhere.
  - tremon 5y ago
    Maybe I'm dense, but wouldn't that solution still use 1U more space than the PCIe card mentioned in TFA?
  - vetinari 5y ago
    In your own rack, you would do exactly that. But if you paying per U in colo, this card can save you one slot.
    - walrus01 5y ago
      If you have enough traffic to need multiple SFP28 interfaces in colo and can't pay $150-250/mo extra to put in place a real hardware router, or stop paying by the 1U increment and get 1/4, 1/3 or 1/2 of of a cabinet, priorities and risk tolerance are misaligned in my opinion. if you have >10Gbps traffic flows and are putting the router and other hosting environment/linux things all together in one 1U piece of hardware as a single x86-64 server, that's a "too many eggs in one basket" problem. also worth noting that many colo/hosting ISPs won't offer 25GbE circuits on SFP28 anyways, you can buy either a 10GbE transit link or 100GbE, or maybe 2x10GbE bundled together in a 802.3ad or similar.
      - vetinari 5y ago
        In this case, I was thinking about moving a currently half a rack worth of equipment from premises to colo, as the (internal) users are mostly on WFH anyway. They would not generate 1 Gbps of external traffic, not even in spikes. Currently, as it is, it makes more sense to stay on premises, but if some increase of density happened, it could make some sense. However, it is not going to happen, it would be somewhere at bottom with priority. It was just an exercise, what could be done.
stingraycharles 5y ago
As someone who’s a home networking enthusiast, and has too much Mikrotik gear at home, I can kind of understand what they’re coming from. RouterOS has the usability of “enterprise-grade” network equipment (meaning it’s arcane and non-intuitive), but at the same time has lots and lots of half-working features. I simply cannot believe how terrible their IPv6 support is (still no connection tracking!), and plenty of weird glitches, etc. But! Their hardware is very reasonably priced, and an excellent gateway to “real” networking equipment for the hobbyist. It’s unfair to compare it against Juniper and the likes: yes, it’s much better, but yes, the products are also 10x - 100x as expensive. While everything that’s done in RouterOS can also be done under vanilla Linux, I buy Mikrotik precisely because I don’t want to build a custom Linux router. I want something that comes with a GUI, and I won’t have to spend too much time setting up. Having said that, I would absolutely kill for an “escape” Linux shell. I know that RED supports ECN in Linux, please allow me to use it!
- blibble 5y ago
  > I simply cannot believe how terrible their IPv6 support is (still no connection tracking!) I see a list of connections under "IPv6 firewall" under the connections tab? > and plenty of weird glitches this bit however I agree with
  - stingraycharles 5y ago
    I don’t think that contrack based mangle rules work, though. If it does, it must be a recent fix (I’m on ROS7.1)
    - iso1210 5y ago
      7.x is still effectively in beta, there are many features that don't work yet, last time I checked neither multicast nor bfd were working.
      - walrus01 5y ago
        the idea that somebody thought to ship to production release a router operating system with broken bfd is amazing.
oarsinsync 5y ago
As a network engineer who’s worked on Cisco, Juniper, Foundry, Brocade, Extreme, HP, Dell, and even Netgear, let me assure you that while the urban legend is that “JunOS is IOS done right”, the reality is that they’re all terrible in their own ways. JunOS is generally better than IOS(-XR), but it’s still got its sharp edges. VyOS / Vyatta are poor enough clones that they will bite and seriously suck to anyone who’s actually got real JunOS experience. Let’s be real. The goal in improving network configuration standards is to suck less. That’s it. Everything in networks sucks. Anyone who tells you otherwise either lacks experience in general, lacks experience suffering at the bleeding edge, or lacks my cynicism and genuinely sees the world as a better place than I do (I envy them for any of the above)
- walrus01 5y ago
  I don't disagree with any of this - have been using JunOS since the M40 was the absolute apex of service provider core router technology. Lots and lots of weird bugs in various versions of IOS and JunOS on all their platorms. Big difference between what you might get spending $15,000 for a Juniper MX204 running JunOS and a Mikrotik $800 router. I mentally categorize Mikrotik RouterOS and similar ultra low cost things in the same tier as VyOS. It's cheap but there are tradeoffs to going cheap. One has to understand the risks and tradeoffs of running a lot of your traffic or important things through cheap routers. Sometimes it's a risk worth taking. Foundry, as we've seen, was a straight knockoff of the IOS 12.2/12.4 CLI and interface. Used plenty of Foundry switches in a previous role. Everything does suck. Some things suck less. Sometimes you can pay money to get things that suck less.
  - oarsinsync 5y ago
    > Everything does suck. Some things suck less. Sometimes you can pay money to get things that suck less. And sometimes you pay more money and you're the one being made to do the sucking :-\
    - elzbardico 5y ago
      Oh, hello and welcome to Oracle Clients Anonymous.
d00bianista 5y ago
We're all different, I find Cisco and Vyatta awkward because of different reasons. RouterOS is not the best there is but it's less awkward, in my opinion.

techsupporter 5y ago

> make a real router of an x86-64 system rather than putting a mikrotik pci-e card into it (wtf, why?) I'd recommend they go with vyatta or VyOS instead One thing I've been looking for is a hardware box that can replicate what Ubiquiti's EdgeRouter Infinity does: a handful of 10Gbps SFP+ ports (sorry, I know that the term is "cages" but I just can't) and a couple of copper 1Gbps ports. So far I haven't found anything but I feel like my search will get motivated in the next couple of years since it feels like Ubiquiti has forgotten that EdgeRouter exists. Do you have any rack form factor x86-type systems you like for VyOS?

walrus01 5y ago
When space permits I prefer full-size 1U systems that have dual/hotswap power supplies and room for three low profile pci-e slots, such as a Dell R630/R640 or similar. With Intel chipset 4-port 10GbE SFP+ NICs this would max out at twelve ports plus whatever is on the motherboard daughtercard for network interfaces (2 x 10GbE + 2 x 1GbE copper, or whatever). for smaller or shallow stuff, supermicro, msi, tyan, asus
logifail 5y ago
> a hardware box [with] a handful of 10Gbps SFP+ [..] and a couple of copper 1Gbps ports I have a couple of (fanless!) CRS305-1G-4S+IN[0] at home, one in my study and one in the utility room. They each connect with 10GbE fibre (or DAC) to ConnectX-3 cards in my PCs and servers. [0] https://mikrotik.com/product/crs305_1g_4s_in https://mikrotik.com/product/crs305_1g_4s_in
- techsupporter 5y ago
  I appreciate the recommendation but that's kind of a gap from the EdgeRouter Infinity (ER-8-XG). The Infinity has 8x10Gbps SFP+ ports, a single copper 1Gbps port, 16GB of RAM, and a multi-core processor because it's designed as an inexpensive core router for a mid-sized network. Where I work, we use one of them as our main router with multiple peering sessions and two transit uplinks. According to Cacti, right now we're pushing about 30Gbps through the router. That's what I'm looking to eventually replace, if Ubiquiti doesn't start up with software updates to the EdgeRouter line again. But I think that's the problem: the EdgeRouter line is so amazingly inexpensive for all of the power you get, there's no financial incentive for Ubiquiti to invest in it and all of the players with the "proper" routers--the Junipers and Ciscos and the like--start at three times the price of an ER-8-XG.
  - logifail 5y ago
    > that's kind of a gap from the EdgeRouter Infinity (ER-8-XG) Indeed, not least on price. How much was your ER-8-XG? My CRS305-1G-4S+IN were about USD180 each. EDIT: If there were a silent version of the CRS326-24S+2Q+RM[0][1] I'd have bought one already... "The MikroTik CRS326-24S+2Q+RM is an insane switch. Its specs are relatively mundane by modern standards. It has 24x SFP+ 10GbE ports and 2x QSFP+ 40GbE ports making it not even as powerful as mainstream previous-generation switches like the QCT QuantaMesh T3048-LY8 that we installed in our lab years ago. Instead what makes the switch insane is that it offers all of that performance at $475" [0] https://mikrotik.com/product/crs326_24s_2q_rm https://mikrotik.com/product/crs326_24s_2q_rm [1] https://www.servethehome.com/mikrotik-crs326-24s2qrm-review-insane-switch/ https://www.servethehome.com/mikrotik-crs326-24s2qrm-review-...
    - Alupis 5y ago
      For what it's worth - there is a healthy "modding" community for some of these Mikrotik switches. People convert them into fanless/silent units pretty regularly, or swap the fans for higher flow / lower rpm fans, etc.
    - walrus01 5y ago

barbazoo 5y ago

> putting a mikrotik pci-e card into it (wtf, why?) It's in the first sentence of the post: > Save space in your server room

walrus01 5y ago
if you want a mikrotik, buy a mikrotik hardware 1U router, despite the many issues with them the one thing they do have going for them are low power consumption and small space use. an actual ccr2004 1U box is not that large and can be mounted almost anywhere.
- tremon 5y ago
  Maybe I'm dense, but wouldn't that solution still use 1U more space than the PCIe card mentioned in TFA?
- vetinari 5y ago
  In your own rack, you would do exactly that. But if you paying per U in colo, this card can save you one slot.
  - walrus01 5y ago
    If you have enough traffic to need multiple SFP28 interfaces in colo and can't pay $150-250/mo extra to put in place a real hardware router, or stop paying by the 1U increment and get 1/4, 1/3 or 1/2 of of a cabinet, priorities and risk tolerance are misaligned in my opinion. if you have >10Gbps traffic flows and are putting the router and other hosting environment/linux things all together in one 1U piece of hardware as a single x86-64 server, that's a "too many eggs in one basket" problem. also worth noting that many colo/hosting ISPs won't offer 25GbE circuits on SFP28 anyways, you can buy either a 10GbE transit link or 100GbE, or maybe 2x10GbE bundled together in a 802.3ad or similar.
    - vetinari 5y ago
      In this case, I was thinking about moving a currently half a rack worth of equipment from premises to colo, as the (internal) users are mostly on WFH anyway. They would not generate 1 Gbps of external traffic, not even in spikes. Currently, as it is, it makes more sense to stay on premises, but if some increase of density happened, it could make some sense. However, it is not going to happen, it would be somewhere at bottom with priority. It was just an exercise, what could be done.

stingraycharles 5y ago

As someone who’s a home networking enthusiast, and has too much Mikrotik gear at home, I can kind of understand what they’re coming from. RouterOS has the usability of “enterprise-grade” network equipment (meaning it’s arcane and non-intuitive), but at the same time has lots and lots of half-working features. I simply cannot believe how terrible their IPv6 support is (still no connection tracking!), and plenty of weird glitches, etc. But! Their hardware is very reasonably priced, and an excellent gateway to “real” networking equipment for the hobbyist. It’s unfair to compare it against Juniper and the likes: yes, it’s much better, but yes, the products are also 10x - 100x as expensive. While everything that’s done in RouterOS can also be done under vanilla Linux, I buy Mikrotik precisely because I don’t want to build a custom Linux router. I want something that comes with a GUI, and I won’t have to spend too much time setting up. Having said that, I would absolutely kill for an “escape” Linux shell. I know that RED supports ECN in Linux, please allow me to use it!

blibble 5y ago
> I simply cannot believe how terrible their IPv6 support is (still no connection tracking!) I see a list of connections under "IPv6 firewall" under the connections tab? > and plenty of weird glitches this bit however I agree with
- stingraycharles 5y ago
  I don’t think that contrack based mangle rules work, though. If it does, it must be a recent fix (I’m on ROS7.1)
  - iso1210 5y ago
    7.x is still effectively in beta, there are many features that don't work yet, last time I checked neither multicast nor bfd were working.
    - walrus01 5y ago
      the idea that somebody thought to ship to production release a router operating system with broken bfd is amazing.

oarsinsync 5y ago

As a network engineer who’s worked on Cisco, Juniper, Foundry, Brocade, Extreme, HP, Dell, and even Netgear, let me assure you that while the urban legend is that “JunOS is IOS done right”, the reality is that they’re all terrible in their own ways. JunOS is generally better than IOS(-XR), but it’s still got its sharp edges. VyOS / Vyatta are poor enough clones that they will bite and seriously suck to anyone who’s actually got real JunOS experience. Let’s be real. The goal in improving network configuration standards is to suck less. That’s it. Everything in networks sucks. Anyone who tells you otherwise either lacks experience in general, lacks experience suffering at the bleeding edge, or lacks my cynicism and genuinely sees the world as a better place than I do (I envy them for any of the above)

walrus01 5y ago
I don't disagree with any of this - have been using JunOS since the M40 was the absolute apex of service provider core router technology. Lots and lots of weird bugs in various versions of IOS and JunOS on all their platorms. Big difference between what you might get spending $15,000 for a Juniper MX204 running JunOS and a Mikrotik $800 router. I mentally categorize Mikrotik RouterOS and similar ultra low cost things in the same tier as VyOS. It's cheap but there are tradeoffs to going cheap. One has to understand the risks and tradeoffs of running a lot of your traffic or important things through cheap routers. Sometimes it's a risk worth taking. Foundry, as we've seen, was a straight knockoff of the IOS 12.2/12.4 CLI and interface. Used plenty of Foundry switches in a previous role. Everything does suck. Some things suck less. Sometimes you can pay money to get things that suck less.
- oarsinsync 5y ago
  > Everything does suck. Some things suck less. Sometimes you can pay money to get things that suck less. And sometimes you pay more money and you're the one being made to do the sucking :-\
  - elzbardico 5y ago
    Oh, hello and welcome to Oracle Clients Anonymous.

d00bianista 5y ago

We're all different, I find Cisco and Vyatta awkward because of different reasons. RouterOS is not the best there is but it's less awkward, in my opinion.