7 ms·

Cumulus Linux – first impressions

51 points by liotier 12y ago

liotier 12y ago
TL;DR : Linux all the way, ready for large-scale provisioning automation.
dmix 12y ago
The use of Linux and commodity hardware is interesting but when I read some of the sales copy on the Cumulus website I was reminded of why I hate enterprise software. > The emerging software-defined data center (SDDC) paradigm involves automated control of all network, server, storage and application resources, resulting in a cloud operating system. Unified visibility is essential, enabling the cloud operating system to efficiently allocate resources, detect problems and ensure consistent performance.
- jeffmcjunkin 12y ago
  My interpretation: "You can manage your network hardware in Ansible and Git. We'll assume that everything else is using it, too." Still pretty attractive, and a lot less Buzzword Bingo.
  - StavrosK 12y ago
    Oh man, I just had flashbacks of pages and pages of enterprise lingo that were much less clear than your sentence. I think Ansible suffered from that at one point, too.
- kijiki 12y ago
  Sometimes you gotta play the marketing game, for better or for worse. If you're interested in having a substantive technical discussion about Cumulus, drop me a line: nolan@cumulusnetworks.com
Thaxll 12y ago
What is the performance, does it bypass the kernel for the network stack or it's just a linux bundled with compatible hardware?
- wmf 12y ago
  Forwarding is done in hardware so it's line rate. Those swp interfaces aren't real interfaces; they're just proxies.
Someone1234 12y ago
I think it is very safe to say that most dedicated "smart" network hardware is going to disappear in the next 5-15 years to be replaced by virtual machines acting as several pieces of network equipment. You just patch in all the cables straight onto effectively a Hypervisor, and then you generate virtual switches, routers, firewalls, and so on completely via the VM management console. You're already seeing some of Cisco's smaller competition go this way. It saves physical space, works on standard hardware, and is easier to centrally manage (as you aren't physically moving wires after initial install). It will be interesting to see if Cisco jumps aboard this train or continues to pretend like the sands underneath it aren't shifting. I'm sure there would be a market for VMs running Cisco's IOS, it is still by far the most popular network operating system.
- freehunter 12y ago
  Cisco already has virtual switches. My company just bought a Cisco v1000 virtual switch. You can run your entire company from just one physical piece of hardware with Hyper-V and a Cisco V switch installed.
- m-app 12y ago
  "I think it is very safe to say that most dedicated "smart" network hardware is going to disappear in the next 5-15 years to be replaced by virtual machines acting as several pieces of network equipment." I agree with this idea in that most of the intelligence will be pushed out towards the edges of the network and overlay networks will make a lot of the physical network invisible to the applications and even the management. However, this does not mean that the networking hardware can all be just dumb devices. First of all, this kind of imperative networking will not scale to large data center implementations and secondly dumb devices will become useless once disconnected from their controller. "It will be interesting to see if Cisco jumps aboard this train or continues to pretend like the sands underneath it aren't shifting. I'm sure there would be a market for VMs running Cisco's IOS, it is still by far the most popular network operating system." The way Cisco is getting into the SDN market is by leveraging Application Centric Infrastructure (ACI) to define the complete DC (Compute, Storage and Network) with templates and letting the whole system configure itself. The switches supporting this infrastructure are based on merchant silicon in combination with Cisco ASIC's to provide the most optimal performance at these intelligent edges. Add to that the available open API's and the investment in OpenStack and you have a rock-solid and cost-effective solution for the future of the DC and DevOps. Cisco ACI: http://cisco.com/go/aci http://cisco.com/go/aci I would love to talk about it more if you want. These are exciting times. Hit me up on email or IRC. (Disclaimer: I work for Cisco but these thought are still my own, yada yada)
hhw 12y ago
If everything is more or less stock Linux, what value does Cumulus actually add for it to be worth the licensing costs? This all seems to be a giant step backwards. JunOS is now the gold standard for its CLI and configuration, due to being highly structured and well organised. Being able to validate configurations before committing them, and automatic rollbacks with commit confirm is a much improved way of doing things over IOS. Given that Juniper hardware can often be bought significantly discounted, the cost savings of white box gear are going to be small, if at all. Thus far, they seem to be sold in such small volumes that comparable Juniper or Cisco gear can easily be purchased for less. Also, how exactly is Cumulus separating the control plane and data plane? From the look of things, SDN is going to end up being relegated to just easy automation instead, like how Cloud has just been relegated to VMs with flexible billing instead of abstracting underlying infrastructure.
- wmf 12y ago
  In theory, the value of Cumulus is (1) support and (2) their proprietary switchd. (Edit: I also second the other comments that Cumulus licensing appears to be really cheap. But if you're looking for "CentSwitch" you're out of luck.) Also, how exactly is Cumulus separating the control plane and data plane? They're not. Cumulus is kind of anti-SDN.
  - drdaeman 12y ago
    "Proprietary" means "if you ever have issues with that that our support won't be able to solve for you in meaningful time span, you're basically out of luck". We're running a gateway router on Vyatta/VyOS and with every single hardware and software change we had some issues. Most were relatively minor (like default settings being not appropriate for our workload), already documented/solved by someone on the 'net. Some had required reading source code and debugging/profiling to see what goes on. Must also admit, some were beyond my understanding and were mysteriously solved with some shamanism and voodoo magic (like randomly tinkering with firmware and driver version combos).
  - kijiki 12y ago
    I wouldn't characterize us as "anti-SDN", instead I would say that Cumulus Linux is not an SDN solution by itself. We work closely with all the major SDN vendors for use-cases that are virtualization-centric or cloud-centric. That said, there are use-cases like Hadoop clusters that have very little need for an SDN layer, just very fast L3 IP connectivity.
pyvpx 12y ago
but is it faster than VALE/netgraph?
- wmf 12y ago
  Yes, hardware forwarding is ~100x faster than optimized software forwarding.
  - pyvpx 12y ago
    I missed the switchd/proprietary ASIC part. oops!
voidlogic 12y ago
>These appear as Ethernet interfaces normally do on Linux, visible with ip link show. (ifconfig has long been deprecated by the community in favor of the iproute2 family of tools.) I am amused when I read things like this, 99% of the Linux savvy folks I know use ifconfig.
- LukeShu 12y ago
  In my experience: we use ifconfig out of familiarity; I know that I "should" be using 'ip' (and I do for scripts).
  - a3176082 12y ago
    I don't think you should. Every UNIX operating system has ifconfig. Writing a nonstandard tool instead of fixing the standard one is very antisocial behaviour from the Linux community, mirroring that of Microsoft. Have a look for example at how excellent the ifconfig program for OpenBSD is. These days Linux has become so popular, that it starts to pretend other operating systems and standards are no longer relevant.
- kijiki 12y ago
  I'm the CTO of Cumulus, and I still sometimes use ifconfig. My fingers are just too programmed at this point. That said, ifconfig doesn't support easy addition of multiple IP addresses to a single interface. You have to create 1 alias interface per IP, which gets unwieldy fast.

51 points by liotier 12y ago

TL;DR : Linux all the way, ready for large-scale provisioning automation.

dmix 12y ago

The use of Linux and commodity hardware is interesting but when I read some of the sales copy on the Cumulus website I was reminded of why I hate enterprise software. > The emerging software-defined data center (SDDC) paradigm involves automated control of all network, server, storage and application resources, resulting in a cloud operating system. Unified visibility is essential, enabling the cloud operating system to efficiently allocate resources, detect problems and ensure consistent performance.

jeffmcjunkin 12y ago
My interpretation: "You can manage your network hardware in Ansible and Git. We'll assume that everything else is using it, too." Still pretty attractive, and a lot less Buzzword Bingo.
- StavrosK 12y ago
  Oh man, I just had flashbacks of pages and pages of enterprise lingo that were much less clear than your sentence. I think Ansible suffered from that at one point, too.
kijiki 12y ago
Sometimes you gotta play the marketing game, for better or for worse. If you're interested in having a substantive technical discussion about Cumulus, drop me a line: nolan@cumulusnetworks.com

Thaxll 12y ago

What is the performance, does it bypass the kernel for the network stack or it's just a linux bundled with compatible hardware?

wmf 12y ago
Forwarding is done in hardware so it's line rate. Those swp interfaces aren't real interfaces; they're just proxies.

Someone1234 12y ago

I think it is very safe to say that most dedicated "smart" network hardware is going to disappear in the next 5-15 years to be replaced by virtual machines acting as several pieces of network equipment. You just patch in all the cables straight onto effectively a Hypervisor, and then you generate virtual switches, routers, firewalls, and so on completely via the VM management console. You're already seeing some of Cisco's smaller competition go this way. It saves physical space, works on standard hardware, and is easier to centrally manage (as you aren't physically moving wires after initial install). It will be interesting to see if Cisco jumps aboard this train or continues to pretend like the sands underneath it aren't shifting. I'm sure there would be a market for VMs running Cisco's IOS, it is still by far the most popular network operating system.

freehunter 12y ago
Cisco already has virtual switches. My company just bought a Cisco v1000 virtual switch. You can run your entire company from just one physical piece of hardware with Hyper-V and a Cisco V switch installed.
m-app 12y ago
"I think it is very safe to say that most dedicated "smart" network hardware is going to disappear in the next 5-15 years to be replaced by virtual machines acting as several pieces of network equipment." I agree with this idea in that most of the intelligence will be pushed out towards the edges of the network and overlay networks will make a lot of the physical network invisible to the applications and even the management. However, this does not mean that the networking hardware can all be just dumb devices. First of all, this kind of imperative networking will not scale to large data center implementations and secondly dumb devices will become useless once disconnected from their controller. "It will be interesting to see if Cisco jumps aboard this train or continues to pretend like the sands underneath it aren't shifting. I'm sure there would be a market for VMs running Cisco's IOS, it is still by far the most popular network operating system." The way Cisco is getting into the SDN market is by leveraging Application Centric Infrastructure (ACI) to define the complete DC (Compute, Storage and Network) with templates and letting the whole system configure itself. The switches supporting this infrastructure are based on merchant silicon in combination with Cisco ASIC's to provide the most optimal performance at these intelligent edges. Add to that the available open API's and the investment in OpenStack and you have a rock-solid and cost-effective solution for the future of the DC and DevOps. Cisco ACI: http://cisco.com/go/aci http://cisco.com/go/aci I would love to talk about it more if you want. These are exciting times. Hit me up on email or IRC. (Disclaimer: I work for Cisco but these thought are still my own, yada yada)

hhw 12y ago

If everything is more or less stock Linux, what value does Cumulus actually add for it to be worth the licensing costs? This all seems to be a giant step backwards. JunOS is now the gold standard for its CLI and configuration, due to being highly structured and well organised. Being able to validate configurations before committing them, and automatic rollbacks with commit confirm is a much improved way of doing things over IOS. Given that Juniper hardware can often be bought significantly discounted, the cost savings of white box gear are going to be small, if at all. Thus far, they seem to be sold in such small volumes that comparable Juniper or Cisco gear can easily be purchased for less. Also, how exactly is Cumulus separating the control plane and data plane? From the look of things, SDN is going to end up being relegated to just easy automation instead, like how Cloud has just been relegated to VMs with flexible billing instead of abstracting underlying infrastructure.

wmf 12y ago
In theory, the value of Cumulus is (1) support and (2) their proprietary switchd. (Edit: I also second the other comments that Cumulus licensing appears to be really cheap. But if you're looking for "CentSwitch" you're out of luck.) Also, how exactly is Cumulus separating the control plane and data plane? They're not. Cumulus is kind of anti-SDN.
- drdaeman 12y ago
  "Proprietary" means "if you ever have issues with that that our support won't be able to solve for you in meaningful time span, you're basically out of luck". We're running a gateway router on Vyatta/VyOS and with every single hardware and software change we had some issues. Most were relatively minor (like default settings being not appropriate for our workload), already documented/solved by someone on the 'net. Some had required reading source code and debugging/profiling to see what goes on. Must also admit, some were beyond my understanding and were mysteriously solved with some shamanism and voodoo magic (like randomly tinkering with firmware and driver version combos).
- kijiki 12y ago
  I wouldn't characterize us as "anti-SDN", instead I would say that Cumulus Linux is not an SDN solution by itself. We work closely with all the major SDN vendors for use-cases that are virtualization-centric or cloud-centric. That said, there are use-cases like Hadoop clusters that have very little need for an SDN layer, just very fast L3 IP connectivity.

pyvpx 12y ago

but is it faster than VALE/netgraph?

wmf 12y ago
Yes, hardware forwarding is ~100x faster than optimized software forwarding.
- pyvpx 12y ago
  I missed the switchd/proprietary ASIC part. oops!

voidlogic 12y ago

>These appear as Ethernet interfaces normally do on Linux, visible with ip link show. (ifconfig has long been deprecated by the community in favor of the iproute2 family of tools.) I am amused when I read things like this, 99% of the Linux savvy folks I know use ifconfig.

LukeShu 12y ago
In my experience: we use ifconfig out of familiarity; I know that I "should" be using 'ip' (and I do for scripts).
- a3176082 12y ago
  I don't think you should. Every UNIX operating system has ifconfig. Writing a nonstandard tool instead of fixing the standard one is very antisocial behaviour from the Linux community, mirroring that of Microsoft. Have a look for example at how excellent the ifconfig program for OpenBSD is. These days Linux has become so popular, that it starts to pretend other operating systems and standards are no longer relevant.
kijiki 12y ago
I'm the CTO of Cumulus, and I still sometimes use ifconfig. My fingers are just too programmed at this point. That said, ifconfig doesn't support easy addition of multiple IP addresses to a single interface. You have to create 1 alias interface per IP, which gets unwieldy fast.