8 ms·

SONiC – An Open Source Network Operating System

116 points by jamieweb 6y ago

liotier 6y ago
Linux-based. How is SONiC positioned vs. Cumulus ?
- iptrans 6y ago
  Cumulus requires a license to run on whitebox hardware. Sonic doesn't.
  - 12bits 6y ago
    Does the Nvidia acquisition of Cumulus make that even worse going forward?
    - wmf 6y ago
      Cumulus was a pure software company; now that they are part of Mellanox there are more business models available such as subsidizing software with hardware.
      - rajnathani 6y ago
        It's also interesting that Nvidia (via Mellanox) announced on their blog their support for SONiC [0] a week after they acquired Cumulus. [0] https://blogs.nvidia.com/blog/2020/05/12/mellanox-integrated-sonic-open-networking/ https://blogs.nvidia.com/blog/2020/05/12/mellanox-integrated...
        wmf 6y ago
        SONIC has a very similar vibe to OpenStack and k8s. It's too big to fail, it has what plants crave, and you're dead if you're not on the bandwagon. Whether the code works doesn't matter.
- wmf 6y ago
  Besides being cheaper and supporting more hardware, SONIC is more like a traditional NOS that runs on Linux but doesn't integrate with it. I would expect SONIC to be more familiar to Cisco CLI jockeys and Cumulus to be more familiar to Debian/Ubuntu sysadmins.
  - dahfizz 6y ago
    You are correct that sonic does a poor job of integrating switch management features, but it's nothing like other CLI platforms. All configurations go in a single JSON file, which is used to configure the docker containers that manage the switching hardware.
  - myrandomcomment 6y ago
    Incorrect. In Cumulus you have NCLU which is the Cumulus cli that just takes the commands from the bash prompt. If you know traditional Cisco you can sort it pretty quickly. They start with “net” so to show something “net show interfaces”. Here is a cheatsheet: https://cumulusnetworks.com/learn/resources/cheatsheets/nclu https://cumulusnetworks.com/learn/resources/cheatsheets/nclu Also Cumulus runs FRR (as does sonic) but on Cumulus you can do sudo vtysh and pretty much be at a CLI for routing like you are at an IOS prompt. Sonic sticks all configuration in different docker containers in Json files and can be a real pain. Also not all commands are hitless, ie some will restart forwarding. That is being worked on. SONiC is pretty much “what MS wanted for large scale ops” and still is rough around the edges for enterprise IT. There are a number of companies that support Sonic in production enterprise such as Dell and Apstra.
Cyph0n 6y ago
Shameless plug: the Cisco 8000 series [1] recently got support for SONiC [2]. Right now, I believe it’s limited to fixed systems, but work is ongoing to get SONiC running on the distributed chassis. Exciting stuff! [1]: https://www.cisco.com/c/en/us/products/routers/8000-series-routers/index.html https://www.cisco.com/c/en/us/products/routers/8000-series-r... [2]: https://blogs.cisco.com/sp/cisco-goes-sonic-on-cisco-8000 https://blogs.cisco.com/sp/cisco-goes-sonic-on-cisco-8000
iptrans 6y ago
Obligatory link to tutorial on how to run Sonic on an Arista switch: https://www.servethehome.com/get-started-with-40gbe-sdn-with-microsoft-azure-sonic-for-under-1k/ https://www.servethehome.com/get-started-with-40gbe-sdn-with...
austincheney 6y ago
How does this compare with existing switch OSs from Cisco and Juniper? Linux seems like it would be a bit heavy for a switch.
- wmf 6y ago
  SONIC is less mature but it's free and open source with the benefits and drawbacks that entails. For example, SONIC runs on a variety of different hardware so you could take advantage of that diversity without having to learn different NOSes. Ten years ago switches were using 800 MHz single-core PowerPCs which was adequate to run Linux (although many were using VxWorks or whatever). Now the $400 switches are still wimpy but more expensive "disaggregated" switches are using Atoms or low-end Xeons.
- iptrans 6y ago
  Adding to what wmf said, JunOS is based on BSD and JunoOS evolved will be based on Linux. Most network equipment run their control plane using standard Intel CPUs, so running Linux isn't much of a strech.
- monocasa 6y ago
  Eh, it's probably not that bad. On the low end, Linux isn't a slouch with XDP, etc. On the high end, it's game over if your kernel is in the data plane at all anyway, and Linux is a great option for the control plane of a switch.
- tw04 6y ago
  Cisco is already Linux based. IOS-XE and NX-OS. https://en.m.wikipedia.org/wiki/Cisco_IOS_XE https://en.m.wikipedia.org/wiki/Cisco_IOS_XE https://en.m.wikipedia.org/wiki/Cisco_NX-OS https://en.m.wikipedia.org/wiki/Cisco_NX-OS
  - 12bits 6y ago
    I like this comparison https://medium.com/@Meela349588204/using-cows-to-explain-the-differences-of-the-cisco-operating-systems-876d14145827 https://medium.com/@Meela349588204/using-cows-to-explain-the...
dahfizz 6y ago
FWIW, I had an absolutely terrible time setting up sonic on an edgecore switch which was "supported". Everything was a battle. The configuration is very finicky and not well documented, and every time I went to edgecore or sonic for support, they would blame the other party. We did eventually get everything working, but it took hours of tinkering on the configuration. There's all sorts of undocumented magic about how ports, interfaces, lanes, etc all have to line up with each other. If it doesn't like how you've configured BGP, it will silently fail and refuse to link. And God help you if you have breakout cables. And for some reason, all the services are run in docker containers, which makes it even more painful to debug. Reapplying configurations is slow and buggy in itself, so you have to reboot the switch constantly. Ultimately I find sonic hard to recommend. Yes it's free, but I'm really not sure that's a benefit. Any home enthusiast will buy a consumer grade switch with an OS preinstalled, or use OpenWRT. Any business using this for production would be wise to go with a stable product with good support, like Cumulus. As much as I hate how Cisco does business, at least their products will mostly work how you need out of the box.
badrabbit 6y ago
Arista's EOS is open source and up to the task, it would be awesome if there was a fork of it supporting whitebox gear.
- wmf 6y ago
  No, EOS is not even close to open source. You might be able to run cEOS on a whitebox.
  - badrabbit 6y ago
    Maybe something changed but I rember downloading the source and also running it in a vm.
    - myrandomcomment 6y ago
      EOS core code never been open source. It it is built on Fedora Linux. They do have vEOS and cEOS offerings but there is closed source Arista code in all of them.
musicale 6y ago
Not really a "network" operating system since it doesn't seem to manage a network consisting of multiple devices the way something like ONOS, OpenDaylight or FAUCET would. More like a "switch" operating system - it looks like SONiC is Microsoft's answer to ONL, Stratum, OpenSwitch, Cumulus et al.. Basically open source software to run on your cheap whitebox (or expensive greybox) switch.
Conan_Kudo 6y ago
So my problem with SONiC, Cumulus, and all these other "network operating system" Linux platforms is that they all seem to be designed to be fiefdoms and tend to be really stale. In my view, they bring almost nothing of value to the table. Let's talk a bit more about SONiC... For one, SONiC literally lists that pull requests that aren't already planned and approved will not be accepted[1]. This defeats a good chunk of the value of having a community project. People will want to contribute and extend your platform in ways you never thought of, and they'll do it in a completely decentralized fashion. Another issue I see is that SONiC literally holds back everything to an old Linux kernel and ships random BSP blobs that are unvetted. This is a nasty combination for anyone who wants to consider their NOS trusted or secure. They're on a 4.9.x kernel, and while that is still maintained, it is far from the best option if you want to take advantage of innovation in Linux networking. I'm also generally confused on why this whole project isn't just "let's get the networking tools and hardware support stuff into standard Linux distributions and leverage their tooling and communities". This was also a problem I had with Cumulus. When I tore apart Cumulus, I figured out that it was less than a dozen unique tools and a distribution rebuilt for 32-bit MIPS and PowerPC. It was pretty trivial to rebase to standard Fedora or Debian and get a better platform out of it. And finally, I don't really think this provides any real innovation. It's not really different from Cumulus, Open Network Linux, and others. And ONL actually is using more up to date kernels (5.4.x as of right now!) and offers better networking tools! What I would love to see is all these people who keep doing this crap working in the actual Linux distribution communities to build and integrate with upstream projects so that everyone downstream gets all kinds of flexibility. Imagine if you had a flavor of Fedora CoreOS for your network gear! The immutable OS, updated with RPM-OSTree, fresh software stack, and broad hardware support, all in one neat package. If we treated the network gear like weaker servers, instead of specialty equipment, there's so many more interesting things you can do! [1]: https://github.com/Azure/SONiC/wiki/Sonic-Roadmap-Planning https://github.com/Azure/SONiC/wiki/Sonic-Roadmap-Planning
- turtlecrush1337 6y ago
  >This was also a problem I had with Cumulus. When I tore apart Cumulus, I figured out that it was less than a dozen unique tools and a distribution rebuilt for 32-bit MIPS and PowerPC. Almost all of which are open source (or at least "source available"), with the exception of switchd, which cannot be open sourced because it links with proprietary asic sdk's. I don't see how having very few custom tools over a vanilla Linux distribution is a bad thing. >It was pretty trivial to rebase to standard Fedora or Debian and get a better platform out of it. If you enable upstream Debian apt sources in your sources.list then it effectively is standard Debian - plus switchd. Of course it is entirely possible to take all of the components of Cumulus Linux and use them on a separate operating system - enter sonic, vyos, etc - so if you build out such a system which can also drive ASICs and that you prefer over Cumulus, you can take full advantage of all of Cumulus's open source contributions. >What I would love to see is all these people who keep doing this crap working in the actual Linux distribution communities to build and integrate with upstream projects so that everyone downstream gets all kinds of flexibility If I read you correctly, Cumulus works upstream as much as it can: ~/linux$ git log --author "cumulusnetworks.com" --oneline | wc -l 773 ~/ifupdown2$ git log --author "cumulusnetworks.com" --oneline | wc -l 1265 ~/frr$ git log --author "cumulusnetworks.com" --oneline | wc -l 8107 I like to believe Cumulus is quite active in the communities of projects it uses. I feel I may have misunderstood your point, though. >If we treated the network gear like weaker servers, instead of specialty equipment, there's so many more interesting things you can do! I completely agree, that's the dream! Disclaimer: I work at Cumulus
teleforce 6y ago
If you want an overview of of the state-of-the-art open source networking stack, you can check this last year FOSDEM 2019 presentation by one of the developers of BISDN [1]. Somehow I don't think they mentioned SONiC in their presentation though. I teach computer networking lab but for now we need to resort using dual boot Linux Switch Appliance (LISA) that uses custom kernel and another vanilla kernel running Quagga on multi-port Ethernet embedded PC. The good thing is that both LISA and Quagga use CLI environment similar to familiar Cisco switch/router IOS. I really wish there is an open source alternative offering that can seamlessly support layer 2 and layer 3 without dual boot, perhaps using Software Defined Networking (SDN) concept with an intuitive CLI. Recently the new Shortest Path Bridging (SPB) has been integrated with 802.1Q since 2018 (bye-bye TRILL). I reckon any reasonably good Linux based open source layer 2 and 3 network OS player will be extremely popular overnight for enterprise, consumer and education. Together with eBPF this thing should be flying on the new off-the-shelf whitebox supporting 40Gbps multi-port Ethernet. Imagine a LAN or Metro LAN party with this beast :-) [1]https://archive.fosdem.org/2019/schedule/event/from_closed_to_open_source_switch/ https://archive.fosdem.org/2019/schedule/event/from_closed_t...

116 points by jamieweb 6y ago

liotier 6y ago

Linux-based. How is SONiC positioned vs. Cumulus ?

iptrans 6y ago
Cumulus requires a license to run on whitebox hardware. Sonic doesn't.
- 12bits 6y ago
  Does the Nvidia acquisition of Cumulus make that even worse going forward?
  - wmf 6y ago
    Cumulus was a pure software company; now that they are part of Mellanox there are more business models available such as subsidizing software with hardware.
    - rajnathani 6y ago
      It's also interesting that Nvidia (via Mellanox) announced on their blog their support for SONiC [0] a week after they acquired Cumulus. [0] https://blogs.nvidia.com/blog/2020/05/12/mellanox-integrated-sonic-open-networking/ https://blogs.nvidia.com/blog/2020/05/12/mellanox-integrated...
      - wmf 6y ago
        SONIC has a very similar vibe to OpenStack and k8s. It's too big to fail, it has what plants crave, and you're dead if you're not on the bandwagon. Whether the code works doesn't matter.
wmf 6y ago
Besides being cheaper and supporting more hardware, SONIC is more like a traditional NOS that runs on Linux but doesn't integrate with it. I would expect SONIC to be more familiar to Cisco CLI jockeys and Cumulus to be more familiar to Debian/Ubuntu sysadmins.
- dahfizz 6y ago
  You are correct that sonic does a poor job of integrating switch management features, but it's nothing like other CLI platforms. All configurations go in a single JSON file, which is used to configure the docker containers that manage the switching hardware.
- myrandomcomment 6y ago
  Incorrect. In Cumulus you have NCLU which is the Cumulus cli that just takes the commands from the bash prompt. If you know traditional Cisco you can sort it pretty quickly. They start with “net” so to show something “net show interfaces”. Here is a cheatsheet: https://cumulusnetworks.com/learn/resources/cheatsheets/nclu https://cumulusnetworks.com/learn/resources/cheatsheets/nclu Also Cumulus runs FRR (as does sonic) but on Cumulus you can do sudo vtysh and pretty much be at a CLI for routing like you are at an IOS prompt. Sonic sticks all configuration in different docker containers in Json files and can be a real pain. Also not all commands are hitless, ie some will restart forwarding. That is being worked on. SONiC is pretty much “what MS wanted for large scale ops” and still is rough around the edges for enterprise IT. There are a number of companies that support Sonic in production enterprise such as Dell and Apstra.

Cyph0n 6y ago

Shameless plug: the Cisco 8000 series [1] recently got support for SONiC [2]. Right now, I believe it’s limited to fixed systems, but work is ongoing to get SONiC running on the distributed chassis. Exciting stuff! [1]: https://www.cisco.com/c/en/us/products/routers/8000-series-routers/index.html https://www.cisco.com/c/en/us/products/routers/8000-series-r... [2]: https://blogs.cisco.com/sp/cisco-goes-sonic-on-cisco-8000 https://blogs.cisco.com/sp/cisco-goes-sonic-on-cisco-8000

iptrans 6y ago

Obligatory link to tutorial on how to run Sonic on an Arista switch: https://www.servethehome.com/get-started-with-40gbe-sdn-with-microsoft-azure-sonic-for-under-1k/ https://www.servethehome.com/get-started-with-40gbe-sdn-with...

austincheney 6y ago

How does this compare with existing switch OSs from Cisco and Juniper? Linux seems like it would be a bit heavy for a switch.

wmf 6y ago
SONIC is less mature but it's free and open source with the benefits and drawbacks that entails. For example, SONIC runs on a variety of different hardware so you could take advantage of that diversity without having to learn different NOSes. Ten years ago switches were using 800 MHz single-core PowerPCs which was adequate to run Linux (although many were using VxWorks or whatever). Now the $400 switches are still wimpy but more expensive "disaggregated" switches are using Atoms or low-end Xeons.
iptrans 6y ago
Adding to what wmf said, JunOS is based on BSD and JunoOS evolved will be based on Linux. Most network equipment run their control plane using standard Intel CPUs, so running Linux isn't much of a strech.
monocasa 6y ago
Eh, it's probably not that bad. On the low end, Linux isn't a slouch with XDP, etc. On the high end, it's game over if your kernel is in the data plane at all anyway, and Linux is a great option for the control plane of a switch.
tw04 6y ago
Cisco is already Linux based. IOS-XE and NX-OS. https://en.m.wikipedia.org/wiki/Cisco_IOS_XE https://en.m.wikipedia.org/wiki/Cisco_IOS_XE https://en.m.wikipedia.org/wiki/Cisco_NX-OS https://en.m.wikipedia.org/wiki/Cisco_NX-OS
- 12bits 6y ago
  I like this comparison https://medium.com/@Meela349588204/using-cows-to-explain-the-differences-of-the-cisco-operating-systems-876d14145827 https://medium.com/@Meela349588204/using-cows-to-explain-the...

dahfizz 6y ago

FWIW, I had an absolutely terrible time setting up sonic on an edgecore switch which was "supported". Everything was a battle. The configuration is very finicky and not well documented, and every time I went to edgecore or sonic for support, they would blame the other party. We did eventually get everything working, but it took hours of tinkering on the configuration. There's all sorts of undocumented magic about how ports, interfaces, lanes, etc all have to line up with each other. If it doesn't like how you've configured BGP, it will silently fail and refuse to link. And God help you if you have breakout cables. And for some reason, all the services are run in docker containers, which makes it even more painful to debug. Reapplying configurations is slow and buggy in itself, so you have to reboot the switch constantly. Ultimately I find sonic hard to recommend. Yes it's free, but I'm really not sure that's a benefit. Any home enthusiast will buy a consumer grade switch with an OS preinstalled, or use OpenWRT. Any business using this for production would be wise to go with a stable product with good support, like Cumulus. As much as I hate how Cisco does business, at least their products will mostly work how you need out of the box.

badrabbit 6y ago

Arista's EOS is open source and up to the task, it would be awesome if there was a fork of it supporting whitebox gear.

wmf 6y ago
No, EOS is not even close to open source. You might be able to run cEOS on a whitebox.
- badrabbit 6y ago
  Maybe something changed but I rember downloading the source and also running it in a vm.
  - myrandomcomment 6y ago
    EOS core code never been open source. It it is built on Fedora Linux. They do have vEOS and cEOS offerings but there is closed source Arista code in all of them.

musicale 6y ago

Not really a "network" operating system since it doesn't seem to manage a network consisting of multiple devices the way something like ONOS, OpenDaylight or FAUCET would. More like a "switch" operating system - it looks like SONiC is Microsoft's answer to ONL, Stratum, OpenSwitch, Cumulus et al.. Basically open source software to run on your cheap whitebox (or expensive greybox) switch.

Conan_Kudo 6y ago

So my problem with SONiC, Cumulus, and all these other "network operating system" Linux platforms is that they all seem to be designed to be fiefdoms and tend to be really stale. In my view, they bring almost nothing of value to the table. Let's talk a bit more about SONiC... For one, SONiC literally lists that pull requests that aren't already planned and approved will not be accepted[1]. This defeats a good chunk of the value of having a community project. People will want to contribute and extend your platform in ways you never thought of, and they'll do it in a completely decentralized fashion. Another issue I see is that SONiC literally holds back everything to an old Linux kernel and ships random BSP blobs that are unvetted. This is a nasty combination for anyone who wants to consider their NOS trusted or secure. They're on a 4.9.x kernel, and while that is still maintained, it is far from the best option if you want to take advantage of innovation in Linux networking. I'm also generally confused on why this whole project isn't just "let's get the networking tools and hardware support stuff into standard Linux distributions and leverage their tooling and communities". This was also a problem I had with Cumulus. When I tore apart Cumulus, I figured out that it was less than a dozen unique tools and a distribution rebuilt for 32-bit MIPS and PowerPC. It was pretty trivial to rebase to standard Fedora or Debian and get a better platform out of it. And finally, I don't really think this provides any real innovation. It's not really different from Cumulus, Open Network Linux, and others. And ONL actually is using more up to date kernels (5.4.x as of right now!) and offers better networking tools! What I would love to see is all these people who keep doing this crap working in the actual Linux distribution communities to build and integrate with upstream projects so that everyone downstream gets all kinds of flexibility. Imagine if you had a flavor of Fedora CoreOS for your network gear! The immutable OS, updated with RPM-OSTree, fresh software stack, and broad hardware support, all in one neat package. If we treated the network gear like weaker servers, instead of specialty equipment, there's so many more interesting things you can do! [1]: https://github.com/Azure/SONiC/wiki/Sonic-Roadmap-Planning https://github.com/Azure/SONiC/wiki/Sonic-Roadmap-Planning

turtlecrush1337 6y ago
>This was also a problem I had with Cumulus. When I tore apart Cumulus, I figured out that it was less than a dozen unique tools and a distribution rebuilt for 32-bit MIPS and PowerPC. Almost all of which are open source (or at least "source available"), with the exception of switchd, which cannot be open sourced because it links with proprietary asic sdk's. I don't see how having very few custom tools over a vanilla Linux distribution is a bad thing. >It was pretty trivial to rebase to standard Fedora or Debian and get a better platform out of it. If you enable upstream Debian apt sources in your sources.list then it effectively is standard Debian - plus switchd. Of course it is entirely possible to take all of the components of Cumulus Linux and use them on a separate operating system - enter sonic, vyos, etc - so if you build out such a system which can also drive ASICs and that you prefer over Cumulus, you can take full advantage of all of Cumulus's open source contributions. >What I would love to see is all these people who keep doing this crap working in the actual Linux distribution communities to build and integrate with upstream projects so that everyone downstream gets all kinds of flexibility If I read you correctly, Cumulus works upstream as much as it can: ~/linux$ git log --author "cumulusnetworks.com" --oneline | wc -l 773 ~/ifupdown2$ git log --author "cumulusnetworks.com" --oneline | wc -l 1265 ~/frr$ git log --author "cumulusnetworks.com" --oneline | wc -l 8107 I like to believe Cumulus is quite active in the communities of projects it uses. I feel I may have misunderstood your point, though. >If we treated the network gear like weaker servers, instead of specialty equipment, there's so many more interesting things you can do! I completely agree, that's the dream! Disclaimer: I work at Cumulus

teleforce 6y ago

If you want an overview of of the state-of-the-art open source networking stack, you can check this last year FOSDEM 2019 presentation by one of the developers of BISDN [1]. Somehow I don't think they mentioned SONiC in their presentation though. I teach computer networking lab but for now we need to resort using dual boot Linux Switch Appliance (LISA) that uses custom kernel and another vanilla kernel running Quagga on multi-port Ethernet embedded PC. The good thing is that both LISA and Quagga use CLI environment similar to familiar Cisco switch/router IOS. I really wish there is an open source alternative offering that can seamlessly support layer 2 and layer 3 without dual boot, perhaps using Software Defined Networking (SDN) concept with an intuitive CLI. Recently the new Shortest Path Bridging (SPB) has been integrated with 802.1Q since 2018 (bye-bye TRILL). I reckon any reasonably good Linux based open source layer 2 and 3 network OS player will be extremely popular overnight for enterprise, consumer and education. Together with eBPF this thing should be flying on the new off-the-shelf whitebox supporting 40Gbps multi-port Ethernet. Imagine a LAN or Metro LAN party with this beast :-) [1]https://archive.fosdem.org/2019/schedule/event/from_closed_to_open_source_switch/ https://archive.fosdem.org/2019/schedule/event/from_closed_t...