6 ms·
A joke answer and a serious question: A: "Use BSD" Q: Why is there such a strong focus on trying to get Linux network performance when (I think) everyone agre
by jedberg 11y ago
A joke answer and a serious question:
A: "Use BSD"
Q: Why is there such a strong focus on trying to get Linux network performance when (I think) everyone agrees BSD is better at networking? What does Linux offer beyond the network that BSD doesn't when it comes to applications that demand the fastest networks?
ps. I think the markdown filter is broken, I can't make a literal asterisk with a backslash. Anyone know how HN lets you make an inline asterisk?
- majke 11y agoI have no idea. How many UDP packets can FreeBSD pass to application? I don't think there is any particular problem with Linux. The core problem of "slow" performance is mostly due to limitations of BSD API (recvmmsg is a good example of a workaround), and due to the feature richness of Linux. Also, doing the math: 2GHz / 350k pps = 5714 cycles per packet. 6k cycles to deliver the packet from NIC to one core, then pass it between cores and copy over to application. That ain't bad really.
- jgrahamc 11y agoMarkdown? This is HN where we use whatever PG* decides we're allowed to use. https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc *Paul Graham
- jedberg 11y agoSo in other words it's impossible to have two inline asterisks without a space after the first one. Oh well.
- elevensies 11y ago**
- jedberg 11y agoYou indented, which means there is no formatting at all. You cheated. :)
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- sliverstorm 11y agoThere was a trick wrapping things in equals signs or something like that, but I can't recall...
- fapjacks 11y agoThe Underhanded HN Contest.
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- scarmig 11y agoWhy is the BSD network stack superior, anyway? I hear it repeated a lot, and I'm just surprised Linux lags with all the attention it gets. Is it something related to kqueue vs epoll?
- makmanalp 11y agoThe easy answer probably has to do with comfort and familiarity - no one wants to be in that situation where their software package isn't ported to BSD or the BSD port is lagging several version behind, or have a BSD-specific bug and have to track it down with the developers. Again, not saying these are necessarily realistic, but it's the thing that comes to mind first.
- feld 11y agoBut thanks to the ports tree and the Porter's Handbook, it's easier to solve this type of problem on FreeBSD than on Linux. And the chance of a Linux distro having a newer version of something important is unlikely unless a new distro was just released and picked the latest release of a particular software to standardize their package on for the lifetime of this new OS. Either way -- FreeBSD will continue (or be capable of) getting updates to track upstream and your Linux distro will only be backporting security fixes.
- acdha 11y ago> But thanks to the ports tree and the Porter's Handbook, it's easier to solve this type of problem on FreeBSD than on Linux. > > Either way -- FreeBSD will continue (or be capable of) getting updates to track upstream and your Linux distro will only be backporting security fixes. In most cases what actually happens is that people use the main distribution repo for the 99% of packages which are stable and when you need something newer you add external apt/yum/etc. sources for those specific projects. In the Ubuntu world, there's a huge ecosystem supporting this style where you upload source packages and they'll build and host the binary packages for you: https://help.launchpad.net/Packaging/PPA https://help.launchpad.net/Packaging/PPA This approach gives you the speed, reliability and security benefits of binary packages with the currency of ports and, more importantly, allows you to opt-in only where you specifically know you need new features.
- feld 11y agoAnd how can you trust those third party repos? That's the hard part. At least FreeBSD's way guarantees your packages are built from source code that matches the checksum of what upstream has released. If it doesn't match because of a compromise or upstream re-rolled their tarball it is discovered very, very quickly.
- deleted 11y ago[deleted]
- acdha 11y agoReading things like http://blog.erratasec.com/2013/02/custom-stack-it-goes-to-11.html http://blog.erratasec.com/2013/02/custom-stack-it-goes-to-11... suggests that it's probably simply the case that the OS itself is sort of a side-issue once you need performance since you're going to be bypassing the normal network stack anyway. At that point ease of management or package installation probably matters more to developers and it might simply come down to things like driver support and other stability/performance issues where Linux has gotten a LOT of highly-specialized attention from hardware vendors and the HPC world. Back when 10G hardware was just starting to enter the market, we bought cards which came with a Linux driver but it took awhile before that was ported to FreeBSD and longer still before the latency had been optimized as much.
- chetanahuja 11y ago"Why is there such a strong focus on trying to get Linux network performance when (I think) everyone agrees BSD is better at networking?" Setting aside the fact that a typical fanboi comment is at the top of HN post, are you seriously contending that since one OS does a thing there's no need for other OS's to do the same thing?
- jalons 11y agoTools. Compatibility. Resembles systems already in place (i.e., back when BSD failed to get SMP support "soon" there wasn't an option if your workspace was CPU bound). Numbers (it's popular in HPC/HFT because it's popular). We're not consuming/using the built in network stacks anyway, we're using the OS as a content delivery system. Get us something we can get to the cores on, we're going to pin our applications directly to those cores keeping the kernel relegated to scheduling tasks on whichever NUMA core is further away from the PCIe bus running into the CPU. We'll have the CPU's we're using pegged in a constant spinlock anyway which would make the scheduler think really, really hard about running tasks there. We don't use realtime kernels as it's better for us to pay the price on the occasional spike outlier than to raise the baseline latency up. Due to my own unfamiliarity, I don't know what BSD's equivalent to isolcpus is. I don't know how to taskset on a BSD. I don't know if the infinibad/ethernet controller's firmware/bypass software works. I don't know how BSD's scheduler works (not that we usually care, but there are times when one can't avoid work needing to be scheduled, things like rpc calls to shut down an app, ssh if you can spare the clock cycles for key verification, etc). Would dtrace come in handy? Most definitely. Is that enough for us to abandon what we know works? Not yet.
- jedberg 11y agoThis is probably the best argument: Familiarity. However, we're talking about very specific needs: super high performance networking. If you have that specific of a need, wouldn't you want something unfamiliar if it solves the problem best?
- jebblue 11y agoIf it's truly better and the only difference is remove Linux and install BSD then what is BSD doing that is better/different/messed up that the packets can flow faster in BSD than in Linux?
- Matumio 11y agoTalking about unfamiliarity and specific needs: FPGAs are much better suited than CPUs for processing minimum-sized frames at wirespeed. They can still forward all unhandled frames to a CPU. Yes, it's a lot of development effort compared to a CPU-only solution, but considering all the kernel-optimizing-multicore-cleverness from OP I would say we are approaching the break-even point.
- xorcist 11y agoDefine "better". The very few times in my life that networking performance mattered, I found Linux (2.2+) to be hard to beat as a general purpose stack. If you need something specific with better performance than that, you should probably look at moving more of the network stack to your application layer.
- jemfinch 11y agoWho, in 2015, agrees that BSD is better at networking? I remember these claims being made in the late 90s, and perhaps they were true back then, but it's been 15 years, and I would be surprised if Linux hasn't caught up by virtue of its faster development pace, greater mindshare, and increased corporate/datacenter usage. So, in all seriousness: what recent, well argued essays/papers can you refer me to so I can understand the claim that BSD networking is still better than Linux in 2015?
- jedberg 11y agoThere was a good discussion about it on reddit about 10 months back: http://www.reddit.com/r/linux/comments/2d5wzg/linux_network_stack/ http://www.reddit.com/r/linux/comments/2d5wzg/linux_network_... It was also linked to from HN: https://news.ycombinator.com/item?id=8167126 https://news.ycombinator.com/item?id=8167126 Both have some pretty good sources (and some not so good sources too)
- vezzy-fnord 11y agoFacebook made the rounds last year for a job posting that stated the goal was "for the Linux kernel network stack to rival or exceed that of FreeBSD": http://www.phoronix.com/scan.php?page=news_item&px=MTc1NjY http://www.phoronix.com/scan.php?page=news_item&px=MTc1NjY
- teacup50 11y agohttp://info.iet.unipi.it/~luigi/netmap/ http://info.iet.unipi.it/~luigi/netmap/ Shipped in FreeBSD by default, developed for FreeBSD first -- and that's just the bleeding edge side of things.
- snoman 11y agoSome of the very same reasons you give for the proposition that network performance having caught up can be used to argue that it may have slowed down (ie feature creep and bloat). So the other question is: who, in 2015 disagrees and what recent, well argued, essays/papers can you refer us to that might demonstrate that anything has changed at all.
- the-dude 11y ago
- the8472 11y agoMaybe one factor is subtle differences in system call behavior, cascading down the whole stack? E.g. on freebsd I recently ran into an ENOBUFS that I've never seen on linux. Although man pages said that they could happen.
- 0xbadcafebee 11y agoReplace "Linux" with "Windows", and "BSD" with "Linux". Obviously it provides nothing extra, and certainly nothing better, when it comes to a server role. Other than support contracts. And support for 3rd-party apps. And hardware drivers/compatibility. And users that know how to operate it. And brand name recognition. And size of software development community, libraries, tools. And industry reliance on non-portable software (Docker). Answer to your question: technical superiority is dwarfed by a more popular product.
- vacri 11y ago> And industry reliance on non-portable software (Docker). Docker is the new shiny kid on the block, but the industry is far from reliant on it. It's also a bit misleading to call the software non-portable in a Linux v FreeBSD debate - it's not that the software is non-portable, it's that it uses a feature that is not available in FreeBSD. In a contest about bragging rights, that's significant.
- trentnelson 11y agoAre you saying the Linux network stack is superior to contemporary Windows? If so, thanks for the aneurism. Windows is decades ahead, thanks to a fundamentally more performant kernel and driver architecture, the key parts of which have been present since NT. (And VMS, if you want to get picky.)
- bcook 11y agoLinux supports more scheduling/QoS algorithms. Most of the recent (~10 years?) academic papers on packet queuing algorithms implement their algo in Linux.
- ninjakeyboard 11y agoIs there any comparison you can provide? (purely out of personal interest?)
- nodata 11y agoExactly! How dare people try and improve Linux when they could use something else instead!