4 ms·
As with all things, you would really need to benchmark the system, preferably with real load, both ways to know for sure. But that takes a lot of time, especial
by toast0 5y ago
As with all things, you would really need to benchmark the system, preferably with real load, both ways to know for sure. But that takes a lot of time, especially if you're going to put in the time to tweak both systems.
People can do amazing stuff with enough time in both FreeBSD and Linux. I honestly think most server applications wouldn't be held back by either OS. You need your application to be really lightweight and focused before the OS makes a big difference, and even then, the differences only show if you're maxing out the hardware.
I worked at WhatsApp, and enjoyed working with FreeBSD there, and clearly it worked for us. Linux in FB datacenters also worked, but the server components were a lot different so there was never an apples to apples comparison. I run FreeBSD on my personal servers because I enjoyed working with it at Yahoo and then WhatsApp; but my personal servers don't have any performance needs. Sure, the networking stuff is nice (and it was nice to work with in the kernel), but what I like most about FreeBSD is the lack of churn. I can look at old administrative recipies and all the commands still work. I can expect (and mostly get) that when I upgrade, everything will keep working, and maybe a little better; occassionally, a lot better.
- waynesonfire 5y ago> but what I like most about FreeBSD is the lack of churn I agree completely. As I mature in this field this becomes an ever important characteristic of the technology I adopt. Erlang also shares this property.
- toast0 5y agoI feel like Erlang churn is more acceptable because they're usually honest in naming. You get pg2 which does what pg did, but differntly. And now pg (which could have been pg3, but the old pg had been gone long enough to reclaim the name). Similar with phash2, although phash may live for much longer.
- tiffanyh 5y agoHi toast0 IIRC, I saw a presentation by someone (Rick?) where at your previous employer - you guys slimmed FreeBSD down to be unbelievably minimal such that only 2-3 total services ran on the entire server. Was that done for performance reason? Or for hardening reasons? If someone wanted to do that today with FreeBSD: would you recommend it and how would you go about doing it (NanoBSD)?
- toast0 5y agoProbably Rick, he did a bunch of presentations. We ran a pretty vanilla FreeBSD. IIRC, we disabled locales, and had 5-10 patches depending on exactly when and what bottlenecks we were running into. We didn't go out of our way to turn off daemons, we just didn't start anything we didn't need. Out of the box, you get sshd, crond, getty, ntpd, syslogd, our system activity report, and that's really all you need to administer the box. Then on our Erlang machines, we'd run one big Erlang process. Unless something went seriously wrong, only Erlang ever had real work to do. I don't know that there was a performance benefit, you can run a lot of daemons that are waiting on sockets and don't use meaningful amounts of ram before noticing a performance hit. Hardening has something to do with it, can't get into our smtpd if we're not running it. But mostly it kept things simple and reduced work. No extra daemons means no configuration of them and no update headaches. Even with that limited set, I had to go around and reconfigure sshd sometimes and ntp rather more than I'd have liked, and we used caveman automation, so stuff that needs super user is extra painful. On hardening, we didn't exclude things from FreeBSD's base, even though that would probably be a good idea. Some things are there for reasonable reasons, but had no relevance to us, and less installed stuff would be better, but tradeoffs. We did run stud (now known as hitch, which we needed when Erlang TLS was too slow) in a totally locked down jail though; statically configured binary, only executable in the jail, etc.
- tiffanyh 5y agoThanks so much. Really appreciate the insight. Would you use FreeBSD for new messaging/web-service today or Linux? (I ask given you have first hand knowledge of how FreeBSD performed vs Linux after your organization made the migration)
- toast0 5y agoPersonally, I would use FreeBSD. But, to be honest, that's 99% because of familiarity and only 1% because I hope the performance is better. Some of the familiarity is just knowing how the system works and how to do routine tasks easily. Some of it is knowing what manual pages will give me the answers I need and being familiar with the writing so I can look it up. But also, I know a lot of important sysctls to adjust behaviors and where to look if there's something that needs adjusting that I think might have a knob. And, I'm fairly handy at digging through the kernel now that I've done it enough. The Linux kernel is organized differently, and I'd need to develop that skill/knowledge. Getting changes upstreamed is a challenge and I've had some success with FreeBSD, but Linux kernel would be new processes, and some things that are really core to a usable system (for example ifconfig/ip) are developed seperately, so have their own processes, so potentially more processes to learn. Mostly our software ran fine in Linux and the things that bothered me the most were mostly artifacts of how Facebook runs their servers. That said, some things in Linux bother me: why does ss exist as a faster netstat instead of just making netstat faster? I don't find the new init framework compelling on servers (I don't hotplug anything, and I don't think init should restart crashed server processes) and it actively hinders me on desktop/laptop, so it's frustrating that the ecosystem has embraced it (but I'm trying to avoid an offtopic rant, email in profile if you want a rant). We had a couple migration issues, but nothing serious. epoll and kqueue are similar, but not similar enough that we didn't need to tweak some things in Erlang to make it work better, but the chat machines in FB have much less ram because we had to select from FB's menu of server hardware and more nodes with less ram in each makes the most sense on their menu, so I can't say for sure if epoll scales as well as kqueue. I understand epoll takes more syscalls to do the same work (for network I/O anyway), so that's probably worse, but it likely doesn't make a big difference, but then there's also io_uring which I haven't evaluated and might be better than either epoll or kqueue. I think Linux's default memory overcommit behavior is problematic for a single process server, but it's hard to disable it because the ecosystem has formed around it. Tools allocate huge memory because 'it doesn't matter, it only really allocates when I write to it', and like, I understand how that's useful, but it's impossible for a program to handle out of memory when the trigger is a write to allocated space; alloc failures are hard to handle, but with careful design you can get a fairly consistent crash report most of the time.
- GoblinSlayer 5y ago> when I upgrade, everything will keep working FreeBSD doesn't promise compatibility between major versions, it's a breaking change, e.g. in FreeBSD 12 they revamped some system types and like half of system interface became incompatible with FreeBSD 11.
- toast0 5y agoIt depends on the specifics, but if you install the -compat packages and enable the compat kernel flags, broadly speaking old software continues to work across major versions. Anyway, just because I expect it to happen, doesn't mean it happens (and I acknowledged that). Things I've run into include when they made the CPU masks bigger, old binaries couldn't set CPU affinity (even when the old code had a mask big enough for the current CPU; we had a local patch for this, since upstream didn't seem interested in making it right). Updating to 13 at home with the approved process (install kernel, reboot, install userland, reboot (maybe), do ports/pkg update), resulted in bad network configuration because 12.x ifconfig didn't fully work with 13.0 kernel, but I believe that's been addressed; and I never hit similar issues in prod because we would install both the new kernel and new userland before rebooting, because redundant servers means you can live dangerously (if the first server fails to start, you can improve the upgrade process on the next ones). This one wasn't terrible for me, but I can imagine someone having a very negative experience and holding a grudge if they had a less accessible machine and specific network configs that made the machine unreachable after the first reboot, my machine was accessible but misbehaving and I've got easy access to the console I rolled back and later did a more cautious update. FreeBSD 11 (+/- 1) changed page inactivity policies in an unexpected way and resulted in undesired swapping for our use cases and becauae it was a slow process it wasn't obvious. That release also negatively impacted our heavy disk I/O systems in a way I didn't have time to get to the bottom of, accurate benchmarking was time consuming and user impacting and migration was in progress, so those systems were forbidden to upgrade. They didn't need the positive networking changes because of their use case, so it was acceptable.