3 ms·

Hi toast0 IIRC, I saw a presentation by someone (Rick?) where at your previous employer - you guys slimmed FreeBSD down to be unbelievably minimal such that on

Hi toast0 IIRC, I saw a presentation by someone (Rick?) where at your previous employer - you guys slimmed FreeBSD down to be unbelievably minimal such that only 2-3 total services ran on the entire server. Was that done for performance reason? Or for hardening reasons? If someone wanted to do that today with FreeBSD: would you recommend it and how would you go about doing it (NanoBSD)?

toast0 5y ago
Probably Rick, he did a bunch of presentations. We ran a pretty vanilla FreeBSD. IIRC, we disabled locales, and had 5-10 patches depending on exactly when and what bottlenecks we were running into. We didn't go out of our way to turn off daemons, we just didn't start anything we didn't need. Out of the box, you get sshd, crond, getty, ntpd, syslogd, our system activity report, and that's really all you need to administer the box. Then on our Erlang machines, we'd run one big Erlang process. Unless something went seriously wrong, only Erlang ever had real work to do. I don't know that there was a performance benefit, you can run a lot of daemons that are waiting on sockets and don't use meaningful amounts of ram before noticing a performance hit. Hardening has something to do with it, can't get into our smtpd if we're not running it. But mostly it kept things simple and reduced work. No extra daemons means no configuration of them and no update headaches. Even with that limited set, I had to go around and reconfigure sshd sometimes and ntp rather more than I'd have liked, and we used caveman automation, so stuff that needs super user is extra painful. On hardening, we didn't exclude things from FreeBSD's base, even though that would probably be a good idea. Some things are there for reasonable reasons, but had no relevance to us, and less installed stuff would be better, but tradeoffs. We did run stud (now known as hitch, which we needed when Erlang TLS was too slow) in a totally locked down jail though; statically configured binary, only executable in the jail, etc.
- tiffanyh 5y ago
  Thanks so much. Really appreciate the insight. Would you use FreeBSD for new messaging/web-service today or Linux? (I ask given you have first hand knowledge of how FreeBSD performed vs Linux after your organization made the migration)
  - toast0 5y ago
    Personally, I would use FreeBSD. But, to be honest, that's 99% because of familiarity and only 1% because I hope the performance is better. Some of the familiarity is just knowing how the system works and how to do routine tasks easily. Some of it is knowing what manual pages will give me the answers I need and being familiar with the writing so I can look it up. But also, I know a lot of important sysctls to adjust behaviors and where to look if there's something that needs adjusting that I think might have a knob. And, I'm fairly handy at digging through the kernel now that I've done it enough. The Linux kernel is organized differently, and I'd need to develop that skill/knowledge. Getting changes upstreamed is a challenge and I've had some success with FreeBSD, but Linux kernel would be new processes, and some things that are really core to a usable system (for example ifconfig/ip) are developed seperately, so have their own processes, so potentially more processes to learn. Mostly our software ran fine in Linux and the things that bothered me the most were mostly artifacts of how Facebook runs their servers. That said, some things in Linux bother me: why does ss exist as a faster netstat instead of just making netstat faster? I don't find the new init framework compelling on servers (I don't hotplug anything, and I don't think init should restart crashed server processes) and it actively hinders me on desktop/laptop, so it's frustrating that the ecosystem has embraced it (but I'm trying to avoid an offtopic rant, email in profile if you want a rant). We had a couple migration issues, but nothing serious. epoll and kqueue are similar, but not similar enough that we didn't need to tweak some things in Erlang to make it work better, but the chat machines in FB have much less ram because we had to select from FB's menu of server hardware and more nodes with less ram in each makes the most sense on their menu, so I can't say for sure if epoll scales as well as kqueue. I understand epoll takes more syscalls to do the same work (for network I/O anyway), so that's probably worse, but it likely doesn't make a big difference, but then there's also io_uring which I haven't evaluated and might be better than either epoll or kqueue. I think Linux's default memory overcommit behavior is problematic for a single process server, but it's hard to disable it because the ecosystem has formed around it. Tools allocate huge memory because 'it doesn't matter, it only really allocates when I write to it', and like, I understand how that's useful, but it's impossible for a program to handle out of memory when the trigger is a write to allocated space; alloc failures are hard to handle, but with careful design you can get a fairly consistent crash report most of the time.

toast0 5y ago

Probably Rick, he did a bunch of presentations. We ran a pretty vanilla FreeBSD. IIRC, we disabled locales, and had 5-10 patches depending on exactly when and what bottlenecks we were running into. We didn't go out of our way to turn off daemons, we just didn't start anything we didn't need. Out of the box, you get sshd, crond, getty, ntpd, syslogd, our system activity report, and that's really all you need to administer the box. Then on our Erlang machines, we'd run one big Erlang process. Unless something went seriously wrong, only Erlang ever had real work to do. I don't know that there was a performance benefit, you can run a lot of daemons that are waiting on sockets and don't use meaningful amounts of ram before noticing a performance hit. Hardening has something to do with it, can't get into our smtpd if we're not running it. But mostly it kept things simple and reduced work. No extra daemons means no configuration of them and no update headaches. Even with that limited set, I had to go around and reconfigure sshd sometimes and ntp rather more than I'd have liked, and we used caveman automation, so stuff that needs super user is extra painful. On hardening, we didn't exclude things from FreeBSD's base, even though that would probably be a good idea. Some things are there for reasonable reasons, but had no relevance to us, and less installed stuff would be better, but tradeoffs. We did run stud (now known as hitch, which we needed when Erlang TLS was too slow) in a totally locked down jail though; statically configured binary, only executable in the jail, etc.

tiffanyh 5y ago
Thanks so much. Really appreciate the insight. Would you use FreeBSD for new messaging/web-service today or Linux? (I ask given you have first hand knowledge of how FreeBSD performed vs Linux after your organization made the migration)
- toast0 5y ago
  Personally, I would use FreeBSD. But, to be honest, that's 99% because of familiarity and only 1% because I hope the performance is better. Some of the familiarity is just knowing how the system works and how to do routine tasks easily. Some of it is knowing what manual pages will give me the answers I need and being familiar with the writing so I can look it up. But also, I know a lot of important sysctls to adjust behaviors and where to look if there's something that needs adjusting that I think might have a knob. And, I'm fairly handy at digging through the kernel now that I've done it enough. The Linux kernel is organized differently, and I'd need to develop that skill/knowledge. Getting changes upstreamed is a challenge and I've had some success with FreeBSD, but Linux kernel would be new processes, and some things that are really core to a usable system (for example ifconfig/ip) are developed seperately, so have their own processes, so potentially more processes to learn. Mostly our software ran fine in Linux and the things that bothered me the most were mostly artifacts of how Facebook runs their servers. That said, some things in Linux bother me: why does ss exist as a faster netstat instead of just making netstat faster? I don't find the new init framework compelling on servers (I don't hotplug anything, and I don't think init should restart crashed server processes) and it actively hinders me on desktop/laptop, so it's frustrating that the ecosystem has embraced it (but I'm trying to avoid an offtopic rant, email in profile if you want a rant). We had a couple migration issues, but nothing serious. epoll and kqueue are similar, but not similar enough that we didn't need to tweak some things in Erlang to make it work better, but the chat machines in FB have much less ram because we had to select from FB's menu of server hardware and more nodes with less ram in each makes the most sense on their menu, so I can't say for sure if epoll scales as well as kqueue. I understand epoll takes more syscalls to do the same work (for network I/O anyway), so that's probably worse, but it likely doesn't make a big difference, but then there's also io_uring which I haven't evaluated and might be better than either epoll or kqueue. I think Linux's default memory overcommit behavior is problematic for a single process server, but it's hard to disable it because the ecosystem has formed around it. Tools allocate huge memory because 'it doesn't matter, it only really allocates when I write to it', and like, I understand how that's useful, but it's impossible for a program to handle out of memory when the trigger is a write to allocated space; alloc failures are hard to handle, but with careful design you can get a fairly consistent crash report most of the time.