3 ms·

Thanks so much. Really appreciate the insight. Would you use FreeBSD for new messaging/web-service today or Linux? (I ask given you have first hand knowledge

Thanks so much. Really appreciate the insight. Would you use FreeBSD for new messaging/web-service today or Linux? (I ask given you have first hand knowledge of how FreeBSD performed vs Linux after your organization made the migration)

toast0 5y ago
Personally, I would use FreeBSD. But, to be honest, that's 99% because of familiarity and only 1% because I hope the performance is better. Some of the familiarity is just knowing how the system works and how to do routine tasks easily. Some of it is knowing what manual pages will give me the answers I need and being familiar with the writing so I can look it up. But also, I know a lot of important sysctls to adjust behaviors and where to look if there's something that needs adjusting that I think might have a knob. And, I'm fairly handy at digging through the kernel now that I've done it enough. The Linux kernel is organized differently, and I'd need to develop that skill/knowledge. Getting changes upstreamed is a challenge and I've had some success with FreeBSD, but Linux kernel would be new processes, and some things that are really core to a usable system (for example ifconfig/ip) are developed seperately, so have their own processes, so potentially more processes to learn. Mostly our software ran fine in Linux and the things that bothered me the most were mostly artifacts of how Facebook runs their servers. That said, some things in Linux bother me: why does ss exist as a faster netstat instead of just making netstat faster? I don't find the new init framework compelling on servers (I don't hotplug anything, and I don't think init should restart crashed server processes) and it actively hinders me on desktop/laptop, so it's frustrating that the ecosystem has embraced it (but I'm trying to avoid an offtopic rant, email in profile if you want a rant). We had a couple migration issues, but nothing serious. epoll and kqueue are similar, but not similar enough that we didn't need to tweak some things in Erlang to make it work better, but the chat machines in FB have much less ram because we had to select from FB's menu of server hardware and more nodes with less ram in each makes the most sense on their menu, so I can't say for sure if epoll scales as well as kqueue. I understand epoll takes more syscalls to do the same work (for network I/O anyway), so that's probably worse, but it likely doesn't make a big difference, but then there's also io_uring which I haven't evaluated and might be better than either epoll or kqueue. I think Linux's default memory overcommit behavior is problematic for a single process server, but it's hard to disable it because the ecosystem has formed around it. Tools allocate huge memory because 'it doesn't matter, it only really allocates when I write to it', and like, I understand how that's useful, but it's impossible for a program to handle out of memory when the trigger is a write to allocated space; alloc failures are hard to handle, but with careful design you can get a fairly consistent crash report most of the time.
- tiffanyh 5y ago
  That's super interesting and insightful to hear. I really appreciate the time you spent detailing out your prespetive! I too love FreeBSD (and don't have much in-depth Linux experience). You hit on a topic I'm also interested in as well, which is io_uring. A week doesn't go by that it seems I bump into an article/post about it. Unforunately, I haven't found any solid comparison yet on kqueue vs io_uring (or at least - aren't written by someone in the Linux community who doesn't understand FreeBSD well ... which can be frustrating). Nonetheless, again - really appreciate your comment.
  - toast0 5y ago
    Well, thank you for asking nice questions and reading my long posts. ;) I must have missed some of the weekly io_uring posts, but it sure does look interesting. But I'm not working on anything performance driven at the moment, so not going to dig into it, certainly not enough to try to setup a good comparison for the three options. A tricky bit would be trying to get the test scenario to try to avoid the IP stack, so differences between IP handling on the two OSes isn't significant in the results. Comparing OS TCP performance is fun too, but we'd want to compare one thing at a time.

toast0 5y ago

Personally, I would use FreeBSD. But, to be honest, that's 99% because of familiarity and only 1% because I hope the performance is better. Some of the familiarity is just knowing how the system works and how to do routine tasks easily. Some of it is knowing what manual pages will give me the answers I need and being familiar with the writing so I can look it up. But also, I know a lot of important sysctls to adjust behaviors and where to look if there's something that needs adjusting that I think might have a knob. And, I'm fairly handy at digging through the kernel now that I've done it enough. The Linux kernel is organized differently, and I'd need to develop that skill/knowledge. Getting changes upstreamed is a challenge and I've had some success with FreeBSD, but Linux kernel would be new processes, and some things that are really core to a usable system (for example ifconfig/ip) are developed seperately, so have their own processes, so potentially more processes to learn. Mostly our software ran fine in Linux and the things that bothered me the most were mostly artifacts of how Facebook runs their servers. That said, some things in Linux bother me: why does ss exist as a faster netstat instead of just making netstat faster? I don't find the new init framework compelling on servers (I don't hotplug anything, and I don't think init should restart crashed server processes) and it actively hinders me on desktop/laptop, so it's frustrating that the ecosystem has embraced it (but I'm trying to avoid an offtopic rant, email in profile if you want a rant). We had a couple migration issues, but nothing serious. epoll and kqueue are similar, but not similar enough that we didn't need to tweak some things in Erlang to make it work better, but the chat machines in FB have much less ram because we had to select from FB's menu of server hardware and more nodes with less ram in each makes the most sense on their menu, so I can't say for sure if epoll scales as well as kqueue. I understand epoll takes more syscalls to do the same work (for network I/O anyway), so that's probably worse, but it likely doesn't make a big difference, but then there's also io_uring which I haven't evaluated and might be better than either epoll or kqueue. I think Linux's default memory overcommit behavior is problematic for a single process server, but it's hard to disable it because the ecosystem has formed around it. Tools allocate huge memory because 'it doesn't matter, it only really allocates when I write to it', and like, I understand how that's useful, but it's impossible for a program to handle out of memory when the trigger is a write to allocated space; alloc failures are hard to handle, but with careful design you can get a fairly consistent crash report most of the time.

tiffanyh 5y ago
That's super interesting and insightful to hear. I really appreciate the time you spent detailing out your prespetive! I too love FreeBSD (and don't have much in-depth Linux experience). You hit on a topic I'm also interested in as well, which is io_uring. A week doesn't go by that it seems I bump into an article/post about it. Unforunately, I haven't found any solid comparison yet on kqueue vs io_uring (or at least - aren't written by someone in the Linux community who doesn't understand FreeBSD well ... which can be frustrating). Nonetheless, again - really appreciate your comment.
- toast0 5y ago
  Well, thank you for asking nice questions and reading my long posts. ;) I must have missed some of the weekly io_uring posts, but it sure does look interesting. But I'm not working on anything performance driven at the moment, so not going to dig into it, certainly not enough to try to setup a good comparison for the three options. A tricky bit would be trying to get the test scenario to try to avoid the IP stack, so differences between IP handling on the two OSes isn't significant in the results. Comparing OS TCP performance is fun too, but we'd want to compare one thing at a time.