5 ms·

Where did you see that WhatsApp is converting to Linux? At MeetBSD Rick Reed claimed they had full autonomy from Facebook on that kind of thing (http://www.sli

by kev009 11y ago

Where did you see that WhatsApp is converting to Linux? At MeetBSD Rick Reed claimed they had full autonomy from Facebook on that kind of thing (http://www.slideshare.net/iXsystems/rick-reed-600-m-unsuspecting-freebsd-users http://www.slideshare.net/iXsystems/rick-reed-600-m-unsuspec...), and FB was looking at them to learn how they did so much with so little engineers and servers. Netflix is doing 100Gbit HTTP and > 60Gbit HTTPS on a single socket Xeon E5 with FreeBSD. These servers are about 1/3 North American Internet by volume. In comparison to OCA team their AWS team sounds like a total tire fire to me in terms of engineer count and monthly spend. I run the OS team at Limelight Networks, one of the largest CDNs. We can peak beyond 10Tbit/s on our current network and have to deal with a more demanding/unknown workload compared to NF OCA. This is all courtesy of FreeBSD. My entire management chain agrees that the cost of entry is lower to influencing and driving development in FreeBSD vs other kernels, and we collaborate with the other well known players on projects that are mutually beneficial such as TCP and filesystems. Companies switching from FreeBSD to Linux usually are victim of the decision from bad VP-level management looking for a scapegoat rather than solving actual business and culture problems. Yahoo began transitioning to RHEL in 2005 and AFAIK are still at least two years from conceivably being off of FreeBSD. A 12 year _free_ OS transition is an embarrassing use of engineering effort and total management failure. There are a lot of companies satisfied with FreeBSD, especially in the appliance space and small business, but they are not vocal because it just works and is often rebranded.

keithpeter 11y ago
Most interesting, thanks for commenting. This phrase caught my eye... "My entire management chain agrees that the cost of entry is lower to influencing and driving development in FreeBSD vs other kernels [...]" Is that simply a reflection of the smaller size of the FreeBSD project or does it stem from the 'tight core' model where the kernel and base system are co-developed?
- kev009 11y ago
  Both.. Facebook advertised "write your own ticket" not so long ago for Linux kernel developers. You are guaranteed to be going up against Google if you want Linux kernel talent that has significant influence. Xref Dan Luu's http://danluu.com/startup-tradeoffs/ http://danluu.com/startup-tradeoffs/ and multiply by 2 to see how much that will cost you. This kind of thing even lures ardent FreeBSD people away, for instance Justin Gibbs, President and Founder of The FreeBSD Foundation currently works at Facebook.. there are many other BSD people stuck in the Borg too. Aside from that, the tight core and stable ABIs and KPIs are much nicer to base things like an appliance off of that may have a long development and longer support lifecycle on. I do some firmware hacking and analysis sometimes for fun, and Linux firmware are always abhorrent in terms of security, quality, license compliance. IBM's disk storage subsystems as one example. I really can't fathom why anyone would base a closed source product on Linux other than wanton incompetence, and I'm not throwing that with tongue in cheek nor lightheartedly.
  - kchoudhu 11y ago
    What are your thoughts on the 2.5 year support windows for "Extended Support" releases? (I have a view here, but I don't want to bias your response one way or another).
    - kev009 11y ago
      IMHO If you are doing a product, you should be doing your own RelEng and support. It takes about two engineers for good coverage.
      - kchoudhu 11y ago
        And your internal releng process should be handling security issues as they come up?
        kev009 11y ago
        That may start to get rough with MFCs as the branch petrifies, but yes it should fall within the same crosschecked two engineer budget for a reasonable time span. You should seek commit bits to maintain said branches if you rely on them, because you're probably not alone.
    - corv 11y ago
      FreeBSD is switching to a 5 year window.
      - kchoudhu 11y ago
        For point releases? Or for ABI compatibility? Edit: Went digging. Here's the link: https://lists.freebsd.org/pipermail/freebsd-announce/2015-February/001624.html https://lists.freebsd.org/pipermail/freebsd-announce/2015-Fe... It's for ABI compatibility. It certainly adds a certain amount of certainty to the planning process. Right now, I'm running off 11.0-CURRENT in preparation for a deployment or around August 2016, which gives me certainty of support until mid-2021.
akerro 11y ago
A bif off-topic, what do you think about this: http://lukewolf.blogspot.co.uk/2015/02/a-prediction-2020-year-of-pc-bsd-on.html http://lukewolf.blogspot.co.uk/2015/02/a-prediction-2020-yea...
- kev009 11y ago
  Honestly, meh. I was moderately early to the Linux game (1998) and bandwagon really didn't do proportional good. I use FreeBSD because I can understand and influence it.
ksec 11y ago
Thx, I cant edit my original post, but my intention / worry was if FreeBSD get enough cooperate usage that is being sustainable in the long term future. Looks like it is doing fine for now. Off-Topic: No idea why Limelight fell off my CDN radar long time ago, ( I got focused on EdgeCast instead ), since Limelight seems not want to list their price in /GB pricing ( Likely an Enterprise Only Strategy ), any chance for a different sub brand that offer something similar to CDN in pay per use cases?
- kev009 11y ago
  As far as I understand, we are more handshake deal based. A lot of our traffic is special events where a customer blows up the Internet for a day and needs a lot of help in mitigating problems from that.
VLM 11y ago
Based on your HN username, you're probably not the guy who did the presentation at BSDCan 2015 on the topic of "FreeBSD Operations at Limelight Networks, An Overview of Operating at Internet Scale". Regardless, that was a good and influential presentation, worth the time to google and watch. I think that specific presentation would answer a lot of OPs questions, probably faster and more accurately than a web discussion board. I got a lot of good ideas from that presentation about freebsd deployment at my workplace. Specifically I seem to recall something along the lines of, its 2016 so start using zabbix instead of nagios, which was pretty good advice, or at least its been interesting to experiment with so far. I'd extend the remarks from the presentation that much as its possible to write Perl in any language, its possible to do everything linux-style but with a freebsd kernel, whereas the real benefits grow when things start smelling more freebsd-ish and less linux-ish. For example the *bsds are much closer to the proven successful unix philosophy of architecture and design, whereas the linuxes in general have been slowly generally moving away from that successful architecture. I'd throw out a semi-dissenting opinion that freebsd use is much like whiteboard algo interview questions... even if it has nothing to do with the job or problem, the folks who can do it, are of a class above the folks who can't, so naturally the average freebsd admin being considerably more inherently skilled than the average linux admin results in freebsd deployments requiring 1/10th the number of sysadmin-hours etc etc. There are actual technical improvements and architectural benefits to freebsd, but the main contributor of the often reported 10x benefit to freebsd is the skill, experience, and raw horsepower of the generally superior users. If freebsd were as popular as linux, that would disappear and freebsd users would only be maybe 2x as productive as linux instead of the existing 10x figures. Naturally the guy writing a playbook for ansible will be much more productive than the guy trying to use the gnome GUI to do sysadmin-ish stuff, but there's nothing inherently bsd-ish or linux-ish about it. (edited to add, yes, i336_, that is exactly the video I recommend)
- i336_ 11y ago
  The landing page for that link: https://www.bsdcan.org/2015/schedule/events/534.en.html https://www.bsdcan.org/2015/schedule/events/534.en.html Video part 1: https://www.youtube.com/watch?v=4l2rlRjkGhk https://www.youtube.com/watch?v=4l2rlRjkGhk Video part 2: https://www.youtube.com/watch?v=K1-ZyiY5z48 https://www.youtube.com/watch?v=K1-ZyiY5z48 The above two videos cued up in a giant "BSDCan" playlist, as linked from the first page: https://www.youtube.com/watch?v=4l2rlRjkGhk&list=PLWW0CjV-TafY0NqFDvD4k31CtnX-CGn8f&index=15 https://www.youtube.com/watch?v=4l2rlRjkGhk&list=PLWW0CjV-Ta...
beagle3 11y ago
> Netflix is doing 100Gbit HTTP and > 60Gbit HTTPS on a single socket Xeon E5 with FreeBSD. These servers are about 1/3 North American Internet by volume. In comparison to OCA team their AWS team sounds like a total tire fire to me in terms of engineer count and monthly spend. Have any idea how many cores and what kind of network hardware they use push those impressive figures? I would guess there is some network offload involved, as this comparable to memory bandwidth of a single core of that Xeon if I'm not mistaken.
- emaste 11y ago
  There's a little bit of detail in this press release from Mellanox: http://ir.mellanox.com/releasedetail.cfm?ReleaseID=959350 http://ir.mellanox.com/releasedetail.cfm?ReleaseID=959350
- bsdimp 11y ago
  For those numbers, it was around 18 real cores with hyper threading enabled. Crypto is a great workload for hyper threads. These systems are currently being built up and aren't yet in production. These are with last year's CPUs. https://openconnect.netflix.com/en/hardware/ https://openconnect.netflix.com/en/hardware/ has the currently deployed system specs.

by kev009 11y ago

keithpeter 11y ago

Most interesting, thanks for commenting. This phrase caught my eye... "My entire management chain agrees that the cost of entry is lower to influencing and driving development in FreeBSD vs other kernels [...]" Is that simply a reflection of the smaller size of the FreeBSD project or does it stem from the 'tight core' model where the kernel and base system are co-developed?

kev009 11y ago
Both.. Facebook advertised "write your own ticket" not so long ago for Linux kernel developers. You are guaranteed to be going up against Google if you want Linux kernel talent that has significant influence. Xref Dan Luu's http://danluu.com/startup-tradeoffs/ http://danluu.com/startup-tradeoffs/ and multiply by 2 to see how much that will cost you. This kind of thing even lures ardent FreeBSD people away, for instance Justin Gibbs, President and Founder of The FreeBSD Foundation currently works at Facebook.. there are many other BSD people stuck in the Borg too. Aside from that, the tight core and stable ABIs and KPIs are much nicer to base things like an appliance off of that may have a long development and longer support lifecycle on. I do some firmware hacking and analysis sometimes for fun, and Linux firmware are always abhorrent in terms of security, quality, license compliance. IBM's disk storage subsystems as one example. I really can't fathom why anyone would base a closed source product on Linux other than wanton incompetence, and I'm not throwing that with tongue in cheek nor lightheartedly.
- kchoudhu 11y ago
  What are your thoughts on the 2.5 year support windows for "Extended Support" releases? (I have a view here, but I don't want to bias your response one way or another).
  - kev009 11y ago
    IMHO If you are doing a product, you should be doing your own RelEng and support. It takes about two engineers for good coverage.
    - kchoudhu 11y ago
      And your internal releng process should be handling security issues as they come up?
      - kev009 11y ago
        That may start to get rough with MFCs as the branch petrifies, but yes it should fall within the same crosschecked two engineer budget for a reasonable time span. You should seek commit bits to maintain said branches if you rely on them, because you're probably not alone.
  - corv 11y ago
    FreeBSD is switching to a 5 year window.
    - kchoudhu 11y ago
      For point releases? Or for ABI compatibility? Edit: Went digging. Here's the link: https://lists.freebsd.org/pipermail/freebsd-announce/2015-February/001624.html https://lists.freebsd.org/pipermail/freebsd-announce/2015-Fe... It's for ABI compatibility. It certainly adds a certain amount of certainty to the planning process. Right now, I'm running off 11.0-CURRENT in preparation for a deployment or around August 2016, which gives me certainty of support until mid-2021.

akerro 11y ago

A bif off-topic, what do you think about this: http://lukewolf.blogspot.co.uk/2015/02/a-prediction-2020-year-of-pc-bsd-on.html http://lukewolf.blogspot.co.uk/2015/02/a-prediction-2020-yea...

kev009 11y ago
Honestly, meh. I was moderately early to the Linux game (1998) and bandwagon really didn't do proportional good. I use FreeBSD because I can understand and influence it.

ksec 11y ago

Thx, I cant edit my original post, but my intention / worry was if FreeBSD get enough cooperate usage that is being sustainable in the long term future. Looks like it is doing fine for now. Off-Topic: No idea why Limelight fell off my CDN radar long time ago, ( I got focused on EdgeCast instead ), since Limelight seems not want to list their price in /GB pricing ( Likely an Enterprise Only Strategy ), any chance for a different sub brand that offer something similar to CDN in pay per use cases?

kev009 11y ago
As far as I understand, we are more handshake deal based. A lot of our traffic is special events where a customer blows up the Internet for a day and needs a lot of help in mitigating problems from that.

VLM 11y ago

Based on your HN username, you're probably not the guy who did the presentation at BSDCan 2015 on the topic of "FreeBSD Operations at Limelight Networks, An Overview of Operating at Internet Scale". Regardless, that was a good and influential presentation, worth the time to google and watch. I think that specific presentation would answer a lot of OPs questions, probably faster and more accurately than a web discussion board. I got a lot of good ideas from that presentation about freebsd deployment at my workplace. Specifically I seem to recall something along the lines of, its 2016 so start using zabbix instead of nagios, which was pretty good advice, or at least its been interesting to experiment with so far. I'd extend the remarks from the presentation that much as its possible to write Perl in any language, its possible to do everything linux-style but with a freebsd kernel, whereas the real benefits grow when things start smelling more freebsd-ish and less linux-ish. For example the *bsds are much closer to the proven successful unix philosophy of architecture and design, whereas the linuxes in general have been slowly generally moving away from that successful architecture. I'd throw out a semi-dissenting opinion that freebsd use is much like whiteboard algo interview questions... even if it has nothing to do with the job or problem, the folks who can do it, are of a class above the folks who can't, so naturally the average freebsd admin being considerably more inherently skilled than the average linux admin results in freebsd deployments requiring 1/10th the number of sysadmin-hours etc etc. There are actual technical improvements and architectural benefits to freebsd, but the main contributor of the often reported 10x benefit to freebsd is the skill, experience, and raw horsepower of the generally superior users. If freebsd were as popular as linux, that would disappear and freebsd users would only be maybe 2x as productive as linux instead of the existing 10x figures. Naturally the guy writing a playbook for ansible will be much more productive than the guy trying to use the gnome GUI to do sysadmin-ish stuff, but there's nothing inherently bsd-ish or linux-ish about it. (edited to add, yes, i336_, that is exactly the video I recommend)

i336_ 11y ago
The landing page for that link: https://www.bsdcan.org/2015/schedule/events/534.en.html https://www.bsdcan.org/2015/schedule/events/534.en.html Video part 1: https://www.youtube.com/watch?v=4l2rlRjkGhk https://www.youtube.com/watch?v=4l2rlRjkGhk Video part 2: https://www.youtube.com/watch?v=K1-ZyiY5z48 https://www.youtube.com/watch?v=K1-ZyiY5z48 The above two videos cued up in a giant "BSDCan" playlist, as linked from the first page: https://www.youtube.com/watch?v=4l2rlRjkGhk&list=PLWW0CjV-TafY0NqFDvD4k31CtnX-CGn8f&index=15 https://www.youtube.com/watch?v=4l2rlRjkGhk&list=PLWW0CjV-Ta...

beagle3 11y ago

> Netflix is doing 100Gbit HTTP and > 60Gbit HTTPS on a single socket Xeon E5 with FreeBSD. These servers are about 1/3 North American Internet by volume. In comparison to OCA team their AWS team sounds like a total tire fire to me in terms of engineer count and monthly spend. Have any idea how many cores and what kind of network hardware they use push those impressive figures? I would guess there is some network offload involved, as this comparable to memory bandwidth of a single core of that Xeon if I'm not mistaken.

emaste 11y ago
There's a little bit of detail in this press release from Mellanox: http://ir.mellanox.com/releasedetail.cfm?ReleaseID=959350 http://ir.mellanox.com/releasedetail.cfm?ReleaseID=959350
bsdimp 11y ago
For those numbers, it was around 18 real cores with hyper threading enabled. Crypto is a great workload for hyper threads. These systems are currently being built up and aren't yet in production. These are with last year's CPUs. https://openconnect.netflix.com/en/hardware/ https://openconnect.netflix.com/en/hardware/ has the currently deployed system specs.