4 ms·

We did this with FreeBSD several years ago at Netflix. We used to run heavily modified -stable kernels. Our tree was ahead of FreeBSD-current in a lot of ways

We did this with FreeBSD several years ago at Netflix. We used to run heavily modified -stable kernels. Our tree was ahead of FreeBSD-current in a lot of ways (TCP, ktls, VM) but way behind in other ways. Every year or two we'd rebase to a new -stable version, and that would be months of work. Often, there would be some bug introduced upstream many months prior that took weeks to track down because there was just so much to bisect through. Because our tree was based on -stable, pushing changes upstream often involved porting them to a substantially different codebase in -current, and being unable to test the ported code at scale in production. Since we've moved to tracking -current, we rebase every few weeks. We catch any newly introduced bugs from upstream almost immediately, since there are only a few weeks of changes, not a few years, to sort through. Its far easier to contribute code upstream. In fact, we do most development upstream and wait until the next rebase to pick our changes up (or cherry pick them if it is something urgent). All in all, life is much, much, much better on the bleeding edge.

hinkley 5y ago
Do you run all service on the new kernel or is there a roll out process where some services are canaries? I have one service that is low traffic that I use to test out new base images and core api changes. It’s easy to generate synthetic traffic to see how it behaves, as a lot of its traffic comes from the build process of more complex services.
- drewg123 5y ago
  All machines (can) run the same services, and we roll an entirely new self contained OS image for every update. That goes through multiple expanding levels of canaries prior to a fleet wide rollout. Bugs that escape the canaries are quite rare. One of the last ones I recall was a release that contained a firmware update for a NIC that had a super rare bug that would lock up the NIC, and we only started to see it when a sizable percent of the fleet was running that firmware.
capableweb 5y ago
> All in all, life is much, much, much better on the bleeding edge. Same feeling here, but on a personal level (meaning, using a desktop computer for professional work). Started using Ubuntu many years ago, ran updates maybe once every month. Every now and then something breaks, but always unsure of what/how/when, so digging through things that changed always took long time. Started using Arch Linux and few years ago and also started doing updates at least once per day. Now it's way easier to discover what particular change broke something, as the releases are way smaller. I attribute this more to how often I do the updates now VS before, and less to which OS you use. Same can be achieved with Debian Testing et al.
_-david-_ 5y ago
Any chance there is a presentation about this?

hinkley 5y ago

Do you run all service on the new kernel or is there a roll out process where some services are canaries? I have one service that is low traffic that I use to test out new base images and core api changes. It’s easy to generate synthetic traffic to see how it behaves, as a lot of its traffic comes from the build process of more complex services.

drewg123 5y ago
All machines (can) run the same services, and we roll an entirely new self contained OS image for every update. That goes through multiple expanding levels of canaries prior to a fleet wide rollout. Bugs that escape the canaries are quite rare. One of the last ones I recall was a release that contained a firmware update for a NIC that had a super rare bug that would lock up the NIC, and we only started to see it when a sizable percent of the fleet was running that firmware.

capableweb 5y ago

> All in all, life is much, much, much better on the bleeding edge. Same feeling here, but on a personal level (meaning, using a desktop computer for professional work). Started using Ubuntu many years ago, ran updates maybe once every month. Every now and then something breaks, but always unsure of what/how/when, so digging through things that changed always took long time. Started using Arch Linux and few years ago and also started doing updates at least once per day. Now it's way easier to discover what particular change broke something, as the releases are way smaller. I attribute this more to how often I do the updates now VS before, and less to which OS you use. Same can be achieved with Debian Testing et al.

_-david-_ 5y ago

Any chance there is a presentation about this?