12 ms·

Bottlerocket: An operating system designed for hosting containers

280 points by ecliptik 7y ago

jahkeup 7y ago
Project is up on GitHub along with its other components! https://github.com/bottlerocket-os https://github.com/bottlerocket-os
ranman 7y ago
I’m here for the puns like updog: https://github.com/bottlerocket-os/bottlerocket/blob/develop/sources/updater/updog/README.md https://github.com/bottlerocket-os/bottlerocket/blob/develop...
- jhaynes 7y ago
  Glad you're enjoying them! We have a glossary just in case: https://github.com/bottlerocket-os/bottlerocket/blob/develop/GLOSSARY.md https://github.com/bottlerocket-os/bottlerocket/blob/develop... (my personal favorite is Laika, the first dog in space and our pre-init binary)
  - paulannesley 7y ago
    Honestly, reading the glossary gives me Urbit [1] vibes :( > bork: A setting generator called by sundog to generate the random seed for updog, determining where the host falls in the update order. [1]: https://urbit.org/docs/glossary/ https://urbit.org/docs/glossary/
    - alkonaut 7y ago
      Agree. It's cool so long as the number of names is small, and the names actually are a pun on the function and not just e.g. names of planets. If "updog" is what brings something "up" that's a good name. WiX (windows installer creation) has a multi-phase command line interface where the compiler/linker/.. has different names indicating the order they are applied: candle, light, smoke... Also a working system I guess.
      - steveklabnik 7y ago
        "bork" is dog-talk for "bark", and so something that randomly gets an updog going being named such makes sense too, it's just a slightly more obscure joke.
jrhill85 7y ago
The subject says "for hosting containers" but the README says "for AWS EKS Kubernetes" which sounds a little less general... How tied in to the AWS model is this? Are the places that would need to be expanded known? Also at least at a glance, this is a neat use of real-world Rust
- jhaynes 7y ago
  While our first variant is focused on Kubernetes and EKS, we have designed Bottlerocket in a way that new variants can be built that work with other orchestrators, or even without one (we have ECS support on our roadmap already). Also, we really enjoyed working in Rust for big chunks of this!
  - steveklabnik 7y ago
    > Also, we really enjoyed working in Rust for big chunks of this! Glad to hear it! Please reach out if you need anything :)
  - bogomipz 7y ago
    >"While our first variant is focused on Kubernetes and EKS..." So is the idea that people create a Bottlerocket AMI and use that as their EKS worker node images? Is that correct?
    - jhaynes 7y ago
      Correct! Or you can spin up one of the AMIs we've already built. You can find current AMIs via public SSM parameters: https://github.com/bottlerocket-os/bottlerocket/blob/develop/QUICKSTART.md#finding-an-ami https://github.com/bottlerocket-os/bottlerocket/blob/develop...
- steveklabnik 7y ago
  > This is a reflection of what we've learned building operating systems and services at Amazon. I don't know how actually tied in it is, but it's not totally surprising to me that it's built for AWS infrastructure. That being said: > To start, we're focusing on use of Bottlerocket as a host OS in AWS EKS Kubernetes clusters. We’re excited to get early feedback and to continue working on more use cases! > > Bottlerocket is architected such that different cloud environments and container orchestrators can be supported in the future. Very exciting!
rhatr 7y ago
This looks pretty sweet. It seems to be a continuation of the same trend of "just enough Linux to run containerd" that CoreOS started, linuxkit continued, then Project EVE expanded to cover virtualization. It is also interesting to note that every step on that journey seems to have picked the coolest runtime to implement it in (C/early Go, established Go, and now Rust)
whalesalad 7y ago
A link to the actual source code (90% Rust) and README: https://github.com/bottlerocket-os/bottlerocket https://github.com/bottlerocket-os/bottlerocket And here is a post from AWS with more technical details: https://aws.amazon.com/blogs/aws/bottlerocket-open-source-os-for-container-hosting/ https://aws.amazon.com/blogs/aws/bottlerocket-open-source-os...
- joana035 7y ago
  The blog post statements looks like FUD, and of course, are easy to echo. Is the telemetry enabled by default?
- whalesalad 7y ago
  > To improve security, there's no SSH server in a Bottlerocket image, and not even a shell. Consider my interest to be piqued.
- lysium 7y ago
  Thanks! I miss the part where it explains how it works. Is it all just containers?
  - geofft 7y ago
    It's buried a bit in the GitHub docs, but it looks like it runs two containerds. One is for Kubernetes. The other is for running administrative containers, including the API server for normal interaction with the host, a server that implements AWS's remote command protocol (enabled by default), and a container that runs sshd (disabled by default). Packages are built with RPM but RPM isn't used at runtime. Instead, the system is image-based and reboots for updates.
- mxie-ca 7y ago
  From the AWS Compliance Shared Responsibility Model: https://aws.amazon.com/compliance/shared-responsibility-model/ https://aws.amazon.com/compliance/shared-responsibility-mode... Operating system maintenance falls under the Customer responsibility side. With using this new OS, would this responsibility shift back to AWS?
lysium 7y ago
So, the main feature is that updates happen for all packages at once and not for each package individually. Sounds interesting, even for non cloud setups. How does that work? The explaining image does not explain that. How is that different from rolling back on file system level?
- tjkirch 7y ago
  The update system is image-based; when an update is downloaded, it's written out to an alternate set of partitions, and then it can flip over to those partitions with a reboot. That makes it easy to roll back with the same kind of single flip, too. It's different than filesystem-level rollbacks because it's all-or-nothing, so you don't have to worry about update failures after a few packages, and because all of the components in a given image are guaranteed to be tested together, whereas with package-based systems, your combination of packages may have never been used together by anyone else. In addition, for builders, it's easier to sign, distribute, and verify a single image.
  - yjftsjthsd-h 7y ago
    How does this compare to something like nix or Fedora Silverblue?
    - arianvanp 7y ago
      Nix has upside that you just need to flip a symlink to do the same . Downside is that you don't have thinks like dm-verity that can prove that your update wasn't tampered with. In nix the nix store is remounted over itself read-only, but nothing stops someone from ripping out the disk and flipping bits. This is not possible with these kind of 2-partition schemes if you have dm-verity set up
      - rkangel 7y ago
        Isn't Nix a Merkle-Tree system of hashes? Doesn't that allow you to easily verify everything that you're running?
eikenberry 7y ago
Looks like it is free software. Dual MIT/Apache license. https://github.com/bottlerocket-os/bottlerocket https://github.com/bottlerocket-os/bottlerocket
Hello71 7y ago
tl;dr buildroot rewritten in Rust
deleted 7y ago
[deleted]
jontro 7y ago
Their take on using cargo for packaging is quite interesting: https://github.com/bottlerocket-os/bottlerocket/tree/develop/packages https://github.com/bottlerocket-os/bottlerocket/tree/develop...
- joana035 7y ago
  They are using tradicional RPM for packaging. The cargo.toml workspaces relates more to make IMHO.
  - DevoidSimo 7y ago
    Correct, that toml file allows you to build everything from that level, and open editors on that folder with the language server functioning. RLS requires a cargo.toml file in the root of the workspace even if it just points at other directories
  - orf 7y ago
    Wouldn’t something like Bazel make more sense? Using cargo to track inter-dependencies seems a bit weird to be honest.
    - merb 7y ago
      well it's interesting since they only need to learn the rust toolchain, not something else.
rhatr 7y ago
Started to dig into the implementation. First impressions so far: loving all the Rust harness - really nicely done and way better than buildroot/yocto/etc for creating tight, single-purpose linux images. Speaking of tight, here comes the bad news: really NOT loving all the over-engineered upstream components like D-Bus and systemd that seem to be there by default. In that sense #linuxkit with its Alpine base and strong attention to how bit the image is still comes way on top. One more thing on the good side: the TUF implementation in Rust seems really interesting. I'll be digging some more and may actually steal it for linuxkit (and by extension Project EVE) Fun fact: a lot of the patches you will find in more system level packages like grub seems to trace their lineage to CoreOS (and potentially Project EVE) but I haven't seen acknowledgments anywhere. This is of course all fine from licensing perspective -- but I still would be curious to know whether it is indeed where it was taken from.
- mwcampbell 7y ago
  > way better than buildroot/yocto/etc for creating tight, single-purpose linux images Can you say more about the advantages over Buildroot and Yocto?
  - rhatr 7y ago
    I'll take Buildroot, and Bitbake first (Yocto itself is slight different we'll get to that later). Both Buildroot, Bitbake share the ultimate goal with all these news systems like linuxkit and bottle rocket. They all aim at producing single-image, pruprose built linux distros based on the very same usual suspects of upstream components. So the question really two-fold: out-of-the box availability of said upstream components (this is where we should stop talking about Bitbake and talk about Yocto -- Buildroot kind of commingles the two) AND the usability of the build harness itself. So on #1, if you look very casually at Buildroot and Yocto -- they will clearly come on top over this next generation of systems. It appears they have WAY more upstream components already available for you to chose from. Compared to them the list here looks almost laughable https://github.com/bottlerocket-os/bottlerocket/tree/develop/packages https://github.com/bottlerocket-os/bottlerocket/tree/develop... and https://github.com/linuxkit/linuxkit/tree/master/pkg https://github.com/linuxkit/linuxkit/tree/master/pkg The problem though is combinatoric explosion of how you can compose all these upstream components. The canonical example here is the choice of your init system. You pick one -- and your choice in everything else gets severely restricted. So to some extent that apparent embarrassment of riches that Buildroot and Yocto offer is misleading. These next generation systems, on the other hand, don't pretend that you can build a host OS in any shape or form you want (hence very few base packages) but rather that you build "just enough of Linux to run containerd" -- the rest of what you would typically put into your baseOS goes into various containers. This is a very different approach to constructing the bootable system, but subtly so -- which I don't think a lot of people on either side of this debate appreciate. I honestly think that what makes Yocto and Buildroot difficult is that they want to be all things to all people and they want it at the level of baseOS -- complexity-wise, this is a wrong approach these days. That scores one point for these next generation systems in my book. The question #2 is not even a comparison. In Buildroot most of the integration/package logic is implemented in Makefiles and usability-wise (if you're trying to actively change the system or add a new package) it falls apart pretty quickly (it is still great if you're just using what's already there btw). In Bitbake -- the codebases is REALLY complex Python which suffers from the same issue. Contrast it with Linuxkit/Project EVE where all that logic is golang and bottle rocket which uses Rust and ask yourself whether you would rather debug a complex issue with a dozen of Makefiles all full of non-trivial recipes or look over go/rust codebase (yes, I know all these things are turing complete and thus equivalent -- but life is to short to debug Makefile). If you don't quite believe me, there's been a number of studies about using Buildroot and Yocto for building containers. Pretty much all of them came back with the same conclusion -- the usability aspect of extending them makes it a non-starter. Here's the one from the last Kubecon that VMWare guys did: https://blogs.vmware.com/opensource/2020/02/27/distribution-tools-container-creation/ https://blogs.vmware.com/opensource/2020/02/27/distribution-...
peterwwillis 7y ago
So it's AWS's version of CoreOS? I wish somebody'd take some VC or R&D money and build distributed computing features into the kernel itself, so we could quit wasting our collective engineering talent, time, money and energy on distributed applications that run on non-distributed-operating-systems. It's like nobody wants to work on creating a round wheel, so instead we're spending all our time building custom roads for square wheels.
- johnmarcus 7y ago
  Erlang is probably the closest thing to this, but unfortunately everything it runs also needs to be written in erlang. If erlang could run containers as processes it would take over the world, all of the features from kubernetes has been baked in since the late 90's.
  - peterwwillis 7y ago
    The closest things really are distributed operating systems from the 70's and 80's, and cluster operating systems from the 90's, and 00's. Unfortunately, all of them were either research projects, proprietary products, or patches that never made it into the mainline kernel. Nobody has since tried to get the functionality into mainline, so people keep hacking together these non-standard pseudo-operating-systems and jumbles of disparate applications. You could eliminate 80% of the need for K8s by adding OS primitives to connect and operate namespaces and control groups between nodes, as well as native i/o (block, file, "N-way pipes") between nodes. Once that was done, systemd (or something like it) could manage services across an entire cluster. Applications could communicate between arbitrary nodes without any added functionality. Virtually all of the complexity would be in the kernel and systemd, so apps could be simpler and we wouldn't need 100 layers of userspace junk just to keep an app running on 3 nodes.
islanderfun 7y ago
Fascinating project! Anyone know of possible overlap with Firecracker? Really digging these Rust projects.
- jhaynes 7y ago
  No overlap per se, but we are looking at how to integrate Firecracker as a potential target for "container" launches: https://github.com/orgs/bottlerocket-os/projects/1#card-33865383 https://github.com/orgs/bottlerocket-os/projects/1#card-3386...
throw2345 7y ago
why would you create yet another "operating system" a.k.a linux that is specifically built to run container and k8s workloads? another coreos? isn't the world's software insecure enough without the constant barrage of clueless container folks making it more insecure?
deleted 7y ago
[deleted]
bbgm 7y ago
I am from AWS. Could we please change the title to say “Bottlerocket from AWS“. Like Firecracker [1] it’s explicitly not AWS branded. 1. https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker
- jsolson 7y ago
  You're adorable :)
  - bbgm 7y ago
    Always. I just wanted to do it before msw :)
    - jsolson 7y ago
      Honestly this reply was worth the karma loss. Hi Matt!
      - _msw_ 7y ago
        wave :-) We know this is how you show you care. I don't think there was any reason to downvote or flag. But not everyone on the Orange Site knows one another, so I can see how it could be misinterpreted.
- wikibob 7y ago
  You’ll have better luck emailing: mailto:hn@ycombinator.com
- ignoramous 7y ago
  Speaking of branding, I've some questions: 1. When are services branded as AWS (AWS Fargate) vs Amazon (Amazon DynamoDB)? 2. Is BottleRocket a nod to SkyRocket [0] or a movie of the same name? 3. Why is it called Fargate [1]? [0] https://en.wikipedia.org/wiki/Skyrocket https://en.wikipedia.org/wiki/Skyrocket [1] https://youtube.com/watch?v=ye3-gUwu9tI&t=44m28s https://youtube.com/watch?v=ye3-gUwu9tI&t=44m28s
ajxs 7y ago
This is somewhat false advertising. This is not an operating system in the sense of being a new kernel. It looks like it's a set of build tools for building a Linux distribution.
- tasogare 7y ago
  An operating system is not defined only by its kernel. It's kernel + APIs + user land. If one of those components is changed radically, it's indeed a new operating system. That's why Ubuntu and Debian are distinct, even if both are based on Linux.
  - phist_mcgee 7y ago
    Stallman is that you?
  - ajxs 7y ago
    You're correct. Debian and Ubuntu are technically different operating-systems. Due to the similarity of 'operating-systems' built around the Linux kernel they're typically referred to as 'Linux distributions'. In the context of software development, if you tell someone you're developing a new operating-system you're probably going to conjure up images of writing a new kernel. If you tell people you're developing a new Linux distro, this is closer to what they'll imagine.
    - akvadrako 7y ago
      It’s not that simple. I wouldn’t use the word distribution for GUIX, NixOS, Android and several OSes that use the Linux kernel.
      - ajxs 7y ago
        This is true. Android is a good example of where the line between a Linux distribution and an outright OS blurs. I'm not convinced that this really affects my overall point all that much though.
    - 7y ago
nstart 7y ago
Opening line from their announcement blog post: >It is safe to say that our industry has decided that containers are now the chosen way to package and scale applications. Curious how the HN community feels about that statement. Not so much about the truth of the statement but about the fact that containers are becoming the de facto method of packaging applications.
- emptysea 7y ago
  For application code the ecosystem around containers is pretty handy. For load balancers, databases, and other stateful stuff I still run the binaries on VMs.
- deleted 7y ago
  [deleted]
- unlinked_dll 7y ago
  You need to prefix "applications" with "web" or "cloud."
  - zokier 7y ago
    Flatpaks and snaps, while maybe not as popular, are also containers
    - unlinked_dll 7y ago
      They're also pretty terrible ways to distribute a native application and can be crippled due to their containerization.
    - AnIdiotOnTheNet 7y ago
      Unfortunately, they're over-complicated as a means of application packaging and distribution, at least in my opinion. Now AppImage, that's pretty great. It doesn't really bring any of the security benefits of containerization, but that can be tacked on separately.
  - akvadrako 7y ago
sciurus 7y ago
Is this intended to become AWS's version of GCP's Container-Optimized OS? https://cloud.google.com/container-optimized-os https://cloud.google.com/container-optimized-os
zapita 7y ago
How does this compare to Linuxkit? At first glance it seems almost identical but I may be missing something. https://www.github.com/linuxkit/linuxkit https://www.github.com/linuxkit/linuxkit
- lifty 7y ago
  Linuxkit allows you to build your own appliance like OS, while Bottlerocket is more of an end user project. A project that is more similar to Bottlerocket is https://www.talos.dev https://www.talos.dev or https://www.projectatomic.io https://www.projectatomic.io
  - zapita 7y ago
    I see. Thank you.
dpeterson 7y ago
Question for any Amazon folks here that may know. Is this akin to something like Atomic or CoreOS that is used within Openshift as the Master or Worker node OS or is this more like the UBI (Universal Base Image) that can be used as the base image of a container via "FROM" within a dockerfile?
- antonvs 7y ago
  You don't need Amazon folks, just read the material. This is not a container base image, it's a container host OS. It is somewhat similar to Atomic or CoreOS, but in some ways it seems to be a bit more of a radical redesign than those.
dirtydroog 7y ago
Remove SSH? Over my dead body! What is container specific about all this? It just seems to be minimal images?
- orf 7y ago
  If you need to SSH into your cattle you’re either not in position to benefit from something like Bottlerocket or you’re doing things wrong.
bogomipz 7y ago
Does anyone if or where Bottlerocket is being used at AWS? Is it used to run their Fargate/ECS/EKS service?
mwcampbell 7y ago
I'm curious about why Bottlerocket is building base packages like glibc, bash, util-linux, etc. from source, rather than just pulling binary RPMs from CentOS or Amazon Linux.
- tjkirch 7y ago
  Amazon Linux and CentOS are general-purpose distributions and need more features built into the base packages than we do for Bottlerocket. We’re able to simplify the spec files and produce smaller RPMs with only the content and dependencies necessary for our more narrow goals.
jpeeler 7y ago
It looks like this supports automated updates within a specific time window, but it's not clear to me how the "waves" are defined. (Note that this is something that is currently lacking in Fedora CoreOS: https://github.com/coreos/zincati/issues/34. https://github.com/coreos/zincati/issues/34.) I do wonder if the dual partition approach was deemed more stable than using OSTree or why the latter wasn't used.
eliaspro 7y ago
Any insights on the design decision to go with Wicked instead of systemd-networkd which is already provided by systemd and better integrated with the remaining systemd tooling/components/conventions?

280 points by ecliptik 7y ago

Project is up on GitHub along with its other components! https://github.com/bottlerocket-os https://github.com/bottlerocket-os

ranman 7y ago

I’m here for the puns like updog: https://github.com/bottlerocket-os/bottlerocket/blob/develop/sources/updater/updog/README.md https://github.com/bottlerocket-os/bottlerocket/blob/develop...

jhaynes 7y ago
Glad you're enjoying them! We have a glossary just in case: https://github.com/bottlerocket-os/bottlerocket/blob/develop/GLOSSARY.md https://github.com/bottlerocket-os/bottlerocket/blob/develop... (my personal favorite is Laika, the first dog in space and our pre-init binary)
- paulannesley 7y ago
  Honestly, reading the glossary gives me Urbit [1] vibes :( > bork: A setting generator called by sundog to generate the random seed for updog, determining where the host falls in the update order. [1]: https://urbit.org/docs/glossary/ https://urbit.org/docs/glossary/
  - alkonaut 7y ago
    Agree. It's cool so long as the number of names is small, and the names actually are a pun on the function and not just e.g. names of planets. If "updog" is what brings something "up" that's a good name. WiX (windows installer creation) has a multi-phase command line interface where the compiler/linker/.. has different names indicating the order they are applied: candle, light, smoke... Also a working system I guess.
    - steveklabnik 7y ago
      "bork" is dog-talk for "bark", and so something that randomly gets an updog going being named such makes sense too, it's just a slightly more obscure joke.

jrhill85 7y ago

The subject says "for hosting containers" but the README says "for AWS EKS Kubernetes" which sounds a little less general... How tied in to the AWS model is this? Are the places that would need to be expanded known? Also at least at a glance, this is a neat use of real-world Rust

jhaynes 7y ago
While our first variant is focused on Kubernetes and EKS, we have designed Bottlerocket in a way that new variants can be built that work with other orchestrators, or even without one (we have ECS support on our roadmap already). Also, we really enjoyed working in Rust for big chunks of this!
- steveklabnik 7y ago
  > Also, we really enjoyed working in Rust for big chunks of this! Glad to hear it! Please reach out if you need anything :)
- bogomipz 7y ago
  >"While our first variant is focused on Kubernetes and EKS..." So is the idea that people create a Bottlerocket AMI and use that as their EKS worker node images? Is that correct?
  - jhaynes 7y ago
    Correct! Or you can spin up one of the AMIs we've already built. You can find current AMIs via public SSM parameters: https://github.com/bottlerocket-os/bottlerocket/blob/develop/QUICKSTART.md#finding-an-ami https://github.com/bottlerocket-os/bottlerocket/blob/develop...
steveklabnik 7y ago
> This is a reflection of what we've learned building operating systems and services at Amazon. I don't know how actually tied in it is, but it's not totally surprising to me that it's built for AWS infrastructure. That being said: > To start, we're focusing on use of Bottlerocket as a host OS in AWS EKS Kubernetes clusters. We’re excited to get early feedback and to continue working on more use cases! > > Bottlerocket is architected such that different cloud environments and container orchestrators can be supported in the future. Very exciting!

rhatr 7y ago

This looks pretty sweet. It seems to be a continuation of the same trend of "just enough Linux to run containerd" that CoreOS started, linuxkit continued, then Project EVE expanded to cover virtualization. It is also interesting to note that every step on that journey seems to have picked the coolest runtime to implement it in (C/early Go, established Go, and now Rust)

whalesalad 7y ago

A link to the actual source code (90% Rust) and README: https://github.com/bottlerocket-os/bottlerocket https://github.com/bottlerocket-os/bottlerocket And here is a post from AWS with more technical details: https://aws.amazon.com/blogs/aws/bottlerocket-open-source-os-for-container-hosting/ https://aws.amazon.com/blogs/aws/bottlerocket-open-source-os...

joana035 7y ago
The blog post statements looks like FUD, and of course, are easy to echo. Is the telemetry enabled by default?
whalesalad 7y ago
> To improve security, there's no SSH server in a Bottlerocket image, and not even a shell. Consider my interest to be piqued.
lysium 7y ago
Thanks! I miss the part where it explains how it works. Is it all just containers?
- geofft 7y ago
  It's buried a bit in the GitHub docs, but it looks like it runs two containerds. One is for Kubernetes. The other is for running administrative containers, including the API server for normal interaction with the host, a server that implements AWS's remote command protocol (enabled by default), and a container that runs sshd (disabled by default). Packages are built with RPM but RPM isn't used at runtime. Instead, the system is image-based and reboots for updates.
mxie-ca 7y ago
From the AWS Compliance Shared Responsibility Model: https://aws.amazon.com/compliance/shared-responsibility-model/ https://aws.amazon.com/compliance/shared-responsibility-mode... Operating system maintenance falls under the Customer responsibility side. With using this new OS, would this responsibility shift back to AWS?

lysium 7y ago

So, the main feature is that updates happen for all packages at once and not for each package individually. Sounds interesting, even for non cloud setups. How does that work? The explaining image does not explain that. How is that different from rolling back on file system level?

tjkirch 7y ago
The update system is image-based; when an update is downloaded, it's written out to an alternate set of partitions, and then it can flip over to those partitions with a reboot. That makes it easy to roll back with the same kind of single flip, too. It's different than filesystem-level rollbacks because it's all-or-nothing, so you don't have to worry about update failures after a few packages, and because all of the components in a given image are guaranteed to be tested together, whereas with package-based systems, your combination of packages may have never been used together by anyone else. In addition, for builders, it's easier to sign, distribute, and verify a single image.
- yjftsjthsd-h 7y ago
  How does this compare to something like nix or Fedora Silverblue?
  - arianvanp 7y ago
    Nix has upside that you just need to flip a symlink to do the same . Downside is that you don't have thinks like dm-verity that can prove that your update wasn't tampered with. In nix the nix store is remounted over itself read-only, but nothing stops someone from ripping out the disk and flipping bits. This is not possible with these kind of 2-partition schemes if you have dm-verity set up
    - rkangel 7y ago
      Isn't Nix a Merkle-Tree system of hashes? Doesn't that allow you to easily verify everything that you're running?

eikenberry 7y ago

Looks like it is free software. Dual MIT/Apache license. https://github.com/bottlerocket-os/bottlerocket https://github.com/bottlerocket-os/bottlerocket

Hello71 7y ago

tl;dr buildroot rewritten in Rust

deleted 7y ago

[deleted]

jontro 7y ago

Their take on using cargo for packaging is quite interesting: https://github.com/bottlerocket-os/bottlerocket/tree/develop/packages https://github.com/bottlerocket-os/bottlerocket/tree/develop...

joana035 7y ago
They are using tradicional RPM for packaging. The cargo.toml workspaces relates more to make IMHO.
- DevoidSimo 7y ago
  Correct, that toml file allows you to build everything from that level, and open editors on that folder with the language server functioning. RLS requires a cargo.toml file in the root of the workspace even if it just points at other directories
- orf 7y ago
  Wouldn’t something like Bazel make more sense? Using cargo to track inter-dependencies seems a bit weird to be honest.
  - merb 7y ago
    well it's interesting since they only need to learn the rust toolchain, not something else.

rhatr 7y ago

Started to dig into the implementation. First impressions so far: loving all the Rust harness - really nicely done and way better than buildroot/yocto/etc for creating tight, single-purpose linux images. Speaking of tight, here comes the bad news: really NOT loving all the over-engineered upstream components like D-Bus and systemd that seem to be there by default. In that sense #linuxkit with its Alpine base and strong attention to how bit the image is still comes way on top. One more thing on the good side: the TUF implementation in Rust seems really interesting. I'll be digging some more and may actually steal it for linuxkit (and by extension Project EVE) Fun fact: a lot of the patches you will find in more system level packages like grub seems to trace their lineage to CoreOS (and potentially Project EVE) but I haven't seen acknowledgments anywhere. This is of course all fine from licensing perspective -- but I still would be curious to know whether it is indeed where it was taken from.

mwcampbell 7y ago
> way better than buildroot/yocto/etc for creating tight, single-purpose linux images Can you say more about the advantages over Buildroot and Yocto?
- rhatr 7y ago
  I'll take Buildroot, and Bitbake first (Yocto itself is slight different we'll get to that later). Both Buildroot, Bitbake share the ultimate goal with all these news systems like linuxkit and bottle rocket. They all aim at producing single-image, pruprose built linux distros based on the very same usual suspects of upstream components. So the question really two-fold: out-of-the box availability of said upstream components (this is where we should stop talking about Bitbake and talk about Yocto -- Buildroot kind of commingles the two) AND the usability of the build harness itself. So on #1, if you look very casually at Buildroot and Yocto -- they will clearly come on top over this next generation of systems. It appears they have WAY more upstream components already available for you to chose from. Compared to them the list here looks almost laughable https://github.com/bottlerocket-os/bottlerocket/tree/develop/packages https://github.com/bottlerocket-os/bottlerocket/tree/develop... and https://github.com/linuxkit/linuxkit/tree/master/pkg https://github.com/linuxkit/linuxkit/tree/master/pkg The problem though is combinatoric explosion of how you can compose all these upstream components. The canonical example here is the choice of your init system. You pick one -- and your choice in everything else gets severely restricted. So to some extent that apparent embarrassment of riches that Buildroot and Yocto offer is misleading. These next generation systems, on the other hand, don't pretend that you can build a host OS in any shape or form you want (hence very few base packages) but rather that you build "just enough of Linux to run containerd" -- the rest of what you would typically put into your baseOS goes into various containers. This is a very different approach to constructing the bootable system, but subtly so -- which I don't think a lot of people on either side of this debate appreciate. I honestly think that what makes Yocto and Buildroot difficult is that they want to be all things to all people and they want it at the level of baseOS -- complexity-wise, this is a wrong approach these days. That scores one point for these next generation systems in my book. The question #2 is not even a comparison. In Buildroot most of the integration/package logic is implemented in Makefiles and usability-wise (if you're trying to actively change the system or add a new package) it falls apart pretty quickly (it is still great if you're just using what's already there btw). In Bitbake -- the codebases is REALLY complex Python which suffers from the same issue. Contrast it with Linuxkit/Project EVE where all that logic is golang and bottle rocket which uses Rust and ask yourself whether you would rather debug a complex issue with a dozen of Makefiles all full of non-trivial recipes or look over go/rust codebase (yes, I know all these things are turing complete and thus equivalent -- but life is to short to debug Makefile). If you don't quite believe me, there's been a number of studies about using Buildroot and Yocto for building containers. Pretty much all of them came back with the same conclusion -- the usability aspect of extending them makes it a non-starter. Here's the one from the last Kubecon that VMWare guys did: https://blogs.vmware.com/opensource/2020/02/27/distribution-tools-container-creation/ https://blogs.vmware.com/opensource/2020/02/27/distribution-...

peterwwillis 7y ago

So it's AWS's version of CoreOS? I wish somebody'd take some VC or R&D money and build distributed computing features into the kernel itself, so we could quit wasting our collective engineering talent, time, money and energy on distributed applications that run on non-distributed-operating-systems. It's like nobody wants to work on creating a round wheel, so instead we're spending all our time building custom roads for square wheels.

johnmarcus 7y ago
Erlang is probably the closest thing to this, but unfortunately everything it runs also needs to be written in erlang. If erlang could run containers as processes it would take over the world, all of the features from kubernetes has been baked in since the late 90's.
- peterwwillis 7y ago
  The closest things really are distributed operating systems from the 70's and 80's, and cluster operating systems from the 90's, and 00's. Unfortunately, all of them were either research projects, proprietary products, or patches that never made it into the mainline kernel. Nobody has since tried to get the functionality into mainline, so people keep hacking together these non-standard pseudo-operating-systems and jumbles of disparate applications. You could eliminate 80% of the need for K8s by adding OS primitives to connect and operate namespaces and control groups between nodes, as well as native i/o (block, file, "N-way pipes") between nodes. Once that was done, systemd (or something like it) could manage services across an entire cluster. Applications could communicate between arbitrary nodes without any added functionality. Virtually all of the complexity would be in the kernel and systemd, so apps could be simpler and we wouldn't need 100 layers of userspace junk just to keep an app running on 3 nodes.

islanderfun 7y ago

Fascinating project! Anyone know of possible overlap with Firecracker? Really digging these Rust projects.

jhaynes 7y ago
No overlap per se, but we are looking at how to integrate Firecracker as a potential target for "container" launches: https://github.com/orgs/bottlerocket-os/projects/1#card-33865383 https://github.com/orgs/bottlerocket-os/projects/1#card-3386...

throw2345 7y ago

why would you create yet another "operating system" a.k.a linux that is specifically built to run container and k8s workloads? another coreos? isn't the world's software insecure enough without the constant barrage of clueless container folks making it more insecure?

deleted 7y ago

[deleted]

bbgm 7y ago

I am from AWS. Could we please change the title to say “Bottlerocket from AWS“. Like Firecracker [1] it’s explicitly not AWS branded. 1. https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker

jsolson 7y ago
You're adorable :)
- bbgm 7y ago
  Always. I just wanted to do it before msw :)
  - jsolson 7y ago
    Honestly this reply was worth the karma loss. Hi Matt!
    - _msw_ 7y ago
      wave :-) We know this is how you show you care. I don't think there was any reason to downvote or flag. But not everyone on the Orange Site knows one another, so I can see how it could be misinterpreted.
wikibob 7y ago
You’ll have better luck emailing: mailto:hn@ycombinator.com
ignoramous 7y ago
Speaking of branding, I've some questions: 1. When are services branded as AWS (AWS Fargate) vs Amazon (Amazon DynamoDB)? 2. Is BottleRocket a nod to SkyRocket [0] or a movie of the same name? 3. Why is it called Fargate [1]? [0] https://en.wikipedia.org/wiki/Skyrocket https://en.wikipedia.org/wiki/Skyrocket [1] https://youtube.com/watch?v=ye3-gUwu9tI&t=44m28s https://youtube.com/watch?v=ye3-gUwu9tI&t=44m28s

ajxs 7y ago

This is somewhat false advertising. This is not an operating system in the sense of being a new kernel. It looks like it's a set of build tools for building a Linux distribution.

tasogare 7y ago
An operating system is not defined only by its kernel. It's kernel + APIs + user land. If one of those components is changed radically, it's indeed a new operating system. That's why Ubuntu and Debian are distinct, even if both are based on Linux.
- phist_mcgee 7y ago
  Stallman is that you?
- ajxs 7y ago
  You're correct. Debian and Ubuntu are technically different operating-systems. Due to the similarity of 'operating-systems' built around the Linux kernel they're typically referred to as 'Linux distributions'. In the context of software development, if you tell someone you're developing a new operating-system you're probably going to conjure up images of writing a new kernel. If you tell people you're developing a new Linux distro, this is closer to what they'll imagine.
  - akvadrako 7y ago
    It’s not that simple. I wouldn’t use the word distribution for GUIX, NixOS, Android and several OSes that use the Linux kernel.
    - ajxs 7y ago
      This is true. Android is a good example of where the line between a Linux distribution and an outright OS blurs. I'm not convinced that this really affects my overall point all that much though.
  - 7y ago

nstart 7y ago

Opening line from their announcement blog post: >It is safe to say that our industry has decided that containers are now the chosen way to package and scale applications. Curious how the HN community feels about that statement. Not so much about the truth of the statement but about the fact that containers are becoming the de facto method of packaging applications.

emptysea 7y ago
For application code the ecosystem around containers is pretty handy. For load balancers, databases, and other stateful stuff I still run the binaries on VMs.
deleted 7y ago
[deleted]
unlinked_dll 7y ago
You need to prefix "applications" with "web" or "cloud."
- zokier 7y ago
  Flatpaks and snaps, while maybe not as popular, are also containers
  - unlinked_dll 7y ago
    They're also pretty terrible ways to distribute a native application and can be crippled due to their containerization.
  - AnIdiotOnTheNet 7y ago
    Unfortunately, they're over-complicated as a means of application packaging and distribution, at least in my opinion. Now AppImage, that's pretty great. It doesn't really bring any of the security benefits of containerization, but that can be tacked on separately.
- akvadrako 7y ago

sciurus 7y ago

Is this intended to become AWS's version of GCP's Container-Optimized OS? https://cloud.google.com/container-optimized-os https://cloud.google.com/container-optimized-os

zapita 7y ago

How does this compare to Linuxkit? At first glance it seems almost identical but I may be missing something. https://www.github.com/linuxkit/linuxkit https://www.github.com/linuxkit/linuxkit

lifty 7y ago
Linuxkit allows you to build your own appliance like OS, while Bottlerocket is more of an end user project. A project that is more similar to Bottlerocket is https://www.talos.dev https://www.talos.dev or https://www.projectatomic.io https://www.projectatomic.io
- zapita 7y ago
  I see. Thank you.

dpeterson 7y ago

Question for any Amazon folks here that may know. Is this akin to something like Atomic or CoreOS that is used within Openshift as the Master or Worker node OS or is this more like the UBI (Universal Base Image) that can be used as the base image of a container via "FROM" within a dockerfile?

antonvs 7y ago
You don't need Amazon folks, just read the material. This is not a container base image, it's a container host OS. It is somewhat similar to Atomic or CoreOS, but in some ways it seems to be a bit more of a radical redesign than those.

dirtydroog 7y ago

Remove SSH? Over my dead body! What is container specific about all this? It just seems to be minimal images?

orf 7y ago
If you need to SSH into your cattle you’re either not in position to benefit from something like Bottlerocket or you’re doing things wrong.

bogomipz 7y ago

Does anyone if or where Bottlerocket is being used at AWS? Is it used to run their Fargate/ECS/EKS service?

mwcampbell 7y ago

I'm curious about why Bottlerocket is building base packages like glibc, bash, util-linux, etc. from source, rather than just pulling binary RPMs from CentOS or Amazon Linux.

tjkirch 7y ago
Amazon Linux and CentOS are general-purpose distributions and need more features built into the base packages than we do for Bottlerocket. We’re able to simplify the spec files and produce smaller RPMs with only the content and dependencies necessary for our more narrow goals.

jpeeler 7y ago

It looks like this supports automated updates within a specific time window, but it's not clear to me how the "waves" are defined. (Note that this is something that is currently lacking in Fedora CoreOS: https://github.com/coreos/zincati/issues/34. https://github.com/coreos/zincati/issues/34.) I do wonder if the dual partition approach was deemed more stable than using OSTree or why the latter wasn't used.

eliaspro 7y ago

Any insights on the design decision to go with Wicked instead of systemd-networkd which is already provided by systemd and better integrated with the remaining systemd tooling/components/conventions?