5 ms·

Ex-Amazon SDE here. A lot of this has changed. First, there is a movement to build a lot of services in Native AWS instead of MAWS/Apollo. Apollo doesn't req

by anon20190326 8y ago

Ex-Amazon SDE here. A lot of this has changed. First, there is a movement to build a lot of services in Native AWS instead of MAWS/Apollo. Apollo doesn't require copying configs anymore; you can have the config exist as part of the package you are deploying. Generally, that's a best practice. Pipelines can be configured as code too. There is a centralized log service which requires onboarding. It does require some commandline tools, but it works. The logs get stored on S3, IIRC. If containers suits your needs, you'd be hard-pressed to find someone telling you not to use it. Generally, though, you would want to use bare metal for Amazon's scale. There is also a change to how NPM is being used at Amazon. It was a lot easier towards the end of my tenure, and was probably as close as it would get when working with Amazon's build systems. Amazonians are generally conservative and don't use the latest and greatest unless it solves an actual customer need. Customer Obsession is still the defining leadership principle.

konspence 8y ago
Any take on how this looks for someone who doesn't work at Amazon? I know Apollo is their deployment tool. No idea how they plays into a Kubernetes or even container stack.
- coreyoconnor 8y ago
  Conceptually, there is a lot of overlap. Apollo had a few layers to it but would, more or less, matched kubernetes plus additional tooling. Dare I say closer to openshift? Brazil, the build tool, infrastructure and repository was similar to using nixpkgs. Brazil had some features beyond nixpkgs that were absolutely essential for an business like Amazon, but otherwise the goals and even techniques overlapped. I onboarded very, very quickly by already knowing nixpkgs and container management. The concepts translated well. A few aspects that didn't either were things Amazon handled that other tools have yet to encounter: Cool to learn about. Or historical bits that didn't matter: fine to ignore.
  - ex_amazon_sde 8y ago
    > Apollo had a few layers to it but would, more or less, matched kubernetes plus additional tooling Have you looked at the codebase? Apollo and Brazil are tiny compared to Kubernetes & friends. You could reimplement them with a small script on top of traditional Linux package management (modulo the web UI).
    - Pfhreak 8y ago
      As someone who worked on Apollo, I'd challenge this. Without divulging any details, Apollo is a non trivial system.
      - amazon_throw 8y ago
        Yeah, I work adjacent to the Apollo org (and have worked near Pfrheak, too. Hi!). I would love to see a HackerNews commenter's idea of what it would take to re-implement Apollo. I'll bring popcorn.
m0zg 8y ago
> you would want to use bare metal for Amazon's scale Containers are "bare metal".
- banku_brougham 8y ago
  Yeah i was confused by this too. Parent possibly thinking of virtual machines?
  - m0zg 8y ago
    Fun fact: all of Google (including Google Cloud) is run in containers. IIRC even VMs are run in containers. The VMs, of course, could contain customer containers, too. It's literally containers "all the way down".
- jhayward 8y ago
  > Containers are "bare metal" Please explain this use of jargon. Are you saying that a Docker container is "bare metal" in some culture? A VM is bare metal? What do you mean by container, and bare metal?
  - m0zg 8y ago
    Containers are just namespaces for things within the Linux kernel. Unlike with VMs, you're not running separate instances of the OS, it's all run by just one kernel instance, and that kernel usually runs directly on the hardware, that is, on "bare metal". That "ubuntu" base image you can spin up does not actually run the Ubuntu kernel. As a result, bare metal containers incur none of the "virtualization penalty" that VMs do. Cloud is, in fact, abnormal in terms of how containers are used in that it wraps them into VMs. That's in part because VMs were already there and they simplify provisioning of something that looks like "machines" from the outside, and in part because containers do not offer much in the way of security guarantees (precisely because all containers running on the same host share the kernel), whereas VMs do.
    - jhayward 8y ago

konspence 8y ago

Any take on how this looks for someone who doesn't work at Amazon? I know Apollo is their deployment tool. No idea how they plays into a Kubernetes or even container stack.

coreyoconnor 8y ago
Conceptually, there is a lot of overlap. Apollo had a few layers to it but would, more or less, matched kubernetes plus additional tooling. Dare I say closer to openshift? Brazil, the build tool, infrastructure and repository was similar to using nixpkgs. Brazil had some features beyond nixpkgs that were absolutely essential for an business like Amazon, but otherwise the goals and even techniques overlapped. I onboarded very, very quickly by already knowing nixpkgs and container management. The concepts translated well. A few aspects that didn't either were things Amazon handled that other tools have yet to encounter: Cool to learn about. Or historical bits that didn't matter: fine to ignore.
- ex_amazon_sde 8y ago
  > Apollo had a few layers to it but would, more or less, matched kubernetes plus additional tooling Have you looked at the codebase? Apollo and Brazil are tiny compared to Kubernetes & friends. You could reimplement them with a small script on top of traditional Linux package management (modulo the web UI).
  - Pfhreak 8y ago
    As someone who worked on Apollo, I'd challenge this. Without divulging any details, Apollo is a non trivial system.
    - amazon_throw 8y ago
      Yeah, I work adjacent to the Apollo org (and have worked near Pfrheak, too. Hi!). I would love to see a HackerNews commenter's idea of what it would take to re-implement Apollo. I'll bring popcorn.

m0zg 8y ago

> you would want to use bare metal for Amazon's scale Containers are "bare metal".

banku_brougham 8y ago
Yeah i was confused by this too. Parent possibly thinking of virtual machines?
- m0zg 8y ago
  Fun fact: all of Google (including Google Cloud) is run in containers. IIRC even VMs are run in containers. The VMs, of course, could contain customer containers, too. It's literally containers "all the way down".
jhayward 8y ago
> Containers are "bare metal" Please explain this use of jargon. Are you saying that a Docker container is "bare metal" in some culture? A VM is bare metal? What do you mean by container, and bare metal?
- m0zg 8y ago
  Containers are just namespaces for things within the Linux kernel. Unlike with VMs, you're not running separate instances of the OS, it's all run by just one kernel instance, and that kernel usually runs directly on the hardware, that is, on "bare metal". That "ubuntu" base image you can spin up does not actually run the Ubuntu kernel. As a result, bare metal containers incur none of the "virtualization penalty" that VMs do. Cloud is, in fact, abnormal in terms of how containers are used in that it wraps them into VMs. That's in part because VMs were already there and they simplify provisioning of something that looks like "machines" from the outside, and in part because containers do not offer much in the way of security guarantees (precisely because all containers running on the same host share the kernel), whereas VMs do.
  - jhayward 8y ago