3 ms·

What? Why should EC2 instances be single-purpose? Amazon offers a wide variety of massive instance sizes with 160+gb of RAM and 30+ cores. It's extremely common

What? Why should EC2 instances be single-purpose? Amazon offers a wide variety of massive instance sizes with 160+gb of RAM and 30+ cores. It's extremely common to run software like mesos, kubernetes, docker, etc on these. Dedicating an instance per app is extremely cost-ineffective.

subway 10y ago
EC2 instances should be single-purpose (Or, if you want to mux containers onto the instance and retain per-container/job IAM role isolation, use ECS) if you're developing for AWS as a platform. I'm a huge fan of k8s, and have respect for Mesos, but these are largely alternatives to the model provided by EC2/ECS/IAM. In a perfect world, any service would cleanly interoperate with any other service. Unfortunately we don't live in a perfect world. If you want to take advantage of `advanced` features in a given platform, you have to understand the drawbacks and limitations of those features, and what it means when they aren't available on another platform. To me, the greatest tragedy in the way EC2 operates is that it looks/tastes/smells like a `server`, but it's far more akin to a process.
- jamiesonbecker 10y ago
  Well.. an EC2 instance running Linux is not a process or even a container, even if functionally it's easier to treat it like one. It is a full virtual server with its own Linux kernel and operating system: it has to be updated, secured, and maintained just like any other Linux server. Most Linux distributions on an EC2 instance have dozens of processes already running out of the box. I understand your point -- that ideally a single instance can be treated as a single functional point from the point of view of the application, and I agree, but not from a point of view of security. As you know, in any larger environment, there are likely many additional support applications running on that server: things like app server monitoring, file integrity, logging, management, security checks, remote data access or local databases, etc. Those must not all be treated with the same levels of security and access. (i.e., why would rsyslog or systemd need access to all objects in our S3 bucket or be able to delete instances or any of the other rights that might legitimately be granted to an instance via an IAM instance role?) To treat security for all of these processes as if they're all part of the same app tosses out decades of operating system development and security principles and places your single function app, as well as that of your entire environment, at grave risk. I.e., there's a reason why a typical Linux distribution has about 50 accounts right out of the box and everything doesn't just run as root. If you are developing or deploying microservices or containers and don't want to be burdened by the security requirements, then there are alternatives at AWS like ECS and Lambda that you should seriously consider.
  - subway 10y ago
    Amusing examples, systemd/rsyslog, as both at least briefly execute as root, with rsyslog being relied upon to willingly drop its own privs (Not to mention being slowly replaced by systemd-journald, which runs as root), and systemd always running as root (ya know, since it's init, and all). It really sounds like we have vastly different ideas about what kinds of processes belong in an EC2 instance, as well as the ideal life-cycle of an EC2 instance. I tend to adopt a strategy of relatively short-lived EC2 instances that get killed and replaced frequently. Persistence that depends on a single instance surviving is avoided at all costs, in favor of persistence distributed across a number of instances (or punted out to Dynamo/S3/RDS). You're absolutely right that there is a reason why the typical Linux distro has 50 accounts out of the box -- it was built with traditional multi-user system security models in mind. I sure as hell appreciate it on workstations and traditional stateful hosts. That said, eschewing the traditional security model in favor of an alternative model does not make your environment inherently more or less safe -- there are going to be pros and cons to both approaches (in terms of both security and functionality).

subway 10y ago

EC2 instances should be single-purpose (Or, if you want to mux containers onto the instance and retain per-container/job IAM role isolation, use ECS) if you're developing for AWS as a platform. I'm a huge fan of k8s, and have respect for Mesos, but these are largely alternatives to the model provided by EC2/ECS/IAM. In a perfect world, any service would cleanly interoperate with any other service. Unfortunately we don't live in a perfect world. If you want to take advantage of `advanced` features in a given platform, you have to understand the drawbacks and limitations of those features, and what it means when they aren't available on another platform. To me, the greatest tragedy in the way EC2 operates is that it looks/tastes/smells like a `server`, but it's far more akin to a process.

jamiesonbecker 10y ago
Well.. an EC2 instance running Linux is not a process or even a container, even if functionally it's easier to treat it like one. It is a full virtual server with its own Linux kernel and operating system: it has to be updated, secured, and maintained just like any other Linux server. Most Linux distributions on an EC2 instance have dozens of processes already running out of the box. I understand your point -- that ideally a single instance can be treated as a single functional point from the point of view of the application, and I agree, but not from a point of view of security. As you know, in any larger environment, there are likely many additional support applications running on that server: things like app server monitoring, file integrity, logging, management, security checks, remote data access or local databases, etc. Those must not all be treated with the same levels of security and access. (i.e., why would rsyslog or systemd need access to all objects in our S3 bucket or be able to delete instances or any of the other rights that might legitimately be granted to an instance via an IAM instance role?) To treat security for all of these processes as if they're all part of the same app tosses out decades of operating system development and security principles and places your single function app, as well as that of your entire environment, at grave risk. I.e., there's a reason why a typical Linux distribution has about 50 accounts right out of the box and everything doesn't just run as root. If you are developing or deploying microservices or containers and don't want to be burdened by the security requirements, then there are alternatives at AWS like ECS and Lambda that you should seriously consider.
- subway 10y ago
  Amusing examples, systemd/rsyslog, as both at least briefly execute as root, with rsyslog being relied upon to willingly drop its own privs (Not to mention being slowly replaced by systemd-journald, which runs as root), and systemd always running as root (ya know, since it's init, and all). It really sounds like we have vastly different ideas about what kinds of processes belong in an EC2 instance, as well as the ideal life-cycle of an EC2 instance. I tend to adopt a strategy of relatively short-lived EC2 instances that get killed and replaced frequently. Persistence that depends on a single instance surviving is avoided at all costs, in favor of persistence distributed across a number of instances (or punted out to Dynamo/S3/RDS). You're absolutely right that there is a reason why the typical Linux distro has 50 accounts out of the box -- it was built with traditional multi-user system security models in mind. I sure as hell appreciate it on workstations and traditional stateful hosts. That said, eschewing the traditional security model in favor of an alternative model does not make your environment inherently more or less safe -- there are going to be pros and cons to both approaches (in terms of both security and functionality).