4 ms·

Huh? Yes Aurora is highly proprietary, but S3 is a commodity in many use cases.

by gkop 5y ago

oneplane 5y ago
Commodity in usage, yes, but not on the service side. Doing S3 properly is really hard, having a service with a 'compatible API' doesn't make it the same as S3. Same goes for EKS (Yes, it looks and smells like a normal API server) and RDS. If you never use anything that ties this stuff together (like IAM), then there is practically no point in using AWS at all, it's way too expensive to use as a 'dumb pipe'. That said, I'm personally a fan of using things that would technically also work outside of AWS, but not shying away from the AWS-specific optimisations. I'm not going to use Kubernetes on AWS without EKS for example (plain EC2). Or Kubernetes on AWS without the Amazon's CNI, that's not a thing I'd enjoy in production either. When managed Kafka was announced, that was another great example of "nice, now I don't have to mess with EC2 node contents anymore", but I still can if I want to, including outside of AWS. The most AWS-native thing (or two things) that make most other offerings not work out all that great is the integrated IAM and Security Groups. Kludging those together with some crappy port to AD or NSX is just a waste of time and sanity.
- dilyevsky 5y ago
  Doing EB-scale s3 is really hard, throwing a few hundred terrabytes (or even couple petabytes) into s3 compatible storage on top of k8s is not that hard (getting easier every day). There’s numerous options available today even with enterprise support > The most AWS-native thing (or two things) that make most other offerings not work out all that great is the integrated IAM and Security Groups SGs are easily covered by kubernetes network policies. What do you need iam for if you’re not using any manager stuff? Sounds like circular argument to me (“you need managed bc you need iam bc you need managed…)
  - oneplane 5y ago
    You need SGs because not everything in the world lives inside a K8S cluster or flows through a CNI. And you need IAM because if you have 10 users and 10 resources but not every user is allowed to perform all actions on all resources you need policies. Policies apply (generally) on roles, actions and resources. IAM that only works on some random resources but not on others are pointless, unless you have a very small setup (i.e. less than 100 resources, less than a small office of users), but at such a small scale, you shouldn't even be thinking about AWS (unless you are on track to scale 1000x). S3 isn't just "I have many files and it needs a lot of disk space". Like I wrote, if you just use base S3 API primitives then no, you're not really using anything AWS-specific, and making a comparison between something that does more than you need seems odd to me. Edge storage, cold storage, cheaper storage based on reductions of redundancy or access times, mirroring, versioning, multi-level IAM (integrated with the rest of your IAM) are all things that make the difference between "look at my virtual USB disk" (most MinIO configurations out there, the unsupported ones that is) and "durable and reliable multi-level object storage, configured for my needs".
    - dilyevsky 5y ago
      I’m sure you can find a feature that minio/ceph/swift don’t support out of the box just as i can probably find something that’s very easy to do do/add in ceph/seaweedfs but will be on the back burner of your cloud tam for a few quarters/years or do it at a cost that can’t be beat by any volume commits. At the end of the day we are payed to solve business problems with set constraints so statements like “you need s3 bc everything else is hard” are kind of myopic
      - oneplane 5y ago
        Unless you are in the business of reinventing wheels and building knowledge silos that will never transfer, none of what you wrote applies in general commercial business.
- Scarbutt 5y ago
  If you never use anything that ties this stuff together (like IAM), then there is practically no point in using AWS at all, it's way too expensive to use as a 'dumb pipe'. Must cheap providers have crappy networks compared to AWS.
  - oneplane 5y ago
    That's true. And sometimes you get an odd mix of good network quality but crappy features, or the other way around and that isn't amazing to work with either.

Huh? Yes Aurora is highly proprietary, but S3 is a commodity in many use cases.

by gkop 5y ago

Huh? Yes Aurora is highly proprietary, but S3 is a commodity in many use cases.

Commodity in usage, yes, but not on the service side. Doing S3 properly is really hard, having a service with a 'compatible API' doesn't make it the same as S3. Same goes for EKS (Yes, it looks and smells like a normal API server) and RDS. If you never use anything that ties this stuff together (like IAM), then there is practically no point in using AWS at all, it's way too expensive to use as a 'dumb pipe'. That said, I'm personally a fan of using things that would technically also work outside of AWS, but not shying away from the AWS-specific optimisations. I'm not going to use Kubernetes on AWS without EKS for example (plain EC2). Or Kubernetes on AWS without the Amazon's CNI, that's not a thing I'd enjoy in production either. When managed Kafka was announced, that was another great example of "nice, now I don't have to mess with EC2 node contents anymore", but I still can if I want to, including outside of AWS. The most AWS-native thing (or two things) that make most other offerings not work out all that great is the integrated IAM and Security Groups. Kludging those together with some crappy port to AD or NSX is just a waste of time and sanity.

dilyevsky 5y ago
Doing EB-scale s3 is really hard, throwing a few hundred terrabytes (or even couple petabytes) into s3 compatible storage on top of k8s is not that hard (getting easier every day). There’s numerous options available today even with enterprise support > The most AWS-native thing (or two things) that make most other offerings not work out all that great is the integrated IAM and Security Groups SGs are easily covered by kubernetes network policies. What do you need iam for if you’re not using any manager stuff? Sounds like circular argument to me (“you need managed bc you need iam bc you need managed…)
- oneplane 5y ago
  You need SGs because not everything in the world lives inside a K8S cluster or flows through a CNI. And you need IAM because if you have 10 users and 10 resources but not every user is allowed to perform all actions on all resources you need policies. Policies apply (generally) on roles, actions and resources. IAM that only works on some random resources but not on others are pointless, unless you have a very small setup (i.e. less than 100 resources, less than a small office of users), but at such a small scale, you shouldn't even be thinking about AWS (unless you are on track to scale 1000x). S3 isn't just "I have many files and it needs a lot of disk space". Like I wrote, if you just use base S3 API primitives then no, you're not really using anything AWS-specific, and making a comparison between something that does more than you need seems odd to me. Edge storage, cold storage, cheaper storage based on reductions of redundancy or access times, mirroring, versioning, multi-level IAM (integrated with the rest of your IAM) are all things that make the difference between "look at my virtual USB disk" (most MinIO configurations out there, the unsupported ones that is) and "durable and reliable multi-level object storage, configured for my needs".
  - dilyevsky 5y ago
    I’m sure you can find a feature that minio/ceph/swift don’t support out of the box just as i can probably find something that’s very easy to do do/add in ceph/seaweedfs but will be on the back burner of your cloud tam for a few quarters/years or do it at a cost that can’t be beat by any volume commits. At the end of the day we are payed to solve business problems with set constraints so statements like “you need s3 bc everything else is hard” are kind of myopic
    - oneplane 5y ago
      Unless you are in the business of reinventing wheels and building knowledge silos that will never transfer, none of what you wrote applies in general commercial business.
Scarbutt 5y ago
If you never use anything that ties this stuff together (like IAM), then there is practically no point in using AWS at all, it's way too expensive to use as a 'dumb pipe'. Must cheap providers have crappy networks compared to AWS.
- oneplane 5y ago
  That's true. And sometimes you get an odd mix of good network quality but crappy features, or the other way around and that isn't amazing to work with either.