4 ms·

This is awful. It completely ignores the economics of leveraging commodity services and re-usable skills while ignoring the true places where you can get maxima

This is awful. It completely ignores the economics of leveraging commodity services and re-usable skills while ignoring the true places where you can get maximal value if you are willing to accept vendor lock in - the high level services. This is more or less the equivalent of mandating that everyone use VMS once UNIX started to commodify, or use Windows for all of your servers once Linux has taken over the market. Using EKS + Fargate instead of ECS + fargate provides no savings over EKS, All it serves to do is to lock you into a single hyper-scaler infrastructure, at the same time as K8s is forcing he cloud vendors into commoditization. Want to use AWS effectively? Depend on high level services like Glue, Athena, Kinesis Firehose, Sagemaker. Want to piss away any chance to run your business effectively? Leverage ECS. If you are a one-man shop, and you know ECS or are willing to depend on underbaked solutions because they solve a problem, more power to you. I suspect that you may benefit over your career by investing in more universal skill sets (alternative, you may benefit from hyper-specializing on AWS toolsets as well(.

glogla 5y ago
I would very much say it's the other way around. AWS is good at giving you commodity infrastructure at scale. They are really good at "stupid" services like S3 or EC2 or RDS. The high-level services meanwhile? I worked with quite a few of them and they're mostly shit. Athena is has account-wide limit of 5 concurrent queries which cannot be increased. Even one larger dashboard will overload it. Redshift have the same limit of 30 concurrent queries. That's good enough for casual use but not suitable for larger company. Glue Catalog does not scale at all, and having more than few objects will break it, and you will very soon end up begging for more API limit due to throttling exceptions. Kinesis has very strange limits (in messages per second) that make it really expensive for use cases where traffic is peaky - which is quite a few streaming use cases. Just like you wouldn't use Amazon.com to buy high-quality, important goods (because you're pretty likely to get something broken or fake), don't use AWS "high level" services. Amazon is company focused on scaling commodity use cases, not at engineering excellency.
- pilate 5y ago
  I recently had my Athena limit increased to 25 for my account, so that’s no longer true.
  - glogla 5y ago
    If they actually can increase the concurrency now, that's good news! We asked for that as well, and at first the support said they did that, but turns out they increased the Athena queue length instead of concurrency - so the queries wouldn't fail (until the queue is full, anyway) but the throughput was still the same. Then the team of AWS account managers and solutions architects (who assured us many times that the concurrency can be increased) get into a email discussion with the support and Athena team, before coming back and telling us it cannot be increased. No apology or anything. Recommended us to run Presto on EMR instead. It was a sobering experience in how AWS operates.
- finnh 5y ago
  > Amazon is company focused on scaling commodity use cases, not at engineering excellency. I think of it slightly differently. S3 is clearly the product of engineering excellency (as is DynamoDb). But I think there are A teams and B teams at AWS, as you would expect from any company so large. And I get the sense that at least some of the newer, less proven solutions were written by the B teams. This likely includes some k8s-market-share-grabby offerings.
  - glogla 5y ago
    I heard same story told slightly differently - for services that process your requests when you go to amazon.com to buy stuff, you can expect engineering excellency. That would be dynamo, s3, ec2, networking (that whole side of AWS - hypervisors, networks, etc., I'd call engineering marvel). But that would not be software they build "for someone else", like Redshift and Glue and Sagemaker and whole lot of other stuff. There's a whole lot of half-baked services in the ~200 they have.

glogla 5y ago

I would very much say it's the other way around. AWS is good at giving you commodity infrastructure at scale. They are really good at "stupid" services like S3 or EC2 or RDS. The high-level services meanwhile? I worked with quite a few of them and they're mostly shit. Athena is has account-wide limit of 5 concurrent queries which cannot be increased. Even one larger dashboard will overload it. Redshift have the same limit of 30 concurrent queries. That's good enough for casual use but not suitable for larger company. Glue Catalog does not scale at all, and having more than few objects will break it, and you will very soon end up begging for more API limit due to throttling exceptions. Kinesis has very strange limits (in messages per second) that make it really expensive for use cases where traffic is peaky - which is quite a few streaming use cases. Just like you wouldn't use Amazon.com to buy high-quality, important goods (because you're pretty likely to get something broken or fake), don't use AWS "high level" services. Amazon is company focused on scaling commodity use cases, not at engineering excellency.

pilate 5y ago
I recently had my Athena limit increased to 25 for my account, so that’s no longer true.
- glogla 5y ago
  If they actually can increase the concurrency now, that's good news! We asked for that as well, and at first the support said they did that, but turns out they increased the Athena queue length instead of concurrency - so the queries wouldn't fail (until the queue is full, anyway) but the throughput was still the same. Then the team of AWS account managers and solutions architects (who assured us many times that the concurrency can be increased) get into a email discussion with the support and Athena team, before coming back and telling us it cannot be increased. No apology or anything. Recommended us to run Presto on EMR instead. It was a sobering experience in how AWS operates.
finnh 5y ago
> Amazon is company focused on scaling commodity use cases, not at engineering excellency. I think of it slightly differently. S3 is clearly the product of engineering excellency (as is DynamoDb). But I think there are A teams and B teams at AWS, as you would expect from any company so large. And I get the sense that at least some of the newer, less proven solutions were written by the B teams. This likely includes some k8s-market-share-grabby offerings.
- glogla 5y ago
  I heard same story told slightly differently - for services that process your requests when you go to amazon.com to buy stuff, you can expect engineering excellency. That would be dynamo, s3, ec2, networking (that whole side of AWS - hypervisors, networks, etc., I'd call engineering marvel). But that would not be software they build "for someone else", like Redshift and Glue and Sagemaker and whole lot of other stuff. There's a whole lot of half-baked services in the ~200 they have.