4 ms·
> spend weeks arguing with AWS support Yup. We've run into that repeatedly. The "we didn't notice anything on our side, please send more screenshots and logs"
by falcolas 8y ago
> spend weeks arguing with AWS support
Yup. We've run into that repeatedly. The "we didn't notice anything on our side, please send more screenshots and logs" gets really old when working with "managed services".
Network packet losses/truncations, EKS control plane failures, cloudformation stacks getting stuck in really wierd states, inconsistent cloudformation implementations for new and existing services, ENI weirdness in containers...
Managed services just feel like they aren't - more and more every day. Amazon (or any other cloud) will never have the same investment in your availability and infrastructure as you will.
- rabi_penguin 8y agoRunning into one of those exact issues (cloudformation stacks getting stuck in really weird states) right now. A real pain in the ass, because I have zero control or visibility into it.
- toomuchtodo 8y agoIs switching to Terraform an option?
- falcolas 8y agoBased on our own research - probably. But it requires tooling to integrate into a CI/CD pipeline, conversion of existing CF stack definitions to Terraform definitions, and a lossy/downtime laden migration. For a team with the goals of IAC, hands-off, and high-availability infrastructure, migrating is a pain.
- ReidZB 8y agoWith Terraform's import functionality, you might be able to migrate into Terraform without any downtime. I believe there are even utilities to translate CloudFormation code straight into Terraform, though I've never tried them. It would definitely be an annoying migration, even if import works for your use-case, but honestly, dealing with CloudFormation is so irritating on a day-to-day basis for me that I'd consider it worth it.
- isatty 8y agoCloudformation is a tragedy in itself.
- TheCraiggers 8y agoIt's infinitely better than what Azure provides, IMO.
- falcolas 8y agoThe idea behind CloudFormation is great, though. Platform-native IAC with promised first party support for all future projects, and backfilling existing services. Plus, it supports deep integrations with first-party supported configuration management services. The problem is that the reality has not lived up to the promise, and "first party support" means "only the first party can support".
- devonkim 8y agoCloudFormation is a declarative state management language and framework just like Terraform. The problem is that CF abstracts said state away from you to the degree that you can't hack around it and you wind up forcibly deleting resources if you try to use it like a configuration management framework. With CF Custom Resources you can add all sorts of other stuff and that's pretty cool at least.
- tech_tuna 8y agoI am a Terraform convert, I agree.
- ReidZB 8y agoMy experience with AWS support for performance issues is similar. My ticket for an Aurora RDS performance issue ended up in a kind of purgatory where they continued to ask for more information and more tests on my side, continued, to infinity. Once one support engineer was satisfied with my results, the ticket would go dark for months, and when I would follow up, another (new) engineer would ask for a whole new round of tests, logs, etc., typically for the latest release. After a few rounds of back-and-forth, I eventually gave up. (I even gave them permission to copy test snapshot data, at their request, and sample queries to reproduce the issue. They could have done all the testing they wanted in-house. But that might have actually required them to spend time on their side to diagnose and fix something, I guess.) It wasn't a critical issue, exactly, and we worked around it in code. I mostly just wanted to report a problem.