5 ms·

author here - curious about interesting AWS issues you've encountered and lessons learned from them

spaghetti-guy 7y ago
Had a similar issue with CloudTrail logs. These are delivered to your S3 bucket but the objects are owned by AWS's 'CloudTrail account'. This means you can't drop the logs into your security account and then query them with Athena from another account. Took ages to figure that out.
- jasonpeacock 7y ago
  I'm doing/fixing the same thing with the Cost Usage Reports. It feels like a common pattern to consolidate your logs into a single account for analysis, I wish they made it more straightforward to setup.
  - spydum 7y ago
    Isn’t that one part of what control tower does now?
    - kevinslin 7y ago
      yep. control tower creates centralized logging, scp rules and more. what used to take a dev a couple weeks to set up can now be done with a few clicks
  - spaghetti-guy 7y ago
    We haven't really dug in to analyzing those outside of cost explorer yet but that makes sense. One workaround I've heard of is S3 replication. I've not tried it yet but apparently the replica has the 'correct' ownership.
time0ut 7y ago
Not a lot of issues really. Lots of lessons learned and small features I wish existed. Here's a classic: I had a lambda fed by an even stream hanging off a Dynamo table. The lambda would delete the record after doing a bit of stuff. I didn't realize on my first pass that this would trigger the lambda again... Wasn't a big deal as I noticed the behavior pretty quickly.
- kevinslin 7y ago
  yeah, that's the classic infinite lambda recursion gotcha: https://sourcebox.be/serverless-a-lesson-learned-the-hard-way/ https://sourcebox.be/serverless-a-lesson-learned-the-hard-wa...
jeffnappi 7y ago
An issue I've encountered with static website hosting on CloudFront+S3 is that if you want `index.html` to be served at the root path of a URL - e.g. https://www.yoursite.com/folder/ https://www.yoursite.com/folder/ vs https://www.yoursite.com/folder/index.html https://www.yoursite.com/folder/index.html, you must use the s3 website endpoint rather than the S3 Origin. And in that case, unfortunately, it is not possible to use Origin Access Identity.
- kevinslin 7y ago
  good news is that you can remedy that now by using lambda@edge to rewrite requests - there's even an aws blog post about it: https://aws.amazon.com/blogs/compute/implementing-default-directory-indexes-in-amazon-s3-backed-amazon-cloudfront-origins-using-lambdaedge/ https://aws.amazon.com/blogs/compute/implementing-default-di...
  - fishtacos 7y ago
    From the blog post above: "It does not work on any subdirectory (such as http://www.example.com/about/ http://www.example.com/about/). If you were to attempt to request this URL through CloudFront, CloudFront would do a S3 GetObject API call against a key that does not exist."

Had a similar issue with CloudTrail logs. These are delivered to your S3 bucket but the objects are owned by AWS's 'CloudTrail account'. This means you can't drop the logs into your security account and then query them with Athena from another account. Took ages to figure that out.

jasonpeacock 7y ago
I'm doing/fixing the same thing with the Cost Usage Reports. It feels like a common pattern to consolidate your logs into a single account for analysis, I wish they made it more straightforward to setup.
- spydum 7y ago
  Isn’t that one part of what control tower does now?
  - kevinslin 7y ago
    yep. control tower creates centralized logging, scp rules and more. what used to take a dev a couple weeks to set up can now be done with a few clicks
- spaghetti-guy 7y ago
  We haven't really dug in to analyzing those outside of cost explorer yet but that makes sense. One workaround I've heard of is S3 replication. I've not tried it yet but apparently the replica has the 'correct' ownership.

time0ut 7y ago

Not a lot of issues really. Lots of lessons learned and small features I wish existed. Here's a classic: I had a lambda fed by an even stream hanging off a Dynamo table. The lambda would delete the record after doing a bit of stuff. I didn't realize on my first pass that this would trigger the lambda again... Wasn't a big deal as I noticed the behavior pretty quickly.

kevinslin 7y ago
yeah, that's the classic infinite lambda recursion gotcha: https://sourcebox.be/serverless-a-lesson-learned-the-hard-way/ https://sourcebox.be/serverless-a-lesson-learned-the-hard-wa...

jeffnappi 7y ago

An issue I've encountered with static website hosting on CloudFront+S3 is that if you want `index.html` to be served at the root path of a URL - e.g. https://www.yoursite.com/folder/ https://www.yoursite.com/folder/ vs https://www.yoursite.com/folder/index.html https://www.yoursite.com/folder/index.html, you must use the s3 website endpoint rather than the S3 Origin. And in that case, unfortunately, it is not possible to use Origin Access Identity.

kevinslin 7y ago
good news is that you can remedy that now by using lambda@edge to rewrite requests - there's even an aws blog post about it: https://aws.amazon.com/blogs/compute/implementing-default-directory-indexes-in-amazon-s3-backed-amazon-cloudfront-origins-using-lambdaedge/ https://aws.amazon.com/blogs/compute/implementing-default-di...
- fishtacos 7y ago
  From the blog post above: "It does not work on any subdirectory (such as http://www.example.com/about/ http://www.example.com/about/). If you were to attempt to request this URL through CloudFront, CloudFront would do a S3 GetObject API call against a key that does not exist."