3 ms·
Run SQL queries using Athena and a Dynamodb crawler in Glue. It's not that complicated.
by dkobia 8y ago
Run SQL queries using Athena and a Dynamodb crawler in Glue. It's not that complicated.
- wh-uws 8y agoLink to a tutorial that is as simple with code examples / screenshots as this one? I ask because I have looked and haven't found one that I could follow without having to dedicate several hours to it. To the point where I want to move off of dynamodb to aurora because I can't believe how hard it is to get simple insights like how many rows exist out of dynamo (I inherited the dynamo choice would not have used it myself)
- iamsomewalrus 8y agoI’ll have a blog post out in AWS Big Data in a few weeks but the parent is right in that it’s not terribly difficult. Should take about an hour with testing to get a Pyspark script together to read in a DynamoDB table and write it out to S3. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#aws-glue-programming-etl-connect-dynamodb https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programm... You’ll then need to crawl the S3 data to add it to your Glue catalog and then you can query it with Athena.
- alexbilbie 8y agoIf you're running this regularly (e.g. once an hour) to dump tables the cost of Glue can really add up due a minimum runtime length. We switched to a scheduled Fargate task to dump data from DynamoDB into S3 as parquet files. It's really reliable, costs us ~$4/month and completely configurable.
- teej 8y agoHow are you getting data from DynamoDB to S3 so Athena can read it?
- vonholstein 8y agoETL job in AWS glue with DynamoDB as source and S3 as sink is one option. https://aws.amazon.com/about-aws/whats-new/2018/07/aws-glue-now-supports-reading-from-amazon-dynamodb-tables/ https://aws.amazon.com/about-aws/whats-new/2018/07/aws-glue-...
- ralusek 8y agoYou could have a lambda hook to trigger on dynamo write, and copy the dynamo change to s3 in the lambda.