5 ms·

Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.

eggman222 8y ago
Hi, thank you for taking the time. I have a couple questions: * Why is the DB scrape written as json instead of directly proto/avro/parquet? Isn't it a lot more costly to store and to handle? * How many events can aws lambda scale to in this kind of architecture?
- iamsomewalrus 8y ago
  Did we just become best friends with our “I am the Walrus” references? The DB scrape uses a template from Data Pipeline that under the hood uses the Dynamo DB Scan API. Not really surprising, but that API uses JSON. I wanted to use as much off the shelf software as I could to get data into Athena. In this architecture Lambda is only used to listen to the SNS topic that fires when the Data Pipeline job succeeds or fails so we’re not pushing the limits of Lambda at all. You’d probably hit an EC2 limit Witt Data Pipeline before hitting your Lambda limit on the account.
mooreds 8y ago
Happy to post it, but you can thank Corey Quinn from https://lastweekinaws.com/ https://lastweekinaws.com/ for pointing me to it.
- iamsomewalrus 8y ago
  I did! I'm a big fan of the Last Week in AWS newsletter. https://twitter.com/iamsomewalrus/status/1016368717880389632 https://twitter.com/iamsomewalrus/status/1016368717880389632
mooreds 8y ago
Why didn't you push the data into a traditional data warehouse or sql database?
- iamsomewalrus 8y ago
  Data will eventually end up there. Our product and business teams are heavy users of Redshift. As mentioned in another comment I’ve found having Dynamo snapshots in Athena really useful as an oncall to sanity check snapshots (what was the state of Harry Potter 3 months ago compared to now?) and to answer product questions that can only be answered from the raw production data.
  - ngould 8y ago
    This is the first time I've come across the approach of storing database snapshots and saving them in a data lake. Do you find those snapshots are useful/used for analytics or data science end-uses, or are they more there for debugging and answering one-off questions?
    - iamsomewalrus 8y ago
      I have personally only used them for debugging one off questions. That was my original intent. We do have teams that are considering using the snapshots for ML problems.
- ecnahc515 8y ago
  For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).
  - iamsomewalrus 8y ago
    :this: A concept that's underlying the move to a datalake architecture (read: keeping your data in its rawest form, and its transforms in S3 or HDFS) is decoupling your compute from storage. Motivating example: you have huge tables in Redshift that are either infrequently accessed or the usefulness of the data decays over time (website logs, customer order information). In this scenario you're paying a lot just to keep data in Redshift (storage) but a large subset of the data is laying dormant (no compute). If you're bought into the Redshift ecosystem this is where Redshift Spectrum comes in. If you're a smaller company you could just store the data in S3 and "spin up" the compute when you need it (Athena, Glue jobs, or Elastic Map Reduce clusters).
bcbrown 8y ago
I'd love to hear more about "A serverless Apache Spark environment". How is that set up? How long do those jobs tend to run for? Are those written in Java/Scala or Python? What are the pros and cons for choosing to go serverless instead of a dedicated cluster or ephemeral on-demand clusters?
- iamsomewalrus 8y ago
  The AWS Glue service provides a serverless Spark environment for running jobs. Here's a link: https://docs.aws.amazon.com/glue/latest/dg/author-job.html https://docs.aws.amazon.com/glue/latest/dg/author-job.html * The default timeout is ~ 48 hours and you pay per Data Processing Unit (DPU) that you've provisioned the Job. * Currently it supports Python and Scala. As far as I'm aware you can't run Java jobs directly, but you can upload JAR libraries and use them in your code. Re serverless vs dedicated / ephemeral clusters: Like with any serverless runtime environment you are trading convenience (across a few dimensions) for flexibility. The Glue environment runs in a few limited runtimes and uses a specific version of Spark that you have no control over updating. Given that it's pretty quick to author a job, you can set the required DPU and Glue handles that, and you don't have to worry about sizing the cluster for your data size. For me most of my jobs fit within those constraints. At some point on the cost curve it may make sense for you to move all of your jobs from Glue into a dedicated cluster on EMR. You may also get there sooner if you need to use specific frameworks or libraries.
RangerScience 8y ago
Heya! I have been working on almost exactly the same thing for the last couple months :) I haven't had a chance to actually read your post yet, but I'm really looking forward to it.
- RangerScience 8y ago
  Okay, neat! Had a chance to skim. It's really cool that you have the Cloud Formation code up and accessible (as well as the Glue script). But, is the lamdba accessible from anywhere? (or did I just miss the link?) We used Terraform instead of CloudFormation, although there are a few places it didn't/doesn't cover. We're also doing protobuf -> parquet instead of JSON; Scala instead of Python, and one of our major feature/issues is that the incoming data is out-of-order; Firehose outputs to partitions based on when the event arrived, and we're repartitioning based on when the event occurred (according to a timestamp in the message) I see you ran into the capitalization issue too :) I ran across docs somewhere that said something along the lines of Glue downcasing column names, which definitely fits observed behavior. I'm hoping to turn what we've done into a similar post, although I think it'll be a month or two before I can get to that. Edit: Oh, I also got a lot of mileage out of the Zeppelin Notebooks. Way better than a raw dev endpoint, but watch out on the cost for both :) The notebooks can be provisioned with almost a single click from the Dev endpoint console, but they changed the recipe halfway through my work on it, and now you have to also SSH into the box and run a script to setup some of the security. :/ Still totally worth it, tho.
  - iamsomewalrus 8y ago
    The Lambda is included in the DynamoDB exports CloudFormation templates. It's embedded in the file template itself. Thumbs up on the Glue Dev endpoint. It's been killer. I had trouble setting up a Notebook (I wanted to get fancy with Docker) and I usually use the Python repl link that's provided. I'm working on a follow up post that removes the Data Pipeline -> Lambda and uses the new Glue DynamoDB integration. Looking forward to your blog post!
    - RangerScience 8y ago
rapathak 8y ago
It's a great post. Thanks for taking the time to write it!
welder 8y ago
iamsomewalrus: Since you're using the same s3 key prefix for all dbexport data does that slow down your Athena queries because of key map partitioning? [1] Also, do you ever see 503 Slowdown errors from Athena requests to s3? [1]: https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/ https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-...

Hi, thank you for taking the time. I have a couple questions: * Why is the DB scrape written as json instead of directly proto/avro/parquet? Isn't it a lot more costly to store and to handle? * How many events can aws lambda scale to in this kind of architecture?

iamsomewalrus 8y ago
Did we just become best friends with our “I am the Walrus” references? The DB scrape uses a template from Data Pipeline that under the hood uses the Dynamo DB Scan API. Not really surprising, but that API uses JSON. I wanted to use as much off the shelf software as I could to get data into Athena. In this architecture Lambda is only used to listen to the SNS topic that fires when the Data Pipeline job succeeds or fails so we’re not pushing the limits of Lambda at all. You’d probably hit an EC2 limit Witt Data Pipeline before hitting your Lambda limit on the account.

mooreds 8y ago

Happy to post it, but you can thank Corey Quinn from https://lastweekinaws.com/ https://lastweekinaws.com/ for pointing me to it.

iamsomewalrus 8y ago
I did! I'm a big fan of the Last Week in AWS newsletter. https://twitter.com/iamsomewalrus/status/1016368717880389632 https://twitter.com/iamsomewalrus/status/1016368717880389632

mooreds 8y ago

Why didn't you push the data into a traditional data warehouse or sql database?

iamsomewalrus 8y ago
Data will eventually end up there. Our product and business teams are heavy users of Redshift. As mentioned in another comment I’ve found having Dynamo snapshots in Athena really useful as an oncall to sanity check snapshots (what was the state of Harry Potter 3 months ago compared to now?) and to answer product questions that can only be answered from the raw production data.
- ngould 8y ago
  This is the first time I've come across the approach of storing database snapshots and saving them in a data lake. Do you find those snapshots are useful/used for analytics or data science end-uses, or are they more there for debugging and answering one-off questions?
  - iamsomewalrus 8y ago
    I have personally only used them for debugging one off questions. That was my original intent. We do have teams that are considering using the snapshots for ML problems.
ecnahc515 8y ago
For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).
- iamsomewalrus 8y ago
  :this: A concept that's underlying the move to a datalake architecture (read: keeping your data in its rawest form, and its transforms in S3 or HDFS) is decoupling your compute from storage. Motivating example: you have huge tables in Redshift that are either infrequently accessed or the usefulness of the data decays over time (website logs, customer order information). In this scenario you're paying a lot just to keep data in Redshift (storage) but a large subset of the data is laying dormant (no compute). If you're bought into the Redshift ecosystem this is where Redshift Spectrum comes in. If you're a smaller company you could just store the data in S3 and "spin up" the compute when you need it (Athena, Glue jobs, or Elastic Map Reduce clusters).

bcbrown 8y ago

I'd love to hear more about "A serverless Apache Spark environment". How is that set up? How long do those jobs tend to run for? Are those written in Java/Scala or Python? What are the pros and cons for choosing to go serverless instead of a dedicated cluster or ephemeral on-demand clusters?

iamsomewalrus 8y ago
The AWS Glue service provides a serverless Spark environment for running jobs. Here's a link: https://docs.aws.amazon.com/glue/latest/dg/author-job.html https://docs.aws.amazon.com/glue/latest/dg/author-job.html * The default timeout is ~ 48 hours and you pay per Data Processing Unit (DPU) that you've provisioned the Job. * Currently it supports Python and Scala. As far as I'm aware you can't run Java jobs directly, but you can upload JAR libraries and use them in your code. Re serverless vs dedicated / ephemeral clusters: Like with any serverless runtime environment you are trading convenience (across a few dimensions) for flexibility. The Glue environment runs in a few limited runtimes and uses a specific version of Spark that you have no control over updating. Given that it's pretty quick to author a job, you can set the required DPU and Glue handles that, and you don't have to worry about sizing the cluster for your data size. For me most of my jobs fit within those constraints. At some point on the cost curve it may make sense for you to move all of your jobs from Glue into a dedicated cluster on EMR. You may also get there sooner if you need to use specific frameworks or libraries.

RangerScience 8y ago

Heya! I have been working on almost exactly the same thing for the last couple months :) I haven't had a chance to actually read your post yet, but I'm really looking forward to it.

RangerScience 8y ago
Okay, neat! Had a chance to skim. It's really cool that you have the Cloud Formation code up and accessible (as well as the Glue script). But, is the lamdba accessible from anywhere? (or did I just miss the link?) We used Terraform instead of CloudFormation, although there are a few places it didn't/doesn't cover. We're also doing protobuf -> parquet instead of JSON; Scala instead of Python, and one of our major feature/issues is that the incoming data is out-of-order; Firehose outputs to partitions based on when the event arrived, and we're repartitioning based on when the event occurred (according to a timestamp in the message) I see you ran into the capitalization issue too :) I ran across docs somewhere that said something along the lines of Glue downcasing column names, which definitely fits observed behavior. I'm hoping to turn what we've done into a similar post, although I think it'll be a month or two before I can get to that. Edit: Oh, I also got a lot of mileage out of the Zeppelin Notebooks. Way better than a raw dev endpoint, but watch out on the cost for both :) The notebooks can be provisioned with almost a single click from the Dev endpoint console, but they changed the recipe halfway through my work on it, and now you have to also SSH into the box and run a script to setup some of the security. :/ Still totally worth it, tho.
- iamsomewalrus 8y ago
  The Lambda is included in the DynamoDB exports CloudFormation templates. It's embedded in the file template itself. Thumbs up on the Glue Dev endpoint. It's been killer. I had trouble setting up a Notebook (I wanted to get fancy with Docker) and I usually use the Python repl link that's provided. I'm working on a follow up post that removes the Data Pipeline -> Lambda and uses the new Glue DynamoDB integration. Looking forward to your blog post!
  - RangerScience 8y ago

rapathak 8y ago

It's a great post. Thanks for taking the time to write it!

welder 8y ago

iamsomewalrus: Since you're using the same s3 key prefix for all dbexport data does that slow down your Athena queries because of key map partitioning? [1] Also, do you ever see 503 Slowdown errors from Athena requests to s3? [1]: https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/ https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-...