13 ms·

AWS Kinesis with Lambdas: Lessons Learned

112 points by omn1 8y ago

cagenut 8y ago
The article doesn't appear to get into the end-destination of this data, it just says "to AWS". My initial thought is: restore last nights backup to another mysql instance on aws and then let it catchup on the binlog? But I guess the unstated assumption is that their goal is to also transform to some other datastore.
- toomuchtodo 8y ago
  You could also have replicas in AWS replicating off your on prem DBs in real time. Is Kafka offering something as the message bus/data transfer mechanism that replication and ETL off the AWS replica can’t?
  - thecopy 8y ago
    See Resume Driven Development: https://rdd.io/ https://rdd.io/
    - deleted 8y ago
      [deleted]
  - xenji_fm 8y ago
    Disclaimer: I work for trivago and I am partially responsible for the architecture behind trivago's Kafka use. I'd like to invite you to watch this talk and get a few more insights why we use Kafka in the ways we do: https://www.youtube.com/watch?v=cU0BCVl4bjo https://www.youtube.com/watch?v=cU0BCVl4bjo Let me try to come up with a TL;DR here: trivago comes from a complete on-premise, central database point of view. Change Data Capture via Debezium into Kafka enables a lot of migration strategies into different directions (e.g. Cloud) in the first place, while not having the need to change everything on the spot. It seems like a common pattern to compare Kafka with a pure MQ technolgy. Kafka can also serve as a persistent data storage and a source of truth for data. I hope this makes the picture a bit more clear to you. Feel free to ask if I missed something.
    - deleted 8y ago
      [deleted]
    - wenc 8y ago
      Thanks for heads up about Debezium. That is very cool. Debezium seems to be a production version of Martin Kleppmann's CDC-to-Kafka POC, Bottled Water [1]. Database replication is the killer app for CDC, but CDC can be used for so much more than replication, like event-based alerting, triggering, etc. [1] https://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/ https://www.confluent.io/blog/bottled-water-real-time-integr...
      - gunnarmorling 8y ago
        (Disclaimer: Debezium lead here) While the basic idea of using PG logical decoding for CDC is the same, Debezium is a completely different code base than Bottled Water. Also we provide connectors for a variety of databases (MySQL, Postgres, MongoDB; Oracle and MongoDB connectors are in the workings). If you like, you can also use Debezium independently of Kafka by embedding it as a library into your own application, e.g. if you don't need to persist change events or want to connect it to other streaming solutions than Kafka. In terms of CDC use cases, I keep seeing more and more the longer I work on it. Besides replication e.g. updates of full-text search indexes and caches, propagatating data between microservices, facilitating the extraction of microservices from monoliths (by streaming changes from writes to the old monoliths to new microservices), maintaining read models in CQRS architectures, life-updating UIs (by streaming data changes to Web Sockets clients) etc. I touch on a few in my Debezium talk (https://speakerdeck.com/gunnarmorling/data-streaming-for-microservices-using-debezium?slide=5 https://speakerdeck.com/gunnarmorling/data-streaming-for-mic...).
        abledon 8y ago
        This is why Hacker news is epic ... engineers reply includes technology, child reply is author of said technology
        wenc 8y ago
        Thanks for the explanation. Any plans to support SQL Server? (SQL Server is prevalent in the enterprise world)
- adzicg 8y ago
  The key benefit of using kinesis paired up it with lambda is controlled sequencing and parallelisation at the same time. Kinesis streams guarantee a single Lambda per shard iterator. For example, at MindMup, we’re using a similar setup to the one described in the article for appending events to user files. It’s critical that we don’t get two updates going over eachother in a single file, which is tricky to do with lambdas only because there’s no guarantee how many lambdas will kick off if updates come concurrently from different users for the same file. With Kinesis, we just use the file ID as the sharding key, so no more than a single lambda ever works on a single user file, but that we can have multiple lambdas in parallel working on different files.
  - djhworld 8y ago
    > in the article for appending events to user files How are you appending to files from Lambda? EFS? > With Kinesis, we just use the file ID as the sharding key, so no more than a single lambda ever works on a single user Is there a risk of "heavy" users causing hot shards?
- nikhilkuria 8y ago
  Your assumption is correct. One of our goals was to transform and restructure the data we have on our on-site premise MySQL. This is where the Lambdas come in. The Kinesis streams trigger the lambda, and the Lambdas filter and process the record and save them to a database on AWS.
sixdimensional 8y ago
I'm not being negative about this, it's a cool setup. But just to be clear, the pattern (not marketing terms) is doing change data capture (essentially the database transaction log) to a message queue, with message/job processors that can take any action, including writing the messages to other databases. Kinesis is SQLStream underneath, which is probably why the lifetime of messages is limited - it's not originally intended to be Kafka or a durable message queue. EDIT: Note above, when SQLStream first came out it didn't seem intended as a long term store. That was like really early on when I saw it at Strata. It looks like they made the storage engine pluggable and Kafka is an option too, so my statement above is likely incorrect. Lambda is being used as a distributed message/job processor, much like any worker process processing a queue would be scaled up.
- otterley 8y ago
  > Kinesis is SQLStream underneath That's the first I've heard of this. Any citations you can provide to substantiate this claim? My understanding is that SQLStream is an event streaming processor, which would make it a potential Kafka consumer, not a basis for a durable message queue.
  - sixdimensional 8y ago
    Amazon Web Services Has Licensed SQLstream Technology for Amazon ... https://sqlstream.com/amazon-web-services-has-licensed-sqlstream-technology-for-amazon-kinesis-analytics-service-launched-today/ https://sqlstream.com/amazon-web-services-has-licensed-sqlst...
    - ryanworl 8y ago
      Kinesis Analytics uses Kinesis Streams as a data source. They are not the same service.
      - sixdimensional 8y ago
        I see your point, although I sense they are quite closely integrated? Additional info.. https://www.datanami.com/2016/08/17/sql-service-heart-amazon-kinesis-analytics/ https://www.datanami.com/2016/08/17/sql-service-heart-amazon...
otterley 8y ago
I'm curious about the economics of this design. Real-time stream consumption implies that the consumer is always running, and if you need to run software 24x7, running it on EC2 instances is likely to be far cheaper than running Lambda functions continuously.
- xenji_fm 8y ago
  Depending on the requirements, one can run the sink (we use Kafka Connect for bridging between Kafka and Kinesis) on-premise or as Docker container in ecs or on ec2. That gives you a lot of flexibility to optimize the parameters like costs, throughput, liveness, etc. Edit: fixed a comma.
- teej 8y ago
  The lambdas are only triggered when there is data in the stream, so the "consumer" in this case is not always running. Under the hood there is a process that polls Kinesis to check for records but it's completely managed by AWS and you aren't charged for it. Plus, in any real world scenario, if you're running 24x7 your load isn't evenly distributed throughout the day. Which means you're setting up autoscaling (w/ added time+complexity) or provisioning for peak, wiping out your cost savings. In my experience if your data volumes are low enough like in the article, a Kinesis+Lambda setup is stress free and quick to implement. That makes it worth the cost over raw instances.
  - djhworld 8y ago
    It also depends on what your business problem is. If it's just doing simple ETL then it's probably OK, but if you need to do aggregations on the data, you're going to have a bad time, or end up implementing some sort of ersatz map-reduce framework in lambda.
- nikhilkuria 8y ago
  Yeah, it all depends on how much load or incoming records you expect. In our case, we use the pipeline to import inventory, say images from from our partners. So, we might have a few million images coming in many times a month. Lambda is super efficient in this case.
mpd 8y ago
With so much overlap in the functionality and use cases of Kafka and Kinesis, it's not clear why they increase their surface area by using both. Is Kinesis' write latency better than it was? IIRC it wrote to 3 data centers synchronously, which led to some pretty bad performance. This was almost 2 years ago though.
- xenji_fm 8y ago
  Kinesis works better for people that have opt'ed in for the whole AWS toolbox, esp. when you do not want to maintain your own Kafka cluster on ec2. Unfortunately, I cannot say anything about the write latency, because I have no in-depth knowledge about the inner workings and what is "fast enough" for them.
- nikhilkuria 8y ago
  Exactly. The biggest motivation to use Kinesis was the other services we could plug in from 'AWS ecosystem'. Lambdas have Kinesis triggers available natively. We did not have any concern with the latency of Kinesis. Now I'm curious, do you have any literature on the bad performance history?
  - mpd 8y ago
    Just past experience. In the end, the low data retention time was the real reason Kinesis didn't fit our needs, but the difference in write performance between the two was pretty stark. Read performance was fine for what we needed.
stlava 8y ago
The post is good but just scratches the surface on running Kinesis Streams / Lambda at scale. Here are a few additional things I found while running Kinesis as a data ingestion pipeline: - Only write logs out that matter. Searching logs in cloudwatch is already a major PITA. Half the time I just scan the logs manually because search never returns. Also, the fewer println statements you have the quicker your function will be. - Lambda is cheap, reporting function metrics to cloudwatch from a lambda is not. Be very careful about using this. - Having metrics from within your lambda is very helpful. We keep track of spout lag (delta of when event got to kineis and when it was read by the lambda), source lag (delta of when the event was emitted and when it was read by the lambda), number of events processed (were any dropped due to validation errors?). - Avoid using the kinesis auto scaler tool. In theory it's a great idea but in practice we found that scaling a stream with 60+ shards causes issues with api limits. (maybe this is fixed now...) - Have plenty of disk space on whatever is emitting logs. You don't want to run into the scenario where you can't push logs to kinesis (eg throttling) and they start filling up your disks. - Keep in mind that you have to balance our emitters, lambda, and your downstream targets. You don't want too few / too many shards. You don't want to have 100 lambda instances hitting a service with 10 events each invocation. - Lambda deployment tools are still young but find one that works for you. All of them have tradeoffs in how they are configured and how they deploy. There are some good tidbits in the Q&A section from my re:Invent talk [1]. Also, for anyone wanting to use lambda but not wanting to re-invent checkout Bender [2]. Note I'm the author. [1] https://www.youtube.com/watch?v=AaRawf9vcZ4 https://www.youtube.com/watch?v=AaRawf9vcZ4 [2] https://github.com/Nextdoor/bender https://github.com/Nextdoor/bender edit: formatting
- djhworld 8y ago
  For metrics we find writing a well defined, formatted message to stdout* and then using Cloudwatch Logs Metric filters works pretty well. * e.g. "ETLLambda METRIC RECORDS_IN 948575"
  - phillipwills 8y ago
    I used the aws sdk to write a simple NodeJS script to pull logs for a date range, then I can grep the output...
dbllxr 8y ago
> For us, increasing the memory for a Lambda from 128 megabytes to 2.5 gigabytes gave us a huge boost. > The number of Lambda invocations shot up almost 40x. One thing I've learned from talking to AWS support is that increasing memory also gets you more vCPUs per container. ----- Serverless is great in scaling and handling bursts, but you may find it VERY difficult in terms of testing and debugging. A while back I started using an open source tool called localstack[1] to mirror some AWS services locally. Despite some small discrepancies in certain APIs (which are totally expected), it's made testing a lot easier for me. Something worth looking into if testing serverless code is causing you headaches. [1]https://github.com/localstack/localstack https://github.com/localstack/localstack
- nikhilkuria 8y ago
  Thanks! For testing on our local machines, we use SAM Local, https://aws.amazon.com/blogs/aws/new-aws-sam-local-beta-build-and-test-serverless-applications-locally/ https://aws.amazon.com/blogs/aws/new-aws-sam-local-beta-buil... . This has very similar capabilities as local stack. Still, there is always a delay from a service being released by AWS to it being available on tools like Serverless or AWS local.
djhworld 8y ago
We used to run an architecture similar to this a few years ago, I work for a broadcaster and unfortunately it failed badly during a big event. The Kinesis stream was adequately scaled, but the poller between Kinesis -> Lambda just couldn't cope. This was discovered after lots of support calls with AWS. It might be better these days I don't know, we moved to using Apache Flink + Apache Beam, which has a lot more features and allows us to do stuff like grouping by a window, aggregation etc.
abledon 8y ago
Anyone else reading this post with the trivago repetitive jingle on loop in the back of their mind? Din dan din dan dan —
sheeshkebab 8y ago
Copying a billion records screams for streaming? do you mean like a couple of hundred gigs that could be copied overnight (and maybe transformed with some script)?
- nikhilkuria 8y ago
  Rather than doing a one-time copy of the data, we have applications continually writing to our on-premise MySQL. This data has to reactively reach our datastore on AWS. Also, this is not a 1-1 copy. There are several transformations and cross-validations in play.
dmlittle 8y ago
> For us, increasing the memory for a Lambda from 128 megabytes to 2.5 gigabytes gave us a huge boost. I thought the maximum memory limit for Lamda was 3008 MB and that you couldn't bump this limit through a service request. Anyone knows if you can request to bump the memory limit or the uncompressed deployment package limit (250 MB)?
- dmlittle 8y ago
  And here I am realizing the stupidity of my question... (3008 MB is close to 3GB)

112 points by omn1 8y ago

The article doesn't appear to get into the end-destination of this data, it just says "to AWS". My initial thought is: restore last nights backup to another mysql instance on aws and then let it catchup on the binlog? But I guess the unstated assumption is that their goal is to also transform to some other datastore.

toomuchtodo 8y ago
You could also have replicas in AWS replicating off your on prem DBs in real time. Is Kafka offering something as the message bus/data transfer mechanism that replication and ETL off the AWS replica can’t?
- thecopy 8y ago
  See Resume Driven Development: https://rdd.io/ https://rdd.io/
  - deleted 8y ago
    [deleted]
- xenji_fm 8y ago
  Disclaimer: I work for trivago and I am partially responsible for the architecture behind trivago's Kafka use. I'd like to invite you to watch this talk and get a few more insights why we use Kafka in the ways we do: https://www.youtube.com/watch?v=cU0BCVl4bjo https://www.youtube.com/watch?v=cU0BCVl4bjo Let me try to come up with a TL;DR here: trivago comes from a complete on-premise, central database point of view. Change Data Capture via Debezium into Kafka enables a lot of migration strategies into different directions (e.g. Cloud) in the first place, while not having the need to change everything on the spot. It seems like a common pattern to compare Kafka with a pure MQ technolgy. Kafka can also serve as a persistent data storage and a source of truth for data. I hope this makes the picture a bit more clear to you. Feel free to ask if I missed something.
  - deleted 8y ago
    [deleted]
  - wenc 8y ago
    Thanks for heads up about Debezium. That is very cool. Debezium seems to be a production version of Martin Kleppmann's CDC-to-Kafka POC, Bottled Water [1]. Database replication is the killer app for CDC, but CDC can be used for so much more than replication, like event-based alerting, triggering, etc. [1] https://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/ https://www.confluent.io/blog/bottled-water-real-time-integr...
    - gunnarmorling 8y ago
      (Disclaimer: Debezium lead here) While the basic idea of using PG logical decoding for CDC is the same, Debezium is a completely different code base than Bottled Water. Also we provide connectors for a variety of databases (MySQL, Postgres, MongoDB; Oracle and MongoDB connectors are in the workings). If you like, you can also use Debezium independently of Kafka by embedding it as a library into your own application, e.g. if you don't need to persist change events or want to connect it to other streaming solutions than Kafka. In terms of CDC use cases, I keep seeing more and more the longer I work on it. Besides replication e.g. updates of full-text search indexes and caches, propagatating data between microservices, facilitating the extraction of microservices from monoliths (by streaming changes from writes to the old monoliths to new microservices), maintaining read models in CQRS architectures, life-updating UIs (by streaming data changes to Web Sockets clients) etc. I touch on a few in my Debezium talk (https://speakerdeck.com/gunnarmorling/data-streaming-for-microservices-using-debezium?slide=5 https://speakerdeck.com/gunnarmorling/data-streaming-for-mic...).
      - abledon 8y ago
        This is why Hacker news is epic ... engineers reply includes technology, child reply is author of said technology
      - wenc 8y ago
        Thanks for the explanation. Any plans to support SQL Server? (SQL Server is prevalent in the enterprise world)
adzicg 8y ago
The key benefit of using kinesis paired up it with lambda is controlled sequencing and parallelisation at the same time. Kinesis streams guarantee a single Lambda per shard iterator. For example, at MindMup, we’re using a similar setup to the one described in the article for appending events to user files. It’s critical that we don’t get two updates going over eachother in a single file, which is tricky to do with lambdas only because there’s no guarantee how many lambdas will kick off if updates come concurrently from different users for the same file. With Kinesis, we just use the file ID as the sharding key, so no more than a single lambda ever works on a single user file, but that we can have multiple lambdas in parallel working on different files.
- djhworld 8y ago
  > in the article for appending events to user files How are you appending to files from Lambda? EFS? > With Kinesis, we just use the file ID as the sharding key, so no more than a single lambda ever works on a single user Is there a risk of "heavy" users causing hot shards?
nikhilkuria 8y ago
Your assumption is correct. One of our goals was to transform and restructure the data we have on our on-site premise MySQL. This is where the Lambdas come in. The Kinesis streams trigger the lambda, and the Lambdas filter and process the record and save them to a database on AWS.

sixdimensional 8y ago

I'm not being negative about this, it's a cool setup. But just to be clear, the pattern (not marketing terms) is doing change data capture (essentially the database transaction log) to a message queue, with message/job processors that can take any action, including writing the messages to other databases. Kinesis is SQLStream underneath, which is probably why the lifetime of messages is limited - it's not originally intended to be Kafka or a durable message queue. EDIT: Note above, when SQLStream first came out it didn't seem intended as a long term store. That was like really early on when I saw it at Strata. It looks like they made the storage engine pluggable and Kafka is an option too, so my statement above is likely incorrect. Lambda is being used as a distributed message/job processor, much like any worker process processing a queue would be scaled up.

otterley 8y ago
> Kinesis is SQLStream underneath That's the first I've heard of this. Any citations you can provide to substantiate this claim? My understanding is that SQLStream is an event streaming processor, which would make it a potential Kafka consumer, not a basis for a durable message queue.
- sixdimensional 8y ago
  Amazon Web Services Has Licensed SQLstream Technology for Amazon ... https://sqlstream.com/amazon-web-services-has-licensed-sqlstream-technology-for-amazon-kinesis-analytics-service-launched-today/ https://sqlstream.com/amazon-web-services-has-licensed-sqlst...
  - ryanworl 8y ago
    Kinesis Analytics uses Kinesis Streams as a data source. They are not the same service.
    - sixdimensional 8y ago
      I see your point, although I sense they are quite closely integrated? Additional info.. https://www.datanami.com/2016/08/17/sql-service-heart-amazon-kinesis-analytics/ https://www.datanami.com/2016/08/17/sql-service-heart-amazon...

otterley 8y ago

I'm curious about the economics of this design. Real-time stream consumption implies that the consumer is always running, and if you need to run software 24x7, running it on EC2 instances is likely to be far cheaper than running Lambda functions continuously.

xenji_fm 8y ago
Depending on the requirements, one can run the sink (we use Kafka Connect for bridging between Kafka and Kinesis) on-premise or as Docker container in ecs or on ec2. That gives you a lot of flexibility to optimize the parameters like costs, throughput, liveness, etc. Edit: fixed a comma.
teej 8y ago
The lambdas are only triggered when there is data in the stream, so the "consumer" in this case is not always running. Under the hood there is a process that polls Kinesis to check for records but it's completely managed by AWS and you aren't charged for it. Plus, in any real world scenario, if you're running 24x7 your load isn't evenly distributed throughout the day. Which means you're setting up autoscaling (w/ added time+complexity) or provisioning for peak, wiping out your cost savings. In my experience if your data volumes are low enough like in the article, a Kinesis+Lambda setup is stress free and quick to implement. That makes it worth the cost over raw instances.
- djhworld 8y ago
  It also depends on what your business problem is. If it's just doing simple ETL then it's probably OK, but if you need to do aggregations on the data, you're going to have a bad time, or end up implementing some sort of ersatz map-reduce framework in lambda.
nikhilkuria 8y ago
Yeah, it all depends on how much load or incoming records you expect. In our case, we use the pipeline to import inventory, say images from from our partners. So, we might have a few million images coming in many times a month. Lambda is super efficient in this case.

mpd 8y ago

With so much overlap in the functionality and use cases of Kafka and Kinesis, it's not clear why they increase their surface area by using both. Is Kinesis' write latency better than it was? IIRC it wrote to 3 data centers synchronously, which led to some pretty bad performance. This was almost 2 years ago though.

xenji_fm 8y ago
Kinesis works better for people that have opt'ed in for the whole AWS toolbox, esp. when you do not want to maintain your own Kafka cluster on ec2. Unfortunately, I cannot say anything about the write latency, because I have no in-depth knowledge about the inner workings and what is "fast enough" for them.
nikhilkuria 8y ago
Exactly. The biggest motivation to use Kinesis was the other services we could plug in from 'AWS ecosystem'. Lambdas have Kinesis triggers available natively. We did not have any concern with the latency of Kinesis. Now I'm curious, do you have any literature on the bad performance history?
- mpd 8y ago
  Just past experience. In the end, the low data retention time was the real reason Kinesis didn't fit our needs, but the difference in write performance between the two was pretty stark. Read performance was fine for what we needed.

stlava 8y ago

The post is good but just scratches the surface on running Kinesis Streams / Lambda at scale. Here are a few additional things I found while running Kinesis as a data ingestion pipeline: - Only write logs out that matter. Searching logs in cloudwatch is already a major PITA. Half the time I just scan the logs manually because search never returns. Also, the fewer println statements you have the quicker your function will be. - Lambda is cheap, reporting function metrics to cloudwatch from a lambda is not. Be very careful about using this. - Having metrics from within your lambda is very helpful. We keep track of spout lag (delta of when event got to kineis and when it was read by the lambda), source lag (delta of when the event was emitted and when it was read by the lambda), number of events processed (were any dropped due to validation errors?). - Avoid using the kinesis auto scaler tool. In theory it's a great idea but in practice we found that scaling a stream with 60+ shards causes issues with api limits. (maybe this is fixed now...) - Have plenty of disk space on whatever is emitting logs. You don't want to run into the scenario where you can't push logs to kinesis (eg throttling) and they start filling up your disks. - Keep in mind that you have to balance our emitters, lambda, and your downstream targets. You don't want too few / too many shards. You don't want to have 100 lambda instances hitting a service with 10 events each invocation. - Lambda deployment tools are still young but find one that works for you. All of them have tradeoffs in how they are configured and how they deploy. There are some good tidbits in the Q&A section from my re:Invent talk [1]. Also, for anyone wanting to use lambda but not wanting to re-invent checkout Bender [2]. Note I'm the author. [1] https://www.youtube.com/watch?v=AaRawf9vcZ4 https://www.youtube.com/watch?v=AaRawf9vcZ4 [2] https://github.com/Nextdoor/bender https://github.com/Nextdoor/bender edit: formatting

djhworld 8y ago
For metrics we find writing a well defined, formatted message to stdout* and then using Cloudwatch Logs Metric filters works pretty well. * e.g. "ETLLambda METRIC RECORDS_IN 948575"
- phillipwills 8y ago
  I used the aws sdk to write a simple NodeJS script to pull logs for a date range, then I can grep the output...

dbllxr 8y ago

> For us, increasing the memory for a Lambda from 128 megabytes to 2.5 gigabytes gave us a huge boost. > The number of Lambda invocations shot up almost 40x. One thing I've learned from talking to AWS support is that increasing memory also gets you more vCPUs per container. ----- Serverless is great in scaling and handling bursts, but you may find it VERY difficult in terms of testing and debugging. A while back I started using an open source tool called localstack[1] to mirror some AWS services locally. Despite some small discrepancies in certain APIs (which are totally expected), it's made testing a lot easier for me. Something worth looking into if testing serverless code is causing you headaches. [1]https://github.com/localstack/localstack https://github.com/localstack/localstack

nikhilkuria 8y ago
Thanks! For testing on our local machines, we use SAM Local, https://aws.amazon.com/blogs/aws/new-aws-sam-local-beta-build-and-test-serverless-applications-locally/ https://aws.amazon.com/blogs/aws/new-aws-sam-local-beta-buil... . This has very similar capabilities as local stack. Still, there is always a delay from a service being released by AWS to it being available on tools like Serverless or AWS local.

djhworld 8y ago

We used to run an architecture similar to this a few years ago, I work for a broadcaster and unfortunately it failed badly during a big event. The Kinesis stream was adequately scaled, but the poller between Kinesis -> Lambda just couldn't cope. This was discovered after lots of support calls with AWS. It might be better these days I don't know, we moved to using Apache Flink + Apache Beam, which has a lot more features and allows us to do stuff like grouping by a window, aggregation etc.

abledon 8y ago

Anyone else reading this post with the trivago repetitive jingle on loop in the back of their mind? Din dan din dan dan —

sheeshkebab 8y ago

Copying a billion records screams for streaming? do you mean like a couple of hundred gigs that could be copied overnight (and maybe transformed with some script)?

nikhilkuria 8y ago
Rather than doing a one-time copy of the data, we have applications continually writing to our on-premise MySQL. This data has to reactively reach our datastore on AWS. Also, this is not a 1-1 copy. There are several transformations and cross-validations in play.

dmlittle 8y ago

> For us, increasing the memory for a Lambda from 128 megabytes to 2.5 gigabytes gave us a huge boost. I thought the maximum memory limit for Lamda was 3008 MB and that you couldn't bump this limit through a service request. Anyone knows if you can request to bump the memory limit or the uncompressed deployment package limit (250 MB)?

dmlittle 8y ago
And here I am realizing the stupidity of my question... (3008 MB is close to 3GB)