2 ms·
Compare pricing on RDS, if doing it yourself is hurting. AWS also has Kinesis, which is deliberately intended to be a sort of event drain. Under the hood it us
by jacques_chester 11y ago
Compare pricing on RDS, if doing it yourself is hurting.
AWS also has Kinesis, which is deliberately intended to be a sort of event drain. Under the hood it uses S3 and they publish an API and an agent that handles all the retry / checkpoint logic for you.
- josefdlange 11y agoThis. I know of a number of modern AWS and Amazon services that use Kinesis under-the-hood, from past internal experience (left a year ago, though, so there might be more, too!)
- takno 11y agoAbsolutely this. If you're in the AWS ecosystem then move away from the bare-ish metal and onto the tools they provide for doing this stuff. MongoDB in particular really doesn't seem to be relevant to the problem at all. Throw all the data into Kinesis, and then push the bits you need into Redshift the other end. If you want to keep a lot of data for ad-hoc queries but only use a subset for regular queries then consider setting up separate dense compute and dense storage Redshift clusters as appropriate. As far as pricing goes there's a hell of a discount for going reserved once you've got things set up the way you want them. Once you've got that running and the immediate fires are out you may want to look at Spark to pre-compute some of the reports (you can still push the computed data into Redshift tables to save changing the reporting too much), although I don't think spark as a service isn't really there in AWS right now, and running up an EMR cluster is seriously going to cost you. Bigtable provides similar capabilities to Redshift in the google ecosystem, and looks okay although I wasn't blown away by it.