3 ms·

I have a question: if I were to send say 10000 event objects into a Amazon Kinesis Firehose stream it's clear that I should expect them to show up in an S3 buc

I have a question: if I were to send say 10000 event objects into a Amazon Kinesis Firehose stream it's clear that I should expect them to show up in an S3 bucket of my choosing, but should I also expect that my account will not incur any S3 HTTP POST API request fees ? Is dodging those HTTP POST fees the value-add over simply using the S3 HTTP API yourself ?

mkobit 11y ago
Unless I am reading it wrong, it sounds like you need to pay for the requests as well. From https://aws.amazon.com/kinesis/firehose/pricing/ https://aws.amazon.com/kinesis/firehose/pricing/: > Storage > You will be billed separately for charges associated with Amazon S3 and Amazon Redshift usage including storage and read/write requests. However, you will not be billed for data transfer charges for the data that Amazon Kinesis Firehose loads into Amazon S3 and Amazon Redshift. For further details, see Amazon S3 pricing and Amazon Redshift pricing.
- solofounder1 11y ago
  To me this sounds like a way to avoid the hassle of creating a lambda to buffer streaming data into chunks before HTTP POSTing them to the S3 API. But to me it wouldn't make much sense to use Kinesis Firehose unless the fees were cheaper than what it would cost to utilize AWS Lamdba for the same work. I mean it can't be all that many lines of nodejs code to pop events off a stream, flush them into a tempfile in batches, HTTP POST those batches to S3 with appropriate error handling/retry logic. Admittedly I haven't crunched the math but just by eyeballing their pricing I suspect it may be more expensive than using your own lamdba. I wonder if Kinesis Firehose is implemented internally as an AWS Lambda, it wouldn't surprise me. I suppose that eventually the pricing on this service will drop once someone open sources such a lamdba, especially since installing a lambda is so remarkably simple.
  - jhurliman 11y ago
    The only complication with this approach right now is that Lambda scripts can't communicate with anything inside a VPC (such as most Redshift instances). I imagine they will fix this issue in the future though.

mkobit 11y ago

Unless I am reading it wrong, it sounds like you need to pay for the requests as well. From https://aws.amazon.com/kinesis/firehose/pricing/ https://aws.amazon.com/kinesis/firehose/pricing/: > Storage > You will be billed separately for charges associated with Amazon S3 and Amazon Redshift usage including storage and read/write requests. However, you will not be billed for data transfer charges for the data that Amazon Kinesis Firehose loads into Amazon S3 and Amazon Redshift. For further details, see Amazon S3 pricing and Amazon Redshift pricing.

solofounder1 11y ago
To me this sounds like a way to avoid the hassle of creating a lambda to buffer streaming data into chunks before HTTP POSTing them to the S3 API. But to me it wouldn't make much sense to use Kinesis Firehose unless the fees were cheaper than what it would cost to utilize AWS Lamdba for the same work. I mean it can't be all that many lines of nodejs code to pop events off a stream, flush them into a tempfile in batches, HTTP POST those batches to S3 with appropriate error handling/retry logic. Admittedly I haven't crunched the math but just by eyeballing their pricing I suspect it may be more expensive than using your own lamdba. I wonder if Kinesis Firehose is implemented internally as an AWS Lambda, it wouldn't surprise me. I suppose that eventually the pricing on this service will drop once someone open sources such a lamdba, especially since installing a lambda is so remarkably simple.
- jhurliman 11y ago
  The only complication with this approach right now is that Lambda scripts can't communicate with anything inside a VPC (such as most Redshift instances). I imagine they will fix this issue in the future though.