Entirely untested and just spitballing... use Firehose to concat a bunch of small files and store based on size or time passage as a larger file to s3. Then us

Entirely untested and just spitballing... use Firehose to concat a bunch of small files and store based on size or time passage as a larger file to s3. Then use Lambda transform in firehose to index the byte ranges for each record (file)? (Biggest q in the process would be here... you might have to index after it hits s3) S3 also allows the Range request header out of the box as well. Interesting problem, and I can totally related with s3 put costs be astronomical.