If the big issue is querying the data, consider redshift (expensive) or a self-hosted column-store database. Data will need to be loaded in a batch for this. Co

If the big issue is querying the data, consider redshift (expensive) or a self-hosted column-store database. Data will need to be loaded in a batch for this. Column stores will reduce iops through compression, and selective data loading, and because they don't have persistent indexes. To save IOPS on the early part of the process, consider using fast encryption (lz4 or snappy) to compress the records before writing to the file system. This might cut your IOPS in half.