5 ms·

Do you have a reference documentation for S3 data layout?

• richieartoul 3y ago
  (WarpStream founder here) Not currently. One of the things we're looking to do next is make it so any topic can be "automatically" turned into a standard format in S3, something like Parquet/Iceberg/Deltalake so its easier to consume for application that don't particularly care about the Kafka protocol.
  • EamonnMR 3y ago
    That would be a killer feature. Once we switched to MSK back at a previous gig, our biggest care and feeding task was adding new event types to our kafka-to-redshift ETL thing. Well, that and dealing with scaling our http wrapper...
  • bothra90 3y ago
    That would be awesome!