How fixed are your requirements for human readability and ~1MB per file? If you relax those, a data lake formatted in Delta or Iceberg is exactly what you're a

How fixed are your requirements for human readability and ~1MB per file? If you relax those, a data lake formatted in Delta or Iceberg is exactly what you're asking for.