I've been experimenting with taking this self-description paradigm even farther, for a file format I've cooked up for ephemeral data in my search engine. Basic

I've been experimenting with taking this self-description paradigm even farther, for a file format I've cooked up for ephemeral data in my search engine. Basically, since I ended up building a custom library for this, I wanted to solve the portability problem by making it stupidly simple to reverse engineer, so I cooked up a convention where each column (and supporting column) is a file, with a file name that describes its format and role. So a real-world production table looks like this if you ls in the directory (omitting a few columns for brevity): combinedId.0.dat.s64le.bin documentMeta.0.dat.s64le.bin features.0.dat.s32le.bin size.0.dat.s32le.bin termIds.0.dat-len.varint.bin termIds.0.dat.s64le[].zstd termMetadata.0.dat-len.varint.bin termMetadata.0.dat.s8[].zstd The design goal is that just based on an ls output, someone who has never seen the code of the library producing the files should be able to trivially write code that reads it.