3 ms·
> TSV is a terrible format for multi-gigabyte files, because it uses line breaks. Put it into a database if you care about these things. But let's not reinven
by zozbot234 4y ago
> TSV is a terrible format for multi-gigabyte files, because it uses line breaks.
Put it into a database if you care about these things. But let's not reinvent the wheel, and let's not bikeshed binary file formats, of all things. It's a non-starter. Plain text is perfectly fine for long-term storage and interop, converting to a more effective representation is just a one-time cost.
- exmadscientist 4y agoA lot of data set sizes are best measured in terabytes and petabytes. Many of these file formats must target that mentality, because small things grow up and get big whether anyone planned for that or not. People who can get away with plaintext for their needs probably aren't even reading this.