3 ms·
It’s strange that we pretend text is not binary. The truth is our many tools are set up to handle binary text data and these tools are not set up for alternate
by twobitshifter 5y ago
It’s strange that we pretend text is not binary. The truth is our many tools are set up to handle binary text data and these tools are not set up for alternate encodings.
If you grab a text file from a Windows machine and bring it to a mac, you’ll see that txt is far from perfect.
This is a long way of saying that if we develop both the format and the tooling then the distinction of text vs “binary” tabular data goes away.
- hermitcrab 5y agoFair point. But the vast majority of programming tools do already handle UFT8 text (although perhaps don't do a great job with US/RS characters). also you could write an efficient parser for US/RS separated data in 5 minutes. For parquet you would have to integrate with a library (with all it's dependencies and design choices) or spend days writing your own parser.
- willis936 5y agoText is text. It's stored as bits, typically 8 bits per character. It allows arbitrary precision but is very space (and memory throughput) inefficient. Fine, compression works but there are many more bits flying around and you're spending CPU cycles on decompression. What could you do instead? Use appropriate integers, fixed point, or floating point representations. Everybody here knows these things. Nobody is pretending anything. To the point of portability: IEEE-754 is the same everywhere.