3 ms·
If every value is always UTF-8, then you can't embed arbitrary binary, since arbitrary bytes aren't necessarily valid UTF-8.
by recursive 4y ago
If every value is always UTF-8, then you can't embed arbitrary binary, since arbitrary bytes aren't necessarily valid UTF-8.
- tantalor 4y agoSure you can, here is example: 0101010010000111101010101
- recursive 4y agoThat's just text. In one sense, yes, it's arbitrary bits. But it's also clearly text. It's probably not what was referred to. If you have binary data that's encoded as text, it seems obvious that that can be embedded in text. It probably wouldn't be worth mentioning that "Any binary allowed between the quotes."
- mbreese 4y agoWe’re talking about a tabular data file format. If you want to include arbitrary binary, use a binary data file. Or base64 encoded data. Most datasets you’d use data like this for are small enough to fit into memory, so let’s not get carried away. (I happen to use tab delimited files to store data that can’t fit into memory, but that’s okay too)
- recursive 4y agoYes. I think we're agreeing. I was responding to this "Any binary allowed between the quotes.". Binary data can't generally be dropped directly into a text format without some kind of organized encoding.
- mbreese 4y agoYeah, I think so… I thought they meant using a quote as a flag for “binary data lies ahead”, which really seemed odd to me. But — it is completely possible in a custom file type. But yes, if this case, the entire file wouldn’t be UTF8, even if all of the non-quotes data would be. In retrospect, the idea of random binary data enclosed in quotes is what I’m mainly responding to — which I think we can all agree is a bad idea. (If you need to do that, encode it!)