4 ms·

We need binary formats. In this era we are capable for it. Throw away the text formats.

deleted 1y ago
[deleted]
bsder 1y ago
Data will always outlive the program that originally produced it. This is why you should almost always use text formats.
account-5 1y ago
Text is universally accessible and widely supported. Binary has it's benefits, but human facing, it has to be text.
voidfunc 1y ago
Yep, and stuff it into a sqlite db too you have a query interface all built.
- culi 1y ago
  What's a good program that non-technical people can use to write sqlite db data. I think it's a great idea in theory but lacking in support
rr808 1y ago
parquet?
smallerize 1y ago
Ok but how do you type them? How do you search them? How do you copy-and-paste between documents?
- Hashex129542 1y ago
  We are just changing the encoders and decoder program only from text to binary. The front end software remains same for example we are using for CSV. The binary form has lot of benefits than plain text form in editing. For example, when you are replacing UIn8 value from 0 to 100 then you just replacing a byte at a position instead of rewriting whole document.
zzo38computer 1y ago
Which formats are helpful can depend on the use. I think DER (which is a binary format) is not so bad (although I added a few additional types (such as key/value list, BCD string, and TRON string), but not all uses are required to use them). I had also made up Multi-DER, which is simply any number of DER concatenated together (there are formats of JSON like that too). (I had also made up TER which is a text format and a program to convert TER to DER. I also wrote a program to convert JSON to DER. It would also be possible to convert CSV, etc.) It was also my idea of an operating system design, it will have a binary format used for most stuff, similar to DER but different in some ways (including different types are available), which is intended to be interoperable among most of the programs on the system.
- Hashex129542 1y ago
  Same Vibe. Yes I am using the same DER files in my apps. So we can have more distinguished universal value types than just a text. My very next step is OS development too but I'm not sure where to learn the OS in the opcode coding level. I thought to get started with Intel docs for my CPU.
baby_souffle 1y ago
> We need binary formats. In this era we are capable for it. We have them, they're used where appropriate. > Throw away the text formats. I would argue that _most_ of the time tsv or csv are used it's because either: a) the lowest common denominator for interchange. Oh, you don't have _my specific version of $program? How about I give you the data in csv? _everything_ can read that... b) a human is expected to inspect/view/adjust the data and they'd be using a bin -> text tool anyways. The move to binary based log formats (`journald`) is still somewhat controversial. It would have been a non-starter if the tooling to make the binary "readable" wasn't on-par with (or, in a few cases, better!) than the contemporary text based tooling we've been used to for the prior 30+ years..
chthonicdaemon 1y ago
The idea that binary formats are the way because "you're going to use a program to interact with the format anyway" ignores the network effects of having things like text editors and unix commands that handle text as a universal intermediate, while having bespoke programs for every format dooms you to developing a full set of tooling for every format (or more likely, writing code that converts the binary format to text formats). More recently though, consider that LLMs are terrible at emitting binary files, but amazing at emitting text. I can have a GPT spit out a nice diagram in Mermaid, or create calendar entries from a photo of an event program in ical format.
ndsipa_pomu 1y ago
Binary formats are generally poorly defined, buggy as hell and have limited tooling that can deal with them. Also, they're not human readable so when you inevitably hit a problem with a binary format, you can't just eyeball it to see that you've got an "O'Connor" in the names.

deleted 1y ago

[deleted]

Data will always outlive the program that originally produced it. This is why you should almost always use text formats.

account-5 1y ago

Text is universally accessible and widely supported. Binary has it's benefits, but human facing, it has to be text.

voidfunc 1y ago

Yep, and stuff it into a sqlite db too you have a query interface all built.

culi 1y ago
What's a good program that non-technical people can use to write sqlite db data. I think it's a great idea in theory but lacking in support

rr808 1y ago

parquet?

smallerize 1y ago

Ok but how do you type them? How do you search them? How do you copy-and-paste between documents?

Hashex129542 1y ago
We are just changing the encoders and decoder program only from text to binary. The front end software remains same for example we are using for CSV. The binary form has lot of benefits than plain text form in editing. For example, when you are replacing UIn8 value from 0 to 100 then you just replacing a byte at a position instead of rewriting whole document.

zzo38computer 1y ago

Which formats are helpful can depend on the use. I think DER (which is a binary format) is not so bad (although I added a few additional types (such as key/value list, BCD string, and TRON string), but not all uses are required to use them). I had also made up Multi-DER, which is simply any number of DER concatenated together (there are formats of JSON like that too). (I had also made up TER which is a text format and a program to convert TER to DER. I also wrote a program to convert JSON to DER. It would also be possible to convert CSV, etc.) It was also my idea of an operating system design, it will have a binary format used for most stuff, similar to DER but different in some ways (including different types are available), which is intended to be interoperable among most of the programs on the system.

Hashex129542 1y ago
Same Vibe. Yes I am using the same DER files in my apps. So we can have more distinguished universal value types than just a text. My very next step is OS development too but I'm not sure where to learn the OS in the opcode coding level. I thought to get started with Intel docs for my CPU.

baby_souffle 1y ago

> We need binary formats. In this era we are capable for it. We have them, they're used where appropriate. > Throw away the text formats. I would argue that _most_ of the time tsv or csv are used it's because either: a) the lowest common denominator for interchange. Oh, you don't have _my specific version of $program? How about I give you the data in csv? _everything_ can read that... b) a human is expected to inspect/view/adjust the data and they'd be using a bin -> text tool anyways. The move to binary based log formats (`journald`) is still somewhat controversial. It would have been a non-starter if the tooling to make the binary "readable" wasn't on-par with (or, in a few cases, better!) than the contemporary text based tooling we've been used to for the prior 30+ years..

chthonicdaemon 1y ago

The idea that binary formats are the way because "you're going to use a program to interact with the format anyway" ignores the network effects of having things like text editors and unix commands that handle text as a universal intermediate, while having bespoke programs for every format dooms you to developing a full set of tooling for every format (or more likely, writing code that converts the binary format to text formats). More recently though, consider that LLMs are terrible at emitting binary files, but amazing at emitting text. I can have a GPT spit out a nice diagram in Mermaid, or create calendar entries from a photo of an event program in ical format.

ndsipa_pomu 1y ago

Binary formats are generally poorly defined, buggy as hell and have limited tooling that can deal with them. Also, they're not human readable so when you inevitably hit a problem with a binary format, you can't just eyeball it to see that you've got an "O'Connor" in the names.