4 ms·

Structured text is good. Very good, in fact. It might even be idea. structured binary data, less so, at least as a storage format. I want to be able to look at

Structured text is good. Very good, in fact. It might even be idea. structured binary data, less so, at least as a storage format. I want to be able to look at your file format using tools that haven't been specialized to the task. Is that so wrong?

dagw 10y ago
Parsing structured text is slow and inefficient. Also reading out just one part of a data structure stored as text often requires either walking through the file character by character or first slurping the whole thing into memory.
- qwertyuiop924 10y ago
  ...But when your system crashes, having all that data in an easily accessible manner (regardless of what tools you have on hand) is a major win.
  - michrassena 10y ago
    Let's not forget that when your system crashes, all of these easy-to-read text files are actually stored in a binary format, sometimes scattered in pieces, and require special tools to extract.
    - qwertyuiop924 10y ago
      True, but they're less fragile: even if the text is garbled, you might be able to get some information out of it.
TeMPOraL 10y ago
Well, I'm advocating for structured text, not binary. Mostly because I haven't seen a future-proof binary format yet, and editing binary formats indeed would require special tooling. I think - for a data exchange protocol meant to be used between many applications - going structured text instead of binary is a worthwhile tradeoff of little lower efficiency vs. much better accessibility. EDIT: Some comments here are slowly making me like the idea of a standard structured binary more and more.
- ryandrake 10y ago
  Let's say I want to know how many days since an asset on my web server has been modified. With bash + some standard unix tools, from the top of my head I have to do something like this: curl -svo /dev/null http://example.com/file 2>&1 | grep Last-Modified | cut -d ' ' -f 3- And that's just to get the last modified date in text form. Now I'm writing a script that parses that date and gets today's date, convert them to days, and subtract. YUCK! Wouldn't it be nice if your shell could do this? curl(http://example.com/file).response_headers.Last-Modified.subtract(date().now).days I think it would be nice.
  - qwertyuiop924 10y ago
    Yes, but as it stands, most of those would have to be builtins for that to work. I would rather have: curl -j example.com|select headers.Last-Modified|time before now|time to days Where the commands send and receive JSON.
- bandrami 10y ago
  Can you name me a single UNIX configuration file whose format is not "structured"?
  - TeMPOraL 10y ago
    Config files are less of a problem. The issue is with programs, which you want to use with pipes. Each has its own undocumented, arbitrary pseudo-structure with often inconsistent semantics, optimized for viewing by user.
    - bandrami 10y ago
      What programs are you thinking of? Maybe this is my sysadmin bias but about 90% of my UNIX tools usage is on config files...
      - JdeBP 10y ago
        Let me suggest an incendiary example: systemd systemctl status wibble.service : Displays human-readable information, with colours, line drawings, and variable-length free-form text; that is exceedingly hard for a program to parse reliably. Contrast with systemctl show wibble.service : Outputs machine-readable information, in systemd's favourite form of a Windows INI file.
xg15 10y ago
Yes, because it's: a) inefficient b) invites a lot of wrong assumptions about the storage format, e.g. about the underlying grammar, the maximum length of tokens or the possible characters than could occur in the file. c) requires you to solve the same problems over and over again (how to represent nested lists, graphs, binary blobs, escaped special characters, etc) d) encourages authors to roll their own format and not research if there maybe is an existing format that would solve their case. I agree with you, the other extreme - treat binary files as sort of opaque black boxes that you can only access with specialized tools beloniging to your application - is even worse. But I don't see why we can't reach some middle ground: have a well-documented, open, binary format that encodes a very generic data structure (maybe a graph or a SQLite-style database) and a simple schema language that lets you annotate that data structure. Then you can develop generic tools to view/edit the binary format even though every application can use it in their own way.
borplk 10y ago
It's only "specialized" because we haven't been doing it so it's considered special. At some point you have to admit that what's meant for the computer is not always byte by byte the same as what's meant for the human. We try to shove these two together and we screw up both of them. Empower the computer to be the best that it can be by taking the human out. Empower and respect the human by giving him/her their own representation. The "I just want to read the bytes that are on disk" philosophy is inherently limiting and broken when the audience are two very different things (humans vs computers). My argument is that instead of fighting that we must embrace it.
imtringued 10y ago
A text editor is specialised to the task of viewing and modifying text in a certain format.
- qwertyuiop924 10y ago
  Wrong. It's specialized to viewing and modifiying any text, regardless of format. That's a huge difference.

dagw 10y ago

Parsing structured text is slow and inefficient. Also reading out just one part of a data structure stored as text often requires either walking through the file character by character or first slurping the whole thing into memory.

qwertyuiop924 10y ago
...But when your system crashes, having all that data in an easily accessible manner (regardless of what tools you have on hand) is a major win.
- michrassena 10y ago
  Let's not forget that when your system crashes, all of these easy-to-read text files are actually stored in a binary format, sometimes scattered in pieces, and require special tools to extract.
  - qwertyuiop924 10y ago
    True, but they're less fragile: even if the text is garbled, you might be able to get some information out of it.

TeMPOraL 10y ago

Well, I'm advocating for structured text, not binary. Mostly because I haven't seen a future-proof binary format yet, and editing binary formats indeed would require special tooling. I think - for a data exchange protocol meant to be used between many applications - going structured text instead of binary is a worthwhile tradeoff of little lower efficiency vs. much better accessibility. EDIT: Some comments here are slowly making me like the idea of a standard structured binary more and more.

ryandrake 10y ago
Let's say I want to know how many days since an asset on my web server has been modified. With bash + some standard unix tools, from the top of my head I have to do something like this: curl -svo /dev/null http://example.com/file 2>&1 | grep Last-Modified | cut -d ' ' -f 3- And that's just to get the last modified date in text form. Now I'm writing a script that parses that date and gets today's date, convert them to days, and subtract. YUCK! Wouldn't it be nice if your shell could do this? curl(http://example.com/file).response_headers.Last-Modified.subtract(date().now).days I think it would be nice.
- qwertyuiop924 10y ago
  Yes, but as it stands, most of those would have to be builtins for that to work. I would rather have: curl -j example.com|select headers.Last-Modified|time before now|time to days Where the commands send and receive JSON.
bandrami 10y ago
Can you name me a single UNIX configuration file whose format is not "structured"?
- TeMPOraL 10y ago
  Config files are less of a problem. The issue is with programs, which you want to use with pipes. Each has its own undocumented, arbitrary pseudo-structure with often inconsistent semantics, optimized for viewing by user.
  - bandrami 10y ago
    What programs are you thinking of? Maybe this is my sysadmin bias but about 90% of my UNIX tools usage is on config files...
    - JdeBP 10y ago
      Let me suggest an incendiary example: systemd systemctl status wibble.service : Displays human-readable information, with colours, line drawings, and variable-length free-form text; that is exceedingly hard for a program to parse reliably. Contrast with systemctl show wibble.service : Outputs machine-readable information, in systemd's favourite form of a Windows INI file.

xg15 10y ago

Yes, because it's: a) inefficient b) invites a lot of wrong assumptions about the storage format, e.g. about the underlying grammar, the maximum length of tokens or the possible characters than could occur in the file. c) requires you to solve the same problems over and over again (how to represent nested lists, graphs, binary blobs, escaped special characters, etc) d) encourages authors to roll their own format and not research if there maybe is an existing format that would solve their case. I agree with you, the other extreme - treat binary files as sort of opaque black boxes that you can only access with specialized tools beloniging to your application - is even worse. But I don't see why we can't reach some middle ground: have a well-documented, open, binary format that encodes a very generic data structure (maybe a graph or a SQLite-style database) and a simple schema language that lets you annotate that data structure. Then you can develop generic tools to view/edit the binary format even though every application can use it in their own way.

borplk 10y ago

It's only "specialized" because we haven't been doing it so it's considered special. At some point you have to admit that what's meant for the computer is not always byte by byte the same as what's meant for the human. We try to shove these two together and we screw up both of them. Empower the computer to be the best that it can be by taking the human out. Empower and respect the human by giving him/her their own representation. The "I just want to read the bytes that are on disk" philosophy is inherently limiting and broken when the audience are two very different things (humans vs computers). My argument is that instead of fighting that we must embrace it.

imtringued 10y ago

A text editor is specialised to the task of viewing and modifying text in a certain format.

qwertyuiop924 10y ago
Wrong. It's specialized to viewing and modifiying any text, regardless of format. That's a huge difference.