3 ms·

I think there is a key distinction here between messages we pass between applications and data we store on disk. These problems have different tradeoffs, and a

by bo1024 15y ago

I think there is a key distinction here between messages we pass between applications and data we store on disk. These problems have different tradeoffs, and a conflict probably occurs when the same bits are used for both purposes. I think the fundamental difference is that messages seem to have these sorts of properties: - temporary and transient; exist primarily to move information from one program to another - may be merely copies/representations/serializations of information that already exists in a running program - may specify implementation ("the following is a hashmap of strings to strings consisting of k1,v1;k2,v2;,...") - may use a common format/language (e.g. XML) to encode arbitrary data structures - therefore, can use commonly available parsers - not usually read/edited by humans It seems like binary data is very often a good choice here, especially since performance seems to matter. But I think the author's use case is very different. S/he's dealing with code. (Data, code, tomato, tomato.) It has different properties: - permanent: stored to disk, possibly copied to other disks, but not "created and destroyed" - is the real true representation of the actual object in question; as the object is changed, the data is edited/overwritten on disk - usually descriptive of problem rather than giving implementation details ("line (7,2) -> (8,4)" rather than "<class>[line] <member>[start] <value>[<pair>[7,2]],...") - is usually domain-specific rather than an arbitrary common language like XML - thus, tends to require a custom parser to read - primitive values transparently map to changes in the object In this latter case (think HTML, LaTeX, and all programming language files), we get tons of good benefits from plain text. It can be read in any editor, it's universally and trivially portable, it can be manipulated with tools like grep and find/replace, it can be generated or altered with simple scripts and programs, etc. And finally, it's primary purpose is to be compiled into a representation which is presented to the user (such as a webpage or pdf document or so on). Those are the two extremes, I think. Data vs messages. So we can argue about which option is better for cases that seem somewhere in between, but this is the landscape as I understand it.

strags 15y ago
Your comparison of data vs. messages is interesting, although I think what you really should contrast is messages/data vs. markup. HTML and LaTeX are clearly markup languages (or "code"). It's entirely reasonable to expect somebody to edit HTML and LaTeX in a plain old text editor. Totally agree with you on this. Where I disagree, however: The OmniGraffle file in the article, isn't code - it's a serialization of an internal data structure. The primary method for editing this data is not a text editor, nor should it be. While it's cool (I guess) that the author was able to hack around inside it, I don't think XML is a good choice here, for the reasons I described earlier. Nor do I think it's a good choice for most object serializations (including both transient messages, and persistent data). Now, the OmniGraffle file in question is pretty simple, so you could argue that maybe XML isn't so terrible. But, consider cases like http://en.wikipedia.org/wiki/COLLADA http://en.wikipedia.org/wiki/COLLADA . Storing 3d object vertex data in XML is, if you ask me, insane. If you have an object with tens of thousands of vertices, you will never edit this file by hand. What is XML gaining anybody here? Yay - you can use an off-the-shelf XML parser! But you then have to copy XML's graph into your own vertex structures in order to do anything useful! So, you really haven't gained anything except vastly increased memory and CPU usage. See http://collada.org/public_forum/viewtopic.php?f=12&t=25&start=0 http://collada.org/public_forum/viewtopic.php?f=12&t=25&... for some discussion on this.
- bo1024 15y ago
  Cool info, and thanks for the links! I agree about XML. I think one reason XML is annoying is that it lives in both worlds -- human-readable, but able to encode arbitrary machine structure. But with OmniGraffle, for example, it seems like XML is a suboptimal choice because you shouldn't need all that XML structure getting in the way -- if you know what data to put where in the document, why throw all these tags around it? So like you say, XML's main advantage seems to be that you can use an off-the-shelf parser; but it doesn't really seem worth it. However, I still think plain text can be a good choice, I just would be less afraid to define my own specification. But I feel like I would rather use an extreme -- binary data (which would require custom serialization) or plain text (which would require custom parsing) -- over something general-purpose and verbose. Disclaimer: I am not a professional software engineer and do not have experience with large systems. My opinion might change after I got my hands dirty.
  - ajuc 15y ago
    There's no excuse for XML - sexps and json are just as general as XML, and much less verbose.

by bo1024 15y ago

strags 15y ago

Your comparison of data vs. messages is interesting, although I think what you really should contrast is messages/data vs. markup. HTML and LaTeX are clearly markup languages (or "code"). It's entirely reasonable to expect somebody to edit HTML and LaTeX in a plain old text editor. Totally agree with you on this. Where I disagree, however: The OmniGraffle file in the article, isn't code - it's a serialization of an internal data structure. The primary method for editing this data is not a text editor, nor should it be. While it's cool (I guess) that the author was able to hack around inside it, I don't think XML is a good choice here, for the reasons I described earlier. Nor do I think it's a good choice for most object serializations (including both transient messages, and persistent data). Now, the OmniGraffle file in question is pretty simple, so you could argue that maybe XML isn't so terrible. But, consider cases like http://en.wikipedia.org/wiki/COLLADA http://en.wikipedia.org/wiki/COLLADA . Storing 3d object vertex data in XML is, if you ask me, insane. If you have an object with tens of thousands of vertices, you will never edit this file by hand. What is XML gaining anybody here? Yay - you can use an off-the-shelf XML parser! But you then have to copy XML's graph into your own vertex structures in order to do anything useful! So, you really haven't gained anything except vastly increased memory and CPU usage. See http://collada.org/public_forum/viewtopic.php?f=12&t=25&start=0 http://collada.org/public_forum/viewtopic.php?f=12&t=25&... for some discussion on this.

bo1024 15y ago
Cool info, and thanks for the links! I agree about XML. I think one reason XML is annoying is that it lives in both worlds -- human-readable, but able to encode arbitrary machine structure. But with OmniGraffle, for example, it seems like XML is a suboptimal choice because you shouldn't need all that XML structure getting in the way -- if you know what data to put where in the document, why throw all these tags around it? So like you say, XML's main advantage seems to be that you can use an off-the-shelf parser; but it doesn't really seem worth it. However, I still think plain text can be a good choice, I just would be less afraid to define my own specification. But I feel like I would rather use an extreme -- binary data (which would require custom serialization) or plain text (which would require custom parsing) -- over something general-purpose and verbose. Disclaimer: I am not a professional software engineer and do not have experience with large systems. My opinion might change after I got my hands dirty.
- ajuc 15y ago
  There's no excuse for XML - sexps and json are just as general as XML, and much less verbose.