6 ms·

> sqlite files couldn't be more open Yes, actually they could. They could be readable without any software at all. Like XML. All of the improvements suggested

> sqlite files couldn't be more open Yes, actually they could. They could be readable without any software at all. Like XML. All of the improvements suggested in the article could be implemented by extending the XML format to allow including by other XML files by reference.

coldtea 9y ago
>Yes, actually they could. They could be readable without any software at all. Like XML. You can't read XML without software. Perhaps you take editors and cat and less as a given. In which case, nothing prevents you from taking sqlite the binary (or 1000 serializers that can be produced very easily) as a given too.
chii 9y ago
But you could also open the sqlite fine format using an sqlite client. The only difference is that xml is also human "readable". But most xml isn't really readable without a tool anyway.
- mseebach 9y ago
  Yes, and that only difference is important, and it makes XML more open than SQLite. Perhaps not enough to matter (depending on your needs), but it is more open. You may not want to look at XML without a tool, but you could. Or, most likely, you could build a tool to look at the XML. This is a crucial design consideration of XML - even if all XML tooling, open or not, is lost forever, you can still read the XML. Not so with SQLite. Especially for a document format, this point seems important. That said, OpenDocument is zipped, which, while a simpler and even more ubiquitous format, is still a binary format that requires a library to read, so that detracts somewhat from the point. It's difficult to imagine a future where zip libraries survive but SQLite libraries don't.
  - jl6 9y ago
    Strictly speaking you can’t read XML without software either. Even a simple text editor is software of course, but you’ll also find XML files in arbitrary character encodings, and you absolutely do need non-trivial software to parse and display “plain” text in UTF-8.
    - mseebach 9y ago
      You don't need a text editor, you just need to be able to read a sequence of bytes. XML anchors at the almost lowest possible point: you have the stream of characters. It's a pointless exercise to consider a format except on the premise that you have access to data in some way, and sequence of bytes is the simplest possible abstraction of data, a sequence of characters the second simplest. ASCII encoded text is simpler than UTF-8, but there is content that can't be represented in ASCII. Zip comes a bit up the food chain from there, and SQLite comes quite a bit further. XML is about being meaningful without tooling at the lowest possible level. Any added required tooling or documentation by definition makes the data less 'open'.
      - Santosh83 9y ago
        In the event of catastrophic loss of human knowledge to the scale that it makes it difficult or impossible to recreate a ZIP or SQLite driver, I would presume that maintaining the replacing the hardware on which XML/text files are stored would be an even bigger problem. Catastrophic collapse of civilization IMO would pretty much doom all our digital knowledge. Only books can survive such an event.
        mseebach 9y ago
        You don't need a catastrophic loss of human knowledge for this to be a problem. Running a piece of software written in the 80s, even with the source code available, can be quite a challenge. Support for some features of the current SQLite format might deprecated, then removed, over the next many versions. Or perhaps a subtle bug is introduced that renders certain ancient versions unreadable (https://news.ycombinator.com/item?id=15531489 https://news.ycombinator.com/item?id=15531489) and sticks around for long enough that you end up in a situation where you need to invest very significantly in software archaeology.
majewsky 9y ago
> They could be readable without any software at all. They aren't, though. Or can you read ZIP archives without an unpacker?
- geezerjay 9y ago
  Actually, iirc the OpenDocument format also includes an uncompressed, human readable, pure XML format.
  - mkesper 9y ago
    It's the 'flat' versions (fods etc.). Diff can be used on them.
  - _pmf_ 9y ago
    > Actually, iirc the OpenDocument format also includes an uncompressed, human readable, pure XML format. For some values of "readable" and "human".
- masklinn 9y ago
  Writing a zip unpacker — including deflate decompressor — can be wrapped up in half a day by an average developer.
  - wruza 9y ago
    It sounds like good nuclear war proof solution. Maybe it is better to encode documents in Morse, because average radio man could transmit it via key in half a week.
    - masklinn 9y ago
      > It sounds like good nuclear war proof solution. It fits the requirements of reading the file without (external) software since you can trivially reimplement it, and is also a much lower threshold than "you need to reimplement sqlite", which is why pretty much every system out there comes bundled with a zipfile unpacker.
jwr 9y ago
This is a common recurring theme that I find strange. Why do people believe that if something can be browsed with less/Notepad, it is somehow more "readable" than a binary representation? I've seen and worked with tons of XML, and most of it was utterly useless without accompanying documentation, just as a binary format would be. The fact that I could "see" the XML on my screen did not make the job easier. The problem with formats "readable without any software" is that you usually end up with everything represented as strings, with no knowledge of actual types and constraints. So, you still need that documentation, even though the format is "readable without any software".
- mseebach 9y ago
  Well, yes, this applies to reasonably well-designed, expressive XML (which OpenDocument is). You can definitely write XML that is perfectly incomprehensible without extensive references.
- enord 9y ago
  It's not about human readability, it's about separating implementation from representation, so "readable without any software" is a good test for wether it's "readable with any sfotware". If your document editor represents documents as a stack of edit-events (for undo/redo or whatever) and you document every nook and cranny of this "document format", standardize it as "binary event-sourced document format, .esd" then your binary spec will be very close to a full description of your document editor. Everyone who wants to use your format now has to reimplement your editor. Anyone can leak implementation details into any format, and it is to some extent unavoidable, but if you can make it readable (or semi-readable), some thought has at least been put into the representation.
  - _pmf_ 9y ago
    > so "readable without any software" is a good test for wether it's "readable with any sfotware" What an extraordinarily stupid notion. Your filesystem is not readable without a filesystem driver for the specific binary format used to represent it, so you cannot even read your text file without having something in your stack that understands a more structured data organization scheme.
koonsolo 9y ago
First of all, notepad is software. And second, xml is totally useless without software. But you do have a point that in a rare case, you could get your hands on the raw data which makes text easier. But everything is a tradeoff, and the question really is if the tradeoff in this case is worth it, since sqlite is so open and portable.
Chris2048 9y ago
> extending the XML format XML is only trivially "readable without any software at all". The same is true of any binary format - the issue is that it can be read, but not easily processed - which is also the case for XML (you need to integrate an XML parser for that). And if you are including XML libs, why not SQlite libs?
- fian 9y ago
  A printout of XML contains the data in a structured and understandable format. A printout of a binary dump does not provide any clues as to how the data was structured.
  - Chris2048 9y ago
    This is useful if you only need to read the data. If you need to process it in large amounts, its not so useful. And what if you need to traverse graph-like data in an XML format? > a binary dump does not provide any clues as to how the data was structured True, but if that binary format is SQlite, the structure is fully documented. You could work it out mentally just as you could work out any given XML schema.
- geezerjay 9y ago
  The same is not true for any binary file. A hexdump of a plain text file is not as readable as opening the same plain text file with any basic text editor such as any notepad clone. That's the whole point: no one needs to install any software, let alone a specialized client, to read any human-readable format. More to the point, no one needs to install a full blown database engine to read a plain text document. You may argue that specialized tools make the job easier, but you also need to acknowledge that requiring specialized tools just to open a text file is silly.
PeachPlum 9y ago
XML is UTF-8 you already need a decoder http://www.utf8-chartable.de/ http://www.utf8-chartable.de/

coldtea 9y ago

>Yes, actually they could. They could be readable without any software at all. Like XML. You can't read XML without software. Perhaps you take editors and cat and less as a given. In which case, nothing prevents you from taking sqlite the binary (or 1000 serializers that can be produced very easily) as a given too.

chii 9y ago

But you could also open the sqlite fine format using an sqlite client. The only difference is that xml is also human "readable". But most xml isn't really readable without a tool anyway.

mseebach 9y ago
Yes, and that only difference is important, and it makes XML more open than SQLite. Perhaps not enough to matter (depending on your needs), but it is more open. You may not want to look at XML without a tool, but you could. Or, most likely, you could build a tool to look at the XML. This is a crucial design consideration of XML - even if all XML tooling, open or not, is lost forever, you can still read the XML. Not so with SQLite. Especially for a document format, this point seems important. That said, OpenDocument is zipped, which, while a simpler and even more ubiquitous format, is still a binary format that requires a library to read, so that detracts somewhat from the point. It's difficult to imagine a future where zip libraries survive but SQLite libraries don't.
- jl6 9y ago
  Strictly speaking you can’t read XML without software either. Even a simple text editor is software of course, but you’ll also find XML files in arbitrary character encodings, and you absolutely do need non-trivial software to parse and display “plain” text in UTF-8.
  - mseebach 9y ago
    You don't need a text editor, you just need to be able to read a sequence of bytes. XML anchors at the almost lowest possible point: you have the stream of characters. It's a pointless exercise to consider a format except on the premise that you have access to data in some way, and sequence of bytes is the simplest possible abstraction of data, a sequence of characters the second simplest. ASCII encoded text is simpler than UTF-8, but there is content that can't be represented in ASCII. Zip comes a bit up the food chain from there, and SQLite comes quite a bit further. XML is about being meaningful without tooling at the lowest possible level. Any added required tooling or documentation by definition makes the data less 'open'.
    - Santosh83 9y ago
      In the event of catastrophic loss of human knowledge to the scale that it makes it difficult or impossible to recreate a ZIP or SQLite driver, I would presume that maintaining the replacing the hardware on which XML/text files are stored would be an even bigger problem. Catastrophic collapse of civilization IMO would pretty much doom all our digital knowledge. Only books can survive such an event.
      - mseebach 9y ago
        You don't need a catastrophic loss of human knowledge for this to be a problem. Running a piece of software written in the 80s, even with the source code available, can be quite a challenge. Support for some features of the current SQLite format might deprecated, then removed, over the next many versions. Or perhaps a subtle bug is introduced that renders certain ancient versions unreadable (https://news.ycombinator.com/item?id=15531489 https://news.ycombinator.com/item?id=15531489) and sticks around for long enough that you end up in a situation where you need to invest very significantly in software archaeology.

majewsky 9y ago

> They could be readable without any software at all. They aren't, though. Or can you read ZIP archives without an unpacker?

geezerjay 9y ago
Actually, iirc the OpenDocument format also includes an uncompressed, human readable, pure XML format.
- mkesper 9y ago
  It's the 'flat' versions (fods etc.). Diff can be used on them.
- _pmf_ 9y ago
  > Actually, iirc the OpenDocument format also includes an uncompressed, human readable, pure XML format. For some values of "readable" and "human".
masklinn 9y ago
Writing a zip unpacker — including deflate decompressor — can be wrapped up in half a day by an average developer.
- wruza 9y ago
  It sounds like good nuclear war proof solution. Maybe it is better to encode documents in Morse, because average radio man could transmit it via key in half a week.
  - masklinn 9y ago
    > It sounds like good nuclear war proof solution. It fits the requirements of reading the file without (external) software since you can trivially reimplement it, and is also a much lower threshold than "you need to reimplement sqlite", which is why pretty much every system out there comes bundled with a zipfile unpacker.

jwr 9y ago

This is a common recurring theme that I find strange. Why do people believe that if something can be browsed with less/Notepad, it is somehow more "readable" than a binary representation? I've seen and worked with tons of XML, and most of it was utterly useless without accompanying documentation, just as a binary format would be. The fact that I could "see" the XML on my screen did not make the job easier. The problem with formats "readable without any software" is that you usually end up with everything represented as strings, with no knowledge of actual types and constraints. So, you still need that documentation, even though the format is "readable without any software".

mseebach 9y ago
Well, yes, this applies to reasonably well-designed, expressive XML (which OpenDocument is). You can definitely write XML that is perfectly incomprehensible without extensive references.
enord 9y ago
It's not about human readability, it's about separating implementation from representation, so "readable without any software" is a good test for wether it's "readable with any sfotware". If your document editor represents documents as a stack of edit-events (for undo/redo or whatever) and you document every nook and cranny of this "document format", standardize it as "binary event-sourced document format, .esd" then your binary spec will be very close to a full description of your document editor. Everyone who wants to use your format now has to reimplement your editor. Anyone can leak implementation details into any format, and it is to some extent unavoidable, but if you can make it readable (or semi-readable), some thought has at least been put into the representation.
- _pmf_ 9y ago
  > so "readable without any software" is a good test for wether it's "readable with any sfotware" What an extraordinarily stupid notion. Your filesystem is not readable without a filesystem driver for the specific binary format used to represent it, so you cannot even read your text file without having something in your stack that understands a more structured data organization scheme.

koonsolo 9y ago

First of all, notepad is software. And second, xml is totally useless without software. But you do have a point that in a rare case, you could get your hands on the raw data which makes text easier. But everything is a tradeoff, and the question really is if the tradeoff in this case is worth it, since sqlite is so open and portable.

Chris2048 9y ago

> extending the XML format XML is only trivially "readable without any software at all". The same is true of any binary format - the issue is that it can be read, but not easily processed - which is also the case for XML (you need to integrate an XML parser for that). And if you are including XML libs, why not SQlite libs?

fian 9y ago
A printout of XML contains the data in a structured and understandable format. A printout of a binary dump does not provide any clues as to how the data was structured.
- Chris2048 9y ago
  This is useful if you only need to read the data. If you need to process it in large amounts, its not so useful. And what if you need to traverse graph-like data in an XML format? > a binary dump does not provide any clues as to how the data was structured True, but if that binary format is SQlite, the structure is fully documented. You could work it out mentally just as you could work out any given XML schema.
geezerjay 9y ago
The same is not true for any binary file. A hexdump of a plain text file is not as readable as opening the same plain text file with any basic text editor such as any notepad clone. That's the whole point: no one needs to install any software, let alone a specialized client, to read any human-readable format. More to the point, no one needs to install a full blown database engine to read a plain text document. You may argue that specialized tools make the job easier, but you also need to acknowledge that requiring specialized tools just to open a text file is silly.

PeachPlum 9y ago

XML is UTF-8 you already need a decoder http://www.utf8-chartable.de/ http://www.utf8-chartable.de/