3 ms·
> That's CSV/TSV's real shortcoming: about the only generic validation they allow is to make sure the column count is the same for all rows. Once upon a time,
by dossy 4y ago
> That's CSV/TSV's real shortcoming: about the only generic validation they allow is to make sure the column count is the same for all rows.
Once upon a time, when I was doing a lot of data interchange between a wide variety of systems (OS'es, applications, etc.) I considered proposing an "enhanced CSV" (ECSV) where the values did not start on the second row in the file, but instead the second row would be regular expressions that could be used to validate the contents of the columns that followed, and data would start on row 3.
In other words, you might have:
```
ID,NAME,DATE
"/^\d+$/","//","/^\d{4}-\d{2}-\d{2}$/"
867,Alice,1984-01-09
5309,Bob,1981-11-16
```
(Newlines added because HN doesn't speak Markdown, sigh.)
In the end, I think the solution was far simpler: we just exchanged a separate descriptor file that had column names and their corresponding regexp patterns for validation as a separate (versioned) file, in order to save a few bytes inside each file transmission, which was a real savings when you paid per-byte over an EDI network.
- LordDragonfang 4y agoHN actually does support markdown code blocks, you just have to preface lines with four (nope, two)[1] spaces instead of the (newer) triple-backtick codefences. ID,NAME,DATE "/^\d+$/","//","/^\d{4}-\d{2}-\d{2}$/" 867,Alice,1984-01-09 5309,Bob,1981-11-16 [1] apparently it's actually two spaces instead of the normal markdown standard of four, making HN even more non-standard than usual https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
- dossy 4y agoYup, I noticed that and my Level of Care was insufficient to go and edit my comment a second time to fix it. But, thanks for confirming that it could have worked!
- syntheweave 4y agoI spent a while making a binary format for tabularish documents, and even started on an editor for it. What I decided on after some long months of gradual iteration was to give each cell its own header that could contain various forms of type info, flags, and modes, and to define a cell type that described forms of break (space, line, page, etc. - a 16-bit range of break types could be encoded). The document header also described a dictionary mapping for the data so that it could immediately be presented to the editor in a readable form. But now I just use plain old spreadsheets to do things - I obsoleted my own tech, although I like certain things about it. The editing and storage encoding isn't really the problem so much as the integrity of the literals, which a solution like the regex idea could accommodate. I do think that CSV would benefit by having a header area that described the encoding of breaks in cells and lines. Maybe that's the only thing that really needs fixing in it. And if it included arbitrary break levels like my thing and dropped the rectangular row-column shape, it would cover a huge number of documents.