3 ms·
I don't see what this improves over standard JSON. Just add a comma at the end of each line and wrap the whole Lines file in brackets, and you have a valid JSON
by Rangi42 11y ago
I don't see what this improves over standard JSON. Just add a comma at the end of each line and wrap the whole Lines file in brackets, and you have a valid JSON file. You can petition spreadsheet makers to support this subset of JSON files, while still letting any general JSON program read and write them.
As for CSV and TSV, they're a dead-simple format to read and write, and you can enforce a standard way to escape special characters if you aren't dealing with arbitrary user-submitted files. With numeric data escaping isn't even necessary, and if you have to deal with strings that might themselves contain special characters, the ASCII control characters 0x1E and 0x1F work well as alternatives to comma/tab and newline.
I know Python has a csv module, but I've just gotten used to writing these snippets:
def tsv_write(path, headers, data, sep='\t', end='\n'):
with open(path, 'w') as file:
file.write(sep.join(map(str, headers)) + end)
for datum in data:
file.write(sep.join(map(str, datum)) + end)
def tsv_read(path, headers, sep='\t', end='\n'):
with open(path, 'r') as file:
for line in file:
yield line.rstrip(end).split(sep)
(That tsv_read function only works if end == '\n'. Here's a somewhat-inefficient general-purpose alternative, although I've never needed it.)
def read_records(file, end='\n'):
if end == '\n':
for line in file:
yield line
else:
record = []
while True:
c = file.read(1)
if c:
record.append(c)
if c == end or not c:
yield ''.join(record)
record = []
if not c:
break
def tsv_read(path, headers, sep='\t', end='\n'):
with open(path, 'r') as file:
for record in read_records(file, end):
yield record.rstrip(end).split(sep)
- couchand 11y agoThe primary advantage is that you can append to the file and always have a legal file. The secondary advantage is that you can work on a subset of lines (think streams or cat | head | tail) without parsing the whole file. The two together mean you can use this format for rich logging and such.
- DaGardner 11y agoand to use it an a spark context and such
- ma2rten 11y agoThe advantage over standard json is that this is trivial to parse incrementally, i.e. for a dataset that is too big to fit in memory. I also wouldn't recommend you to reimplement a functionality that is already part of the standard library, especially because the standard library function is going to be much faster because it's implemented in C.
- Rangi42 11y agoThanks for that info. I didn't know the module was written in C.
- ma2rten 11y agoOnly a small subset of the python standard library is actually written in python.
- toomim 11y agoActually, the approach with commas and a wrapped brackets are equally trivial to parse incrementally. Just start by erasing the open "[", and then read one row at a time, just like you would do without the "[" and comma.
- smsm42 11y agoI would say any tool that outputs lists of things in JSON and there is reasonable suspicion that the output may be large, should do this. I.e. do one-thing-per-line with opening and closing brackets on separate lines. It is quite annoying to find oneself with huge dataset in JSON which was produced by a naive generator that doesn't do that - when you need to extract something like "first N elements that satisfy condition X" it is annoyingly more work that it should be. By doing this people that don't care still get valid JSON, people that do care save some time.