4 ms·

In a few years we will rediscover column-based, punctuation-free text files that can be parsed with single call to scanf(3).

saghm 3y ago
Maybe I'm reading too generously, but my sense is that the format described in the article is intended to be used for encoding and transferring data to be used in some other application rather than something where you can assume constraints like "no punctuation". I could imagine using the "JSON lines" format for things like encoding a large number of records to insert for test cases or as a way of sharing a test dataset with another developer; I'm not convinced that either of those would work well without any punctuation whatsoever, and even if they could, I suspect I'd still rather pay the cost of slower importing in order to make my testing code (and any manual reading/transformation outside of my code) easy to do without having to add some additional processing step to deal with the punctuation. The best argument against this format to me is that it's possible to just use JSON itself for this by wrapping all of the values in an array. I'm not totally convinced that the quality of life improvements of not needing to add commas along with the single pair of square braces necessitate an entirely new file format, but realistically I think I'll just end up following whatever happens to be supported by the tooling I use for a given project. If "JSON lines" became widely used overnight, I wouldn't be bothered by it, but if I never end up seeing it supported by anything, I doubt I'll miss it.
- Arch485 3y ago
  One particular downside to using an actual array instead of something like JSON lines is that you can't really stream the data. If you have so much data that you need to stream it, you probably shouldn't be using JSON, though...
  - fermisea 3y ago
    Shopify, for instance, uses this format if you request bulk downloads of records, for this exact reason.
  - vsnf 3y ago
    >If you have so much data that you need to stream it, you probably shouldn't be using JSON, though... Why not? I have a bunch of large/real-time files I'd like to read piece by piece and json is a convenient format for parsing it to something useful on the client. I understand using other, more efficient formats, but I think just wanting to stream the data isn't enough of a reason to move to them.
    - Arch485 3y ago
      Well, I said _need_ to stream it. I think that's slightly different: I _want_ to stream data almost all the time, but I _need_ to stream data (e.g. due to RAM limitations) fairly infrequently. If you have so much data (e.g. 8GB+, yes my workstation "only" has 8GB of RAM) that it _must_ be streamed, in my personal opinion you should opt for a better format than JSON. It's likely, too, that with a more specialized/space-efficient format, you may no longer need to stream the data!
  - baq 3y ago
    You can, this has been done since xml was a thing (yes some of us are old). The api to do that is necessarily horrible, so no one sane does, though.
globular-toast 3y ago
The problem is text editor support. People don't trust binary data and want to see it in a text editor. ASCII has four characters reserved for separation so ridiculous formats like xSV where x is an in-band character are unnecessary. It's just the text editor support that is lacking. Even Emacs can't seem to do it well.
- tracker1 3y ago
  I used those characters in the latter 90s when sending query results to a web client. There was a set of parameters with ADO to seriously the data or to a string. It was easy enough to split/map it back in the client. Using hidden frames for jsonp-like request and callback support, that was the fun stuff. Don't miss it. Also don't miss XML or SOAP.
DonHopkins 3y ago
And in a few years plus a nanosecond we will rediscover what a terrible, insecure, bug-prone, foot-gun api scanf(3) is, because of buffer overflow errors, lack of input validation, difficult error handling, clumsy newline handling, and format string vulnerabilities.
- deleted 3y ago
  [deleted]
- enriquto 3y ago
  I said that you can use scanf(3), not that you should. You can use "s.split()" in python, if that makes you feel safer.
  - DonHopkins 3y ago
    Then you are well on your way to rediscovering csv, and you still can't handle self-describing records with different structures, or nested sub-structures, or arbitrary length arrays, or null records, or unicode and control character escapes, or records containing the delimiter. Has it occurred to you that maybe there's a reason people are using jsonl instead of an old existing standard like csv? Not just that they're such dummies they never heard of csv and would take a few years to rediscover it, but an actual reason? Hey, you're the one who chose to recommend scanf, and claimed it would take a few years to rediscover it. I would generally assume people who read hacker news are smarter than that, and are fully aware of the security problems of scanf, and the weaknesses of csv, and are probably not crazy enough to write bulk streaming data parsers in C. And no, s.split() doesn't make me feel safer, or even do rudimentary type conversion like scanf. Pro-tip: use a comma in your password to pollute the csv when your password gets stolen and published on the dark web.
    - smitty1e 3y ago
      .csv is great if you want to browse output in a spreadsheet. Wobbly format? Sure. But if the data are purely internal, why not opt for convenience?

saghm 3y ago

Maybe I'm reading too generously, but my sense is that the format described in the article is intended to be used for encoding and transferring data to be used in some other application rather than something where you can assume constraints like "no punctuation". I could imagine using the "JSON lines" format for things like encoding a large number of records to insert for test cases or as a way of sharing a test dataset with another developer; I'm not convinced that either of those would work well without any punctuation whatsoever, and even if they could, I suspect I'd still rather pay the cost of slower importing in order to make my testing code (and any manual reading/transformation outside of my code) easy to do without having to add some additional processing step to deal with the punctuation. The best argument against this format to me is that it's possible to just use JSON itself for this by wrapping all of the values in an array. I'm not totally convinced that the quality of life improvements of not needing to add commas along with the single pair of square braces necessitate an entirely new file format, but realistically I think I'll just end up following whatever happens to be supported by the tooling I use for a given project. If "JSON lines" became widely used overnight, I wouldn't be bothered by it, but if I never end up seeing it supported by anything, I doubt I'll miss it.

Arch485 3y ago
One particular downside to using an actual array instead of something like JSON lines is that you can't really stream the data. If you have so much data that you need to stream it, you probably shouldn't be using JSON, though...
- fermisea 3y ago
  Shopify, for instance, uses this format if you request bulk downloads of records, for this exact reason.
- vsnf 3y ago
  >If you have so much data that you need to stream it, you probably shouldn't be using JSON, though... Why not? I have a bunch of large/real-time files I'd like to read piece by piece and json is a convenient format for parsing it to something useful on the client. I understand using other, more efficient formats, but I think just wanting to stream the data isn't enough of a reason to move to them.
  - Arch485 3y ago
    Well, I said _need_ to stream it. I think that's slightly different: I _want_ to stream data almost all the time, but I _need_ to stream data (e.g. due to RAM limitations) fairly infrequently. If you have so much data (e.g. 8GB+, yes my workstation "only" has 8GB of RAM) that it _must_ be streamed, in my personal opinion you should opt for a better format than JSON. It's likely, too, that with a more specialized/space-efficient format, you may no longer need to stream the data!
- baq 3y ago
  You can, this has been done since xml was a thing (yes some of us are old). The api to do that is necessarily horrible, so no one sane does, though.

globular-toast 3y ago

The problem is text editor support. People don't trust binary data and want to see it in a text editor. ASCII has four characters reserved for separation so ridiculous formats like xSV where x is an in-band character are unnecessary. It's just the text editor support that is lacking. Even Emacs can't seem to do it well.

tracker1 3y ago
I used those characters in the latter 90s when sending query results to a web client. There was a set of parameters with ADO to seriously the data or to a string. It was easy enough to split/map it back in the client. Using hidden frames for jsonp-like request and callback support, that was the fun stuff. Don't miss it. Also don't miss XML or SOAP.

DonHopkins 3y ago

And in a few years plus a nanosecond we will rediscover what a terrible, insecure, bug-prone, foot-gun api scanf(3) is, because of buffer overflow errors, lack of input validation, difficult error handling, clumsy newline handling, and format string vulnerabilities.

deleted 3y ago
[deleted]
enriquto 3y ago
I said that you can use scanf(3), not that you should. You can use "s.split()" in python, if that makes you feel safer.
- DonHopkins 3y ago
  Then you are well on your way to rediscovering csv, and you still can't handle self-describing records with different structures, or nested sub-structures, or arbitrary length arrays, or null records, or unicode and control character escapes, or records containing the delimiter. Has it occurred to you that maybe there's a reason people are using jsonl instead of an old existing standard like csv? Not just that they're such dummies they never heard of csv and would take a few years to rediscover it, but an actual reason? Hey, you're the one who chose to recommend scanf, and claimed it would take a few years to rediscover it. I would generally assume people who read hacker news are smarter than that, and are fully aware of the security problems of scanf, and the weaknesses of csv, and are probably not crazy enough to write bulk streaming data parsers in C. And no, s.split() doesn't make me feel safer, or even do rudimentary type conversion like scanf. Pro-tip: use a comma in your password to pollute the csv when your password gets stolen and published on the dark web.
  - smitty1e 3y ago
    .csv is great if you want to browse output in a spreadsheet. Wobbly format? Sure. But if the data are purely internal, why not opt for convenience?