3 ms·

Can anyone explain the statement about simplejson for me please? Some libraries manage to skip the token part. (I'm looking at you simplejson, a library t

• SoftwareMaven 14y ago
  When you are trying to parse extremely large documents or you only care about a small subset of a document, needing to load the entire parsed file into memory is problematic. Simplejson doesn't provide an API that allows for any other option. This is in contrast to something like an XML sax parser, that allows you to register for events like "a foobar element was loaded". You get the foobar element while all the other tokens are thrown out the window as soon as they are parsed. The complaint is that, somewhere, under the hood, simplejson is doing that token parsing, but because of their API, a user can't plug into it.
  • masklinn 14y ago
    > The complaint is that, somewhere, under the hood, simplejson is doing that token parsing, but because of their API, a user can't plug into it. I read it exactly the other way around: simplejson skips an internal tokenization step, so even if you fork the library it is pretty much impossible to make it streaming, because there's no token stream to handle stream state.
    • hyperpape 14y ago
      That's more or less right, masklinn. To the above writers, look at the Python source (I can't vouch for how the C implementation does things). It's not beautiful, but it's also not hard to understand. The effective work is all done inside a big function. To customize it, you'd have to split that function into pieces and then glue it back together using a class or your own function coordinating the parts.