5 ms·

This uses std::string all over the place, which allocates memory under the hood. I'm not sure how any JSON parser could avoid memory allocation without a diffi

This uses std::string all over the place, which allocates memory under the hood. I'm not sure how any JSON parser could avoid memory allocation without a difficult to use interface. Numbers, arrays, strings, and objects are unbounded by the JSON spec, so a truly malloc free library would need to provide a kind of streaming interfaces where things are returned in fixed sized chunks. JSON wouldn't be my first choice for data storage in situations where I needed to avoid dynamic memory.

pjungwir 12y ago
jsmn is a nice JSON tokenizer. You malloc a chunk of memory up front, but that's it. It gives you pointers to the tokens and you can do what you like with them. I am using it in an app that has a fixed-size data structure that is populated from a JSON file. So my code is malloc-free also.
- aktau 12y ago
  Link for those interested: https://bitbucket.org/zserge/jsmn/wiki/Home https://bitbucket.org/zserge/jsmn/wiki/Home jsmn is pretty awesome, and contrary to this (std::string, std::vector) actually puts you in charge of allocations. It's also in C. In a similar vein, and also possibly already mentioned: - yayl (C): https://lloyd.github.io/yajl/ https://lloyd.github.io/yajl/ - microjson (C): http://esr.ibiblio.org/?p=6262 http://esr.ibiblio.org/?p=6262
Iftheshoefits 12y ago
It also uses vector, which is almost certainly using malloc and/or placement new even if the SIO somehow really does reserve space for a single object. Its also using the heap unless there is some hidden Allocator object in the code that explicitly is using stack allocated space.
codehero 12y ago
I wrote a JSON parser that avoids using dynamic memory but works a bit differently. It's a callback interface like SAX, but instead of pushing already "typed" data to the user, the user calls a function to interpret the data. In this fashion, if the user wants a float instead of an integer, they call the function for float. Likewise, if the user has some space allocated for string values, the library can convert the already inmemory JSON format string to UTF-8 in the user's buffer. The C interface is purposely ugly but the C++ is much easier to work with. https://codehero.github.io/benejson/ https://codehero.github.io/benejson/
- deleted 12y ago
  [deleted]
jly 12y ago
> JSON wouldn't be my first choice for data storage in situations where I needed to avoid dynamic memory. I work a lot with very low-resource embedded systems and we like to use JSON host-side for structured data configuration, but convert to BSON (http://bsonspec.org http://bsonspec.org) for target-side storage. This can be parsed and iterated with very low-overhead and no dynamic memory. Similar idea to protocol buffers but with JSON-like data. Avoiding a build-time conversion would be nice, though, and this idea is interesting.
areop 12y ago
I wrote a JSON-Parser in C with minimal validation in ~250 LOC that does not allocate memory, but leaves the type check and conversion to the user. https://github.com/vurtun/json https://github.com/vurtun/json
- huhtenberg 12y ago
  That's a very nice code.
- klibertp 12y ago
  This is absolutely beautiful. I didn't know you could get an address of a label in C or to do a goto to an address from variable instead of a literal label. Is this new in C, or was it always there?
  - dietrichepp 12y ago
    It's a GCC extension. https://gcc.gnu.org/onlinedocs/gcc/Labels-as-Values.html https://gcc.gnu.org/onlinedocs/gcc/Labels-as-Values.html
    - techdragon 12y ago
      Is there an equivalent for Clang/LLVM?
      - cjcole 12y ago
        Yes. http://blog.llvm.org/2010/01/address-of-label-and-indirect-branches.html http://blog.llvm.org/2010/01/address-of-label-and-indirect-b... A venerable interpreter implementation trick.
  - codehero 12y ago
    Unfortunately it won't port to IAR, which is a shame. I don't understand why pointers to labels never made it to the C spec; it's one of the (equivaling) features that actually makes assembler look elegant.
chadaustin 12y ago
sajson allocates once: it turns out you can structure your parse tree such that each byte of input only requires one machine word of output. http://chadaustin.me/2013/01/sajson-why-the-parse-tree-is-big-enough/ http://chadaustin.me/2013/01/sajson-why-the-parse-tree-is-bi... https://github.com/chadaustin/sajson https://github.com/chadaustin/sajson
makomk 12y ago
I've got a streaming JSON parser in GCC C around here somewhere that does basically that - numbers and keys are truncated, but everything else is chopped up into chunks and streamed. Someday I ought to clean it up, document it and release it. Runs quite nicely on an Arduino.
matt42 12y ago
The parser does not extensively use std::string. to parse json keys, It uses an internal stringview class mirroring the json string (string_view is candidate to get into the next C++ standard). When I said malloc free, I meant that no intermediate memory is allocated: actually, If your json object only contains int or floats, the parser will not use malloc: it will just traverse the string and fill the scalars of your destination object (thanks to static introspection), a big difference with state of the art json parsers returning an intermediate dynamically allocated tree that you have to interpret after the parsing. Of course, if the destination object stores instances of std::string of std::vector, the iod library will allocated them and fill them with the json values.
gurkendoktor 12y ago
That's correct in the worst case, but in the best case, std::string does not allocate memory anymore (short string optimisation), e.g. libc++: http://stackoverflow.com/a/21710033 http://stackoverflow.com/a/21710033 So it might be "malloc-free" for configuration files where the possible values are limited, etc. :)

pjungwir 12y ago

jsmn is a nice JSON tokenizer. You malloc a chunk of memory up front, but that's it. It gives you pointers to the tokens and you can do what you like with them. I am using it in an app that has a fixed-size data structure that is populated from a JSON file. So my code is malloc-free also.

aktau 12y ago
Link for those interested: https://bitbucket.org/zserge/jsmn/wiki/Home https://bitbucket.org/zserge/jsmn/wiki/Home jsmn is pretty awesome, and contrary to this (std::string, std::vector) actually puts you in charge of allocations. It's also in C. In a similar vein, and also possibly already mentioned: - yayl (C): https://lloyd.github.io/yajl/ https://lloyd.github.io/yajl/ - microjson (C): http://esr.ibiblio.org/?p=6262 http://esr.ibiblio.org/?p=6262

Iftheshoefits 12y ago

It also uses vector, which is almost certainly using malloc and/or placement new even if the SIO somehow really does reserve space for a single object. Its also using the heap unless there is some hidden Allocator object in the code that explicitly is using stack allocated space.

codehero 12y ago

I wrote a JSON parser that avoids using dynamic memory but works a bit differently. It's a callback interface like SAX, but instead of pushing already "typed" data to the user, the user calls a function to interpret the data. In this fashion, if the user wants a float instead of an integer, they call the function for float. Likewise, if the user has some space allocated for string values, the library can convert the already inmemory JSON format string to UTF-8 in the user's buffer. The C interface is purposely ugly but the C++ is much easier to work with. https://codehero.github.io/benejson/ https://codehero.github.io/benejson/

deleted 12y ago
[deleted]

jly 12y ago

> JSON wouldn't be my first choice for data storage in situations where I needed to avoid dynamic memory. I work a lot with very low-resource embedded systems and we like to use JSON host-side for structured data configuration, but convert to BSON (http://bsonspec.org http://bsonspec.org) for target-side storage. This can be parsed and iterated with very low-overhead and no dynamic memory. Similar idea to protocol buffers but with JSON-like data. Avoiding a build-time conversion would be nice, though, and this idea is interesting.

areop 12y ago

I wrote a JSON-Parser in C with minimal validation in ~250 LOC that does not allocate memory, but leaves the type check and conversion to the user. https://github.com/vurtun/json https://github.com/vurtun/json

huhtenberg 12y ago
That's a very nice code.
klibertp 12y ago
This is absolutely beautiful. I didn't know you could get an address of a label in C or to do a goto to an address from variable instead of a literal label. Is this new in C, or was it always there?
- dietrichepp 12y ago
  It's a GCC extension. https://gcc.gnu.org/onlinedocs/gcc/Labels-as-Values.html https://gcc.gnu.org/onlinedocs/gcc/Labels-as-Values.html
  - techdragon 12y ago
    Is there an equivalent for Clang/LLVM?
    - cjcole 12y ago
      Yes. http://blog.llvm.org/2010/01/address-of-label-and-indirect-branches.html http://blog.llvm.org/2010/01/address-of-label-and-indirect-b... A venerable interpreter implementation trick.
- codehero 12y ago
  Unfortunately it won't port to IAR, which is a shame. I don't understand why pointers to labels never made it to the C spec; it's one of the (equivaling) features that actually makes assembler look elegant.

chadaustin 12y ago

sajson allocates once: it turns out you can structure your parse tree such that each byte of input only requires one machine word of output. http://chadaustin.me/2013/01/sajson-why-the-parse-tree-is-big-enough/ http://chadaustin.me/2013/01/sajson-why-the-parse-tree-is-bi... https://github.com/chadaustin/sajson https://github.com/chadaustin/sajson

makomk 12y ago

I've got a streaming JSON parser in GCC C around here somewhere that does basically that - numbers and keys are truncated, but everything else is chopped up into chunks and streamed. Someday I ought to clean it up, document it and release it. Runs quite nicely on an Arduino.

matt42 12y ago

The parser does not extensively use std::string. to parse json keys, It uses an internal stringview class mirroring the json string (string_view is candidate to get into the next C++ standard). When I said malloc free, I meant that no intermediate memory is allocated: actually, If your json object only contains int or floats, the parser will not use malloc: it will just traverse the string and fill the scalars of your destination object (thanks to static introspection), a big difference with state of the art json parsers returning an intermediate dynamically allocated tree that you have to interpret after the parsing. Of course, if the destination object stores instances of std::string of std::vector, the iod library will allocated them and fill them with the json values.

gurkendoktor 12y ago

That's correct in the worst case, but in the best case, std::string does not allocate memory anymore (short string optimisation), e.g. libc++: http://stackoverflow.com/a/21710033 http://stackoverflow.com/a/21710033 So it might be "malloc-free" for configuration files where the possible values are limited, etc. :)