9 ms·
Fastest JSON parser in the world is a D project?
- wfunction 11y ago> Yep, that's right. stdx.data.json's pull parser finally beats the dynamic languages with native efficiency. 'dynamic' is the key word here. The title is misleading.
- vamega 11y agoThis post is about the fast project, which is claimed to beat RapidJson, a very fast C++ json parser.
- easytiger 11y agoIf it was implemented in c++ using identical algorithms it would be no slower.
- fauigerzigerk 11y agoThat's hard to know. It's always possible that one compiler generates faster code than another one for the same algorithm. I wouldn't be surprised if most of the speed difference was down to correctness checks though.
- easytiger 11y agoThere are full threads on why this impl is faster. It's cheating
- jonnybgood 11y agohttps://github.com/kostya/benchmarks#json https://github.com/kostya/benchmarks#json
- syllogism 11y agoThe fast Python json libraries are all C extensions, though. The only "dynamic" thing they have to do is create the actual Python objects when they're done.
- illumen 11y agoMost of python code that takes time has C/asm extensions. It's part of the 'glue code' nature of python. Right tool for the job and all that.
- masklinn 11y agoYou may want to read past that part. The author was just pointing out that only around DMD 2.067 (that's March 2015 or later) did D finally get a JSON library which did genuinely better than those in dynamic language stdlibs, which hadn't been the case beforehand (the stdlib DMD 2.067 rivaling but not being better than Python's json on the author's system). That's setup for the reveal of "fast" which not only does better than dynamic language stdlibs and than the previous fastest D library but does better than any other JSON parser. The point of the historical recap is how fast things improved for the D ecosystem: in 7 months the best-case option went from parsing JSON 2~3 times slower than Ruby to parsing it in half the time of RapidJSON, a 2 orders of magnitude improvement in speed.
- unwind 11y agoI don't read D, but here's the code anyway: https://github.com/mleise/fast/blob/master/source/fast/json.d https://github.com/mleise/fast/blob/master/source/fast/json.....
- simgidacav 11y agoSo, how is it going with D? Last time I gave it a look I really liked it, but I sadly left the place when I hit the multiple standard libraries problems. Anyone using D in real life, among hackers here?
- krautsourced 11y agoSociomantic is using it for their systems, https://www.sociomantic.com/jobs/d-software-developer/ https://www.sociomantic.com/jobs/d-software-developer/
- _ak 11y agoLast time I seen one of their ads, they emphasized that their code base is still D1. I don't know whether that changed since then.
- FraaJad 11y agothey have since moved to D2. One of their developers gave a talk about the Big Migration in D conf. edit: They have series of blog posts documenting the same too - https://www.sociomantic.com/search/tag/dlang https://www.sociomantic.com/search/tag/dlang (6 so far).
- zamalek 11y agoFacebook does [1][2][3]. [1]: http://www.drdobbs.com/mobile/facebook-adopts-d-language/240162694 http://www.drdobbs.com/mobile/facebook-adopts-d-language/240... [2]: https://github.com/facebook/warp https://github.com/facebook/warp [3]: https://github.com/facebook/flint https://github.com/facebook/flint
- TwoBit 11y agoActually Facebook is backing away from D. Not because they dislike it, but simply to converge on C++.
- 11y ago
- grayrest 11y agoHere's the "how so fast" explanation: https://github.com/kostya/benchmarks/pull/46#issuecomment-147932489 https://github.com/kostya/benchmarks/pull/46#issuecomment-14...
- Tinyyy 11y agoI still don’t get it. ELI5?
- Tobani 11y agoThe parser finds the start/end of sub-structures and doesn't necessarily process them all. So for large structures of which you only need a subset, you're only doing the work you need to do. For a large structure in which you need all of the data, there is probably less of gain.
- valarauca1 11y agoThat plus the parsing is done in SIMD so even in UTF-8 you're processing 4 unicode points per CPU cycle [1], instead of 1 in most traditional parsers. [1] This is a board generalization not necessarily true for all SIMD opcodes.
- thechao 11y agoHm. I guess when thinking "SSE parsing" I didn't go with 4/8-wide parsing. I was thinking that they'd be grabbing 16/32-bytes, doing a compare against a fixed constant literal of 16/32-copies of, say, '{', or '}', then extracting the index of the match, exactly. Something like this: d := _mm_loadu_ps <json> b := _mm_loadu_ps <token: '{'> n := _mm_cmpistrc(d, b) You'd have to be clever to skip around strings ("... \"sonuva ... "), but once that's handled, you'd have significant speed ups to scan for ',', '{', '}', etc. I think the double-quote escape might look something like this: d := _mm_loadu_ps <json> q := _mm_loadu_ps <token: '"'> e := _mm_loadu_ps <token: '\\'> n := _mm_cmpistrc(d, q) m := _mm_cmpistrc(d, e) if m+1 == n: branch-to-top ... process ... Looks like cmpistrc has 1/2 reciprocal throughput. If you unrolled the loop 8 deep, you're probably looking at 10c per 16bytes scanned.
- sgt 11y agoOn another note, I see Scala is doing really poorly in the JSON benchmark test: https://github.com/kostya/benchmarks#json https://github.com/kostya/benchmarks#json
- waxjar 11y agoThose aren't very fair to the dynamic languages and JIT compiled languages (this includes Scala). * For the dynamic languages execution time includes the time it takes to lex, parse and interpret the source code. * For language implementations with a JIT execution time includes the time the JIT takes to properly optimise hot code paths. Generally you start benchmarking after a warm up period in such cases. The only fair comparisons are those between ahead of time compiled languages.
- deleted 11y ago[deleted]
- valarauca1 11y agoThe benchmark suited used the STDLIB json parser which is slow as hell. How slow is it? In 2011 there was a proposal to deprecate it, which never happened [1]. Why? STDLIB parser loads the file line-by-line, then copies line by line into a new buffer that joins the strings together. Then the parser is called. The Scala community normally circumvents by using non-standard community developed solutions. Furthermore JIT vs Compiled language benchmarks are pretty unfair to JIT'd languages. Especially for the JVM which doesn't start to compile sections until >10,000 calls. [1] https://groups.google.com/forum/m/#!msg/scala-user/P7-8PEUUj6A/NonA0_IIovsJ https://groups.google.com/forum/m/#!msg/scala-user/P7-8PEUUj...
- z3t4 11y agoThe D people laugh at the dynamic type languages, yet they have "auto" in-front of all variables ;)
- qznc 11y agoInfered type is different than dynamic type.
- coldtea 11y agoAuto is just a keyword to invoke/assist type inference in the parser -- the objects are still statically typed. C++ 11x (or however it's called) uses the same. You were probably thinking of something like the "dynamic" type that C# has.
- thechao 11y agoSince Jaakko, my advisor, was one of the core authors for 'auto' in C++11: the use of 'auto' for type inference has been in C since the '70s. C++11 just generalized the feature.
- coldtea 11y agoHuh? Maybe it's a joke, but "auto" in C has nothing to do with auto in C++. In C auto just defines the storage location (and is kinda useless), not the type (which you still need to define).
- thechao 11y agoKind of serious-kind of joking: auto x = 3; "Infers" 'int'. Everything else is an error.
- Drup 11y agoAnd apparently, you have no idea what type inference is. ;)
- geofft 11y ago
- _Codemonkeyism 11y agoSounds to me like VW or Nvidia: (Why so fast) "On the downside I did not validate the unused side-structures. I think it is not necessary to validate data you are not using. So basically I only scan them so much as to find where they end. Granted it is a bit of optimization for a benchmark, but is actually handy in real-life as well."
- nailer 11y agoWhat was the nvidia thing? I recall http://techreport.com/review/3089/how-ati-drivers-optimize-quake-iii http://techreport.com/review/3089/how-ati-drivers-optimize-q... , but that was ATI.
- icebraining 11y agohttp://www.geek.com/games/futuremark-confirms-nvidia-is-cheating-in-benchmark-553361/ http://www.geek.com/games/futuremark-confirms-nvidia-is-chea...
- kal31dic 11y agoNot really, because the only people that benefited from VW were Vw managers, whereas as a user I am quite happy to trade off speed for not validating useless fields when processing terabytes of JSON. And he is quite open about the tradeoff, so it's a perfect analogy except for what's different!
- tacone 11y ago> Yep, that's right. stdx.data.json's pull parser finally beats the dynamic languages with native efficiency. (I used the default options here that provide you with an Exception and line number on errors.) Nice to see that it only took a couple of years for the D community to beat in speed the scripting languages.
- nkozyra 11y agoIt's literally the next paragraph that extends this: "A few days ago I decided to get some practical use out of my pet project 'fast' by implementing a JSON parser myself, that could rival even the by then fastest JSON parser, RapidJSON. The result can be seen in the benchmark results right now: https://github.com/kostya/benchmarks#json https://github.com/kostya/benchmarks#json fast: 0.34s, 226.7Mb (GDC) RapidJSON: 0.79s, 687.1Mb (GCC)"
- ma2rten 11y agoPython's json parser is written in C.
- merb 11y agoI don't know but the benchmarks are aweful, since scala and python using their built in Json Parsers and not the fastest parsers available and he is comparing some other super fast libraries against D and says "hoho" it's so fast. Benchmarks are for the poor people who can't live in the real world.
- dbcfd 11y agoBenchmarks would be fine if they could somehow actually compare performance between languages. I have yet to see any benchmarks between languages that are even close to being results that would actually be seen.
- kal31dic 11y agoA strange way of putting it. He started out using the standard library for D with dmd compiler, and then people submitted pull requests to give better information about choices and he incorporated these. Anyone else is free to do the same. If you know of a faster parser in any language, I would love to hear, because getting the job done well matters more to me.
- dbcfd 11y agoAlthough D may have a fast json parser, this benchmark is a horrible comparison. Really need a better way to do cross language comparisons.
- mtanski 11y agoIf I had a nickel for every time I've seen a language benchmark be a very specialized contrived problem (in this case specific JSON with specific access pattern) I'd have a lot of nickels.
- EvenThisAcronym 11y agoThe repo owner is accepting pull requests for the benchmarks so at least it's fixable.
- dbcfd 11y agoIt's a lot of work to fix this benchmark, which is fairly contrived to begin with. To start, have to adjust every sample to accept a warmup time, then run time (which is likely multiple samples), measuring results in that run time, both speed and performance. And you also have to be careful that the compiler is then not optimizing out the repetitions in the runtime, while still allowing optimizations that would produce the best performance.