jkeiser

4 ms·

1.
by jkeiser 6y ago
We have started experimenting how to do it with simdjson, though :) https://github.com/simdjson/simdjson/pull/947
2.
by jkeiser 6y ago
It's kind of a "continued learning" requirement, a little. Work doesn't always have the problems (or funding for them) that grow you in the right directions. Open source is also a sort of portfolio for many people, I thi
3.
by jkeiser 6y ago
What I've learned from this (as a simdjson author) is that we need to update the quick start in the README to have -O3. I was so psyched about the fact that we now compiled warning-free without any parameters ... that I didn't sto
4.
by jkeiser 6y ago
simdjson doesn't care what is in the padding and won't modify it; it just needs the buffer the string lives in to have 32 extra addressable (allocated) bytes. It doesn't ever use the bytes to make decisions, but it may read
5.
Simdjson 0.3: fighting global warming with ridiculously fast JSON parsing(johnkeiser.com)
2 points by jkeiser 7y ago|0 comments
6.
by jkeiser 7y ago
Yep. Most of stage 2's branchiness essentially comes from "is the next thing an array? object? string? number? Handle it differently if so." Making it so you can handle all the brackets at once, all the strings at once, all t
7.
by jkeiser 7y ago
Agreed, we're thinking it through too. Most parsing of JSON could be done almost without any second stage at all, or at least with a direct-to-user-struct/array, if you had the right API.
8.
by jkeiser 7y ago
Worth pointing out: I thought it was just the SIMD that made it fast when I first got involved. It turns out that while it helps, it's just a tool that helps to achieve the real gain: eliminating branches (if statements) that make the
9.
by jkeiser 7y ago
Streaming and selective parsing are good things, and something we're looking into for the next set of features. Note that there are real speed gains to be had by not being selective. The CPU is normally sprinting ahead, executing fut
10.
by jkeiser 7y ago
simdjson does not modify the input, fyi. Rather than \0, which isn't really a part of JSON, the SIMD instructions search for whitespace and JSON structure like commas, colons, etc.
11.
by jkeiser 7y ago
simdjson's spinup overhead is minimal, it runs at a pretty steady rate no matter the size of document: https://github.com/simdjson/simdjson/blob/master/doc/growing... It's also still faste
12.
by jkeiser 7y ago
Yep, simdjson is a fully compliant, validating JSON parser, up to and including full UTF-8 validation. That's part of what makes its speed so eerie.
13.
by jkeiser 7y ago
Yep. You can see the raw numbers at https://simdjson.org/about/ . simdjson UTF8-validation, exact numbers: 2.5 GB/s RapidJSON insitu, UTF8-validation: 0.41 GB/s RapidJSON insitu, UTF8-validation, exact numbers