6 ms·

According to the OjC README: > No official benchmarks are available but in informal tests Oj is 30% faster that [sic] simdjson. Source: https://github.com/ohl

According to the OjC README: > No official benchmarks are available but in informal tests Oj is 30% faster that [sic] simdjson. Source: https://github.com/ohler55/ojc/blob/master/README.md#benchmarks https://github.com/ohler55/ojc/blob/master/README.md#benchma... If you have benchmarks that show otherwise, that would be great for the discussion here, but your point appears to have already been addressed?

svnpenn 6y ago
Thats one person saying it is faster. Without a reproducible test, thats pretty much worthless.
- coder543 6y ago
  You don’t see the irony in that statement? You’re one person saying simdjson is faster, and you are also lacking a reproducible test. That doesn’t make your side of the argument particularly compelling either. I’ve never used either library, so I don’t care which one is faster, but I would generally trust the author of the library to have tested the performance at some point more than I would trust someone who isn’t the author to comment on the performance of code they’ve never tested. The author of that library isn’t present to make their case, so you’re the only one of the two who could provide evidence right now. EDIT: the author is present now: https://news.ycombinator.com/item?id=23738512 https://news.ycombinator.com/item?id=23738512
  - the-dude 6y ago
    Your comment would improve without the last sentence.
    - coder543 6y ago
      I was iterating on the comment rapidly after posting to try to make my point in the most neutral way possible, so I’m not sure which “last sentence” you saw, but hopefully it is better now.
      - the-dude 6y ago
        This is much better, no need for ad hominems. But please keep the editorializing in check too, it is etiquette to mention edits, and continually editting breaks continuity in the discussion. As you pointed out yourself.
  - nemothekid 6y ago
    >You don’t see the irony in that statement? You’re one person saying simdjson is faster, and you are also lacking a reproducible test. That doesn’t make your side of the argument particularly compelling either. simdjson is a massively popular library (~9.9k stars, the third most popular json library on GitHub), that is well known to beat the pants off of every other json library. Likewise the simdjson library has reproducible benchmarks in its own repo with charts and the like. So yes, I would consider that statement pretty much "worthless" given the popularity of simdjson.
  - glangdale 6y ago
    Given how things turned out, this might be a good lesson as to whether "I measured my own library informally on some benchmarks that I couldn't be bothered to describe to you and it WINS!" is trustworthy in future. When we introduced simdjson, we went out of our way to explain (a) why we thought it might be faster than other libraries and (b) how people could replicate our measurements or find fault in our methodology. There were zero performance claims, public or private about the performance of simdjson before this material was available. In my long experience, undocumented and non-reproducible performance claims suffer from the fact that people generally stop benchmarking when they get the result that they wanted to hear, even if the absolute numbers involved are unreasonable. It's very easy to make mistakes in performance measurement (compiling simdjson with anything short of -O2 or -O3, doing repeated runs of a tiny input and thus getting perfect branch prediction behavior), but people generally fix these mistakes only on the side that makes their stuff look good. Back when I worked on the Hyperscan project ( https://github.com/intel/hyperscan https://github.com/intel/hyperscan ), we had an API call that allocated scratch memory (necessary due to Hyperscan's rather constrained approach to memory usage). I lost count of the number of times we had users run this allocation (which only ever needed to be done once per thread, effectively) inside their measurement loop. And whenever we were competing against an incumbent system and the people doing the measurement were the authors of that incumbent system, it was amazing how quickly they'd accept the "truth" of their benchmark run.
    - coder543 6y ago
      I think it’s worth clarifying that I said I would “generally trust {author} more than {random internet person who hasn’t even tested author’s library}”. Are you saying that you trust random internet person more by default? Trusting one more than the other is not the same as blindly trusting one and never running my own tests. The author who put in the effort to develop and benchmark their library deserves more benefit of the doubt than some drive-by commenter, in my opinion. If I needed a high performance JSON parsing library, I would definitely set up a benchmark representative of my use case, and test a few different libraries. JSON libraries are usually pretty ergonomic and easy to swap between, so it shouldn’t be an arduous task. Your very first two comments in this thread were far more insightful than {random internet person} just saying over and over that simdjson can’t possibly be slower, and those comments of yours on their own merit would have removed the need for me to leave any comments whatsoever to defend the original HN post, even without needing to observe that you’re an expert in this field, which just lends further credibility to your comments. I think the rest of the discussion that occurred was really interesting to watch!
glangdale 6y ago
I find it surprising that a fairly straightforward parser runs "30% faster" than simdjson (full disclosure: I wrote the original simdjson code) while parsing things byte by byte with copious conditional branches. This looks exactly like the approach used in most JSON parsers, which generally were beaten by simdjson by an order of magnitude or more. "Informal tests", especially ones with zero description as to what was being tested or even ballpark numbers as to how fast this system supposedly run, may not entirely address this point. We wrote a paper and anyone curious about our methodology can download and run simdjson with a few clicks and verify our performance claims, which (I hope) are well-described in the paper and line up with the arguments we make in favour of the design. Given that no indication of what was being measured is provided, it's kind of hard to tell what this performance claim even is, or how to verify it.
- beached_whale 6y ago
  Are there benchmarks with simdjson getting the JSON reified into native data structures outside the parser?
  - glangdale 6y ago
    No. This is an interesting problem but a non-goal for the C++ version of simdjson, which produces usable "native data structures" albeit not outside the parser (speaking informally - the structures aren't really "in the parser" but they are specific to simdjson if you see what I mean). The reification problem is awkward but it's a different problem for each language. I did daydream of breaking some API boundaries and using SIMD to magic up some Python structures directly, but this would be incredibly brittle. Someone would probably use it but the chance of getting burned by this is... high.
    - jkeiser 6y ago
      We have started experimenting how to do it with simdjson, though :) https://github.com/simdjson/simdjson/pull/947 https://github.com/simdjson/simdjson/pull/947
    - 6y ago
peterohler 6y ago
I'll get the code up on the OjC repo in the next day or two. You are right, there should be reproducible benchmarks to back up the statement.
- peterohler 6y ago
  Benchmarks added at https://github.com/ohler55/ojc/blob/master/test/simdbench/README.md https://github.com/ohler55/ojc/blob/master/test/simdbench/RE...
  - hyperman1 6y ago
    i see you are using strlen in the inner loop in simdjson which is an extra O(n) operation. UPDATE and your test data is quite short. You're maybe mostly measuring startup overhead?
    - peterohler 6y ago
      I tried to make the test realistic. Agreed that in a benchmark where the string is the same every time the length could be hard coded but in a real world scenario the length of a string is not known until you get the length. As for the length of the test, I also tried with 10 times the number of iterations with the same results but then the simbbench takes almost a minute to complete. I figured 5 seconds was good enough to get the results and someone can bump up the iterations if they want to.
      - hyperman1 6y ago
        Sorry, this is not realistic in C. The moment you get the json data, you have to also get the length, or you could not allocate enough memory for it. Also, number of iterations is not the issue. Length of the input is. What happens if you r json is a few kb, is bigger than l1/l2 cache, ....
        peterohler 6y ago
        Have you tried running the tests with a pre-calculated length? When I made that change there was no noticeable difference. The iterations/msec went from 195 to 192 which is not statistically significant. I have no doubt a more complete set of benchmarks could be made with various sized JSON and from file as well as string. This was put together quickly to get an answer to the questions raised in this post. Since it does seem to be a topic of interest I'll expand and cleanup the tests in the future but for now it does give at least one data point. You might notice that the somdjson::dom::parser is reused. Without that optimization to allow warming up the iteration/msec was only 160.

svnpenn 6y ago

Thats one person saying it is faster. Without a reproducible test, thats pretty much worthless.

coder543 6y ago
You don’t see the irony in that statement? You’re one person saying simdjson is faster, and you are also lacking a reproducible test. That doesn’t make your side of the argument particularly compelling either. I’ve never used either library, so I don’t care which one is faster, but I would generally trust the author of the library to have tested the performance at some point more than I would trust someone who isn’t the author to comment on the performance of code they’ve never tested. The author of that library isn’t present to make their case, so you’re the only one of the two who could provide evidence right now. EDIT: the author is present now: https://news.ycombinator.com/item?id=23738512 https://news.ycombinator.com/item?id=23738512
- the-dude 6y ago
  Your comment would improve without the last sentence.
  - coder543 6y ago
    I was iterating on the comment rapidly after posting to try to make my point in the most neutral way possible, so I’m not sure which “last sentence” you saw, but hopefully it is better now.
    - the-dude 6y ago
      This is much better, no need for ad hominems. But please keep the editorializing in check too, it is etiquette to mention edits, and continually editting breaks continuity in the discussion. As you pointed out yourself.
- nemothekid 6y ago
  >You don’t see the irony in that statement? You’re one person saying simdjson is faster, and you are also lacking a reproducible test. That doesn’t make your side of the argument particularly compelling either. simdjson is a massively popular library (~9.9k stars, the third most popular json library on GitHub), that is well known to beat the pants off of every other json library. Likewise the simdjson library has reproducible benchmarks in its own repo with charts and the like. So yes, I would consider that statement pretty much "worthless" given the popularity of simdjson.
- glangdale 6y ago
  Given how things turned out, this might be a good lesson as to whether "I measured my own library informally on some benchmarks that I couldn't be bothered to describe to you and it WINS!" is trustworthy in future. When we introduced simdjson, we went out of our way to explain (a) why we thought it might be faster than other libraries and (b) how people could replicate our measurements or find fault in our methodology. There were zero performance claims, public or private about the performance of simdjson before this material was available. In my long experience, undocumented and non-reproducible performance claims suffer from the fact that people generally stop benchmarking when they get the result that they wanted to hear, even if the absolute numbers involved are unreasonable. It's very easy to make mistakes in performance measurement (compiling simdjson with anything short of -O2 or -O3, doing repeated runs of a tiny input and thus getting perfect branch prediction behavior), but people generally fix these mistakes only on the side that makes their stuff look good. Back when I worked on the Hyperscan project ( https://github.com/intel/hyperscan https://github.com/intel/hyperscan ), we had an API call that allocated scratch memory (necessary due to Hyperscan's rather constrained approach to memory usage). I lost count of the number of times we had users run this allocation (which only ever needed to be done once per thread, effectively) inside their measurement loop. And whenever we were competing against an incumbent system and the people doing the measurement were the authors of that incumbent system, it was amazing how quickly they'd accept the "truth" of their benchmark run.
  - coder543 6y ago
    I think it’s worth clarifying that I said I would “generally trust {author} more than {random internet person who hasn’t even tested author’s library}”. Are you saying that you trust random internet person more by default? Trusting one more than the other is not the same as blindly trusting one and never running my own tests. The author who put in the effort to develop and benchmark their library deserves more benefit of the doubt than some drive-by commenter, in my opinion. If I needed a high performance JSON parsing library, I would definitely set up a benchmark representative of my use case, and test a few different libraries. JSON libraries are usually pretty ergonomic and easy to swap between, so it shouldn’t be an arduous task. Your very first two comments in this thread were far more insightful than {random internet person} just saying over and over that simdjson can’t possibly be slower, and those comments of yours on their own merit would have removed the need for me to leave any comments whatsoever to defend the original HN post, even without needing to observe that you’re an expert in this field, which just lends further credibility to your comments. I think the rest of the discussion that occurred was really interesting to watch!

glangdale 6y ago

I find it surprising that a fairly straightforward parser runs "30% faster" than simdjson (full disclosure: I wrote the original simdjson code) while parsing things byte by byte with copious conditional branches. This looks exactly like the approach used in most JSON parsers, which generally were beaten by simdjson by an order of magnitude or more. "Informal tests", especially ones with zero description as to what was being tested or even ballpark numbers as to how fast this system supposedly run, may not entirely address this point. We wrote a paper and anyone curious about our methodology can download and run simdjson with a few clicks and verify our performance claims, which (I hope) are well-described in the paper and line up with the arguments we make in favour of the design. Given that no indication of what was being measured is provided, it's kind of hard to tell what this performance claim even is, or how to verify it.

beached_whale 6y ago
Are there benchmarks with simdjson getting the JSON reified into native data structures outside the parser?
- glangdale 6y ago
  No. This is an interesting problem but a non-goal for the C++ version of simdjson, which produces usable "native data structures" albeit not outside the parser (speaking informally - the structures aren't really "in the parser" but they are specific to simdjson if you see what I mean). The reification problem is awkward but it's a different problem for each language. I did daydream of breaking some API boundaries and using SIMD to magic up some Python structures directly, but this would be incredibly brittle. Someone would probably use it but the chance of getting burned by this is... high.
  - jkeiser 6y ago
    We have started experimenting how to do it with simdjson, though :) https://github.com/simdjson/simdjson/pull/947 https://github.com/simdjson/simdjson/pull/947
  - 6y ago

peterohler 6y ago

I'll get the code up on the OjC repo in the next day or two. You are right, there should be reproducible benchmarks to back up the statement.

peterohler 6y ago
Benchmarks added at https://github.com/ohler55/ojc/blob/master/test/simdbench/README.md https://github.com/ohler55/ojc/blob/master/test/simdbench/RE...
- hyperman1 6y ago
  i see you are using strlen in the inner loop in simdjson which is an extra O(n) operation. UPDATE and your test data is quite short. You're maybe mostly measuring startup overhead?
  - peterohler 6y ago
    I tried to make the test realistic. Agreed that in a benchmark where the string is the same every time the length could be hard coded but in a real world scenario the length of a string is not known until you get the length. As for the length of the test, I also tried with 10 times the number of iterations with the same results but then the simbbench takes almost a minute to complete. I figured 5 seconds was good enough to get the results and someone can bump up the iterations if they want to.
    - hyperman1 6y ago
      Sorry, this is not realistic in C. The moment you get the json data, you have to also get the length, or you could not allocate enough memory for it. Also, number of iterations is not the issue. Length of the input is. What happens if you r json is a few kb, is bigger than l1/l2 cache, ....
      - peterohler 6y ago
        Have you tried running the tests with a pre-calculated length? When I made that change there was no noticeable difference. The iterations/msec went from 195 to 192 which is not statistically significant. I have no doubt a more complete set of benchmarks could be made with various sized JSON and from file as well as string. This was put together quickly to get an answer to the questions raised in this post. Since it does seem to be a topic of interest I'll expand and cleanup the tests in the future but for now it does give at least one data point. You might notice that the somdjson::dom::parser is reused. Without that optimization to allow warming up the iteration/msec was only 160.