Any company that wants to show benchmarks should, at a minimum show: 1. The hardware on which the tests were run 2. The software (OS) on which the test

Any company that wants to show benchmarks should, at a minimum show: 1. The hardware on which the tests were run 2. The software (OS) on which the tests were run 3. The design of their benchmarking utility 4. How much memory/CPU was available for each test (i.e. how much crap is running in the background) 5. The version of each thing (parser) they are testing 6. How many trials they ran for each test (no reason for a test like this to not have several hundred or thousand) 7. The average and standard deviation for each thing (parser) they are comparing 8. The data ran for the tests. (Some data suites may favour certain parsers over others). Best case: running different kinds of data suites in separate tests, to see if some things (parsers) handle certain kinds of data better than others. (i.e. there might not be a global optimum parser) Additionally, they should also run the parser on a few passes prior to starting benchmarking to "warm it up". This reduces variance in the runs, especially via things like initialization/memory allocation in the first run. Without most (or all!) of that kind of information, one can't make a reasonable judgement.