4 ms·

I wonder why they used a SQL fuzzer as opposed to the nist compatibility guidelines. I've had the honor (torture) of implementing SQL 4 times and it was the onl

I wonder why they used a SQL fuzzer as opposed to the nist compatibility guidelines. I've had the honor (torture) of implementing SQL 4 times and it was the only way I could effectively test compliance.

kqr 7y ago
Because of interaction effects. You can have something that passes all compatibility requirement examples yet fails in weird ways for long, strange queries that combine various features in ways not found in compatibility requirement examples, simply because these examples cannot possibly span the full, infinite input space. As a simple example, maybe your system handles select queries fine, and it handles all legal field names, but some bug causes it to do the wrong thing when the third field in your select list is called "ape" because that matches the name of some internal bookkeeping structure you have. Does the code have to be badly designed for that to even be possible? Yes, but that is also the type of code we tend to find bugs in.
- alexnewman 7y ago
  So, I don't want to sound snarky, because what you are saying is not false, it just misses something. The nist compatibility guidelines aren't just about syntactic guidelines, they are about checking to make sure the isolation levels work correctly. The tests are meant to be extensive and reproducible. I'm not saying they are perfect, but I would suggest databases have been testing these particular issues for a long time. Another good tool to checkout is jepsen. Anyway, Nist compatibility guidelines are underused and as a result open source databases miss out.
  - kqr 7y ago
    I agree both would be a good idea. I was just suggesting they might not be sufficient alone. I like that you pointed them out! Sorry for coming across a bit contrarian.
    - alexnewman 7y ago
      Ah now I feel like I was wrong to make you feel as though you are contrarian. It was the right balance to bring knowledge to the post. Thanks for your input.

kqr 7y ago

Because of interaction effects. You can have something that passes all compatibility requirement examples yet fails in weird ways for long, strange queries that combine various features in ways not found in compatibility requirement examples, simply because these examples cannot possibly span the full, infinite input space. As a simple example, maybe your system handles select queries fine, and it handles all legal field names, but some bug causes it to do the wrong thing when the third field in your select list is called "ape" because that matches the name of some internal bookkeeping structure you have. Does the code have to be badly designed for that to even be possible? Yes, but that is also the type of code we tend to find bugs in.

alexnewman 7y ago
So, I don't want to sound snarky, because what you are saying is not false, it just misses something. The nist compatibility guidelines aren't just about syntactic guidelines, they are about checking to make sure the isolation levels work correctly. The tests are meant to be extensive and reproducible. I'm not saying they are perfect, but I would suggest databases have been testing these particular issues for a long time. Another good tool to checkout is jepsen. Anyway, Nist compatibility guidelines are underused and as a result open source databases miss out.
- kqr 7y ago
  I agree both would be a good idea. I was just suggesting they might not be sufficient alone. I like that you pointed them out! Sorry for coming across a bit contrarian.
  - alexnewman 7y ago
    Ah now I feel like I was wrong to make you feel as though you are contrarian. It was the right balance to bring knowledge to the post. Thanks for your input.