5 ms·

Specs are important but I'd argue a conformance test is equally or even more important. With a conformance test people can test their implementations and when

Specs are important but I'd argue a conformance test is equally or even more important. With a conformance test people can test their implementations and when ambiguities arrise new tests can be added or old tests fixed. Without, conformance tests different interpretations of a spec lead to divergent behavior. Once that behavior is out there long enough it becomes difficult to fix as people are depended on it and/or its quirks.

mistercow 14y ago
I'm confused. How are you supposed to have a conformance test without first having a spec? To what would the implementations be conforming, if not a spec?
- greggman 14y ago
  Sorry, What I meant to really say is specs are not sufficient. Sure specs are important. But even with specs, without comprehensive conformance tests implementations will diverge. I work on the WebGL spec and tests based off the OpenGL ES spec and tests. Even though the OpenGL ES spec is long and detailed, every edge case for which there is no test is broken or different in one driver or another.
- ZeroGravitas 14y ago
  It's entirely possible to build a conformance test suite around an existing 'black box' of code and accept that whatever it does, even stuff that might be considered a bug, is what any compliant implementation should also do. (You could also fix some of the more egregious bugs, and then expect implementations to follow your new behaviour)
- jamesrom 14y ago
  In an ideal world, the spec would be the test. A spec is just a human readable test. A test is just a machine readable spec.
th 14y ago
> Without, conformance tests different interpretations of a spec lead to divergent behavior. I think you mean either: > Without conformance tests incorrect interpretations of a spec lead to divergent behavior Or > Without conformance tests different interpretations of an ambiguous spec lead to divergent behavior. Conformance tests should test specific behavior defined in the spec. Tests will never (or rarely) prove spec conformance but spec conformance should always prove test conformance.
- lmm 14y ago
  I have yet to encounter a spec that was not in some way ambiguous.
  - jjr 14y ago
    1 + 1 = 2 Is fairly unambiguous until you introduce operator overloading (and why i despise overloading).
    - nollidge 14y ago
      Overloading is a feature of natural language though, and therefore completely unavoidable in writing a spec.
    - JadeNB 14y ago
      > 1 + 1 = 2 Is fairly unambiguous until you introduce operator overloading (and why i despise overloading). What is it specifying—or, perhaps a better question, what would an implementation look like? It might be more precise to say "I have yet to encounter an implemented spec that was not in some way ambiguous."
    - _pferreir_ 14y ago
      > 1 + 1 = 2 Is fairly unambiguous until you introduce operator overloading (and why i despise overloading). It is only unambiguous if you specify the numbering system you are using ;) </pedantic>
      - 14y ago
SideburnsOfDoom 14y ago
> Specs are important but I'd argue a conformance test is equally or even more important. I agree, but the article mentions tests prominently. They didn't miss that out.
- bergie 14y ago
  Here are the tests we wrote for HTML-to-Markdown conversion. Could be useful when switched the other way around :-) https://github.com/bergie/to-markdown/blob/master/test/tests.js https://github.com/bergie/to-markdown/blob/master/test/tests...
  - draegtun 14y ago
    Also worth looking at the tests for Text::Markdown which is the canonical version of Markdown on CPAN. https://metacpan.org/source/BOBTFISH/Text-Markdown-1.000031/t https://metacpan.org/source/BOBTFISH/Text-Markdown-1.000031/...

mistercow 14y ago

I'm confused. How are you supposed to have a conformance test without first having a spec? To what would the implementations be conforming, if not a spec?

greggman 14y ago
Sorry, What I meant to really say is specs are not sufficient. Sure specs are important. But even with specs, without comprehensive conformance tests implementations will diverge. I work on the WebGL spec and tests based off the OpenGL ES spec and tests. Even though the OpenGL ES spec is long and detailed, every edge case for which there is no test is broken or different in one driver or another.
ZeroGravitas 14y ago
It's entirely possible to build a conformance test suite around an existing 'black box' of code and accept that whatever it does, even stuff that might be considered a bug, is what any compliant implementation should also do. (You could also fix some of the more egregious bugs, and then expect implementations to follow your new behaviour)
jamesrom 14y ago
In an ideal world, the spec would be the test. A spec is just a human readable test. A test is just a machine readable spec.

th 14y ago

> Without, conformance tests different interpretations of a spec lead to divergent behavior. I think you mean either: > Without conformance tests incorrect interpretations of a spec lead to divergent behavior Or > Without conformance tests different interpretations of an ambiguous spec lead to divergent behavior. Conformance tests should test specific behavior defined in the spec. Tests will never (or rarely) prove spec conformance but spec conformance should always prove test conformance.

lmm 14y ago
I have yet to encounter a spec that was not in some way ambiguous.
- jjr 14y ago
  1 + 1 = 2 Is fairly unambiguous until you introduce operator overloading (and why i despise overloading).
  - nollidge 14y ago
    Overloading is a feature of natural language though, and therefore completely unavoidable in writing a spec.
  - JadeNB 14y ago
    > 1 + 1 = 2 Is fairly unambiguous until you introduce operator overloading (and why i despise overloading). What is it specifying—or, perhaps a better question, what would an implementation look like? It might be more precise to say "I have yet to encounter an implemented spec that was not in some way ambiguous."
  - _pferreir_ 14y ago
    > 1 + 1 = 2 Is fairly unambiguous until you introduce operator overloading (and why i despise overloading). It is only unambiguous if you specify the numbering system you are using ;) </pedantic>
    - 14y ago

SideburnsOfDoom 14y ago

> Specs are important but I'd argue a conformance test is equally or even more important. I agree, but the article mentions tests prominently. They didn't miss that out.

bergie 14y ago
Here are the tests we wrote for HTML-to-Markdown conversion. Could be useful when switched the other way around :-) https://github.com/bergie/to-markdown/blob/master/test/tests.js https://github.com/bergie/to-markdown/blob/master/test/tests...
- draegtun 14y ago
  Also worth looking at the tests for Text::Markdown which is the canonical version of Markdown on CPAN. https://metacpan.org/source/BOBTFISH/Text-Markdown-1.000031/t https://metacpan.org/source/BOBTFISH/Text-Markdown-1.000031/...