4 ms·

The problem is that the more tools you add, the more mistakes you make in the tools. This is the same debate we had previously about formal methods. They don't

The problem is that the more tools you add, the more mistakes you make in the tools. This is the same debate we had previously about formal methods. They don't eliminate errors, they just move them to the testing systems and where there are several the gaps between them. Communication becomes more difficult between the people involved and you rapidly start looking for the Chief Programmer Solomon who knows everything. then you get communication failures because not everybody is Einstein. So it is always a trade off. And it would help if we stopped calling automated testing as testing and described what it is - a domain specific type system checker. If we realised we always build a system and a compiler for that system we might stop reinventing the wheel so much.

taneq 9y ago
I don't get why automated testing is a "domain specific type system checker". Not saying it's not, I just don't get it. Maybe that's because my automated tests are pretty free-form and tend to be more about throwing mud at an algorithm and seeing if the results stick. Your last line is bang on, though. We build a system, and a compiler (or processor) for that system. It doesn't matter if everything is "data driven" if your data is code and the thing it's driving is an interpreter.
- marcosdumay 9y ago
  > I don't get why automated testing is a "domain specific type system checker". That's because he was talking about formal verification. Testing like the name is normally used is a "adhock domain specific heuristic type system checker".
- megaman22 9y ago
  I can see that label applying if you're using a language that has weak dynamic typing, so that you have to be checking all manner of crap that the compiler should flag for you. Like Python and JS unit tests suites I've seen too many of.
sanxiyn 9y ago
Formal methods, in theory, don't eliminate errors. In practice, though, they do, as dramatically demonstrated by fuzzing CompCert C compiler.
- aidenn0 9y ago
  IIRC they were actually able to get CompCert to generate incorrect code when fuzzing due to a bug in a system header file. Formal methods do (both in theory and practice) eliminate errors that can occur within the scope of what they cover. If there is an error in your model for the underlying system, then formal methods don't save you, but it seems obvious that eliminating one source of bugs reduces overall bugs. Furthermore, if there is an error for your formal model for the underlying system, there would also probably be an error with your informal model that your programmers would have in their head while developing software without formal methods, yielding similar bugs. All software should be tested; that doesn't mean formal methods have no value.
Cthulhu_ 9y ago
Agreed, each tool - process, framework, ritual, management layer, etc - adds more complexity to juggle, which hides the original problem and/or source code that does the actual work. It's why I like the direction the Go language and its proponents point towards, less tooling, simpler language with sane defaults and a complete standard library, and a mindset to go with it (things like "Nope you don't need a web application framework, just use the standard library first")
- jnbiche 9y ago
  I would agree that additional tools like static analyzers and linters add extra work in the beginning, sometimes significant extra work. But once they're up and running, they greatly decrease the amount of work I have to do, in my experience. I do like Go's sane defaults and awesome standard library, although I find it wanting as a language (Rust, which ships w/Cargo, also has sane defaults and is more what I look for in a programming language).

taneq 9y ago

I don't get why automated testing is a "domain specific type system checker". Not saying it's not, I just don't get it. Maybe that's because my automated tests are pretty free-form and tend to be more about throwing mud at an algorithm and seeing if the results stick. Your last line is bang on, though. We build a system, and a compiler (or processor) for that system. It doesn't matter if everything is "data driven" if your data is code and the thing it's driving is an interpreter.

marcosdumay 9y ago
> I don't get why automated testing is a "domain specific type system checker". That's because he was talking about formal verification. Testing like the name is normally used is a "adhock domain specific heuristic type system checker".
megaman22 9y ago
I can see that label applying if you're using a language that has weak dynamic typing, so that you have to be checking all manner of crap that the compiler should flag for you. Like Python and JS unit tests suites I've seen too many of.

sanxiyn 9y ago

Formal methods, in theory, don't eliminate errors. In practice, though, they do, as dramatically demonstrated by fuzzing CompCert C compiler.

aidenn0 9y ago
IIRC they were actually able to get CompCert to generate incorrect code when fuzzing due to a bug in a system header file. Formal methods do (both in theory and practice) eliminate errors that can occur within the scope of what they cover. If there is an error in your model for the underlying system, then formal methods don't save you, but it seems obvious that eliminating one source of bugs reduces overall bugs. Furthermore, if there is an error for your formal model for the underlying system, there would also probably be an error with your informal model that your programmers would have in their head while developing software without formal methods, yielding similar bugs. All software should be tested; that doesn't mean formal methods have no value.

Cthulhu_ 9y ago

Agreed, each tool - process, framework, ritual, management layer, etc - adds more complexity to juggle, which hides the original problem and/or source code that does the actual work. It's why I like the direction the Go language and its proponents point towards, less tooling, simpler language with sane defaults and a complete standard library, and a mindset to go with it (things like "Nope you don't need a web application framework, just use the standard library first")

jnbiche 9y ago
I would agree that additional tools like static analyzers and linters add extra work in the beginning, sometimes significant extra work. But once they're up and running, they greatly decrease the amount of work I have to do, in my experience. I do like Go's sane defaults and awesome standard library, although I find it wanting as a language (Rust, which ships w/Cargo, also has sane defaults and is more what I look for in a programming language).