4 ms·

One area I haven't seen any good solutions for is the interleaving of tests with production code. I think we need better ways to express tests without clutteri

by YZF 9y ago

One area I haven't seen any good solutions for is the interleaving of tests with production code. I think we need better ways to express tests without cluttering the production code and excessive mocking. What I'd like to do, and really can't in any language/tooling that I know of, is to extract some arbitrary subset of the code and surround it with tests or a test harness. I'd also like to specify various injection points for tests right as I'm writing the code in a way that doesn't affect the production code and contributes to its readability. Perhaps this is pieces of server and client code or various services all tested together. When I refactor the code I want the tests to "refactor" with the code. I don't want to need to rewrite my tests... Automatic test generation is another interesting area (beyond the basic table or algorithm stuff we might do in tests today). It's almost like the "problem" isn't the languages themselves, it's the tooling around the languages. I want to be able to do a lot more than "compile", "run", with my code... I can imagine machine learning driven tooling being able to automate a lot of the mechanical aspects of code writing beyond the simple generate a closing bracket that an IDE can do...

enobrev 9y ago
I can imagine something like a code-collapse indicator next to every method in a class (or function or code block, whatever), which would expand tests that can be run on-the-fly - even while you type. Kind of like how comments can be collapsed in some IDEs. Technically, these could very well be simple commented annotations that point to a separate .test file or something, but are shown in-line by your favorite IDE or code-editor.
tomelders 9y ago
I don’t do coding tests. I talk to the candidate about programming and I only require one interview. I haven’t made a hiring mistake in 6 years.
- closeparen 9y ago
  Uh, this is about automated testing of programs, not interviewing.
  - tomelders 9y ago
    I'm an idiot.
- NotSammyHagar 9y ago
  okay. care to give any other info? Usually when someone says I haven't failed in 'some impressive challenge in a long time' it means your challenge wasn't that impressive. How much interviewing have you done, what were the hard calls you were asked to me, whats your scheme? When I was at google, one of the hardest thing I did was look at the marginal intern review scores and tried to pick out the ones from the big pile whom we should look at again and who should not be looked at. There were all kinds of crazy ass stupid interview questions that in my opinion were not very useful to classify capability.
  - coldtea 9y ago
    >okay. care to give any other info? Usually when someone says I haven't failed in 'some impressive challenge in a long time' it means your challenge wasn't that impressive. Or they couldn't recognize failure
- 35bge57dtjku 9y ago
  I'd love to know how you do that, but the answer is always, "I just ask them the right questions and can tell by their answers," which is so vague it's useless.
  - deleted 9y ago
    [deleted]
geogriffin 9y ago
William Byrd's work in program synthesis is an interesting take on this, where an IDE can, guided by test cases written for a function, actually auto-complete code itself whilst writing the function, or tell the programmer when they have something wrong by violating a test case. Of course it is impractical right now, but a good step nonetheless.. It (Barliman) is demoed near the end of this video: https://youtu.be/OyfBQmvr2Hc https://youtu.be/OyfBQmvr2Hc
- zmonx 9y ago
  This is indeed an extremely interesting approach. It was pioneered in the context of logic programming and Prolog by Ehud Shapiro in his 1982 PhD thesis "Algorithmic Program Debugging". The thesis was published as an ACM Distinguished Dissertation by MIT press and is available online from: http://cpsc.yale.edu/sites/default/files/files/tr237.pdf http://cpsc.yale.edu/sites/default/files/files/tr237.pdf Together with Leon Sterling, Ehud Shapiro later wrote a very important introductory Prolog book called "The Art of Prolog".
- mrkgnao 9y ago
  MagicHaskeller[0] (which seems to sadly be offline at present) is a similar project that infers Haskell functions from properties like reverse "abcde" = "edcba" Other, older projects include Exference[1] and Djinn[2], in decreasing order of power. Also, this is really similar to how Idris and Agda work, except they use expressive types to generate the code (using the Emacs modes) rather than test cases. [0]: http://nautilus.cs.miyazaki-u.ac.jp/cgi-bin/MagicHaskeller.cgi http://nautilus.cs.miyazaki-u.ac.jp/cgi-bin/MagicHaskeller.c... [1]: https://github.com/lspitzner/exference/ https://github.com/lspitzner/exference/ [2]: http://www.hedonisticlearning.com/djinn/ http://www.hedonisticlearning.com/djinn/
yogthos 9y ago
I recommend taking a look at Clojure Spec https://clojure.org/about/spec https://clojure.org/about/spec or Racket Contracts https://docs.racket-lang.org/guide/contracts.html https://docs.racket-lang.org/guide/contracts.html
- polymeris 9y ago
  I get clojure.spec is primary a way to define a schema and validate against it, and that you can use it for automatic test generation, too. But I feel I fail to grok the whole extent of the possibilities it offers. Does it address any of the other things YZF mentions? Specially, the "prevent excessive mocking" part?
  - angled 9y ago
    Eiffel? https://www.eiffel.com/values/design-by-contract/introduction/ https://www.eiffel.com/values/design-by-contract/introductio...
  - yogthos 9y ago
    I think this is a good example of Spec being used to guide the solution http://gigasquidsoftware.com/blog/2016/05/29/one-fish-spec-fish/ http://gigasquidsoftware.com/blog/2016/05/29/one-fish-spec-f... I don't think Spec alone removes the need for mocking, using Clean Architecture is probably more important for that https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-architecture.html https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-arc... If you structure your app so that any IO lives at the edges, it becomes possible to test all your business logic without mocking. Spec can help at that point by providing generative testing. Since Spec
rexpop 9y ago
Forgive me, this is going to sound snarky but I promise I am earnest: > I'd also like to specify various injection points for tests right as I'm writing the code You mean, like methods/functions? > extract some arbitrary subset of the code and surround it with tests You mean, like modules/classes?
- TeMPOraL 9y ago
  Kind of, but three big problems in practice are: - runtime state you need to initialize if your code is written in a stateful manner - other methods/functions in different modules/classes the code you want to test calls out to - the fact that method/function and module/class separation is (and IMO always should be) primarily driven by needs of production architecture, not testing, means that it may not be perfect for testing needs Add to that the failure of testing tools and methodologies (like "don't test private methods" meeting "separate out duplicated or complex code into private methods"), and I feel the problems are real. ---- Here's an idea that just popped into my head right now: how about "hash-based testing"? You take a code you want to test, like: private Integer foo(String bar, Frobnicator quux) { String frob = quux.invokeMagic(bar); return memberApi.transform(frob); } and turn it into: Method cut = << private Integer foo(String bar, Frobnicator quux) { String frob = MOCK(quux.invokeMagic(bar)) AS("frob"); return MOCK(memberApi.transform(frob)) AS (frob.length() > 3); }>>; //continue testing cut() The idea being, the compiler or whatever external tooling ensures that the original method in production code, and the inline-modified method in tests, are the same with respect to some equality/hashing function that always treats expressions "foo" and "MOCK(foo) AS(bar)" as equal. This way, you end up being able to mock everything precisely, inline, in whatever way you see fit, with your tooling ensuring that the actual code stays in sync with the test, since whenever the original method changes in any meaningful way, you'll fail the code "equality" test. Might be a stupid idea, I welcome comments. (INB4 testing to implementation instead of the interface - if you have to mock anything, you're already testing to implementation, and this way you can inject testing alterations precisely, instead of having to turn your architecture inside-out to support IoC/DI/whatever the current testing-enabling OOP fad is.)
  - lmm 9y ago
WalterBright 9y ago
It many not be exactly what you want, but D supports unit tests embedded directly in the production code: https://dlang.org/spec/unittest.html It's been a real game changer for us in improving the quality of the code.
tiuPapa 9y ago
Doesn't Rust and cargo do this kinda well?
- zbraniecki 9y ago
  I really like how rust does unit test in line and in docs.
  - tiuPapa 9y ago
    You can also have integration testing.
mpweiher 9y ago
> interleaving of tests with production code > tests without cluttering the production code and excessive mocking IMHO, tests should be an integral part of production code. In MPWTest[1], tests are typically expressed on the class side of the class under test, in a category called Testing, though it's easy to override that. This solves 2 of my major annoyances with xUnit-style testing: dual class/test hierarchies and (with static typing) the need to add public interface for the tests, which don't have privileged access. Mocking in particular and stubbing should be eliminated as much as possible, but this is not a programming language issue[2], more a "not making architectural assumptions too early" issue. [1] https://github.com/mpw/MPWTest https://github.com/mpw/MPWTest [2] http://blog.metaobject.com/2014/05/why-i-don-mock.html http://blog.metaobject.com/2014/05/why-i-don-mock.html
deleted 9y ago
[deleted]

by YZF 9y ago

enobrev 9y ago

I can imagine something like a code-collapse indicator next to every method in a class (or function or code block, whatever), which would expand tests that can be run on-the-fly - even while you type. Kind of like how comments can be collapsed in some IDEs. Technically, these could very well be simple commented annotations that point to a separate .test file or something, but are shown in-line by your favorite IDE or code-editor.

tomelders 9y ago

I don’t do coding tests. I talk to the candidate about programming and I only require one interview. I haven’t made a hiring mistake in 6 years.

closeparen 9y ago
Uh, this is about automated testing of programs, not interviewing.
- tomelders 9y ago
  I'm an idiot.
NotSammyHagar 9y ago
okay. care to give any other info? Usually when someone says I haven't failed in 'some impressive challenge in a long time' it means your challenge wasn't that impressive. How much interviewing have you done, what were the hard calls you were asked to me, whats your scheme? When I was at google, one of the hardest thing I did was look at the marginal intern review scores and tried to pick out the ones from the big pile whom we should look at again and who should not be looked at. There were all kinds of crazy ass stupid interview questions that in my opinion were not very useful to classify capability.
- coldtea 9y ago
  >okay. care to give any other info? Usually when someone says I haven't failed in 'some impressive challenge in a long time' it means your challenge wasn't that impressive. Or they couldn't recognize failure
35bge57dtjku 9y ago
I'd love to know how you do that, but the answer is always, "I just ask them the right questions and can tell by their answers," which is so vague it's useless.
- deleted 9y ago
  [deleted]

geogriffin 9y ago

William Byrd's work in program synthesis is an interesting take on this, where an IDE can, guided by test cases written for a function, actually auto-complete code itself whilst writing the function, or tell the programmer when they have something wrong by violating a test case. Of course it is impractical right now, but a good step nonetheless.. It (Barliman) is demoed near the end of this video: https://youtu.be/OyfBQmvr2Hc https://youtu.be/OyfBQmvr2Hc

zmonx 9y ago
This is indeed an extremely interesting approach. It was pioneered in the context of logic programming and Prolog by Ehud Shapiro in his 1982 PhD thesis "Algorithmic Program Debugging". The thesis was published as an ACM Distinguished Dissertation by MIT press and is available online from: http://cpsc.yale.edu/sites/default/files/files/tr237.pdf http://cpsc.yale.edu/sites/default/files/files/tr237.pdf Together with Leon Sterling, Ehud Shapiro later wrote a very important introductory Prolog book called "The Art of Prolog".
mrkgnao 9y ago
MagicHaskeller[0] (which seems to sadly be offline at present) is a similar project that infers Haskell functions from properties like reverse "abcde" = "edcba" Other, older projects include Exference[1] and Djinn[2], in decreasing order of power. Also, this is really similar to how Idris and Agda work, except they use expressive types to generate the code (using the Emacs modes) rather than test cases. [0]: http://nautilus.cs.miyazaki-u.ac.jp/cgi-bin/MagicHaskeller.cgi http://nautilus.cs.miyazaki-u.ac.jp/cgi-bin/MagicHaskeller.c... [1]: https://github.com/lspitzner/exference/ https://github.com/lspitzner/exference/ [2]: http://www.hedonisticlearning.com/djinn/ http://www.hedonisticlearning.com/djinn/

yogthos 9y ago

I recommend taking a look at Clojure Spec https://clojure.org/about/spec https://clojure.org/about/spec or Racket Contracts https://docs.racket-lang.org/guide/contracts.html https://docs.racket-lang.org/guide/contracts.html

polymeris 9y ago
I get clojure.spec is primary a way to define a schema and validate against it, and that you can use it for automatic test generation, too. But I feel I fail to grok the whole extent of the possibilities it offers. Does it address any of the other things YZF mentions? Specially, the "prevent excessive mocking" part?
- angled 9y ago
  Eiffel? https://www.eiffel.com/values/design-by-contract/introduction/ https://www.eiffel.com/values/design-by-contract/introductio...
- yogthos 9y ago
  I think this is a good example of Spec being used to guide the solution http://gigasquidsoftware.com/blog/2016/05/29/one-fish-spec-fish/ http://gigasquidsoftware.com/blog/2016/05/29/one-fish-spec-f... I don't think Spec alone removes the need for mocking, using Clean Architecture is probably more important for that https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-architecture.html https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-arc... If you structure your app so that any IO lives at the edges, it becomes possible to test all your business logic without mocking. Spec can help at that point by providing generative testing. Since Spec

rexpop 9y ago

Forgive me, this is going to sound snarky but I promise I am earnest: > I'd also like to specify various injection points for tests right as I'm writing the code You mean, like methods/functions? > extract some arbitrary subset of the code and surround it with tests You mean, like modules/classes?

TeMPOraL 9y ago
Kind of, but three big problems in practice are: - runtime state you need to initialize if your code is written in a stateful manner - other methods/functions in different modules/classes the code you want to test calls out to - the fact that method/function and module/class separation is (and IMO always should be) primarily driven by needs of production architecture, not testing, means that it may not be perfect for testing needs Add to that the failure of testing tools and methodologies (like "don't test private methods" meeting "separate out duplicated or complex code into private methods"), and I feel the problems are real. ---- Here's an idea that just popped into my head right now: how about "hash-based testing"? You take a code you want to test, like: private Integer foo(String bar, Frobnicator quux) { String frob = quux.invokeMagic(bar); return memberApi.transform(frob); } and turn it into: Method cut = << private Integer foo(String bar, Frobnicator quux) { String frob = MOCK(quux.invokeMagic(bar)) AS("frob"); return MOCK(memberApi.transform(frob)) AS (frob.length() > 3); }>>; //continue testing cut() The idea being, the compiler or whatever external tooling ensures that the original method in production code, and the inline-modified method in tests, are the same with respect to some equality/hashing function that always treats expressions "foo" and "MOCK(foo) AS(bar)" as equal. This way, you end up being able to mock everything precisely, inline, in whatever way you see fit, with your tooling ensuring that the actual code stays in sync with the test, since whenever the original method changes in any meaningful way, you'll fail the code "equality" test. Might be a stupid idea, I welcome comments. (INB4 testing to implementation instead of the interface - if you have to mock anything, you're already testing to implementation, and this way you can inject testing alterations precisely, instead of having to turn your architecture inside-out to support IoC/DI/whatever the current testing-enabling OOP fad is.)
- lmm 9y ago

WalterBright 9y ago

It many not be exactly what you want, but D supports unit tests embedded directly in the production code: https://dlang.org/spec/unittest.html It's been a real game changer for us in improving the quality of the code.

tiuPapa 9y ago

Doesn't Rust and cargo do this kinda well?

zbraniecki 9y ago
I really like how rust does unit test in line and in docs.
- tiuPapa 9y ago
  You can also have integration testing.

mpweiher 9y ago

> interleaving of tests with production code > tests without cluttering the production code and excessive mocking IMHO, tests should be an integral part of production code. In MPWTest[1], tests are typically expressed on the class side of the class under test, in a category called Testing, though it's easy to override that. This solves 2 of my major annoyances with xUnit-style testing: dual class/test hierarchies and (with static typing) the need to add public interface for the tests, which don't have privileged access. Mocking in particular and stubbing should be eliminated as much as possible, but this is not a programming language issue[2], more a "not making architectural assumptions too early" issue. [1] https://github.com/mpw/MPWTest https://github.com/mpw/MPWTest [2] http://blog.metaobject.com/2014/05/why-i-don-mock.html http://blog.metaobject.com/2014/05/why-i-don-mock.html

deleted 9y ago

[deleted]