3 ms·

Conversely this is what I like least about it. It works nicely for small functions, and even reads like documentation. But past that it will balloon and just b

by norswap 10y ago

Conversely this is what I like least about it. It works nicely for small functions, and even reads like documentation. But past that it will balloon and just bloat the program you're trying to understand. Tests are better left in their own file. That's not to mention test infrastructure (mocks etc) should you need it.

jpolitz 10y ago
Thanks for the feedback! To add to what you said, this is exactly the kind of situation where we design features that _may_ not scale well, but are remarkably useful in teaching contexts. Having test cases inline with code is massively useful: 1. For students not used to thinking about their program being split over several files. If tests must be in their own file, then understanding multi-file programs (modules, essentially), becomes a curricular dependency of writing tests. That is a _lot_ of friction early on – think about an intro student who has a fuzzy notion of a "working directory" or IDE "workspace" in general. Even if the eventual goal is to address these concepts, they shouldn't have to come first just to write a unit test. 2. When live coding in front of a class. Having tests in a separate file is a huge pain because you pay either (a) the cognitive overhead of switching buffers, or (b) precious projector real-estate on having both open. Since live-coding programs that are less than 100 lines long is a major use case for Pyret, this is a big win. As programs scale up, for example in Pyret's compiler or in larger assignments, we end up with a blend of test cases in inline where: blocks, with the majority in separate files in standalone check: blocks. So at scale, things turn into more traditional-looking separate suites.
lliamander 10y ago
For writing production software, I have the same kind of concerns. Plus, what if you want do something more than simple unit tests (property based testing, concolic testing, etc) or if someone comes up with a better unit testing framework? You'll still have to setup those tests in a separate file, and then the in-line tests are just some unused feature of the language. However, for educational purposes, I think it's actually a compelling idea. My perspective is that while unit tests are a great idea, each unit test only provides marginal value. Thus, unit tests need to be as easy and natural to write as possible or developers won't do it. Habits, like writing code that doesn't need much mocking, definitely makes TDD easier, but that often comes after habituating oneself to TDD. For students, I think getting them as early as possible to write unit tests would be a big win. Putting the tests right there in the code with no need to set up test infrastructure or learn a separate test framework will be helpful.
pmarreck 10y ago
I think there is a case to be made that unit tests can live inside the file under test; integration tests can live outside of it. Also, mocks are arguably a test-code-smell side effect of too little isolation: https://www.destroyallsoftware.com/blog/2014/test-isolation-is-about-avoiding-mocks https://www.destroyallsoftware.com/blog/2014/test-isolation-... In my Elixir one-page code experiments, I put the unit test in the same file after the code under test. It makes sure the test only runs when the file is loaded a certain way: https://github.com/pmarreck/elixir-snippets/ https://github.com/pmarreck/elixir-snippets/
chubot 10y ago
It seems like the obvious thing is to allow both tests in the source file and separate test file. I like the idea of short functions with doctest-like tests, but written with the real language, syntax-highlighted, etc. In addition to education, it's good for quick prototyping. It's also nice to see at a glance which functions don't have tests! But I can also see it getting a bit unwieldy for complicated stuff, so in that case you can put it in another file, perhaps with a pointer. I'm guessing you can't stop people from writing tests in separate files, so I suppose Pyret allows that. I guess you could write a unit test framework in Python that does this with a tiny amount of metaprogramming. def Foo(): return 5 def testFoo(t): t.assertEqual(Foo(), 5) So you could just look for all the functions in the file named "testFoo" where Foo is also a function, and then run them with the "t" param. Hm.
- qznc 10y ago
  In Python nose can do that for you: http://nose.readthedocs.io/en/latest/ http://nose.readthedocs.io/en/latest/ In D there is even a keyword for that. It allows to embed the tests into the documentation as examples. This has the nice side effect that your examples are actually tested. https://dlang.org/spec/unittest.html https://dlang.org/spec/unittest.html
shriramkmurthi 10y ago
There's absolutely nothing in Pyret forcing you to write `where` blocks with immediate tests. You can also put your tests in a `check` block, which can float anywhere, and in another module. What we find valuable is to write a _small_ number of _illustrative_ examples, which we call the “sweep”, that summarize the main behavior of the function. Think of it as essential, runnable documentation. When you come to a new codebase (including your own, six months later!), these help you quickly page back in what the function was supposed to do. The sweep is also really useful for peer-review, which is another especially valuable technique in both education and industry. We have done some nice studies on its effectiveness: http://cs.brown.edu/~sk/Publications/Papers/Published/pcgfk-sweep-ess-eg-ifpr/ http://cs.brown.edu/~sk/Publications/Papers/Published/pcgfk-...

by norswap 10y ago

jpolitz 10y ago

Thanks for the feedback! To add to what you said, this is exactly the kind of situation where we design features that _may_ not scale well, but are remarkably useful in teaching contexts. Having test cases inline with code is massively useful: 1. For students not used to thinking about their program being split over several files. If tests must be in their own file, then understanding multi-file programs (modules, essentially), becomes a curricular dependency of writing tests. That is a _lot_ of friction early on – think about an intro student who has a fuzzy notion of a "working directory" or IDE "workspace" in general. Even if the eventual goal is to address these concepts, they shouldn't have to come first just to write a unit test. 2. When live coding in front of a class. Having tests in a separate file is a huge pain because you pay either (a) the cognitive overhead of switching buffers, or (b) precious projector real-estate on having both open. Since live-coding programs that are less than 100 lines long is a major use case for Pyret, this is a big win. As programs scale up, for example in Pyret's compiler or in larger assignments, we end up with a blend of test cases in inline where: blocks, with the majority in separate files in standalone check: blocks. So at scale, things turn into more traditional-looking separate suites.

lliamander 10y ago

For writing production software, I have the same kind of concerns. Plus, what if you want do something more than simple unit tests (property based testing, concolic testing, etc) or if someone comes up with a better unit testing framework? You'll still have to setup those tests in a separate file, and then the in-line tests are just some unused feature of the language. However, for educational purposes, I think it's actually a compelling idea. My perspective is that while unit tests are a great idea, each unit test only provides marginal value. Thus, unit tests need to be as easy and natural to write as possible or developers won't do it. Habits, like writing code that doesn't need much mocking, definitely makes TDD easier, but that often comes after habituating oneself to TDD. For students, I think getting them as early as possible to write unit tests would be a big win. Putting the tests right there in the code with no need to set up test infrastructure or learn a separate test framework will be helpful.

pmarreck 10y ago

I think there is a case to be made that unit tests can live inside the file under test; integration tests can live outside of it. Also, mocks are arguably a test-code-smell side effect of too little isolation: https://www.destroyallsoftware.com/blog/2014/test-isolation-is-about-avoiding-mocks https://www.destroyallsoftware.com/blog/2014/test-isolation-... In my Elixir one-page code experiments, I put the unit test in the same file after the code under test. It makes sure the test only runs when the file is loaded a certain way: https://github.com/pmarreck/elixir-snippets/ https://github.com/pmarreck/elixir-snippets/

chubot 10y ago

It seems like the obvious thing is to allow both tests in the source file and separate test file. I like the idea of short functions with doctest-like tests, but written with the real language, syntax-highlighted, etc. In addition to education, it's good for quick prototyping. It's also nice to see at a glance which functions don't have tests! But I can also see it getting a bit unwieldy for complicated stuff, so in that case you can put it in another file, perhaps with a pointer. I'm guessing you can't stop people from writing tests in separate files, so I suppose Pyret allows that. I guess you could write a unit test framework in Python that does this with a tiny amount of metaprogramming. def Foo(): return 5 def testFoo(t): t.assertEqual(Foo(), 5) So you could just look for all the functions in the file named "testFoo" where Foo is also a function, and then run them with the "t" param. Hm.

qznc 10y ago
In Python nose can do that for you: http://nose.readthedocs.io/en/latest/ http://nose.readthedocs.io/en/latest/ In D there is even a keyword for that. It allows to embed the tests into the documentation as examples. This has the nice side effect that your examples are actually tested. https://dlang.org/spec/unittest.html https://dlang.org/spec/unittest.html

shriramkmurthi 10y ago

There's absolutely nothing in Pyret forcing you to write `where` blocks with immediate tests. You can also put your tests in a `check` block, which can float anywhere, and in another module. What we find valuable is to write a _small_ number of _illustrative_ examples, which we call the “sweep”, that summarize the main behavior of the function. Think of it as essential, runnable documentation. When you come to a new codebase (including your own, six months later!), these help you quickly page back in what the function was supposed to do. The sweep is also really useful for peer-review, which is another especially valuable technique in both education and industry. We have done some nice studies on its effectiveness: http://cs.brown.edu/~sk/Publications/Papers/Published/pcgfk-sweep-ess-eg-ifpr/ http://cs.brown.edu/~sk/Publications/Papers/Published/pcgfk-...