9 ms·

Clojure and Testing

69 points by fogus 13y ago

programminggeek 13y ago
I agree that testing getters/setters and other libraries functionality is a waste of time, but... You know, the "pragmatic" approach that many programmers take - "writing tests is hard and time consuming, so let's write fewer tests" is a recipe for disaster and frankly the idea that if you just think about the problem more, that you're make fewer mistakes is very much along those same lines. Here is the real problem - humans are fallible and we make mistakes. This is why pilots have flight checklists, why manufacturing has quality checks, why engineers have building codes, and why at hospitals checklists and procedures save lives If pilots and mechanics just "thought more about the problem" they would still make mistakes. You might be a brilliant surgeon, but thinking you're too smart or too good to go through a checklist could cost someone their life because of a small oversight. Would you feel good going under the knife if you knew that your life was in the hands of someone who thought they were too smart to scrub in? We are a young profession with very few actual quality standards. Luckily, most of the software that we build isn't critical to people's safety, but I'd be terrified if the software powering medical robots, life support machines, and airplane autopilot systems were built by people who were "too smart" to write tests. When a software bug costs people their lives because some programmer thought writing those tests was "too expensive" or "a waste of time" how smart is that programmer?
- yummyfajitas 13y ago
  I agree that testing getters/setters...is a waste of time, but... But it isn't necessarily. Say I have multiple implementations of a class, e.g.: trait TimeSeries { def datapoints: Seq[DataPoint] } case class SequenceTimeSeries(datapoints: Seq[DataPoint]) extends TimeSeries class ArrayTimeSeries(times: Array[Long], values: Array[Double]) extends TimeSeries It's pretty straightforward and nearly pointless to test that `sequenceTimeSeries.datapoints` does what I expect. On the other hand, applying the same test to `arrayTimeSeries.datapoints` exposed an off by one error in my implementation.
  - tel 13y ago
    Out of curiosity—I'm not so familiar with Scala—how does an off-by-1 arise here? Aren't you just going to zip (times, values)?
    - yummyfajitas 13y ago
      I did this a while back, so I don't recall the exact off-by-one error. But I don't want to to times.zip(values) - that would involve constructing explicitly a List[DataPoint] which is precisely what I'm trying to avoid with the ArrayTimeSeries. Rather, I defined a custom Seq[DataPoint], which had the apply(i: Int) method construct datapoints on the fly from the arrays. (I admit that the error was entirely stupidity on my part and not the result of anything complicated. If I was as smart as Rich Hickey, I probably wouldn't need so many tests.)
      - tel 13y ago
        Ohh, I see, array-of-structures/structure-of-arrays trouble.
  - puredanger 13y ago
    So test that one. Please, I encourage you to engage your brain in the process of choosing what does and does not need to be tested. That is the whole point.
- taeric 13y ago
  My problem is just how adverse to utilizing a proper QA department many developers are. Sure, unit tests and other automated systems can and should be used to help out in creation of a system. However, realize that there are folks who have the sole job of testing a system. Unsurprisingly, many of them are actually very good at it. Instead of spending a lot of time trying to make a more convoluted model just so you can test it, use the real tools at your disposal to accomplish the same goal. I fully grant that the line between "convoluted model" and "adequately testable" system is a tough one to spot. I am open to some good rules in distinguishing them.
  - engrenage 13y ago
    A QA department doesn't locate the bugs for you. They report bugs, and then developers have to repeat the procedure and investigate. This is a costly round-trip. Automated testing is intended to be a much cheaper way to stop bugs from propagating up to QA, and it usually provides much better and more immediate information about the precise nature and location of bugs.
    - taeric 13y ago
      But an automated testing suite does not write itself. Not even in a "statically" typed language. In those worlds, the staffing costs of having a dedicated QA team can be far cheaper than engineering your system to be perfectly modelled for testing. This is especially true in areas that do not already have established models. (And is why early attempts at such endeavors as bridge building focused on prototypes and iterative solutions, and the rigorous mathematical models came later.)
      - engrenage 13y ago
        If you remove the word 'perfectly', you're just saying that a dedicated QA team is cheaper than writing tests. I doubt that very much.
        taeric 13y ago
- puredanger 13y ago
  Would you fly an airline that did 24 hours of verification before every flight thus making flights cost 10x as much? (and same argument for all your other examples...) Our job as professionals is to verify that the software we write does what we expect. Our job is also to manage costs appropriate to the application (not just up-front but over the long haul). For me, that means that we want our tests to maximize value per effort. I am not saying a) write fewer tests b) tests are a waste of time or c) tests are too expensive. I'm saying: a) write better tests, b) some tests are a waste of time and c) let's write tests that do more work for us.
  - pbiggar 13y ago
    [disclaimer: I work at https://circleci.com https://circleci.com, so I do have a vested interest in disagreeing with you ;) ] I think is a strawman. If running a huge suite of automated verification was really cheap, then airlines would run it before each flight.
    - puredanger 13y ago
      Hey Paul, I totally agree that the cost of running big test suites can be mitigated and CircleCI is doing awesome stuff there. That does nothing to address the problem of maintaining a large suite of tests tightly coupled to your code. That is, the cost of updating many example-based tests when you change the code.
      - pbiggar 13y ago
        I guess that depends on how much of your suite is in flux. We have largely example-driven tests, and our suite is pretty big: about 16kLOC. I can't think of a case where a significant amount of time was spent fixing the test suite (and in those cases, it was generally useful to think about why). Also, I haven't seen anything that works better. Generative tests are nice in theory, but I haven't seem the be that useful for anything other than short, pure, functions.
        jamii 13y ago
- kaonashi 13y ago
  Really, what it comes down to is that with a language like Clojure, that has very isolated side effects and works primarily with pure functions, getting to high test coverage requires a lot less testing code.
thurn 13y ago
I mean, there's a spectrum here. I don't think anybody is going to argue that that java.util.LinkedList shouldn't have tests. On the other end of the spectrum, there are things that obviously shouldn't be tested (getters and setters, whether some div has 10px of padding, etc). Tests are are how you lock down your software once it's ready to be frozen. You use them to turn software into hardware, to use an analogy from Steve Yegge [1]. [1]: http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.html http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.ht...
siscia 13y ago
Humm, I completely agree with Alex, however for different reasons. If you write clojure, you use immutable structure (in most of the case, if you don't this comment doesn't apply), and so you end up writting only functions that take some data in, and return some data out. If you are smart enough to work bottom up and you can create different layer of abstraction once a layer is finished and once you are sure that it does what it is suppose to do, then you can really stop to test that layer. (I hope is clear what I mean...) The only test I can see having a real sense in clojure is when you use impure functions (functions that don't only get something in and spit something out, but that also do some work, writting in a DB maybe...) or when you think that your function/abstraction will be re-factor someday-somehow. However in the case of the refactoring, the test need to be written still in a clever way: you have such functions tunnel A->B->C->D where you know that you will refactor C so there is no point in test A, what we really need to test is just C and its direct "neigbourd", or even better making sure that B returns what C needs and that C returns what D needs. Or even better, since a clojure functions is way too long if it is more than 10 lines of code, well maybe all this testing is already too much, maybe the pre-post clojure's condition are enough. (This if we are talking about small functions, if A, B, C and D are abstraction layer you can't just keep all in your mind, so tests really makes sense here, still however doesn't makes sense to test A at all, nor the whole B, nor the whole D)
tel 13y ago
Test when you lay down abstraction. If you know your expectations about some code, then it's worth testing in proportion to the amount of dependency you have on those expectations holding. If you don't yet know the expectations/abstraction you have for some code then you're still deep in the design phase whether or not you like it. Good tests—like good static types or property checks—can be useful to smoke out the properties your abstraction will eventually fulfill. If they slow you down, don't use them. But once you know your abstraction and start depending on it then please test. Most likely a good test will, right there, show you that you don't actually know your abstraction very well.
- puredanger 13y ago
  Good advice. Have you tried using contracts (pre/post conditions, assertions, invariant checks) in your code? I find them to be a useful way to encode those expectations in the code before you test.
  - tel 13y ago
    I'm a huge proponent powerful static checking, so I haven't written much code with contracts. My understanding is that they're quite wonderful for dynamic property elaboration, though, so by that fiat I'd say I'm a fan.
pbiggar 13y ago
Could not disagree more. For reference, my company (https://circleci.com https://circleci.com) probably has one of the largest test suites in Clojure (approx 16kLOC in the test suite, > 5000 assertions). The value is testing is that it saves time in the long run. If you don't have a decent test suite, then as your product matures, you become unable to do anything because each new feature breaks a dozen old things. Alex is right about the opportunity cost (see my own post "Testing is bullshit": http://blog.circleci.com/testing-is-bullshit/ http://blog.circleci.com/testing-is-bullshit/). But not testing is also an opportunity cost. Basically you can spend T minutes writing tests, or D minutes debugging problems that tests would have found. If you don't test at all, then D will rapidly (within weeks possibly) become so high that you can't get anything done. By contrast, T will rarely go above 50% of time spent on a project (and if it is, you need to rethink how you test, unless you're building rockets or medical equipment).
- puredanger 13y ago
  What in particular are you disagreeing with? You seem to be implying that I'm saying not to write tests or that testing isn't valuable but that's not what I said. As I said: "I think tests are highly valuable. I don’t know how I could write high quality code without writing tests." Every Clojure dev I've ever worked with wrote tests and relied on them. My point is that there are costs to consider. I observe that the Clojure community is interested in trying a wide variety of testing approaches that may mitigate some costs of testing, perhaps more so than other communities. I have also worked on a system with a large Clojure test suite (at least 30kLOC), primarily example-based. We found that there was substantial cost (even with many custom-built tools) to maintaining those tests. I've had the same experience maintaing large Java test suites. Recently, I've been doing more with generative testing - I think generative and contracts may give you a better return on your effort than example based testing. But that's a hypothesis I look forward to investigating.
- siscia 13y ago
  What kind of test you run ? Do you test every single pure functions or you test just abstraction layers ? Or what ?
skatenerd 13y ago
When I test at the REPL, I feel like I spend lots of time trying to re-enter updated code. This means I do a lot of: A: Pasting changed code into repl B: Re-typing code C: Hitting "previous command" and "return" The overhead of automating this process (by making a test suite) seems high, but I haven't ever regretted it.
- emidln 13y ago
  You can also get rid of this overhead by using an editor with an integrated repl (emacs+nrepl.el/slime/inferior-lisp-mode, vim+slimv/fireplace, LightTable all do this). I can type ,e to send an updated function to my repl for instance. ,b to evaluate an entire buffer. That said, you can also develop your test suite like this as well (which is a sublime experience, particularly with a generative test tool).
  - jamii 13y ago
    It helps to make a sandbox file that has everything imported already and just reeval the whole file - https://github.com/jamii/strucjure/blob/compiled/test/strucjure/regression/tests.clj https://github.com/jamii/strucjure/blob/compiled/test/strucj... Then you can also run the file as a free regression test - https://github.com/jamii/strucjure/blob/compiled/test/strucjure/regression.clj https://github.com/jamii/strucjure/blob/compiled/test/strucj...
- coolsunglasses 13y ago
  https://github.com/aphyr/prism/ https://github.com/aphyr/prism/
dljsjr 13y ago
Maybe this is unique to the field that I work in, but I can pretty safely say that if our organization didn't strive to maintain 60%-80% unit test coverage at any given time, we'd have been sunk a long time ago. We don't "ship" a product. We're a humanoid robotics research laboratory, we have a fairly large and old codebase for what we do (~1MLOC of Java, plus a whole big whack of C++ for JNI), and since we neither have a customer nor an OSS community then we make no guarantees about our API nor do we frequently generate stable, frozen binary releases. As such, the code is constantly being refactored and evolving; especially our numerical based stuff; things like physics, controls, etc. (we use an in-house simulation environment that one of our PI's created while he was in grad school, so even stuff like the 3D engine or the simulator data viz GUI's can change at any given moment). I realize that this is entirely anecdotal and highly specific to what we do, but without high coverage and solid CI (We use a self-hosted Bamboo instance with nightly and every-commit builds), our code would be in complete and utter disarray. The build routinely breaks a dozen times a day when one of the control theory guys checks in a parameter tweak to our state estimator or introduces a new component to one of our modular controllers (we have test suites that actually run headless versions of our sims and mathematically verify the walking behavior). I guess my point is that, as is the case with anything in programming, choose the right tool for the job. Evaluate what you're doing, what your goals are, the nature of your code-base, and decide (as is brushed over in the source article) whether or not the opportunity cost is worth it. For us, we'd be up shit creek if we didn't have high coverage.
danneu 13y ago
Note: I only have experience with my one-man Clojure projects. My favorite poor-man's test-feedback-loop in Clojure is to use the built-in core.assert function to put tests at the bottom of the source file in combination with my editor's eval-buffer command. Test DSLs, test directory/file boilerplate, and setting up an autorun cycle are the three things that constitute enough of a barrier to prevent me from testing, so those are the three things this example eludes. Here's a file of a `to-slug` function that normalizes drug names into URL slugs. ;; Production code ... (defn to-slug [s] (hyphenize (simplify-name s))) ;; Tests (defn assert-slug [a b] (assert (= (to-slug a) b))) (assert-slug "ocusert pilo-20" "ocusert-pilo") (assert-slug "ogen .625" "ogen") (assert-slug "ogen 1.25" "ogen") (assert-slug "hydralazine hydrochloride, hydrochlorothiazide and reserpine" "hydralazine-hydrochloride") (assert-slug "hydralazine hydrochloride w/ hydrochlorothiazide 100/50" "hydralazine-hydrochloride") ;; Sandbox (to-slug "00-d-amphetamine") Every time I hit C-c C-k (eval-buffer in Emacs w/ nrepl), the asserts are run which will throw an error on failure. And since eval-buffer always returns the result of the last expression evaluated, the above "Sandbox" section represents whatever case I'm currently trying to extend `to-slug` support for. The result of which is returned on every eval-buffer. In this case, `to-slug` doesn't correctly handle "00-d-amphetamine" yet, so as I modify the production code to handle it, I can mash eval-buffer after every change to (1) see if my change introduces regressions and (2) visualize my progress on handling "00-d-amphetamine". Once it works, I add an "00-d-amphetamine" assert-slug and move on to the next case. As to-slug matures, I can then move the asserts to a dedicated test directory if I want to and replace the slime with minimal case tests. -- Things that otherwise slow down my exploration phase. ---- I think core.assert fills the chasm of a function's/file's lifecycle between "I'm not even sure if I want this function or what its input/output will look like" and "my source code definitely needs this and this is its API". In Ruby, I like https://github.com/sconover/wrong https://github.com/sconover/wrong.
- ruricolist 13y ago
  I do this in Common Lisp. It also makes sense as a technique for documentation: why leave examples rusting in comments when you can incorporate them into the program? Add a Quickcheck-ish type-based test case generator and you start to blur the line between tests and types.
i_s 13y ago
I have a suggestion for people who are very pro unit testing for understanding people who do not test as much: try working in areas that are traditionally light on unit testing. For example, graphics, multimedia, and games. People write stuff in these areas all the time that still work, despite minimal automated testing. There are other paths we might take, Frodo.
davidrupp 13y ago
https://twitter.com/davidrupp/status/375290808640884736 https://twitter.com/davidrupp/status/375290808640884736 "Not all who wander are lost."

69 points by fogus 13y ago

I agree that testing getters/setters and other libraries functionality is a waste of time, but... You know, the "pragmatic" approach that many programmers take - "writing tests is hard and time consuming, so let's write fewer tests" is a recipe for disaster and frankly the idea that if you just think about the problem more, that you're make fewer mistakes is very much along those same lines. Here is the real problem - humans are fallible and we make mistakes. This is why pilots have flight checklists, why manufacturing has quality checks, why engineers have building codes, and why at hospitals checklists and procedures save lives If pilots and mechanics just "thought more about the problem" they would still make mistakes. You might be a brilliant surgeon, but thinking you're too smart or too good to go through a checklist could cost someone their life because of a small oversight. Would you feel good going under the knife if you knew that your life was in the hands of someone who thought they were too smart to scrub in? We are a young profession with very few actual quality standards. Luckily, most of the software that we build isn't critical to people's safety, but I'd be terrified if the software powering medical robots, life support machines, and airplane autopilot systems were built by people who were "too smart" to write tests. When a software bug costs people their lives because some programmer thought writing those tests was "too expensive" or "a waste of time" how smart is that programmer?

yummyfajitas 13y ago
I agree that testing getters/setters...is a waste of time, but... But it isn't necessarily. Say I have multiple implementations of a class, e.g.: trait TimeSeries { def datapoints: Seq[DataPoint] } case class SequenceTimeSeries(datapoints: Seq[DataPoint]) extends TimeSeries class ArrayTimeSeries(times: Array[Long], values: Array[Double]) extends TimeSeries It's pretty straightforward and nearly pointless to test that `sequenceTimeSeries.datapoints` does what I expect. On the other hand, applying the same test to `arrayTimeSeries.datapoints` exposed an off by one error in my implementation.
- tel 13y ago
  Out of curiosity—I'm not so familiar with Scala—how does an off-by-1 arise here? Aren't you just going to zip (times, values)?
  - yummyfajitas 13y ago
    I did this a while back, so I don't recall the exact off-by-one error. But I don't want to to times.zip(values) - that would involve constructing explicitly a List[DataPoint] which is precisely what I'm trying to avoid with the ArrayTimeSeries. Rather, I defined a custom Seq[DataPoint], which had the apply(i: Int) method construct datapoints on the fly from the arrays. (I admit that the error was entirely stupidity on my part and not the result of anything complicated. If I was as smart as Rich Hickey, I probably wouldn't need so many tests.)
    - tel 13y ago
      Ohh, I see, array-of-structures/structure-of-arrays trouble.
- puredanger 13y ago
  So test that one. Please, I encourage you to engage your brain in the process of choosing what does and does not need to be tested. That is the whole point.
taeric 13y ago
My problem is just how adverse to utilizing a proper QA department many developers are. Sure, unit tests and other automated systems can and should be used to help out in creation of a system. However, realize that there are folks who have the sole job of testing a system. Unsurprisingly, many of them are actually very good at it. Instead of spending a lot of time trying to make a more convoluted model just so you can test it, use the real tools at your disposal to accomplish the same goal. I fully grant that the line between "convoluted model" and "adequately testable" system is a tough one to spot. I am open to some good rules in distinguishing them.
- engrenage 13y ago
  A QA department doesn't locate the bugs for you. They report bugs, and then developers have to repeat the procedure and investigate. This is a costly round-trip. Automated testing is intended to be a much cheaper way to stop bugs from propagating up to QA, and it usually provides much better and more immediate information about the precise nature and location of bugs.
  - taeric 13y ago
    But an automated testing suite does not write itself. Not even in a "statically" typed language. In those worlds, the staffing costs of having a dedicated QA team can be far cheaper than engineering your system to be perfectly modelled for testing. This is especially true in areas that do not already have established models. (And is why early attempts at such endeavors as bridge building focused on prototypes and iterative solutions, and the rigorous mathematical models came later.)
    - engrenage 13y ago
      If you remove the word 'perfectly', you're just saying that a dedicated QA team is cheaper than writing tests. I doubt that very much.
      - taeric 13y ago
puredanger 13y ago
Would you fly an airline that did 24 hours of verification before every flight thus making flights cost 10x as much? (and same argument for all your other examples...) Our job as professionals is to verify that the software we write does what we expect. Our job is also to manage costs appropriate to the application (not just up-front but over the long haul). For me, that means that we want our tests to maximize value per effort. I am not saying a) write fewer tests b) tests are a waste of time or c) tests are too expensive. I'm saying: a) write better tests, b) some tests are a waste of time and c) let's write tests that do more work for us.
- pbiggar 13y ago
  [disclaimer: I work at https://circleci.com https://circleci.com, so I do have a vested interest in disagreeing with you ;) ] I think is a strawman. If running a huge suite of automated verification was really cheap, then airlines would run it before each flight.
  - puredanger 13y ago
    Hey Paul, I totally agree that the cost of running big test suites can be mitigated and CircleCI is doing awesome stuff there. That does nothing to address the problem of maintaining a large suite of tests tightly coupled to your code. That is, the cost of updating many example-based tests when you change the code.
    - pbiggar 13y ago
      I guess that depends on how much of your suite is in flux. We have largely example-driven tests, and our suite is pretty big: about 16kLOC. I can't think of a case where a significant amount of time was spent fixing the test suite (and in those cases, it was generally useful to think about why). Also, I haven't seen anything that works better. Generative tests are nice in theory, but I haven't seem the be that useful for anything other than short, pure, functions.
      - jamii 13y ago
kaonashi 13y ago
Really, what it comes down to is that with a language like Clojure, that has very isolated side effects and works primarily with pure functions, getting to high test coverage requires a lot less testing code.

thurn 13y ago

I mean, there's a spectrum here. I don't think anybody is going to argue that that java.util.LinkedList shouldn't have tests. On the other end of the spectrum, there are things that obviously shouldn't be tested (getters and setters, whether some div has 10px of padding, etc). Tests are are how you lock down your software once it's ready to be frozen. You use them to turn software into hardware, to use an analogy from Steve Yegge [1]. [1]: http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.html http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.ht...

siscia 13y ago

Humm, I completely agree with Alex, however for different reasons. If you write clojure, you use immutable structure (in most of the case, if you don't this comment doesn't apply), and so you end up writting only functions that take some data in, and return some data out. If you are smart enough to work bottom up and you can create different layer of abstraction once a layer is finished and once you are sure that it does what it is suppose to do, then you can really stop to test that layer. (I hope is clear what I mean...) The only test I can see having a real sense in clojure is when you use impure functions (functions that don't only get something in and spit something out, but that also do some work, writting in a DB maybe...) or when you think that your function/abstraction will be re-factor someday-somehow. However in the case of the refactoring, the test need to be written still in a clever way: you have such functions tunnel A->B->C->D where you know that you will refactor C so there is no point in test A, what we really need to test is just C and its direct "neigbourd", or even better making sure that B returns what C needs and that C returns what D needs. Or even better, since a clojure functions is way too long if it is more than 10 lines of code, well maybe all this testing is already too much, maybe the pre-post clojure's condition are enough. (This if we are talking about small functions, if A, B, C and D are abstraction layer you can't just keep all in your mind, so tests really makes sense here, still however doesn't makes sense to test A at all, nor the whole B, nor the whole D)

tel 13y ago

Test when you lay down abstraction. If you know your expectations about some code, then it's worth testing in proportion to the amount of dependency you have on those expectations holding. If you don't yet know the expectations/abstraction you have for some code then you're still deep in the design phase whether or not you like it. Good tests—like good static types or property checks—can be useful to smoke out the properties your abstraction will eventually fulfill. If they slow you down, don't use them. But once you know your abstraction and start depending on it then please test. Most likely a good test will, right there, show you that you don't actually know your abstraction very well.

puredanger 13y ago
Good advice. Have you tried using contracts (pre/post conditions, assertions, invariant checks) in your code? I find them to be a useful way to encode those expectations in the code before you test.
- tel 13y ago
  I'm a huge proponent powerful static checking, so I haven't written much code with contracts. My understanding is that they're quite wonderful for dynamic property elaboration, though, so by that fiat I'd say I'm a fan.

pbiggar 13y ago

Could not disagree more. For reference, my company (https://circleci.com https://circleci.com) probably has one of the largest test suites in Clojure (approx 16kLOC in the test suite, > 5000 assertions). The value is testing is that it saves time in the long run. If you don't have a decent test suite, then as your product matures, you become unable to do anything because each new feature breaks a dozen old things. Alex is right about the opportunity cost (see my own post "Testing is bullshit": http://blog.circleci.com/testing-is-bullshit/ http://blog.circleci.com/testing-is-bullshit/). But not testing is also an opportunity cost. Basically you can spend T minutes writing tests, or D minutes debugging problems that tests would have found. If you don't test at all, then D will rapidly (within weeks possibly) become so high that you can't get anything done. By contrast, T will rarely go above 50% of time spent on a project (and if it is, you need to rethink how you test, unless you're building rockets or medical equipment).

puredanger 13y ago
What in particular are you disagreeing with? You seem to be implying that I'm saying not to write tests or that testing isn't valuable but that's not what I said. As I said: "I think tests are highly valuable. I don’t know how I could write high quality code without writing tests." Every Clojure dev I've ever worked with wrote tests and relied on them. My point is that there are costs to consider. I observe that the Clojure community is interested in trying a wide variety of testing approaches that may mitigate some costs of testing, perhaps more so than other communities. I have also worked on a system with a large Clojure test suite (at least 30kLOC), primarily example-based. We found that there was substantial cost (even with many custom-built tools) to maintaining those tests. I've had the same experience maintaing large Java test suites. Recently, I've been doing more with generative testing - I think generative and contracts may give you a better return on your effort than example based testing. But that's a hypothesis I look forward to investigating.
siscia 13y ago
What kind of test you run ? Do you test every single pure functions or you test just abstraction layers ? Or what ?

skatenerd 13y ago

When I test at the REPL, I feel like I spend lots of time trying to re-enter updated code. This means I do a lot of: A: Pasting changed code into repl B: Re-typing code C: Hitting "previous command" and "return" The overhead of automating this process (by making a test suite) seems high, but I haven't ever regretted it.

emidln 13y ago
You can also get rid of this overhead by using an editor with an integrated repl (emacs+nrepl.el/slime/inferior-lisp-mode, vim+slimv/fireplace, LightTable all do this). I can type ,e to send an updated function to my repl for instance. ,b to evaluate an entire buffer. That said, you can also develop your test suite like this as well (which is a sublime experience, particularly with a generative test tool).
- jamii 13y ago
  It helps to make a sandbox file that has everything imported already and just reeval the whole file - https://github.com/jamii/strucjure/blob/compiled/test/strucjure/regression/tests.clj https://github.com/jamii/strucjure/blob/compiled/test/strucj... Then you can also run the file as a free regression test - https://github.com/jamii/strucjure/blob/compiled/test/strucjure/regression.clj https://github.com/jamii/strucjure/blob/compiled/test/strucj...
coolsunglasses 13y ago
https://github.com/aphyr/prism/ https://github.com/aphyr/prism/

dljsjr 13y ago

Maybe this is unique to the field that I work in, but I can pretty safely say that if our organization didn't strive to maintain 60%-80% unit test coverage at any given time, we'd have been sunk a long time ago. We don't "ship" a product. We're a humanoid robotics research laboratory, we have a fairly large and old codebase for what we do (~1MLOC of Java, plus a whole big whack of C++ for JNI), and since we neither have a customer nor an OSS community then we make no guarantees about our API nor do we frequently generate stable, frozen binary releases. As such, the code is constantly being refactored and evolving; especially our numerical based stuff; things like physics, controls, etc. (we use an in-house simulation environment that one of our PI's created while he was in grad school, so even stuff like the 3D engine or the simulator data viz GUI's can change at any given moment). I realize that this is entirely anecdotal and highly specific to what we do, but without high coverage and solid CI (We use a self-hosted Bamboo instance with nightly and every-commit builds), our code would be in complete and utter disarray. The build routinely breaks a dozen times a day when one of the control theory guys checks in a parameter tweak to our state estimator or introduces a new component to one of our modular controllers (we have test suites that actually run headless versions of our sims and mathematically verify the walking behavior). I guess my point is that, as is the case with anything in programming, choose the right tool for the job. Evaluate what you're doing, what your goals are, the nature of your code-base, and decide (as is brushed over in the source article) whether or not the opportunity cost is worth it. For us, we'd be up shit creek if we didn't have high coverage.

danneu 13y ago

Note: I only have experience with my one-man Clojure projects. My favorite poor-man's test-feedback-loop in Clojure is to use the built-in core.assert function to put tests at the bottom of the source file in combination with my editor's eval-buffer command. Test DSLs, test directory/file boilerplate, and setting up an autorun cycle are the three things that constitute enough of a barrier to prevent me from testing, so those are the three things this example eludes. Here's a file of a `to-slug` function that normalizes drug names into URL slugs. ;; Production code ... (defn to-slug [s] (hyphenize (simplify-name s))) ;; Tests (defn assert-slug [a b] (assert (= (to-slug a) b))) (assert-slug "ocusert pilo-20" "ocusert-pilo") (assert-slug "ogen .625" "ogen") (assert-slug "ogen 1.25" "ogen") (assert-slug "hydralazine hydrochloride, hydrochlorothiazide and reserpine" "hydralazine-hydrochloride") (assert-slug "hydralazine hydrochloride w/ hydrochlorothiazide 100/50" "hydralazine-hydrochloride") ;; Sandbox (to-slug "00-d-amphetamine") Every time I hit C-c C-k (eval-buffer in Emacs w/ nrepl), the asserts are run which will throw an error on failure. And since eval-buffer always returns the result of the last expression evaluated, the above "Sandbox" section represents whatever case I'm currently trying to extend `to-slug` support for. The result of which is returned on every eval-buffer. In this case, `to-slug` doesn't correctly handle "00-d-amphetamine" yet, so as I modify the production code to handle it, I can mash eval-buffer after every change to (1) see if my change introduces regressions and (2) visualize my progress on handling "00-d-amphetamine". Once it works, I add an "00-d-amphetamine" assert-slug and move on to the next case. As to-slug matures, I can then move the asserts to a dedicated test directory if I want to and replace the slime with minimal case tests. -- Things that otherwise slow down my exploration phase. ---- I think core.assert fills the chasm of a function's/file's lifecycle between "I'm not even sure if I want this function or what its input/output will look like" and "my source code definitely needs this and this is its API". In Ruby, I like https://github.com/sconover/wrong https://github.com/sconover/wrong.

ruricolist 13y ago
I do this in Common Lisp. It also makes sense as a technique for documentation: why leave examples rusting in comments when you can incorporate them into the program? Add a Quickcheck-ish type-based test case generator and you start to blur the line between tests and types.

i_s 13y ago

I have a suggestion for people who are very pro unit testing for understanding people who do not test as much: try working in areas that are traditionally light on unit testing. For example, graphics, multimedia, and games. People write stuff in these areas all the time that still work, despite minimal automated testing. There are other paths we might take, Frodo.

davidrupp 13y ago

https://twitter.com/davidrupp/status/375290808640884736 https://twitter.com/davidrupp/status/375290808640884736 "Not all who wander are lost."