6 ms·

It's funny. Just as the author reached this realisation I've reached the realisation that tests are killing my productivity and are exactly the wrong strategy f

• jacquesm 15y ago
  Funny you should say that :) This is one of the reasons why I don't make testing into a religion but pick a time and a place to add the tests. The 'write your tests first' school is right in that it can create another boost but it comes with the problem that it puts a damper on any exploratory programming. I typically take three tries to get it 'right', and only after the third try with a reasonably stable interface is where I will add tests. Maybe I'm a lousy designer in that I can't nail down an interface on the first try but on non-trivial code this is unfortunately my experience so far. First try: very quick, rough and dirty. Second try: a bit slower, mostly right but still needs major rework. Third try: minor tweaks in the longer term but mostly finalized. The elapsed time is usually something under a day for the first try, a few days to a week for the second and after the third try it goes into 'maintenance' pretty quickly.
  • petenixey 15y ago
    I think that what you say makes a lot of sense and I completely agree with doing code-test-code. True TDD (which I have tried) requires too much constant change to be practical. The churn I was referring to though is less that of short-term interface changes but more to do with long-term multi-month revamping of the product proposition. During these highly exploratory phases, the main risk with errors is not that they multiply but that they never happen because the code isn't run. I always knew this in theory but like all lessons I needed to learn it the hard way to really appreciate.
  • peteretep 15y ago
    > why I don't make testing into a religion but pick a time and a place to add the tests Quite. Testing is great. "Test Driven Development" where expensive Agile Consultants who haven't written code in ten years tell teams they always have to write tests first for everything is bullshit: http://xrl.us/bmebj3 http://xrl.us/bmebj3
    • Drbble 15y ago
      Many successful devs also preach TDD. TDD is great when your client has a high quality spec, or you are being paid for meeting team spec. TDD is less great when your project goal is "build something neat" with a very loose spec and wide tolerances.
      • philwelch 15y ago
        Well it really depends. You can make a religious Process out of TDD where you start with something like Cucumber and work your way down through the design. You can TDD your glue code. You can TDD trivial stuff, like when you put a link in your web app to your new feature, you could write a view test that verifies that the link appears. You can, if you really want to, go through an obnoxious iterative process of writing the same function in a progressively less and less broken state. There might even be value to this approach. I wouldn't know. Or you could do something like this: whenever you have a function that seems to do an actual algorithm, or process some data, or provide a predictable result given an input, you write up a few asserts and programmatically see whether it seems to work immediately after writing it instead of writing it and wasting your time futzing about with it by hand, or tracking down a bug in it after noticing strange results coming out of the whole program. Or you could do something like this: whenever you get a bug report, create a test case that reproduces the bug first. Agile Consultants are like anyone else who tries to sell you enlightenment: they wouldn't eat unless there was actually a kernel to truth behind what they were saying. Writing your tests first is a fantastic idea sometimes.
  • mattmanser 15y ago
    This makes me wonder even more though when you put it together with this part of your post: ...I have good hope that in the long run this will tremendously increase my productivity. Even if in the short term it seemed as though my productivity took a hit (after all, writing code that tests stuff does not add any functionality to the program itself). You seem to be saying that even though you've been doing it for a year, you're still suffering a drop in productivity. Have I misunderstood you? Doesn't that indicate that it's actually not a good idea? Or do you have evidence that there's less customer visible bugs to deal with? (evidence rather than your hopes would be better :) It doesn't really feel like you've communicated why all that extra testing code is so great. And let's not forget that's what it actually is, code which can be wrong itself. And you're not even doing TDD but adding them way down the line!
    • jacquesm 15y ago
      There definitely is a drop in initial productivity, after the exploratory phase is over there is a hick-up in the output because I'm writing test cases. Then, as soon as that is done there is a huge boost in productivity because it becomes much (and I can't emphasize enough how much that really is) easier. So it is more a matter of uneven output than an overall loss, overall it is definitely a net gain. Hard to quantify though because you will never know how much time you would have spent debugging and testing if you had not written those tests first. It is more of a feeling than that it is measurably slower, in fact it is measurably faster but when writing the test cases it feels slower. I hope that clears it up, sorry for the confusion! As for fewer customer visible bugs: yes, absolutely, no doubt about that.
  • chromatic 15y ago
    The 'write your tests first' school is right in that it can create another boost but it comes with the problem that it puts a damper on any exploratory programming. As far back as 1999 I can remember the "write your tests first" people saying "... unless you're doing exploratory programming and are likely to throw away the prototype."
  • chernevik 15y ago
    I am but an egg, but . . . Even if the components built of units change, and thus trach their tests, won't the tests covering the units still be useful? Also: I find that a lot of component changes are interface changes -- the test code has to change to match the new interface and return values, but still remains useful.
• debacle 15y ago
  > I've been testing obsessively for the past 18 months but just realised that I've got a stack of perfectly tested code that's perfectly wrong for what I need it for. I couldn't agree more. TDD has its place, but it so greatly reduces your ability to react to change that it's a non-starter for me.
• nkoren 15y ago
  I think it depends a lot on the nature of the problem that you're tackling. If you know in advance that your task is to turn input 'A' into output 'B', then I think it makes sense to write some of your tests before writing any of your actual code. Most of the projects I'm involved with, however, are highly experimental, both in terms of the inputs and outputs that we're dealing with, as well as the ways we interact with the user. Writing tests or even specifications (except in the most generic sense) upfront makes no sense in this case: what we need to do is quickly make some crude stuff, play around with it, and then make slightly less crude stuff. Wash, rinse, and repeat. After several cycles of this, we know what we actually want to do, and can start buckling down and writing production code. Some of it may or may not be borrowed from the prototyping cycles. In any case, testing becomes CRITICAL at this point.
  • boyter 15y ago
    That's what I use tests to do. Because of the effort of setting up and re-factoring the tests I have a long hard think before I write any code. I consider it using an API before writing it any code. Doing this forces me to consider how I would actually use it and usually my initial assumptions on how the code should work are wrong. I usually end up with the correct solution this way. This is of course for small parts of the application. The overall idea may be wrong, but since everything is now in small reusable components its easy to plug and play to get the correct solution. Probably the same result as rapid iterations, throwing away portions and then building correctly the last time. I wonder which one is more effective or if its dependent on the person.
• drostie 15y ago
  The convergence between Minimum Viable Products and Hacking is a poignant one and suitable for this audience, I suppose. People who develop new security breaks -- let's call them inbreakers because they break in -- are generally either Hackers or Script Kiddies, and not End Users or Software Engineers. I would broadly file computer users into those four categories, and maybe add some other categories on a good day. What does this mean? Let's start with Hackers. It means that by the time the inbreaking software is finished, it's quite possibly buggy, it quite possibly doesn't work on all infectable PCs, it's built to be a proof-of-concept which can be expanded. It's supposed to work, but it's not necessarily supposed to be clean and efficient and well-documented. That's someone else's problem. This is the Hacker mentality. It is the same that you should focus on for your Minimum Viable Products. Get version 0.1 out really quickly: it doesn't have to handle any edge cases, instead focus on just getting the core logic right. Everything else can wait. That is what "hacker" connotes in software circles. It's going to be ugly. It's going to be hacks. In the context of web development you should think, "Version 0.1 won't work on IE 6 or 7. It won't have graphics. It might have seconds of latency. It won't use a database -- I'll just use an array. Unless I'm in PHP and I absolutely have to use a database -- then I'll store everything in a JSON string stored in one text field. Usernames will just be stored in the query string part of the URL. My checkIfUserIsAdmin() function will return true. People can sign up for an email list and that's about it -- no automatic emails, I will solve that later." The core rule for hacking is, if you spend time doing something which isn't fun, what the hell are you doing? The Script Kiddies are the more common inbreakers our applications, but let's broadly define them as "people who use your application warts and all." They know it breaks. That doesn't matter, they're surprised that they can use it at all. They're not relying on your code to provide them anything, and they're not going to program your code either. For web development, these are the folks who use Mailinator and DownForEveryoneOrJustMe, perhaps also, more broadly, you could describe searchers (especially Scroogle users), blog readers, and so forth, as this sort. They're a natural match for hackers: hackers put out code which does the right thing but has warts, script kiddies use bundles of that code to actually compromise lots of systems and don't care necessarily about the warts. End Users, by contrast, are people who want to pay you, or at least consume your ads, for the services you provide. They expect something to Work As Expected. Maybe your boss has to negotiate with them to figure out what exactly the required functionality is, and then a contract has to be worked out. Or maybe you're just large enough that a substantial userbase relies on you -- Gmail and Google aren't technically doing something different from Mailinator and DuckDuckGo, handling emails and searching the web respectively, but you expect your email to Work and if Gmail is down for an hour a day next month you might huffily leave to some other provider. As you start to provide this reliability to attract End Users and Get Money, you'll need to work towards version 1.0 of your product, slowly patching in the things that were missing from the original idea. You'll have to have real logins with salted passwords. Your system won't just concatenate to the Email List -- it will send out live emails and handle bounces. For those types of work, you want Software Engineers. You want someone who can weave edge cases and apply them, you want tests for new code, you want someone who studies the API of all the languages reasonably available to figure out which calls can be made more efficient. I think these are not different people, but different hats. If you're cooking, you wear all of these hats. Sometimes you have a great idea for throwing some ingredients together: hey, I've never tasted chicken with avocados, that might actually work together, let's combine last night's guacamole with the night before's roast chicken and see how it tastes -- hacker. Okay, it tasted good, but to really serve this to my friends, I suppose I'll want a much less ugly presentation -- engineer. Okay, I pick up the knife to slice the avocado -- script kiddie. Then we eat up and my friends compliment me on my crazy ideas -- end users.
• jonstjohn 15y ago
  Excellent points. At the company that I work for, rigorous testing is an absolute necessity. Our web applications are critical to our users' business and any significant bugs in production is not acceptable. We have unit tests, selenium tests AND manual testing that goes on before any release. As you mentioned, in the early stages of a product, bugs in production may be tolerable especially when you're not sure you have a viable product. However, once you hit a significant user base that is paying for the service, service disruptions from bugs can really hurt business. Obviously we're not trying to be NASA, but I would say we spend 25-30% of our development time writing tests.
  • pbiggar 15y ago
    > Obviously we're not trying to be NASA, but I would say we spend 25-30% of our development time writing tests. With absolutely no disrespect intended, this seems low. I've read some studies about testing, and I chat to my customers a lot about it (my startup, Circle, does hosted continuous integration), and I would estimate that most projects spend over 50% of time testing.
    • jonstjohn 15y ago
      I initially wrote 35-40% but wasn't sure if that was realistic. No disrespect taken at all - I'm pushing for a more concerted effort on better unit test coverage. We also have a very large set of Selenium tests that run in a distributed build environment.
• wpietri 15y ago
  To me, there are two different kinds of tests. One answers "Does the code work as I intended?" For that, I write unit tests, integration tests, or end-to-end tests as needed. The other answers "Is this product useful?" For that, we do things like user tests, A/B tests, and landing page tests. I think you need both. If you do too much programmer-focused testing, then you can end up with something that's well-built and useless. If you do too much user testing and iteration, you end up with something that people like, but that is buggy and hard to evolve.
• jes5199 15y ago
  break your code into smaller chunks! If you're testing at a fine-grained level, you can assemble those well-tested components into different products
• pbiggar 15y ago
  It depends on the stage of your company and product, which I think of as a continuum: on the far right is banking software, life support machines, and airline code; on the far left are prototypes and fun hacks. The amount of testing you do should depend on where on that continuum your project lies. Let's say it's a scale from 1 to 10: 1 being no testing, 10 meaning hardcore static analysis and tests for your test's tests). Life support machines are 10, as are aircraft, and please god the software in my car. Most projects start at 1, become roughly 3 in the early stages, and settle on 6 when they become more mature. Using the wrong level at the wrong stage is a recipe for disaster. No tests for your mature product instils fear of change - anything you do will likely break something, and you will have no warning about it. But writing lots of tests for your prototype will kill your productivity.
  • jacquesm 15y ago
    You've nailed it absolutely perfect. Thank you.
• jemfinch 15y ago
  I write tests to confirm basic functionality. Then I wait for the bugs, write tests to tickle them, and then fix them.