5 ms·
Do people not typically modify the tests in the change where they introduce a feature? I have never worked anywhere where I would check in a change without the
by jrockway 8y ago
Do people not typically modify the tests in the change where they introduce a feature?
I have never worked anywhere where I would check in a change without the tests passing. If you are making a huge refactoring, you do it a step at a time, with each incremental change still having passing tests.
When I was at Google I believe that changes were automatically rolled back if they broke enough tests globally. I don't recall anyone ever complaining except that the rollback was slower than they would have preferred. You simply can't check in a library change that breaks half the company, after all.
If people are writing adequate tests and consider commits to be "change lists" instead of just a random thing they want to back up from their local machine... this is a no-brainer. But it does require the culture of having your code reviewer check that they think there's enough code coverage. Otherwise "passing tests" doesn't mean anything.
(Google's tools happily showed lines of code that were covered by tests in the review UI, so you have objective and subjective data as a code reviewer to look at before approving.)
- jonstewart 8y agoYeah, this is how I’ve worked for 16 years. The unit tests always pass. Maybe once or twice a year you can go way out on a limb and do something exploratory where you don’t care about the tests passing for every commit, but... part of TDD and CI is that the tests always pass. In some ways, I think the feature branch->PR model of today is a step backwards from traditional central server single-trunk development of years gone by (CVS/Perforce/SVN). Branches and PRs work better for F/OSS development, and for monster refactoring, but I kinda long for the days of a normal team just committing to master/trunk several times a day.
- jrockway 8y agoI think PRs are logically the same as Perforce changelists. The fact that you can pick apart a PR into multiple commits does pollute the history when you merge, though, unless your tools treat the PR as a single atomic commit (for bisection, etc.) I have never treated the individual commits inside a PR as anything other than "I'd like to backup my workstation to the cloud now", but if people are treating them as something meaningful, I think that's a mistake.
- XorNot 8y agoThe popularity of rebase merging in Git I think is partly to blame here. Rebase merging tries to make everything seem like real history, but no one ever seems to have a solid idea of what that actually is supposed to mean (and CI systems don't make it easy to configure "test all commits" or declare what you want your commit history to mean).
- jrockway 8y agoYup. I always picked "merge" instead of "rebase" in Gerrit because it's not like I tested the result of applying each patch to the master branch. All I did was decide that mine worked and that I wanted to merge my branch with the upstream. People like using clever features to make the history inaccurate, which seems like a bad idea to me. (It's "cleaner", but wrong.)
- int0x80 8y agoIf you internally rebase to cleanup history you have to test again. Ignoring that step is wrong. Local rebase can be good and useful.
- Vinnl 8y agoThere's no such thing as "real history" - even without rebasing, your commits do not reflect the development of the code; they reflect practically arbitrary moments in time at which you chose to commit. The idea behind rebasing is not to make it seem like "real history", but to have all changes in a commit to be related. They're not a reflection of how you built the code, but how you can understand it. They're documentation, and as such, you spend effort on them.
- int0x80 8y agoI don't know if I'm reading your comment correctly, sorry if I'm not... but individual commits within a PR are absolutely not a backup. Every commit in the PR has to be a logical unit, meaningful, atomic and stand on its own. The PR, as a high level feature, is just a grouping. But single commits have to be useful and be a history/build up of how high level features are being developed. This granularity is important to logically isolate the changes and make them bisectable among other things.
- ewjordan 8y agoOnce your test suite is big enough that it won't run within a minute or two, it's kind of hard to keep a "green on each commit" workflow, though. Especially with UI and visual diff tests in the mix, I certainly don't want to tie up my local computer for a full test run when I'm just fixing a small bug; pushing the commits in a branch and letting the full suite run against that while I move on to another task is a much better use of resources, and that's where the PR model seems most useful to me. Are there workflows that avoid this problem but still enforce every commit being green?
- closeparen 8y agoIn Phabricator you make your changes on a local branch, running "arc diff" to post or update a changeset. CI runs on each update to the changeset. You're expected to have CI passing before you "arc land" (squash merge to master and push), and usually before you solicit code review. Commit whatever garbage you want locally and even publish it to the Diff, but your code doesn't touch master in the central repository until reviewed and passing.
- jonstewart 8y agoSure, I separate heavyweight tests out into integration/acceptance tests (the side effects make them slow; the side effects mean they're not really unit tests). So, I don't commit until the unit tests pass, and then a PR triggers heavyweight tests on my build server, to be sure you're clean before the merge to master. But if there's too much divergence between unit test and integration test pass/fail rates, then unit test coverage needs to be improved; obviously, easier said then done when you have old pre-TDD that's tightly coupled and/or code that's interacting with the system and incurring side effects that are hard/expensive to test.
- OJFord 8y ago> But it does require the culture of having your code reviewer check that they think there's enough code coverage. Otherwise "passing tests" doesn't mean anything. Replying just to highlight that it's possible, even common, to have a scenario in which: > "passing tests" doesn't mean anything.
- danShumway 8y agoBack in my last job, management tried to get automated standards for code coverage. This led to people writing unit tests that looked like: //implementation var CONSTANTS = { RED : "red" }: //test test(function () { assert.strictEqual(CONSTANTS.RED, "red"); }); I spent a lot of time arguing (unsuccessfully) that these kinds of tests accomplished literally nothing. I heard everything from, "well, this lets someone know that the constant shouldn't change" to "well, writing it twice will encourage programmers to be more careful." The worst was that people started doing it with translation strings. So we'd send a config file to the doc team, they'd rephrase something to be clearer, or remove a misspelling, and suddenly tests are failing and the translation string can't be committed until the next day. We had a release build delayed over this once. Meanwhile, because the coverage report is already showing the code path that displays the string as green, nobody has written tests to figure out if the dialog containing the string actually shows up on screen. Sometimes I feel like the entire concept of code coverage was just a mistake ;)
- xahrepap 8y agoCode coverage is a very poor measure of quality of your tests. You can have lower coverage and great tests, and high coverage with completely useless tests. I use test coverage as part of my workflow. Run my tests with coverage on and make sure my new code is fully covered. Kinda like a sanity check to make sure I'm not missing anything. Or if I'm stepping into old code and I want to see how well it's tested, I'll run the full suite with coverage on and see how bad it is. The second management starts asking for x% code coverage is when it starts to break down, imo.
- bethly 8y agoThis only comes up if you have data that isn't bundled with logic: as long as there is some logic to test, even enums end up with full coverage. The outstanding issue with Java code coverage tools is switch statements, because there are unhandled paths generated by the compiler, but there is an easy solution: don't use switch statements. Enums are basically always a better option anyway.
- derefr 8y agoNote that this post is talking about the idea as a follow-on to the idea of automatically running tests and then generating a commit, from every filesystem-observable change (https://medium.com/@kentbeck_7670/limbo-on-the-cheap-e4cfae840330 https://medium.com/@kentbeck_7670/limbo-on-the-cheap-e4cfae8...). I'm sure your (manual) commits keep together the changes to the code and the matching changes to the test that keep the test green—but your Ctrl+S'es probably don't. In this strange workflow, the tests are required to be green every time you press Ctrl+S; and any time you press Ctrl+S and the tests aren't green, everything you did since the last time the tests were green disappears from the worktree.
- jonstewart 8y agoOh, that's... odd.
- danShumway 8y agoI just can't imagine how that would scale in any significant way. One of the many points of automated testing is that it's automated. I don't have to mentally run all of my tests in the back of my head to tell whether or not my code is correct. With this setup, I do. I become scared to even hit Ctrl+S in case I made a typo somewhere. Or perhaps even worse, what if I made a typo in my test and just committed a half-finished implementation? I've taken automated testing, one of the most brilliant time-saving inventions in all of programming, and turned it into a chore. And as a programmer, that's supposed to encourage me to do more TDD? I understand that the article is specifically about addressing those kinds of doubts, but... I dunno. I can't imagine ever trying this for any program bigger than a couple thousand lines of code. If the goal is to have tiny changes, why not just count up the changelog and revert it if anyone ever tries to commit more than 25 lines of code? It seems like that would be just as effective, and potentially less annoying.
- pbh101 8y agoThe way I read it, it’s not on every save, but on every time you try running the tests.
- slobotron 8y agoI do this fairly often. I find it can aid communication and recall. Few examples: Say we have a parser and we discover that it chokes on certain input. I'll start a new branch and first commit will be "Failing test case for XXX" to demonstrate the bug. Then I go about fixing it (or sending it off to a colleague who is better suited to do the work). Once it's fixed, it goes into PR and it gets merged into master only if CI likes it. No one is inconvenienced by the fact that that particular test fails in one of the commits... In this case, there are couple of benefits: 1. I get to see the tests go red. Happened to me a few times where I do the test and fix in the same breath cause the fix seems simple, but then turns out the fix did nothing as the test wasn't triggering the bug. I find that I'm less likely to do a commit that says "add failing test" than I am to dive right into fixing the code prior to running the tests. 2. It's easy for anyone to go back in time to see that test indeed got red, that is to verify that provided input indeed demonstrates a bug etc. If fix and failing test case were in the same commit, then one would have to carefully checkout just some files and not others. 3. If bug is being handed off to someone, it's pretty obvious where to start if there is a single failing test when they begin looking into it. In a similar vein, another usecase for a commit that breaks the build is as a means of communication when "architecting" new features and delegating work. We are all developers, so code sometimes comes more easily than prose, and creating a new test case that fails is more expressive and less ambiguous than paragraphs of text in the ticket. Say we get a ticket that says "We are having problems with trolls", and we want to make it a bit harder for them to register (contrived example and silly solution). I can create a new branch, make the following commit and assign to a junior developer: + assert_false( + UserRepository->create( email => 'blah@mailinator.com' ), + 'Do not allow throwaway email addresses to register' + ); This way, whoever gets to work on the feature has a great starting point - ticket explains the business problem, and the failing test puts them right where they need to start looking. It can be a time saver for everyone involved vs me updating the ticket and saying: "Hi DevJunior, I think the way to solve this is to hook into the registration process. The create method in UserRepository is probably the best place to implement this - it should return failure if someone tries to register with @mailinator.com" Lastly, it's great to have a failing test to come back to when needing to leave a certain branch for a while - be it end of the day, or switching focus to more urgent matters. I find I remember quicker where I left off if test suite tells me, than if everything is green and I have to hunt what was the next thing that was needed to complete the feature... Basically, using VCS and the testsuite as a distributed TODO list :)
- craftyguy 8y agoOn some projects (e.g. Mesa), the test projects (piglit, etc) are decoupled from the project. It's not unusual that tests for a feature land after the feature is merged.