4 ms·
One of the greatest lines I have ever read on a blog: It may be hard to imagine writing rock solid one-in-a-million-or-better tests that drive Internet Explore
by patio11 18y ago
One of the greatest lines I have ever read on a blog:
It may be hard to imagine writing rock solid one-in-a-million-or-better tests that drive Internet Explorer to click ajax frontend buttons executing backend apache, php, memcache, mysql, java and solr. I am writing this blog post to tell you that not only is it possible, it’s just one part of my day job.
- ntoshev 18y agoThese are automated functional tests. It would be interesting to know if the author uses unit test too and how he writes tests in general. Also some code:test ratio metrics would be indicative of how much weight does all this testing add.
- jmathes 18y agoI work at IMVU as well, know the author personally, and can tell you that yes, he unit tests too. In fact, like all of us, writes all code test-first. For test strategy in general, you need a whole seminar. There's a good blog on testing over at http://googletesting.blogspot.com/ http://googletesting.blogspot.com/ <- go there! As for code:test ratio, I'm not sure what metric you want, but using just php files, there are ~1k active test files and ~7k php files total. So if you take into account that 1k of that 7k is the test, and some is test-setup and obsolete files we no longer use, it's maybe about 5:1 code:test. However, a lot of the code is third party OS software packages. In-house code is about 1.5:1 test:code, in my experience. These, by the way, are very rough estimates.
- ntoshev 18y agoThanks a lot, this is pretty much what I wanted to know. Looks like there is not too much overhead and the results are amazing.
- plinkplonk 18y ago"writing rock solid one-in-a-million-or-better tests that drive Internet Explorer" I find this unparseable. (English is not my native language). As far as I know "one in a million" means something like "very rare". Help?
- briansmith 18y ago"rock solid" is "very reliable" "one-in-a-million-or-better tests" is "tests which fail less than one in a million times". Our Internet-Explorer-based tests are very reliable; they fail less than once per million executions.
- sbraford 18y agoWhile the "one in a million" better is a cool blurb, what does it really mean? Let's say your team makes 25 commits per day. 25 * ~300 working days = 7,500 commits per year That would take 133+ years to reach 1 in a million. The more interesting metric to me is how often the build gets broken.
- nuclear_eclipse 18y agoThe part you're missing is the 15000 tests, multiplied by a new commit every 9 minutes, which in 8 working hours, is roughly 50 commit-test cycles, so 750,000 tests run in a day's timespan... Edit: of course that assumes a peak commit rate matching or exceeding the commit-test cycle period. The point being that even a considerably low rate of failure in the testing mechanism could manifest itself as a blocked commit-test-deploy cycle at least once a day, hence the importance placed on rock-solid testing systems that should only ever fail when the tested code itself fails.
- TimothyFitz 18y agoWe empirically have on average 70 builds a day. The number is higher than your calculation because we don't all work 9-5, we're commiting frequently from around 8am to 9pm. We also run builds repeatedly overnight to flush out any intermittently failing tests we may have recently introduced. We'll run the builds as fast as they can go from 2am-4am.
- sbraford 18y agoSo how often does a commit get checked in that causes a test (or tests) to fail? It just seemed to me like you were bragging that tests get run over and over again. They only need to get run if any new code is committed, of course. And what kind of commit is being checked in every 9 minutes? How big is the dev team? Seems like an awful lot of commits. Is each one a full-fledged feature / bug fix for the site, or are many 1-line changes to the code?
- edw519 18y agonot only is it possible, it’s just one part of my day job A necessary, but not sufficient, requirement for a nimble start-up.