Long time ago I got attracted to mutation testing (http://en.wikipedia.org/wiki/Mutation_testing). Alas, at that time all the Java mutation testing tools were useless: abandoned and forgotten long time ago. Lately I discovered a new tool - PIT Mutation Testing (http://pitest.org/) - which is actively developed. So I gave it a try. A very limited try (but that is only the beginning!).

The Idea

Does "mutation testing" rings any bells? No? I'm not surprised, so let me briefly describe the idea (there is much more to discuss, but let us keep it short).

Suppose you have some classes and a suite of tests. Now imagine you introduce a change into one of your classes, for example by reverting (negating) one of if conditional expressions. This way you created a mutant of your code. What should happen now if you run all tests once again? Provided that the suite of tests contains a test that examines this class thoroughly, then this test should fail. If no test fails, this means your test suite is not good enough. And that is exactly the idea of mutational testing.

As you can see mutation testing goes much further than code coverage (which is frequently used as a metric of tests quality, which, BTW, is a silly thing to do).

PIT Tryout

To see for myself what PIT can do, I created a toy-project (https://github.com/tomekkaczanowski/pit-testng). When you run it (see README on github), you get the results as presented on the picture below:

You will find more on mutation testing in my book
"Practical Unit Testing
with TestNG and Mockito"

As you can see the report, generated automatically by PIT, presents which mutants of the original class survived when run against my tests.

Conclusions

• PIT is still very young, but already impressive and actively developed
• I haven't tested it yet on any codebase of reasonable size - only on a toy-project
• PIT works with JUnit (its primary testing framework), but recently support for TestNG was added (Yupi-yey!)

Ok, that would be it for now. I'm really excited about PIT, and its further development. I contacted Henry Coles - author of PIT - and hope to somehow help him (or at least encourage him to keep going!). If you also see some value in mutation testing, give PIT a try and join the community.

Stay tuned, I will surely blog some more on PIT.

Mutants

Since this blog post is to large extents devoted to mutants, I couldn't help myself but to put these two pictures. :)

Source: http://verydemotivational.memebase.com/

Source: http://morifake.com/