Few notes on how setting a high code coverage can do more harm than good.

This topic is described in more details in my book:
"Practical Unit Testing
with TestNG and Mockito"


Before we get to the point let me present a short anecdote. I am not sure if it is not a fake. I can't recall where did I hear it, neither am I able to find it on the Internet. Nevertheless, this is a good story, so read on.

Publisher and His Author

Alexandre Dumas (the son) used to publish novels in newspapers. At some point his publisher grew impatient of short length of delivered episodes and enforced a change of the contract they had. He decided to pay his author not by episode but by number of lines. The answer of Mr Dumas was quick. The next episode was full of very short dialogs, e.g.:

  - "Who?"
  - "Me".
  - "You?"
  - "Yes, me."

The publisher was furious and demanded another change in contract. This time he decided to pay by number of syllables. I guess he wasn't very pleased to meet a new character introduced in the next episode by Mr Dumas. This new person was a stammerer. :)

After this the publisher gave up and brought back the contract to its initial form.

Money And Human Nature

I recalled this anecdote to make one point: people tend to do what they are paid for. This has an obvious implication. In order to get exactly what you want you should pay them for doing exactly this. The problem is that sometimes there are no good measures that can be used to make sure that they "do exactly this". Take "quality of tests" for an example. How can you measure automatically that people are doing a good job in this aspect?

The common answer is code coverage. But we all know that this measurement is a weak way to assess the quality of tests.

(As you can see I don't even bother with mutation testing. This is nice for academics but not really useful in real life AFAIK.)

Code Coverage

You set up a high code coverage threshold hoping that this will result in well tested code. It might work with highly skilled and motivated team, which understands the need for high quality of tests. But if the team is inexperienced, or does not really care, or if the team regards such high code coverage as unnecessary for a certain parts of code, than you will get only what you paid for: a high percentage of code coverage. Will you also get good tests? Rather not. Instead, you can expect tests that don't really test anything but increase code coverage metrics.

What Can You Do About It?

Of course you could try to utilize Findbugs or PMD to automatically discover tests without assertions or with useless assertions (e.g. assertTrue(true)). But this can be easily avoided by smart developers (and you probably have only smart people on you team, don't you?).

The second thing you can do is to carry out code reviews of tests. (In fact this should be done by technical team lead anyway.) But this is also reactive, and will not help with negative emotions within team which is not convinced on tests idea.

Third, and probably the best way, is to have a talk with your team. What do they think about tests and code coverage? How skilled they feel they are in this aspect? How do they see it? Does it really make sense to have unit tests coverage of 90%?
Then you can mentor them, yell at them, do some transfers or change jobs. But for heaven's sake, don't impose some code coverage threshold just like that! You can do more harm than good.