Is test-first better than test-last ? Is there a significant difference in the quality of code developed using these approaches ? It's hard to say for sure. My gut feeling (and experience) tells me that test-first is superior. Sometimes I find evidences for this claim. Like the one I present in this blog post.

You will find much more about testing in my book
"Practical Unit Testing
with TestNG and Mockito"


The example below is a simplified version of a real code that we are working on right now.

Let us consider two entities - Document and Metadata. Each Document stores a set of Metadata. The document class offers methods for adding, retrieving and removing metadata.

Let us say that we have a good tests for addition and retrieval of metadata. Now we want to test "remove all metadata" functionality of Document class.

Test-last

Now, let us assume, that deleteAllMetadata method of Document class is already implemented and we want to write a test code.

In fact, this is exactly what happened in our project. The developer was coding test-last, and this is the test that he created:

private IDocument testDoc;

public void testDeleteAllMetadata() {
    testDoc = new Document(DOC_ID);
    Set<IMetadata> metadataSet = new HashSet<IMetadata>();
    Metadata metadata = new Metadata(META_KEY1);
    metadata.addMetadataValue(META_VALUE1);
    testDoc.addAllMetadata(metadataSet);
    testDoc.deleteAllMetadata();
    Set<IMetadata> result = testDoc.getAllMetadata();
    assertTrue(result.isEmpty());
}

The problem is, that this test is wrong. It will pass even if deleteAllMetadata method has no body at all. Why ? If you look closer, you'll notice that even though the metadataSet is added to the document, it contains no metadata from the very beginning (developer forgot to add metadata to metadataSet).

    metadataSet.add(metadata); // THIS LINE WAS MISSING

So, what can we do about this ?

First of all, you should check your tests ! In this particular example, you should definitely check if some metadata are available in the document, before you try to remove them.
Like this:

private IDocument testDoc;

public void testDeleteAllMetadata() {
    testDoc = new Document(DOC_ID);
    Set<IMetadata> metadataSet = new HashSet<IMetadata>();
    Metadata metadata = new Metadata(META_KEY1);
    metadata.addMetadataValue(META_VALUE1);
    testDoc.addAllMetadata(metadataSet);
    Set<IMetadata> result = testDoc.getAllMetadata();
    assertTrue(result.isNotEmpty()); // WOULD FAIL HERE
    testDoc.deleteAllMetadata();
    assertTrue(result.isEmpty());
}

Remember, we already have a good test that checks if the addition and retrieval of metadata works well (test, that checks that if you add XYZ metadata to a document, you can retrieve the same XYZ metadata).
Now, in the test above, I don't want to repeat myself, I don't try to check once again that addition works - I already know it works. This line:

assertTrue(result.isNotEmpty());

does not check my production code, it checks the test code. And by running the test now, you will see that it fails prematurely, which means there is no metadata in the document object, and removing it is useless.

This real-life example shows, that testing after writing the production code can lead to buggy tests. It is kind of natural. The developer is pretty sure that his code works. He writes a test, expecting it to pass. The tests passes (accidentally), so the developer has no clue if there are any problems.

We have also seen, that one can prevent such mistakes by checking (testing) the test code. Can we do better than this using test-first technique ? Let us see.

Test-first

Let us assume, we start once again, but this time following the test-first paradigm.

Naturally, we start with a test.

private IDocument testDoc;

public void testDeleteAllMetadata() {
    testDoc = new Document(DOC_ID);
    Set<IMetadata> metadataSet = new HashSet<IMetadata>();
    Metadata metadata = new Metadata(META_KEY1);
    metadata.addMetadataValue(META_VALUE1);
    testDoc.addAllMetadata(metadataSet);
    testDoc.deleteAllMetadata();
    Set<IMetadata> result = testDoc.getAllMetadata();
    assertTrue(result.isEmpty());
}

The test does not compile. We need to add the deleteAllMetadata method to Document class. So, let us add it.

public void deleteAllMetadata() {
  // TODO implement me, please !
}

Now we run the test, expecting it to go red (the remove functionality is not written yet !) but surprisingly the test passes ! It can mean only one thing - the test is flawed !

Quickly, we add the missing line and repair the test:

private IDocument testDoc;

public void testDeleteAllMetadata() {
    testDoc = new Document(DOC_ID);
    Set<IMetadata> metadataSet = new HashSet<IMetadata>();
    Metadata metadata = new Metadata(META_KEY1);
    metadata.addMetadataValue(META_VALUE1);
    metadataSet.add(metadata); // THIS LINE WAS MISSING
    testDoc.addAllMetadata(metadataSet);
    testDoc.deleteAllMetadata();
    Set<IMetadata> result = testDoc.getAllMetadata();
    assertTrue(result.isEmpty());
}

We will notice with delight now, that the test fails this time. Hip hip, hurray ! Now, we can start working on deleteAllMetadata method implementation till the test passes.

Conclusion

I tried to make two points in this post:

1. test-first will not allow you to make some mistakes
2. you should test your tests !

Call me paranoiac, but even writing test-first, I'd definitely check my tests. I would add this line:

assertTrue(result.isNotEmpty());

to make sure that I'm really testing something.