Testing Basics: Results of Tests

Tests executed by hand can end up with plethora of results ranging from "*#$& exception!!" to "it works... and this exception has always been there". In case of automated tests things look different. There are only few possible outcomes. Let us take a closer look at them.

This topic is described in more details in my book:
"Practical Unit Testing
with TestNG and Mockito"

An automated test ends up in one of two states: as passed or failed. Two other outcomes are less frequent - a test can be skipped or finish with error.

BTW. No matter if you use JUnit or TestNG you will encounter exactly the same results of tests.

Pass

Test that did not throw unexpected exception (or threw expected exception) is considered to pass. Less formally speaking, a passed test is a test which all assertions were met (unmet assertions throw exceptions). This is where you want your tests to be. :)

A passed test is usually marked with green color by IDEs and test reports.

Fail

If you follow TDD, you will encounter failing tests quite often, because that is how the TDD rhythm goes (red -> green -> refactor).If an unexpected exception is thrown then test fails. This happens if some assertion is unmet or you have a bug in your code which result in e.g. ArraysOutOfBoundsException.

Your IDE and test reports will mark such failed test with red color.

Skip

A test can be skipped (which means it was not run at all) in two cases:

  • if another test that it depended on failed,
  • if user decided explicitly that it should be skipped.

Such test is usually marked with yellow color.

Error

Eventually a test can end up as an error if some unexpected condition occurred that interrupted its execution. This is rather an uncommon situation and usually indicates that something is wrong with your test code (not production code!).

Tests that ended with an error are marked with red color.

The source code is available at github. Enjoy !

An Example

Consider the following TestNG test class which contains four test method, each of them ending with different result:

import org.testng.annotations.Test;
import static org.testng.Assert.assertTrue;

@Test
public class TestResults {

    public void passingTest() {
        assertTrue(true);
    }

    public void failingTest() {
        assertTrue(false);
    }

    @Test(dataProvider = "non-existing-data-provider")
    public void errorTest() {
        assertTrue(true);
    }

    @Test(dependsOnMethods = "errorTest")
    public void skippedTest() {
        assertTrue(true);
    }
}

The results, generated with ReporNG, are the following:

 
 
 
This used to be my blog. I moved to http://tomek.kaczanowscy.pl long time ago.

 
 
 

Please comment using