This post presents results of TestNG 2012 Users Survey.

Some facts to begin with:

• the survey was taken in October 2012,
• this was the second edition, first was in 2011 (see here)
• 62 people participated (12 more than in 2011)
• the survey was announced on TestNG mailing list and on my various social sites so I guess most of participants are well-familiar with TestNG

The questions was slightly modified compared to 2011 version.

The result are here: http://kaczanowscy.pl/tomek/sites/default/files/2012_testng_survey_resul...

Now some comments.

...but first, thank you very much for taking part in this survey! :)

Why TestNG?

Similarly to last years result people find TestNG features useful and important and also like the fact that it is built with all kind of tests in mind.

Which version?

Version 6.x rules, and a lot of people are using whatever Cedric releases. This is a proof (IMHO) that released versions are stable enough to grab and use them without waiting for others. Which is a good thing.

What languages?

Java 6 and Java 7 lead the way. There is some interest in TestNG from Groovy and Scala users (there was almost none year ago).

How do you run TestNG tests?

It is not a surprise that IDEs are the most common way of running TestNG tests. From build tools Maven is the most popular with Ant and Gradle behind. This is different than year ago when Maven lead with 54%, Ant was behind with 42% and Gradle was mentioned by only one user.

TestNG killer features

TestNG is so rich of features that we had a lot to choose from. Similarly to the last year most people use groups, data providers and test-dependent methods.

What kind of tests do you run with TestNG?

As expected we use TestNG to run all kind of tests. Not surprises here.

I wish TestNG had...

It is hard to comment on this as people mentioned very different things. Ordering of tests seems to be mentioned most often.

Is TestNG your primary testing framework?

No surprise here. More than 80% uses TestNG as their main testing tool.

Other frameworks

This one is interesting. First of all, JUnit is very popular among TestNG users. But other frameworks are also often used. This definitely results from the presence of other mature JVM languages (i.e. Scala and Groovy).

Why use other frameworks?

Two main reasons for use of other frameworks are:

• because other (legacy) projects/team use it
• because it is better suited for a particular case

Both reasons are valid. Choose the right tool for the job (if you can) and if you can't stick with whatever is being used.

Which kind of assertions do you use?

It seems like in general people care about the way the assertions are written. Many among us use matchers and 25% had tried new assertions recently added to TestNG.

I'm little bit surprised by FEST being more popular than Hamcrest. I always thought the latter is the number one when it comes to matchers. Not sure if this is voting reflects the general trend among Java developers or is it only specific to TestNG users.

Mocking frameworks

The one surprise for me is the number of people not using mocking frameworks at all (almost 20%). On the second though it can be explained by the following:

• some people use hand mocks
• some people use TestNG for integration and end-to-end tests

Code coverage tools

Slightly surprised by the number of JaCoCo users (20%). JaCoCo is a relatively new framework and I haven't really expected it to be so popular.

How do you learn TestNG?

I really like these two answer: "by writing lots of code" and "by using it". That's the spirit! :)

TestNG documentation

The majority of use seems to like the current documentation. Two people gave the lowest mark - I wonder what they miss there (my fault, haven't asked about it).

Who are you?

80% devs, 20% testers and few one-army-men ;)

Free speech

It happened that the last comments in this section is "keep up the good work". And I think this is a very nice ending of this survey. :)

Thank you, see you next year!