Results Of TestNG Users Survey Q4 2011

Interested in TestNG?
See my book
"Practical Unit Testing
with TestNG and Mockito"

Introduction


Below you will find the results of an online survey on TestNG.

I did not find the way to share the results of survey in any civilized manner (Google Docs, aren't you missing and important feature?!) so I need to paste them as screenshots. Sorry. If you want to see original spreadsheet, go to https://docs.google.com/spreadsheet/ccc?key=0Ar3dS2idvVMLdE12WEJRY3VKb3N...

And one more thing - results lie when it comes to the number of "other" answers. Sometimes there was less of them then indicated by charts. Not sure why - maybe because of blank answers.

Results

50 people responded to online survey on TestNG.


There were also few other reasons to use TestNG mentioned by responders:

  • I am forced to use it
  • It allows for sequenced tests
  • It seems to fit best with selenium

"I am forced to use it" is definitely my favorite. :)



Seems like we all like new stuff. No one pointed to anything older than 5.x, and most of use are using 6.x. A lot of people is using the latest version. Nothing surprising here, because in general TestNG is backward compatible, so not a big issue to upgrade to the latest one.

No one "always uses the latest beta", which is rather reasonable. We take beta only when testing some new features.



Java, Java, Java, boring... :) One poor soul still on Java 1.4. Some Groovy users, and no Scala at all.

IDEs rules, and so does Eclipse. More Maven than Ant.

In case of running TestNG tests, some people mentioned also:

  • CI server
  • Jenkins
  • Gradle
  • jruby version of rake via exposed ant task

I'm really disappointed with lack of Gradlers (except for myself!).



Regarding the features of TestNG responders also pointed out the following:
  • Before Class
  • Method Selector

Some comments to the most often used features:

  • (38 + 22) Passing parameters to tests (using testng.xml or DataProviders) is number 1. Not surprised with DataProviders results - they are really cool, and at the same time very easy to use.
  • (38) A little surprise with Groups being so often used. I do not use them for unit tests but only for integration or end-to-end tests, so I did not think they are so popular.
  • (34) Tests/Groups dependencies - definitely one of the selling points of TestNG. Especially useful to implement a "fail fast" approach which is so very useful for integration and end-to-end tests.
  • (22) Custom listeners / reporters. Interesting, so many of use takes adavantage of TestNG hooks allowing to use tweak the output of tests.
  • (20) Parallel tests execution. Yeah. We write a lot of code which is being executed by many threads, so nothing strange that so many of us use this feature.


Here we can see that TestNG is much more than unit testing frameworks.

In case of this question responders mentioned many specific testing frameworks and tools they use:

  • end to end tests with MicroFocus Silk4J
  • DBUnit
  • system test on a telecom SUT
  • DB validation with data providers in groovy
  • we also run tests that exist in other frameworks
  • massive tests, random tests, ad-hoc tests
  • functional system tests
  • regression tests
  • Blitzer to carry out performance tests


Few comments on this:
  • Code coverage tools used only by ~50% of us? I find it surprising. I would expect it to be much more. Is it because a lot of tests we run with TestNG are integration/end-to-end tests which makes it harder to measure code coverage?
  • Test doubles are our friends. Seems like this testing technique is commonly used by TestNG users.
  • Quite a lot of people mentioned ReportNG. Apparently the aesthethic look of the default TestNG reports is not enough.
  • Web tests seems to be very popular with TestNG.

Responders mentioned the following "other" tools they use together with TestNG:

  • Cuanto
  • STAF
  • xpath
  • firebug


JUnit and then almost nothing. The only "other" answer was:
  • NUnit


Nothing beats the original documentation (btw. why 8% of us do not use it?). Other online sources are also popular. In "other" category the folowing responses were given:
  • browsing sources
  • Just reading the mailing list
  • video source
  • Trial and Error


One additional comment regarding the TestNG documentation:
  • So-so, but is most of the time quite updated


Test-first or code-first? Based on the given answers, it seems there are more options:
  • Depends
  • I am a tester so its always code first (which I don't write ;) )
  • Varies with new code, plus a goal to add tests to large existing codebase.

And by the way, test-first is more popular than I expected it to be. Still, less popular than code-first.


And finally the last questions:

Forgotten Questions

Things I should have asked about:

  • do you modify directly testng.xml?
  • are you a boy or a girl?
  • who are you - developer, tester, etc.
  • what do you like / dislike about the TestNG documentation?
  • missing features of TestNG
  • free space for expressing your opinions/expectations/comments on TestNG

Well, next year maybe. :)

Thank You!

Thank you once again for responding to this survey.

Please comment using