Here is a small update on my little fun project.

I released version 0.2 of quant. Now it will recognize all TestNG annotations which do not have a TestNG group (like @BeforeMethod, @BeforeClass etc.). The method ‘reportViolation’ on ClassTester will now report whats wrong with the examined class.

[/java]
 assertFalse(classTester.isInvalidTestClass(),
    classTester.reportViolation());
[java]

Above code will now report:

java.lang.AssertionError:
Report for Class com.maxheapsize.quant.testclasses.SetupMethodWithoutTestGroup
Ignore abstract classes: true
Specified TestGroups :  + testUnitTest
* Methods with wrong test group:
  - testSetUp

In this way you know what went wrong and where to look for it.

Quant is now also available in my repository. To include it add the following code to your pom.xml.

<dependency>
    <groupId>com.maxheapsize</groupId>
    <artifactId>quant</artifactId>
    <version>0.2</version>
</dependency>

For now you need to add my mvn repository to your pom. I’m about to have it mirrored to the official servers.

Have fun.

Did you ever wondered if all tests your team wrote are really running ? How many disabled tests does your code base have? How many public void methods do not have a @Test annotation (or at the class) ?

I saw all of that in the last years.

To overcome this situation I wrote a couple of java classes which will scan your java test sources and will examine the annotations (for TestNG, sorry JUnit). The code will detect if all public void methods do have a @Test annotation (either direct on the method or on the class) or if there are disabled tests. Both signals are most likely a sign of rotten code.

If you have different test groups defined like ‘unitTest’ or ‘integrationTest’ you want to make sure all tests are in at least one test group. This ensures if you run all test groups all tests are executed.

The usage is really simple. To make sure all your tests are ok is to write just another test which will check all tests.

ClassFinder classFinder =
   new ClassFinder.Builder(sourceDirectory).build();
for (Class klass : classFinder.getClassList()) {
  TestClassTester testClassTester =
    new TestNGTestClassTester.Builder(klass).build();
  Assert.assertTrue(testClassTester.allTestMethodsHaveValidTestGroup());
}

You need to specify the source files of your code so that all java files can be scanned. It is possible to exclude certain packages to be checked (via TestNGTestClassTester.Builder(klass).addExcludedPackage("testclasses").build()). After gathering all class names from the sources the code will check for @Test annotations in the test classes. You can also specify the test groups which all test should belong to (via addTestGroup("unitTest") in the builder).

Finding disabled test methods is very simple too (one of the unit tests)

@Test
public void testNoDisabledTest() {
  DisabledTestFinder unitUnderTest =
    new TestNGDisabledTestFinder.Builder(TwoTestGroups.class).build();
  assertFalse(unitUnderTest.hasDisabledTests());
}

This would test a single class (but you can of course feed it with the results of the ClassFinder).

If you combine the possibility to break your build because of defect test classes and a continuous integration system, you can make sure everybody will annotate their classes correctly or never disables tests (or even better use a CI with delayed check in and personal build like Jetbrains Teamcity). Of course you can define your own thresholds for e.g. 30 disabled tests are allowed (I know sometimes you just can’t avoid it).

The code depends on testng and commons-io and is released under the Apache v2 license.

I still need to figure out where to put the maven2 sources. Either Google code or GitHub ? Any ideas?

Update: Code is available at Google Code Hosting.

Things to implement:

• A more flexible exclude patterm, like Apache ant maybe
• Make sure to check @BeforeXXX and @AfterXXX methods as well
• Implement JUnit (maybe )