We have been using our own flavor of Fit for Rules (which is build on top of fit) for about 1 1/2 years now to test our business logic written in JBoss Rules 5. It’s relatively easy to get the Business Analyst on board since he is using his tool (which is Microsoft Excel) to communicate test cases for the rules. So in theory, he writes the tests in Excel, we do the rules coding and voila, all tests turn green.

Reality is, we have to tweak the Excel sheets. We need to put in imports of our fact model, insert facts and create objects within that not so programmer friendly table environment. A couple of days ago we got the request to tweak some rules and we all had to start doing rules again (and we used to use Eclipse for writing rules because that’s the only IDE having a plugin for that).

After half a day of coding Java syntax in Excel sheets we decided that the ramp up time for the not so knowledgeable rules/fit programmers like me is too much. With debugging, copy and paste we spent easily 5-10 times more time on making the tests work than writing the code itself. Test driven design is not really an option here, since you need to know the imports of the rules file to get the sheet even to compile.

So what did we do ? Well why not try to get things working the way we used to do it ? TestNG anyone ?

There are many pros to use unit testing but also some cons. The biggest issue is that we will loose the direct communication to the business analyst. It’s always better if someone else writes the test and I just have to implement the solution. Maybe we can find another solution involving Active Spec or DSL. For now we stick to unit tests and the task the we have to make sure we convert every Excel test case to java code (but hey, that’s what our code reviews are for).

Checkout our current base class for testing our rules:

public abstract class AbstractRulesTest {
   public abstract String[] getRulesFileNames();
   private final String GET_FINDINGS = "import com.maxheapsize.RulesFinding;" +
                                                                           "query \"getAllRulesFindings\"\n" +
                                                                           "   finding : FRulesFinding()\n" +
                                                                           "end";

   private static Logger LOG = Logger.getLogger(AbstractRulesTest.class);

   public final List<FRulesFinding> fireRules(Set factsForWorkingMemory) {
       KnowledgeBase ruleBase = setUpKnowledgeBase();
       return fireRules(ruleBase, factsForWorkingMemory);
   }

   public KnowledgeBase setUpKnowledgeBase() {
       KnowledgeBaseConfiguration configuration = KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
       KnowledgeBase ruleBase = KnowledgeBaseFactory.newKnowledgeBase(configuration);

       KnowledgeBuilder build = KnowledgeBuilderFactory.newKnowledgeBuilder();
       build.add(ResourceFactory.newReaderResource(new StringReader(GET_FINDINGS)), ResourceType.DRL);
       String[] fileNames = getRulesFileNames();
       for (String fileName : fileNames) {
           File userDefinedFile = new File(fileName);
           build.add(ResourceFactory.newFileResource(userDefinedFile), ResourceType.DRL);
       }

       handleBuilderErrors(build);

       ruleBase.addKnowledgePackages(build.getKnowledgePackages());
       return ruleBase;
   }

   private void handleBuilderErrors(KnowledgeBuilder build) {
       if (build.hasErrors()) {
           KnowledgeBuilderErrors knowledgeBuilderErrors = build.getErrors();
           for (KnowledgeBuilderError knowledgeBuilderError : knowledgeBuilderErrors) {
               int[] ints = knowledgeBuilderError.getErrorLines();
               LOG.error("Error at : "+ints[0]+" : "+ints[1]);
               LOG.error(knowledgeBuilderError.getMessage());
           }
       }
   }

   private List<FRulesFinding> fireRules(KnowledgeBase ruleBase, Set facts) {
       List<FRulesFinding> result = new ArrayList<FRulesFinding>();
       StatefulKnowledgeSession statefulSession = ruleBase.newStatefulKnowledgeSession();
       for (Object fact : facts) {
           statefulSession.insert(fact);
       }
       statefulSession.fireAllRules();

       QueryResults results = statefulSession.getQueryResults("getAllRulesFindings");
       try {
           FRulesFinding finding = (FRulesFinding) results.iterator().next().get("finding");
           result.add(finding);
       }
       catch (NoSuchElementException e) {
           result = new ArrayList<FRulesFinding>();
       }
      return result;
   }
}

All my rules insert a RulesFinding (and only one at the moment) into the working memory when triggered. The rest is pretty easy. You subclass it, overwrite getRulesFileNames and call fireRules with a set of objects (your tests) which need be insert into the working memory. To get the finding back you need to execute an already inserted query which needs to have an identifier (line 3, 20, 52, 54). It will contain the result of your rule execution.

Sample code would look like this:

public class RulesTest extends AbstractRulesTest {

   private Set facts;

   @BeforeMethod
   public void setUp() {
       facts = new HashSet();
   }

   @Override
   public String[] getRulesFileNames() {
       return new String[]{
               "src/main/rules/myrules.drl",
               "src/main/rules/generealRules.drl"
           };
  }

   @Test
   public void testDemoRule() {

       FMyFact myFact = new FMyFact();
    myFact.setColor("green");
       facts.add(myFact); // add all your facts here

       List<FRulesFinding> findings = fireRules(facts);
       Assert.assertTrue(findings.size() == 1);
       FRulesFinding finding = findings.get(0);
       Assert.assertTrue(finding.getStatus() == FStatus.OK);
   }
 }

Depending on how you cut your rules you can extract the assertion of the status.

Each test case in Excel now takes about 5-10 lines of java code. Considering we are covering each rule with about 5-15 test cases and boundary conditions this amounts to 75-150 lines of test code. I take that any day over programming in Excel.

Here is a small update on my little fun project.

I released version 0.2 of quant. Now it will recognize all TestNG annotations which do not have a TestNG group (like @BeforeMethod, @BeforeClass etc.). The method ‘reportViolation’ on ClassTester will now report whats wrong with the examined class.

[/java]
 assertFalse(classTester.isInvalidTestClass(),
    classTester.reportViolation());
[java]

Above code will now report:

java.lang.AssertionError:
Report for Class com.maxheapsize.quant.testclasses.SetupMethodWithoutTestGroup
Ignore abstract classes: true
Specified TestGroups :  + testUnitTest
* Methods with wrong test group:
  - testSetUp

In this way you know what went wrong and where to look for it.

Quant is now also available in my repository. To include it add the following code to your pom.xml.

<dependency>
    <groupId>com.maxheapsize</groupId>
    <artifactId>quant</artifactId>
    <version>0.2</version>
</dependency>

For now you need to add my mvn repository to your pom. I’m about to have it mirrored to the official servers.

Have fun.

Did you ever wondered if all tests your team wrote are really running ? How many disabled tests does your code base have? How many public void methods do not have a @Test annotation (or at the class) ?

I saw all of that in the last years.

To overcome this situation I wrote a couple of java classes which will scan your java test sources and will examine the annotations (for TestNG, sorry JUnit). The code will detect if all public void methods do have a @Test annotation (either direct on the method or on the class) or if there are disabled tests. Both signals are most likely a sign of rotten code.

If you have different test groups defined like ‘unitTest’ or ‘integrationTest’ you want to make sure all tests are in at least one test group. This ensures if you run all test groups all tests are executed.

The usage is really simple. To make sure all your tests are ok is to write just another test which will check all tests.

ClassFinder classFinder =
   new ClassFinder.Builder(sourceDirectory).build();
for (Class klass : classFinder.getClassList()) {
  TestClassTester testClassTester =
    new TestNGTestClassTester.Builder(klass).build();
  Assert.assertTrue(testClassTester.allTestMethodsHaveValidTestGroup());
}

You need to specify the source files of your code so that all java files can be scanned. It is possible to exclude certain packages to be checked (via TestNGTestClassTester.Builder(klass).addExcludedPackage("testclasses").build()). After gathering all class names from the sources the code will check for @Test annotations in the test classes. You can also specify the test groups which all test should belong to (via addTestGroup("unitTest") in the builder).

Finding disabled test methods is very simple too (one of the unit tests)

@Test
public void testNoDisabledTest() {
  DisabledTestFinder unitUnderTest =
    new TestNGDisabledTestFinder.Builder(TwoTestGroups.class).build();
  assertFalse(unitUnderTest.hasDisabledTests());
}

This would test a single class (but you can of course feed it with the results of the ClassFinder).

If you combine the possibility to break your build because of defect test classes and a continuous integration system, you can make sure everybody will annotate their classes correctly or never disables tests (or even better use a CI with delayed check in and personal build like Jetbrains Teamcity). Of course you can define your own thresholds for e.g. 30 disabled tests are allowed (I know sometimes you just can’t avoid it).

The code depends on testng and commons-io and is released under the Apache v2 license.

I still need to figure out where to put the maven2 sources. Either Google code or GitHub ? Any ideas?

Update: Code is available at Google Code Hosting.

Things to implement:

• A more flexible exclude patterm, like Apache ant maybe
• Make sure to check @BeforeXXX and @AfterXXX methods as well
• Implement JUnit (maybe )

A couple of days ago I had the case that I needed to test a method with different parameters. I ended up writing a couple of test methods differing only in passing various arguments to the unit under test.

Tonight I was at a TestNG talk and while I knew most of the stuff already the DataProviders (which I heard of before but unfortunately never really payed attention to) really caught me.

Now I can create a DataProvider which generates test data. Each of these data sets will result in call of the test method with the corresponding arguments.

The following code will test a String to Property Converter if it works correctly. It takes two parameters, first the string to be converted, second the result which I will assert.

@Test(dataProvider = "convertTestDataProvider")
public void testConvert(String property, String result)
{
    Properties properties =
      stringPropertyConverter.convertString(property);
    Assert.assertTrue(properties.get("A").equals(result));
  }

Now the DataProvider must return an array of array of objects. TestNG will cast the return values to the method signature of all the tests with the corresponding annotation.

    @DataProvider(name = "convertTestDataProvider")
    public Object[][] convertTestDataProvider()
    {
        return new Object[][]{
                {"A=", ""},
                {"A=1", "1"},
                {"A=2=3", "2=3"},
                {"A=2" + StringPropertyConverter.ideaLineSeperator +
                 "# Comment" +
                 StringPropertyConverter.ideaLineSeperator + "C=1", "2"},
        };
    }

Here I cover 4 test cases. It is very easy to add more tests just by adding one more line with the values to test and the expected result.

TestNG results with DataProvider

This will save me some amount of time.

Good Job Cedrik & friends.

Next step: How to make sure you cover all the relevant test cases (and having a systematic way of getting there). Anybody has an Idea?