Validate Your Tests With Jester

What is Jester?

Jester is available  on sourceforge (http://jester.sourceforge.net/) and on the project page, the tool is described as a "JUnit test tester". I think that describes it very precise. Jester basically tests how many changes to the code are discovered by the tests. 

What does Jester do?

What Jester does is rather simple. The tool scans the source code for possible mutations. For any mutation, Jester modifies the source code and runs the tests again. If at least one test fails, everything is fine. After that, the change is reverted and Jester continues scanning.

What is a mutation?
This  can be configured in a way that if a pattern <A> is found, it is substituted with <B>. Some of the mutations pre-configured are:

• "true" -> "false"
• "false" -> "true"
• "if(" -> "if(false && "
• "if("-> "if(true || "

After Jester is done with scanning for mutations, there are 2 results:

1. A score between 0 and 100. Example: If there were 100 mutations and 30 were not found with the tests, the core is ... 70!!! Suprising, right? :)
2. An HTML report showing all mutations that "survived" the tests.

How to use Jester?

Basically, Jester needs 3 things to know in order to do the job:

1. Where to find the source code?
2. How to build the project?
3. How to run the tests?

So, a bit simplified, you just need a directory and 2 ant scripts. In addition, Jester needs Python to be installed if you want the mentioned HTML report. But this is optional.

Why to use Jester?

I find Jester very useful and interesting for 2 reasons.

Validate your coverage

Of course, to measure code coverage is very useful and I think important as well. BUT it is very hard to interpret. The fact that X% of your code is covered with your tests is good to know. No question about that. What does this mean? I believe this alone means NOTHING. It is too simple to write tests that execute a lot of code, but validate only a little. I don't necessarily mean by accident, but unless you do strict TDD, the coverage will always be a bit better than what you really test.

I believe the Jester score can be some kind of hint on valid your code coverage is. So what means the Jester score?

100 - Most likely any change (bug?) introduced to your existing code will be discovered

0 - You probably can do anything with the code, the tests won't break. I like to call such tests "coverage tests". They are good for generating code coverage, but for nothing else.

Of course, the truth will be somewhere between. At least I find it very interesting to find out to which end I am nearer :)  

Find missing tests

The above mentioned HTML report with all the mutations that "survived" the tests is nothing less than a list of missing tests. If "true" can be changed to "false" without any test to fail, there are 2 possible reasons:

1. A test is missing to test the need for that "true".
2. That "true" is not needed and can be removed.

Summary

When I first heard about Jester, I was very curious about how it would work within the project I work with. Now that it works fine, I really like the insight I get from it. I really can recommend it and would like to hear other experiences.