The JMockit Unit Testing library continues to astound. One new thing I discovered is its Coverage reporting.
Code Coverage Code coverage is simply a measurement of what code has been actually run when tests are executed. There are many such measures, ramifications, and tools. Like testing itself, code coverage measurement is probably not done enough, or misused.
“Code coverage tells you what you definitely haven’t tested, not what you have.” — Mark Simpson in comment
Path Coverage
Plenty of coverage reporting tools out there. What this one also includes is Path coverage. This is different then branch coverage. Paths are possible execution paths from entry points to exit points. If you visualize a methods statements in a directed graph, paths are a enumeration of the possible edges traversed when that method is invoked. So, Path coverages is inclusive of Branch coverage. Well, I’m not a testing expert, so this may be way off.
Very surprising results. For example, you run a coverage report with a tool such as Cobertura or Emma and feel very happy that you exercised every line and branch with your tests. Then you run the same tests but use JMockit Coverage and discover your tests didn’t cover all the paths! Not only that your line coverage wasn’t so great either.
Report
JMockit explicitly gives you a report showing:
Path
Measures how many of the possible execution paths through method/constructor bodies were actually executed by tests.
The percentages are calculated as 100*NPE/NP, where NP is the number of possible paths and NPE the number of fully executed paths.
Line
Measures how much of the executable production code was exercised by tests. An executable line of code contains one or more executable segments.
The percentages are calculated as 100*NE/NS, where NS is the number of segments and NE the number of executed segments.
Data
Measures how many of the instance and static non-final fields were fully exercised by the test run. To be fully exercised, a field must have the last value assigned to it read by at least one test. The percentages are calculated as 100*NFE/NF, where NF is the number of non-final fields and NFE the number of fully exercised fields.
– from the JMockit coverage report HTML page
Other information is found by using the full HTML output option.
Example
A sample JMockit coverage report is here. Of course you can drill down into various parts of the html page. Like when you click on an exercised line you will get a list of what invoked that line.
Worth it?
Are the metrics such as Path coverage that this tool generates accurate? Is JMockit coverage a replacement for other tools such as Cobertura? I don’t know. For most projects, the resources would probably make the use of coverages generated by multiple tools prohibitive.
Evaluation
One possible approach to evaluating coverage tools is to just use actual real results of the target application. Use the list of bugs and correlate to a coverage tool report. Where were the bugs? Which tool gave the least measure for this location?
Some developers refuse or are slow to test units for certain values or situations because, they claim, those values or situations will not occur.
Some reasons given are that the presentation tier or UI will prevent certain values; that other modules in the call chain already have error handling and validation. So why test something that won’t happen?
Let’s take an example. In the method/function below, the type will never be blank or null. Looking at the actual source code for the application will show this. So should a unit test be written that will invoke this method with a null, “”, or ” “?
public boolean service(final String type){
// do stuff
return result;
}
Yes!
1. Things change.
2. Bad stuff happens.
3. Development process will use invalid values.
4. More complete testing. Regression testing, path coverage, etc.
5. The method could now be used in a different call chain.
6. Its a public method, anything can invoke it, even a business partner via some remoting technology.
7. When invoked as part of other unit tests, it could have invalid values.
#3 is, I think, the most important. Did you ever do development and something not work and it turn out to be existing code that did not handle arguments correctly? That is wasted time and an aggravation. Sure, in the current production code, a blank string won’t be used, but during development, especially TDD, you scaffold code. Sometimes you don’t have values yet, so you just use a blank string.
Just the other day I tested a deployed production method that correctly tested the argument for an empty string, “”, and did the right thing. However, the code did not check for a blank string, ” “, and throws an exception. Unit testing would have shown this.
And, yes, this will never happen in our application.
Ok, you still don’t want to test something for null or invalid values? At least put this in writing in the Javadoc: “I (insert your name and address) am a great developer and stake my professional career that this method will never be invoked with invalid values.” The address is needed so us mediocre developers can hunt you down when the server crashes.
So, I was thinking of testing my getter and setters in a JUnit test. Yea, I know it is not recommended, but I thought that I could write a simple reflective iterator that could do it without much fuss.
What I show below is that maybe getter/setters should be tested, especially in Java which really does not have “real” properties. Further, by investigating a potential solution, I show that it doesn’t work, and the reason is that testing IS hard to do.
Anyway, just to save time, and not have to code this, I did a quick search and found that this has been done many times before. So I looked at one of the solutions, Scott’s. Very nicely done! With one line you can test a class’s property getters and setters. Then I noticed a problem.
In his approach he invokes the setter and then the getter, finally comparing that the values are the same. For example, if the property is a boolean, the test class will set it true, then read it and the value better be true. However, what if the value prior to your set was true, the object is initialized with true for the field value?
For example, I took a class and set field x to true via the default constructor, then I modified the setX method to not set the field value.
public class Fubar {
private boolean launch = true;
public void getLaunch(){
return this.launch;
}
public void setLaunch(boolean launch){
/* this.launch = launch; */ // broken!
}
}
In the unit test the getLaunch method still returned true. The getter/setter test did not fail; the test was bogus. Not only that, the fact that the setLaunch method did not work illustrates that sometimes testing setters is warranted.
Thus, the version of the program that controls the sports strategy will ship and the boolean that aborts a play cannot be set to false! (Edited, removed joke about war stuff; you can’t be too careful with all the craziness in the world).
In a recent presentation I gave on unit testing, I said that testing could be difficult. This is a great example of this. In this case, one should have followed analogous patterns in other fields. Thus, from semiconductor testing, for example, you would test gates by not just writing a 1 bit, you have to write 1 and 0 to make sure the gate is working.
Updates
Feb 11, 2012: I’m using Scott’s solution in our test suite. I did modify the code a bit. Turns out that a JavaBean property is not so clear cut. If you have a getter and setter method pair but no actual target field, are those Javabean methods? Looks like the Java bean introspector will report these as bean accessors. Hmm.
Yesterday at work I gave a presentation on Unit Testing. It went well. 160 slides! And, no one passed out and hit the floor.
One thing I mentioned was mocking frameworks and how JMockit is very useful. Perhaps JMockIt represents the state of the art in Java based Mocking tools.
“JMockit allows developers to write unit/integration tests without the testability issues typically found with other mocking APIs. Tests can easily be written that will mock final classes, static methods, constructors, and so on. There are no limitations.” — JMockit
JMockit is ” a collection of tools and APIs for use in developer testing, that is, tests written by developers using a testing framework such as JUnit or TestNG.”
I’ve used it for some tests. Since it uses Java instrumentation it can mock almost anything, especially those legacy untestable great OO classes. Best of all it has a very good tutorial.
The only ‘negative’, so far, is that JMockit does not, afaik, have many developers working on the project. That could also be a plus, of course.
Seems to me there are too many mock frameworks and they do pretty much the same things. Time for consolidation so that an API and a body of practice can shake out?
A simple example is used to show how to use Aspect Oriented Programming to provide fault injection.
Intro
A few months ago I was looking at Java code and thinking there must be a better way to see if this will really work, if there are any issues in how the program would react to exceptional conditions. I could write some unit tests or force some problems to see if the exception handling is correct and the log file output is really useful. But is there another way?
I got a solution after a while of thinking: Insert faults. Now how to do that in an easy and maintainable way? Not a new idea of course. After doing a web search I found some interesting references.
Funny, today at work I was extending an application and one of the support objects was failing. Turned out to be an index out of bounds problem. The method had adequate exception handling but, of course, an index out of bounds scenario was not tested. In this case the problem would not occur in production (hmmm), but it definitely occurred during development, and was not handled correctly.
Exception Handling
There are many references of correct exception handling strategies and best practices. However, when creating applications, the developer must still make critical creative decisions about how to handle an exception. That decision may not be a wise one. And even if it is, may later prove to have been very unwise since the system requirements and implementation may have changed due to maintenance or requirement evolution.
Testing Approaches
Using various testing methodologies a subsystem can be exhaustively tested for exception handling. Yet, most testing methods require that the subsystems be isolated in some manner. This, though very valuable, can give fatal false confidence that the system will behave in a predictable manner in responding to exceptional conditions.
The prime example of this is within the Unit Testing methodologies. Since Unit Tests are tests of isolated components, they do not test actual exception handling in the live system. A perfect component can still contribute to system failure when it is used incorrectly or uses other components incorrectly. For example, a unit test can show that class X’s methods correctly handle and when necessary throw the perfect exception. That means nothing if components that interact with X don’t correctly use those exceptions: they swallow, mask, or mistakenly catch them when there is nothing they can do.
Thus, one needs to use Integration Testing to test assemblages of components. But, we are still back with the same shortcoming as with Unit Testing. Thus we would next need various forms of system testing. And, that system testing must be a White Box test. A Black Box test wherein an interface is exercised with various inputs (such as in web test systems or by QA personnel), will not necessarily invoke the full internal API/SPI between components that could be part of a complicated call chain. Why? Because, the interface, such as a browser client, will have (we hope) the validation, security, and logging layers implemented so that by the time a system test workflow enters the deep system components within the client or middleware, there is no way to influence the target component state into programmatic intentional failure mode.
If there is no way to exercise a run time trajectory, then why bother? The reason is that exceptional conditions are exceptional. Resources can be exhausted, systems not available, and so forth. The rule of Fail Fast may not be enough if collaborating components are not exceptionally responsive.
Fault Injection
To see if a system is responding to exceptional conditions, one must wait for those conditions or create them. Analogously to Fuzz Testing we can dynamically and randomly insert faults. In Fuzz Testing, inputs are manipulated to a system under test. In Fault Injection we have to manipulate inputs inside the real system, within the components themselves. And as in Fuzz Testing, we can employ various strategies to do so, such as randomized faults, etc. Of course, this system is not the deployed production system, it is the real system in a test environment, an internal full deployment.
Some approaches are: source modification, source macros, annotations, dependency injection (IOC), and Aspect Oriented Programming (AOP). Of course, there are many other techniques as found in open/closed projects, commercial products, and the research community.
Fault Injection using AOP
Since using the real system is required, a viable approach is to use the Instrumentation already available in the Java system. We could dynamically insert code that forces exceptions. AOP as implemented by the AspectJ language can already do this.
The major advantage of using AOP is that the source is not changed in any way. AspectJ is the state of the art in the Java ecosystem for using AOP.
Other approaches
Monitoring
BTrace (a Java oriented approach comparable to DTrace) is very interesting. However, since, like DTrace, it is meant to be safe, defect injection may not be doable. But, see this post on an unsafe mode for BTrace.
Source modification
One way of injecting failure is just to edit the source code and add strategic code that will cause the instigating failure. An advanced form of this is Mutation Testing. Code changes could be: arguments with wrong values, nulls, deliberately thrown exceptions, etc. For example, one can simply create a method that throws a runtime exception:
*** DON'T DO THIS ***
public static void REMOVE_XXX_FROM_XXX_SOURCE(){
if(true){
throw new NullPointerException(
"\n\t****** DELIBERATE RUNTIME EXCEPTION*****\n");
}
}
Then insert invocations to this method at strategic points in the code, deploy to a test environment, and see if the expected results are obtained, such as logging output that can identify the cause, or that no side effects are recorded, such as incorrect data storage.
This is a low tech and error prone approach. Most importantly, this is dangerous. In the mad development rush to meet deadlines and also eat lunch, this code could make it into production! Even with that horrific method name, it will wind up in production! One would have to create deployment filters that stop the process if any “fault” inducing code is included. Another reason why this is not a good approach is that it doesn’t scale very well. Forcing exceptions is just one type of verification. Another is setting values outside of expected ranges or states. Thus, one would need many different kinds of source code insertions.
Of course, Unit and Behavior based testing tools sets can supply these required verifications if included into the system as Built-In Self-Tests (BIST).
Built-In Self-Test
In the hardware realm, BIST as found in, for example, IEEE 1149.1 JTAG, has been very successful. On the software side, there is ongoing research on how to implement BIST-like capability. This would make Brad Cox’s concept of the “Software IC” even more powerful.
Macros
A somewhat viable approach to source code fault insertion is instead of inserting the faulting code, insert “include” macros in the original source code that indicates what fault insertion should be done at a location. A fault injection preprocessor can scan and insert the fault framework’s invocations to accomplish the requirement. The source code build process can then simply enable or disable the use of the preprocessor. This would however still require source code modification and an extra maintenance nightmare. When the code changes the macros may also require change.
Annotations
Instead of macros, we could also use Annotations. Annotations could explicitly state the runtime exceptional behavior “service level agreements” at the component level. These SLA could then be systematically manipulated to test if they really hold at the system level.
Dependency Injection
One can also inject exceptional behavior by dependency injection, using programmatic, declarative, or annotations, faulted components could be inserted (or created via Aspect Oriented Programming) into the target system.
AOP Demonstration
In listing one below, an ATM class uses a service to do some banking.
/**
* Driver class for example.
* For a real system see: https://bitbucket.org/aragost/javahg/
* @author jbetancourt
*/
public class ATM {
private BankService service = new BankService(1000);
/**
* Perform a banking action.
*/
public void transaction(){
{
service.deposit(100L);
service.withdraw(200L);
}
service.statement();
}
/** Application entry point */
public static void main(String[] args) {
new ATM().transaction();
}
}
Listing two is the BankService being used. Of course, a real service would be more complex.
/**
* Example service.
*/
public class BankService {
// Use Long to allow example's null for failure
// injection.
private BigDecimal balance;
public BankService(long savings){
this.balance = new BigDecimal(savings);
}
/** */
public void deposit(Long amount){
balance = balance.add(new BigDecimal(amount));
System.out.println("Deposit: " + amount);
}
/** */
public void withdraw(Long amount){
balance = balance.subtract(new BigDecimal(amount));
System.out.println("Withdraw: " + amount);
}
/** */
public void statement() {
System.out.println("Balance: " + balance);
}
} // end BankService
Report: pointcut usage
withdrawCut: true -> true
depositCut: false -> false
Deposit: 100
Exception in thread "main" java.lang.NullPointerException
at BankService.withdraw_aroundBody2(BankService.java:15
at BankService.withdraw_aroundBody3$advice(BankService.java:81)
at BankService.withdraw(BankService.java:1)
at Main.main(Main.java:16)
Exception at the deposit method.
Report:
pointcut usage
withdrawCut: true -> false
depositCut: false -> true
Exception in thread "main" java.lang.NullPointerException
at BankService.deposit_aroundBody0(BankService.java:9)
at BankService.deposit_aroundBody1$advice(BankService.java:81)
at BankService.deposit(BankService.java:1)
at Main.main(Main.java:15)
All pointcuts enabled.
Report:
pointcut usagewithdrawCut: true -> true
depositCut: false -> true
Exception in thread "main" java.lang.NullPointerException
at BankService.deposit_aroundBody0(BankService.java:9)
at BankService.deposit_aroundBody1$advice(BankService.java:81)
at BankService.deposit(BankService.java:1)
at Main.main(Main.java:15)
Implementation
Now to test the exceptional behavior we want to fault the deposit and withdraw methods.
First we have a way of specifying what we want to fault by using a JSON configuration file:
The “cuts” settings indicate which AspectJ “pointcut” to turn on. The “random” setting indicates if we want the exceptions to be randomly inserted into the code base at the enabled pointcuts.
The Abstract failure injection aspect. Subclasses (aspects) will supply the actual pointcuts to specify where to ‘do’ the injection.
/**
* FailureInjectionAspect.aj
* @author jbetancourt
*
*/
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
/**
* Abstract Aspect that determines if failure should occur.
* @author jbetancourt
*
*/
public abstract aspect FailureInjectionAspect {
private Map<String, Boolean> pointcutFlags = new HashMap<String, Boolean>();
private volatile boolean initialized =false;
/**
* Initialize Failure injection aspect.
* @throws Exception
*
*/
protected void init() throws Exception {
Config config = new Config();
config.configure();
pointcutFlags = config.getPointcutFlags();
initialized = true;
}
/**
* Get boolean value for pointcut name.
*
* @param name pointcut name.
* @return true if pointcut enabled.
* @throws IOException
*/
protected boolean isPointcutEnabled(String name) {
if(!initialized){
try {
init();
} catch (Exception e) {
throw new IllegalStateException(
"Could not initialize object",e);
}
}
boolean f = false;
Object val = pointcutFlags.get(name);
if (null != val) {
f = ((Boolean) val).booleanValue();
}
return f;
}
} // end FailureInjectionAspect.aj
Here is a aspect that uses a nulling type of injection.
/**
*
* Example of an aspect for nulling an argument to a method.
* @author jbetancourt
*
*/
public aspect AmountFailureAspect extends FailureInjectionAspect {
/** fault the deposit */
private pointcut depositCut(Long amount) :
execution(public void BankService.deposit(Long))
&& if(AmountFailureAspect.aspectOf().isPointcutEnabled("depositCut"))
&& args(amount)
&& !within(FailureInjectionAspect)
;
/** fault the withdrawal */
private pointcut withdrawCut(Long amount) :
execution(public void BankService.withdraw(Long))
&& if(AmountFailureAspect.aspectOf().isPointcutEnabled("withdrawCut"))
&& args(amount)
&& !within(FailureInjectionAspect)
;
/** Null the amount arg */
void around(Long amount) : depositCut(amount)|| withdrawCut(amount) {
amount = null;
proceed(amount);
}
}
Here is how to read the configuation JSON file. I use the JSON-Simple library.
Using AspectJ is much easier in a supporting IDE like Eclipse. Below is the “mess”, unless you love the CLI, of compiling and running in a command shell. Could be made clean by creating aliases, scripts, etc.
cd src
src>javac -d ..\bin -cp ..\jars\json-simple-1.1.jar *.java
src>java -cp ..\bin Main
Jun 9, 2011 3:18:24 PM Main main
INFO: Starting Main.....
Deposit: 100
Withdraw: 200
Balance: 900
Now we change the flag from false to true in the runtimeInjection.json file:
Compile the aspects using the Aspectj “ajc” compiler. Here we use 1.6 compliance level; destination of output to ..\bin folder; and give the classpath.
Now we run the same Main program. Since the pointcut flag is true, the advise is invoked and the list argument to the service(List) is set to null.
c:\Users\jbetancourt\Documents\projects\dev\AspectsForNullTesting>java -cp "c:\java\aspectj1.6\lib\aspectjrt.jar;c:\java\aspectj1.6\lib\aspectjtools
.jar;c:\java\aspectj1.6\lib\aspectjweaver.jar;jars\json-simple-1.1.jar;bin;." Main
Jun 9, 2011 3:25:04 PM Main main
INFO: Starting Main.....
Pointcut usage
Name Prior Current
withdrawCut: true -> false
depositCut: false -> false
c:\Users\jbetancourt\Documents\projects\dev\AspectsForNullTesting>java -cp "c:\java\aspectj1.6\lib\aspectjrt.jar;c:\java\aspectj1.6\lib\aspectjtools
.jar;c:\java\aspectj1.6\lib\aspectjweaver.jar;jars\json-simple-1.1.jar;bin;." Main
Jun 9, 2011 3:25:10 PM Main main
INFO: Starting Main.....
Pointcut usage
Name Prior Current
withdrawCut: true -> true
depositCut: false -> true
Jun 9, 2011 3:25:10 PM Config exec
INFO: Initialized: true
Exception in thread "main" java.lang.NullPointerException
at BankService.deposit_aroundBody0(BankService.java:19)
at BankService.deposit_aroundBody1$advice(BankService.java:26)
at BankService.deposit(BankService.java:1)
at Main.main(Main.java:16)
c:\Users\jbetancourt\Documents\projects\dev\AspectsForNullTesting>java -cp "c:\java\aspectj1.6\lib\aspectjrt.jar;c:\java\aspectj1.6\lib\aspectjtools
.jar;c:\java\aspectj1.6\lib\aspectjweaver.jar;jars\json-simple-1.1.jar;bin;." Main
Jun 9, 2011 3:25:13 PM Main main
INFO: Starting Main.....
Pointcut usage
Name Prior Current
withdrawCut: true -> true
depositCut: false -> false
Jun 9, 2011 3:25:13 PM Config exec
INFO: Initialized: true
Deposit: 100
Exception in thread "main" java.lang.NullPointerException
at BankService.withdraw_aroundBody2(BankService.java:25)
at BankService.withdraw_aroundBody3$advice(BankService.java:26)
at BankService.withdraw(BankService.java:1)
at Main.main(Main.java:17)
Hero waits. Villain will come, maniacally cackle and talk. Hero can loosen bonds and save the world. Villain comes in, shoots the hero.------4 months ago
Posted by josefB 
