Test Coverage Using JMockit

January 21, 2012

The JMockit Unit Testing library continues to astound. One new thing I discovered is its Coverage reporting.

Code Coverage
Code coverage is simply a measurement of what code has been actually run when tests are executed. There are many such measures, ramifications, and tools. Like testing itself, code coverage measurement is probably not done enough, or misused.

“Code coverage tells you what you definitely haven’t tested, not what you have.” — Mark Simpson in comment

Path Coverage
Plenty of coverage reporting tools out there. What this one also includes is Path coverage. This is different then branch coverage. Paths are possible execution paths from entry points to exit points. If you visualize a methods statements in a directed graph, paths are a enumeration of the possible edges traversed when that method is invoked. So, Path coverages is inclusive of Branch coverage. Well, I’m not a testing expert, so this may be way off.

Very surprising results. For example, you run a coverage report with a tool such as Cobertura or Emma and feel very happy that you exercised every line and branch with your tests. Then you run the same tests but use JMockit Coverage and discover your tests didn’t cover all the paths! Not only that your line coverage wasn’t so great either.

JMockit explicitly gives you a report showing:

Measures how many of the possible execution paths through method/constructor bodies were actually executed by tests.
The percentages are calculated as 100*NPE/NP, where NP is the number of possible paths and NPE the number of fully executed paths.

Measures how much of the executable production code was exercised by tests. An executable line of code contains one or more executable segments.
The percentages are calculated as 100*NE/NS, where NS is the number of segments and NE the number of executed segments.

Measures how many of the instance and static non-final fields were fully exercised by the test run. To be fully exercised, a field must have the last value assigned to it read by at least one test. The percentages are calculated as 100*NFE/NF, where NF is the number of non-final fields and NFE the number of fully exercised fields.

– from the JMockit coverage report HTML page

Other information is found by using the full HTML output option.

A sample JMockit coverage report is here. Of course you can drill down into various parts of the html page. Like when you click on an exercised line you will get a list of what invoked that line.

Worth it?
Are the metrics such as Path coverage that this tool generates accurate? Is JMockit coverage a replacement for other tools such as Cobertura? I don’t know. For most projects, the resources would probably make the use of coverages generated by multiple tools prohibitive.

One possible approach to evaluating coverage tools is to just use actual real results of the target application. Use the list of bugs and correlate to a coverage tool report. Where were the bugs? Which tool gave the least measure for this location?

Further Reading

Eliades Ochoa – Siboney

Use JMockit to Unit test logging output

December 18, 2011

A very simple method of unit testing output logging is presented using a state-based mocking approach and JMockit toolkit.

This post copied to new blog: Use JMockit to Unit test logging output
Java language. Logging frameworks such as Log4j, commons-logging, java.util.logging, and SLF4J. Application logs.

Unit test logging output? Isn’t that going overboard? Most likely. However, there can be some reasons why in certain parts of a code base you’d better:

  1. Audit requirements
  2. Logging adheres to standards such as Common Event Expression (CEE) language
  3. Ability to maintain a system
  4. Log maintenance tools get correct data
  5. Reduce technical debt
  6. SIEM

In listing 1, a simple class uses the java.util.Logger to log. We want to make sure this class will always log this in the future, i.e., a regression test. In order to qualify as a unit test, the system under test (SUT) should be isolated. Thus, parsing an actual logging output file would not be optimal.

Listing 1

import java.util.logging.Level;
import java.util.logging.Logger;

/**  Example class that logs. */
public class Service {
	Logger logger = Logger.getLogger(this.getClass().getName());
	public void serve(){		
		logger.log(Level.INFO,"Hello world!");

In listing 2 we use JMockit to mock the java.util.Logger. This should also work for other logging frameworks. JMockit has two approaches for applying mock techniques: Behavior-based and state-based.

We apply state-based below (just cause that is the one I’m starting to get the hang of). The Arrange, Act, Assert (AAA) pattern is still used, but the Assert step is in the mock object. We apply a simple ‘equals’ test. Of course, based on what we expect in the log message, a regex may be more useful.

The above technique just tests that the log parameters are correct. This doesn’t check that the log output itself is correct. That is a different concern. Since the actual output logging is controlled by various configuration options, a unit test may not make sense. Would that be a functional test?

Listing 2

import static org.hamcrest.CoreMatchers.*;
import static org.junit.Assert.*;

import java.util.logging.Level;
import java.util.logging.Logger;

import mockit.Mock;
import mockit.MockUp;
import mockit.Mockit;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

 * JUnit test for Service class.
 * @author jbetancourt
public class ServiceTest {
	private Service service;

	public void setUp() throws Exception {
		service = new Service();
	public void tearDown() throws Exception {

	public final void we_are_logging_correctly() {	
           // Arrange

           // Act

	private void mockLogger() {

		new MockUp<Logger>() {
			public void log(Level level, String msg) {
				assertThat("Hello world!", is(equalTo(msg)));
				assertThat(level, is(equalTo(Level.INFO)));

Of course, just checking that the message sent to the logger may not be enough. In this case a possible approach is to hook into the logging library to capture the final log output. In the java.util.logging API, one can add a new stream Handler to the Logger instance being used.

In listing 3 below a simple attempt is made to use a stream handler to capture the actual logger output used by java.util.logging.

Listing 3

public class ServiceTest{
  Logger logger = Logger.getLogger(ServiceTest.class.getName());
  private OutputStream logOut;
  private StreamHandler testLogHandler;

  public void setUp() throws Exception {

  /** */
  public final void exception_log_has_all_info(){
    logger.log(Level.WARNING, "Hello world!");
    String captured = logOut.toString();
    Assert.assertTrue(captured.contains("Hello world!");

   Add stream handler to logger.  
   Will take more effort then this, e.g., may not have parent handler.
  protected void setUpLogHandler(Logger logger) {
    logOut = new ByteArrayOutputStream();
    Handler[] handlers = logger.getParent().getHandlers();
    testLogHandler = new StreamHandler(logOut, handlers[0].getFormatter());

* JUnit: 4.*
* JMockit: 0.999.11
* JDK: 1.6*
* Eclipse: IDE 1.7
* Git: 1.76.msysgit.0

Shown was a little technique that may come in handy one day. Though presented in the context of logging, it is really a simple application of state-based mock use.

Further reading

Carlos Santana/Mahavishnu John McLaughlin – The Life Divine

Unit Testing what will never happen?

December 4, 2011

Some developers refuse or are slow to test units for certain values or situations because, they claim, those values or situations will not occur.

Some reasons given are that the presentation tier or UI will prevent certain values; that other modules in the call chain already have error handling and validation. So why test something that won’t happen?

Let’s take an example. In the method/function below, the type will never be blank or null. Looking at the actual source code for the application will show this. So should a unit test be written that will invoke this method with a null, “”, or ” “?

public boolean service(final String type){
    // do stuff
    return result;

1. Things change.
2. Bad stuff happens.
3. Development process will use invalid values.
4. More complete testing. Regression testing, path coverage, etc.
5. The method could now be used in a different call chain.
6. Its a public method, anything can invoke it, even a business partner via some remoting technology.
7. When invoked as part of other unit tests, it could have invalid values.

#3 is, I think, the most important. Did you ever do development and something not work and it turn out to be existing code that did not handle arguments correctly? That is wasted time and an aggravation. Sure, in the current production code, a blank string won’t be used, but during development, especially TDD, you scaffold code. Sometimes you don’t have values yet, so you just use a blank string.

Just the other day I tested a deployed production method that correctly tested the argument for an empty string, “”, and did the right thing. However, the code did not check for a blank string, ” “, and throws an exception. Unit testing would have shown this.

And, yes, this will never happen in our application. :)

Ok, you still don’t want to test something for null or invalid values? At least put this in writing in the Javadoc: “I (insert your name and address) am a great developer and stake my professional career that this method will never be invoked with invalid values.” The address is needed so us mediocre developers can hunt you down when the server crashes.

Further Reading

  1. Duplicate blog post: Unit testing what will never happen
  2. FindBugs and JSR-305

Off topic, some music …
” If ” – Oregon (2009)

Testing getter/setter using JUnit

November 19, 2011

So, I was thinking of testing my getter and setters in a JUnit test. Yea, I know it is not recommended, but I thought that I could write a simple reflective iterator that could do it without much fuss.

What I show below is that maybe getter/setters should be tested, especially in Java which really does not have “real” properties. Further, by investigating a potential solution, I show that it doesn’t work, and the reason is that testing IS hard to do.

Anyway, just to save time, and not have to code this, I did a quick search and found that this has been done many times before. So I looked at one of the solutions, Scott’s. Very nicely done! With one line you can test a class’s property getters and setters. Then I noticed a problem.

In his approach he invokes the setter and then the getter, finally comparing that the values are the same. For example, if the property is a boolean, the test class will set it true, then read it and the value better be true. However, what if the value prior to your set was true, the object is initialized with true for the field value?

For example, I took a class and set field x to true via the default constructor, then I modified the setX method to not set the field value.

public class Fubar {
  private boolean launch = true;
  public void getLaunch(){ 
      return this.launch;

  public void setLaunch(boolean launch){  
      /* this.launch = launch; */ // broken!

Code at: git clone git://gist.github.com/1408493.git gist-1408493

In the unit test the getLaunch method still returned true. The getter/setter test did not fail; the test was bogus. Not only that, the fact that the setLaunch method did not work illustrates that sometimes testing setters is warranted.

Thus, the version of the program that controls the sports strategy will ship and the boolean that aborts a play cannot be set to false! (Edited, removed joke about war stuff; you can’t be too careful with all the craziness in the world).

In a recent presentation I gave on unit testing, I said that testing could be difficult. This is a great example of this. In this case, one should have followed analogous patterns in other fields. Thus, from semiconductor testing, for example, you would test gates by not just writing a 1 bit, you have to write 1 and 0 to make sure the gate is working.

Feb 11, 2012: I’m using Scott’s solution in our test suite. I did modify the code a bit. Turns out that a JavaBean property is not so clear cut. If you have a getter and setter method pair but no actual target field, are those Javabean methods? Looks like the Java bean introspector will report these as bean accessors. Hmm.

Further Reading

  1. Behavior counters for improved JUnit tests
  2. Duplicate of this blog post: Testing getter/setter using JUnit
  3. Should unit tests be written for getter and setters?
  4. Is there a Java unit-test framework that auto-tests getters and setters?
  5. Do You Unit Test Getters and Setters?
  6. Google search

Bill Evans Trio – Nardis – 19 Mar 65 (7 of 11)

Article on Bill Evans: http://www.chuckisraels.com/articleevans.htm


November 19, 2011

Yesterday at work I gave a presentation on Unit Testing. It went well. 160 slides! And, no one passed out and hit the floor.

One thing I mentioned was mocking frameworks and how JMockit is very useful. Perhaps JMockIt represents the state of the art in Java based Mocking tools.

There are plenty of good reasons for using mocks:

“JMockit allows developers to write unit/integration tests without the testability issues typically found with other mocking APIs. Tests can easily be written that will mock final classes, static methods, constructors, and so on. There are no limitations.” — JMockit

JMockit is ” a collection of tools and APIs for use in developer testing, that is, tests written by developers using a testing framework such as JUnit or TestNG.”

I’ve used it for some tests. Since it uses Java instrumentation it can mock almost anything, especially those legacy untestable great OO classes. Best of all it has a very good tutorial.

The only ‘negative’, so far, is that JMockit does not, afaik, have many developers working on the project. That could also be a plus, of course.

Another mock tool is PowerMock.

Seems to me there are too many mock frameworks and they do pretty much the same things. Time for consolidation so that an API and a body of practice can shake out?

Further reading

    Duplicate of this post on new blog: JMockit

  1. Mock object
  2. MockingToolkitComparisonMatrix
  3. Beyond EasyMock and JMock, try JMockIt !
  4. The Difference Between Mocks and Stubs
  5. The JMockit Testing Toolkit
  6. The Concept of Mocking
  7. PowerMock
  8. Unit Testing Using Mocks – Testing Techniques 5
  9. Making a mockery of CQ5 with JMockit

Off topic, some music …

Stefano Cantini – Blowin in the wind

jsUnit based tests won’t run in other folders?

January 16, 2011

This is why software is such a sometimes frustrating undertaking. I once had a jsUnit based test working. It ran fine in FireFox. Today it did not. Funny thing is that the existing tests that come with jsUnit still work. Other browsers fail too. (maybe).

It had to be a typo, bad url, or something. The jsUnit docs say that the problem is that I am not correctly giving the path to the jsUnitCore.js file. I tried everything. But, now I try to get smart. First copy the tests that are working to another folder. I copy failingTest.html to /temp/jsunit, and I also copy jsUnitCore.js there too.
Still doesn’t work.

I edit jsUnitCore.js, and add an alert to show that it is being executed or loaded. To make sure I really did it, I diff:

C:\temp\jsunit>diff failingTest.html \java\jsunit\tests\failingTest.html

<     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<     <script type="text/javascript" src="./jsUnitCore.js"></script>
>     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
>     <link rel="stylesheet" type="text/css" href="../css/jsUnitStyle.css">
>     <script type="text/javascript" src="../app/jsUnitCore.js"></script>

C:\temp\jsunit>diff jsUnitCore.js \java\jsunit\app\jsUnitCore.js

< alert("Maybe a career in sanitation?");

Try the test and I see the message. So, I know that the library is being found and is running. That is weird. Must be the browser. I empty the cache. still nothing. Arrrrr.

Google? I find a reference to jsUnit forum about someone having a similar problem. No advice, but one person gives a url for the solution. Is it a good url, spam, or worse? Nope its legit, and real advice. It is the browser. Thank you Andrew! I should have checked the forum first.

Browsers, bah! I hope they don’t start showing up in embedded systems. That’s probably how WWIII will get started, some Javascript mistyping.

17JAN11: Spoke too soon. FireFox ver 4.0b9 is showing the problem again. I’ll try in my Ubuntu instance too.

JSUnit 2.2 on Firefox 3.07 – FTW


Java Dev Using Embedded Groovy Console in Eclipse

December 12, 2010

In development, simple but powerful tools to get at the essence of a code source in order to understand, test, and extend it is essential. This is even more important in Test Driven Development (TDD). Eclipse’s Java Debugger is for most situations, powerful enough. Eclipse has an Expressions View available in the debug perspective to execute snippets of code. The Java Development Toolkit (JDT) also has a Scrapbook facility that allows the creation, storage, and running of experimental code. And, of course, all the other features of the debugger are excellent.

However, when you need it, it’s possible to embed a script engine and have new ways of analyzing and developing code. In listing 1, an app shows the use of the ConsoleWaiter class. When the code executes the waiter.run() at line 43, it opens the Groovy Console which allows the use Groovy shell scripting in a GUI frame, see figure 1. When the console is closed the app continues executing.

Listing 1
* File: ExampleApp2.java
* @author jbetancourt
* Date: 20101213T1718-5
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

* @author jbetancourt
public class ExampleApp2 {
   static public String greeting = "Hello world!";
   private static final List<String> nodes;

   static {
        nodes = new

   private String title = "Example 2";

   public String getTitle(){
      return title;

   * @param args command line args
   public static void main(String[] args) {
      ExampleApp2 app = new ExampleApp2();

      ConsoleWaiter waiter = new ConsoleWaiter(app);

      waiter.setVar("greet", greeting);
      waiter.setVar("nodes", nodes);
      waiter.setVar("title", app.getTitle());

screen capture of Groovy console

Console screen capture, click to view

Another screen capture, click to view

This is awesome. In one project I had to examine the contents of a Properties object. Did it have an “email” value? I was stumped when using the Eclipse debugger, it did not show all entries in the Map, at the end was “…”. Sure, I could use the Expressions window, but with the Console I could not only do a get(key), but iterate using Groovy style closures and much more.

The magic that enables this is the ConsoleWaiter.groovy class shown below in listing 2 that was written by John Green. Since a Groovy script is a Java class underneath, in Eclipse you can call Groovy from Java easily (some compiler magic).

Listing 2
 * File:  ConsoleWaiter.groovy

import groovy.lang.Binding;
import groovy.ui.Console;

 * Provides a wrapper for the console.
 * Based on source by John Green
 * Adapted from:  http://www.oehive.org/files/ConsoleWaiter.groovy
 * Released under the Eclipse Public License
 * http://www.eclipse.org/legal/epl-v10.html
 * I added methods to allow use from Java.
 * The run() method launches the console and causes this thread
 * to sleep until the console's window is closed.
 * Allows easy interaction with the objects alive at a given
 * point in an application's execution, like in a debugger
 * session.
 * Example 1:
<pre> * new ConsoleWaiter().run()

 * Example 2:
<pre> * def waiter = new ConsoleWaiter()
 * waiter.console.setVariable("node", node)
 * waiter.run()
class ConsoleWaiter {
 Console console
 Object source
 boolean done = false;

 /** */
 public ConsoleWaiter(Console inConsole){
    this.console = inConsole

 /** */
 public ConsoleWaiter(Object source){
    console =
    new Console(getClass().classLoader,
    new Binding())
    this.source = source
    console.setVariable("source", source)

 /** */
 public void setVar(String key, Object value){
    console.setVariable(key, value)

 /** 	 */
 public void setVar(String key, List values){
    console.setVariable(key, values)

 /** 	 */
 public void setVar(String key, Object[] values){
    console.setVariable(key, values)

 /** 	 */
 public void run() {
    // I'm a little surprised that this exit() can be private.
    console.frame.windowClosing = this.&exit
    console.frame.windowClosed = this.&exit
    while (!done) {
       sleep 1000

 /** 	 */
 public boolean isDone(){
    return done;

 /** 	 */
 public void exit(EventObject evt = null) {
    done = true

 /** 	 */
 public Console getConsole(){
    return console;

Eclipse Integration

The disadvantage of this approach is that you have to put extraneous code inside the tests or target class. Not only is this tedious and slow, what if code is accidentally deployed like this? A better approach is to just set a breakpoint in the code, and then have the ability to open a script console at that breakpoint, in the Java Stack Frame, that has access to the execution context and classpath. Is there an Eclipse add-in that does this? If not, there should be.


Shown was a simple example of embedding a Groovy console in Java code to allow scripting. Of course, this is not a new idea. It was even mentioned in an older JVM scripting language, Beanshell. Note that it is possible to instead of using a GUI console, to use the Groovy shell, InteractiveShell class. In the reading list below this approach is taken to allow remote scripting of a server hosted application.


  • Oct 10, 2011:
    Interesting tool that could be relevant: YouDebug.
  • March 20, 2012: Not exactly same scenario, but the concept of an embedded script console is found in many products. Jenkins CI Server has one and it uses Groovy. Jenkins Script Console

Further reading

Use Groovy to find missing lines in a file

December 5, 2010

In a project at work I had to compare two files and generate a list of missing lines so that a spreadsheet could be created. Sounds easy and probably is. Example, file A has 1,2,4. File B has 1,2. So File B is missing 4.

So, if args[] has the file paths, a simple approach is:

MissingLines.groovy script

// MissingLines.groovy

def firstList = new File(args[0]).readLines()

new File(args[1]).readLines().each {

That was easy! :) Testing it is not so easy. :(

Alternative approach using Map

// MissingLines.groovy  using Map

def inFirst = [:]

new File(args[0]).eachLine{
	inFirst.put(it, null)

new File(args[1]).eachLine {

Still easy …

The script above compares two files and prints the lines that appear in the first but not in the second. It uses a simple string compare. To find out whats missing in each file a better recourse is to use a real file diffing tool. This script reads one of the files into memory, so with very large files may balk. How to test a Groovy script?

From a Computer Science point of view this script is suboptimal of course, all of those calls to contains() must be bad. But, this was a script to solve a single task, so a throwaway.

For some reasons using the available tools I could not do it. I tried different file comparison tools, even KDiff3, diff, some editors, and simple command line utilities. Maybe I was having a bad hair day. It’s not that they won’t show the missing lines, they gave no way to reuse that info. Hint to tool builders, allow the user to re-purpose the result.

Yes, I know this can be done in one line on Linux using the incredible Linux utilities and shell, or a Perl script so tiny it can be etched using atoms. And, since I’m running on Windows, can also be done using Cygwin. I searched and saw how. I’m sure I could have figured it out too, using sort, diff, and all that.

So this post is on how to write a simple script that does this using the Groovy JVM based language and how to test it with JUnit. The testing part for a learning exercise; I really used the compare script. But, why is it so hard to do with common tools, oh well. Groovy offers an easy to use syntax and for Java developers, a comfortable super set of standard Java.

Testing it

Below is a test of this script. Note that this is just a JUnit 3 test. JUnit is embedded in Groovy (that is cool). As I wrote above, this simple script does not require a test class. But, it gave me the opportunity to figure out how to test a Groovy script using JUnit. This illustrates:

  • how to test a simple script
  • how to provide command line args
  • how to capture and test the console output.
  • use of the default Eclipse test invocation project location.
Test data
a.txt             b.txt 
-----             ----- 
Alfa              India
Bravo             Juliet
Charlie           Kilo
Delta             Lima
Echo              Mike
Foxtrot           November
Golf              Oscar
Hotel             Papa
Romeo             Quebec

MissingLinesTest.groovy class

// MissingLinesTest.groovy
import java.io.ByteArrayOutputStream;
import java.io.PrintStream;
import groovy.util.GroovyTestCase;
import groovy.lang.GroovyShell;

 * Unit Test the MissingLines.groovy script.
 * Also illustrates techniques
 * 1.  Capturing console out.
 * 2.  Accessing test data via Eclipse's project folder.
 * 3.  Supplying command line args to script.
 * The project structure is:
 * .
 * +---.hg
 * +---.settings
 * +---bin
 * +---docs 
 * +---data
 * \---src
class MissingLinesTest extends GroovyTestCase {
	def basedir = new java.io.File(".").getAbsolutePath()
	def originalOut // console output
	def ByteArrayOutputStream captured // 
	def script = new GroovyShell() //

	protected void setUp() throws Exception {
		originalOut = System.out
		captured = new ByteArrayOutputStream()
		System.setOut(new PrintStream(captured)) // can fail
	/** first test */
	public final void testAB(){		
	/** Second test */
	public final void testBA(){		
	/**  */
	public final void testNothingFellOut(){
		// todo: combining missing from each should give total lines from both files.
	/** avoids repeating test code */
	def callTest(String file1Path, String file2Path, List expected){
		script.run(new File(buildPath("src\\MissingLines.groovy")), [buildPath(file1Path), buildPath(file2Path)] )
		def list = captured.toString().readLines()
		def flag = list.containsAll(expected)
		assertTrue("Did not find missing strings",flag)
	private String buildPath(name){
		return basedir + "\\" + name		
	protected void tearDown() throws Exception {
		System.out = originalOut


Compare the size and complexity of the test class with the target test script. Could it have been smaller too?

Further reading

Use Eclipse project dir for Unit Test data

December 4, 2010

Use the default launch configuration of a unit test in Eclipse to access Unit Test data.

Best practice is to not access test data in relative or absolute paths in the file system. Instead to use classpath resources. But, maybe you have valid reasons to want to do things your own way. Thus, you want to get at test data in a folder within your project without monkeying around with adding file paths to launch configuration and all that stuff.

Ok, who am I to diminish your mojo. Just use the current directory that Eclipse creates. Example, you have this:

class FileCompareTest extends GroovyTestCase {
       /** 	 */
       public void testSomething(){
		def aFileName = 
           // use file in test.

Change it to:

class FileCompareTest extends GroovyTestCase{
       def basedir
       protected void setUp() throws Exception {
           basedir = new java.io.File(".").getAbsolutePath()
       /** 	 */
       public void testSomething(){			
           def aFileName = basedir + "\\data\\alpha.txt"


17JAN11: Could basedir = System.getProperty(“user.dir”); also be used?

Related links

How to default the working directory for JUnit launch configurations in Eclipse?


Get every new post delivered to your Inbox.