Category Archives: JUnit

The Art of Test Driven Development: Running unit tests conditionally

There comes a time where Java’s promise of “write once, run anywhere” breaks down for some of us. In my case and my Windows users, file paths present platform specific challenges. This article will show you how to run unit tests conditionally; only on Windows for example.

I talked about conditionally ignoring tests in JUnit in the past, but I’d like to revisit this topic in a little more detail.

I have an application that uses URIs for file references, but for most sane Windows users, a file reference is something like E:\some\folder\foo.txt, not file:///E:/some/folder/foo.txt. Is it ‘file’, a colon, and two slashes, or three, or four? Darn!

To make life easier for normal and power users, the application supports both Windows paths and URIs. The application converts Windows file paths to URIs internally and everyone’s happy. How do you unit test this? Just as always, you say. The trick comes when your build runs on a farm of Windows and Linux boxes, and some of your carefully crafted tests pass on Windows but fail on Linux for obtuse reasons that are not the point of this write up. The bottom line is that I want to run certain tests on Windows only. Fortunately, JUnit provides a nice way to do this using its Assume class.

Here’s how you do it:

private static final boolean IS_WINDOWS =
  System.getProperty("os.name").startsWith("Windows");

@Test
public void testMeOnWindows() {
  Assume.assumeTrue(IS_WINDOWS);
  …
}

We first set a Boolean constant that indicates that we are running on Windows or not. To do that, we simply lookup the os.name system property and check that it starts with the String “Windows”.

The JUnit Assume class causes JUnit to ignore a test if the given condition fails. In our example, the test method will only proceed on Windows; but the test will not fail.

A failed assumption does not mean you have a bug or that the test failed, it means that JUnit should not continue running the test and not mark it as a failure. JUnit test runners will skip tests will failing assumptions. Your mileage may vary with other test runners.

Assume checks can be for any condition, giving you and your tests a lot of flexibility.

Under the hood

How does JUnit do this? When you call an Assume method like assumeTrue(), the following appens:

public static void assumeTrue(boolean b) {
  assumeThat(b, is(true));
}

JUnit calls the more general assumeThat() method with a Hamcrest matcher from org.hamcrest.CoreMatchers.is(T):

public static <T> void assumeThat(T actual, Matcher<T> matcher) {
  if (!matcher.matches(actual)) {
    throw new AssumptionViolatedException(actual, matcher);
  }
}

The expression matcher.matches(actual) uses Hamcrest to evaluate your condition and returns true if it passes. If your condition fails, the method assumeThat() throws an AssumptionViolatedException; which JUnit uses to implement assumptions. The JUnit guts catch these exceptions and handle them differently than other exceptions by adding the assumption violation to a list it uses in its reports.

There you have it, JUnit assumptions give your tests an extra level of flexibility by allowing you to run tests conditionally.

Happy Coding,
Gary Gregory

The Art of Test Driven Development: Logging JUnit Test Results

JUnit saves your test results to a file (using XML, plain, brief or failure formatters). If you use Maven, you can convert these files into an HTML report easily. However, that means waiting for a build to complete. If you want to see your test results while a Maven or Ant build is running, you must watch the command line.

In an IDE like Eclipse, you can watch the JUnit view to see tests progress. Wouldn’t it be nice to be able to do whatever you want with tests results on the fly?

What we’ll show you here as an alternative is logging your test results while JUnit runs your tests. This gives you all the power of Log4j to direct test results all over the place, from a simple console, to a database, a JMS queue, or any of the long list of appenders Log4j provides. You can provide your own Log4j appender as well for custom processing.

The basic mechanism that lets us do this is JUnit’s TestWatcher rule. Let’s introduce the TestWatcher with a simple example that logs test results to a StringBuilder. The example dumps the builder’s contents on the console when the test finishes.

package com.garygregory.log4j;

import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;

public class StringBuilderTestWatcherTest {
    private static final String EOL = 
        System.getProperty("line.separator");

    private static StringBuilder builder = new StringBuilder();

    @AfterClass
    public static void afterClass() {
        System.out.print(builder);
    }

    @Rule
    public TestWatcher watchman = new TestWatcher() {

        @Override
        protected void failed(Throwable e, Description description) {
            if (description != null) {
                builder.append(description);
            }
            if (e != null) {
                builder.append(' ');
                builder.append(e);
            }
            builder.append(" FAIL");
            builder.append(EOL);
        }

        @Override
        protected void succeeded(Description description) {
            if (description != null) {
                builder.append(description);
            }
            builder.append(" OK");
            builder.append(EOL);
        }
    };

    @Test
    public void testFails() {
        // test
        Assert.fail();
    }

    @Test
    public void testSucceeds() {
        // test
    }
}

In this example, the test class implements the TestWatcher rule directly. JUnit calls each method you annotate with @Test one after the next. JUnit calls the failed() method when a @Test method fails and gives it a description and an exception. JUnit calls the succeeded() method when a @Test method completes normally, no JUnit Assert calls failed, no unexpected exceptions.

The method afterClass() (annotated with JUnit’s @AfterClass) runs after all test methods have run and outputs the contents of the StringBuilder.

The console output is for the StringBuilder example is:

testSucceeds(com.garygregory.log4j.StringBuilderTestWatcherTest) OK
testFails(com.garygregory.log4j.StringBuilderTestWatcherTest) java.lang.AssertionError FAIL

It’s easy to imagine growing and morphing this StringBuilder implementation into all sorts of things: to write the results to a file, with your own format, or send results to a database, and so on.

Instead of going that way though, you should bridge to Log4j and reuse its powerful and rich feature set, which let’s you do all that and much more. The power of Log4j comes from the simplicity of its API combined with rich configurations you create in XML, JSON, or YAML. Log4j 2.4 also provides a programmatic configuration API, but that’s a separate topic.

To log our results to Log4j, we replace the StringBuilder with a Logger:

package com.garygregory.log4j;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;

public class Log4jTestWatcherTest {

    private static Logger logger = LogManager.getLogger();

    @Rule
    public TestWatcher watchman = new TestWatcher() {

        @Override
        protected void failed(Throwable e, Description description) {
            logger.error(description, e);
        }

        @Override
        protected void succeeded(Description description) {
            logger.info(description);
        }
    };

    @Test
    public void testFails() {
        // test
        Assert.fail();
    }

    @Test
    public void testSucceeds() {
        // test
    }
}

In this example, we still embed an implementation of TestWatcher in the test itself, but this time the methods call on its Logger. When a test fails, we log at the ERROR level, and when a test succeeds, we log at the INFO level.

The console output is for the Log4j example is:

2015-09-14 20:25:57,173 INFO [main][com.garygregory.log4j.Log4jTestWatcherTest] testSucceeds(com.garygregory.log4j.Log4jTestWatcherTest)
2015-09-14 20:25:57,175 ERROR [main][com.garygregory.log4j.Log4jTestWatcherTest] testFails(com.garygregory.log4j.Log4jTestWatcherTest)
java.lang.AssertionError
  at org.junit.Assert.fail(Assert.java:86)
  at org.junit.Assert.fail(Assert.java:95)
  at com.garygregory.log4j.Log4jTestWatcherTest.testFails(Log4jTestWatcherTest.java:32)

The next obvious step is to abstract this TestWatcher in a class:

package com.garygregory.log4j;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;

public class Log4jTestWatcher extends TestWatcher {

    private final Logger logger;

    public Log4jTestWatcher() {
        logger = LogManager.getLogger();
    }

    public Log4jTestWatcher(String loggerName) {
        logger = LogManager.getLogger(loggerName);
    }

    @Override
    protected void failed(Throwable e, Description description) {
        logger.error(description, e);
    }

    @Override
    protected void succeeded(Description description) {
        logger.info(description);
    }
}

And reuse it in the test:

package com.garygregory.log4j;

import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TestWatcher;

public class Log4jTestWatcherTest2 {

    @Rule
    public TestWatcher watchman = new Log4jTestWatcher();

    @Test
    public void testFails() {
        // test
        Assert.fail();
    }

    @Test
    public void testSucceeds() {
        // test
    }
}

This is short and sweet, compare it to our first listing!

Boom! All the power of Log4j and JUnit combined.

Happy Coding,
Gary Gregory

The Art of Test Driven Development: Per-Test Logging

Log4j logoAll applications complex enough have a logging subsystem. When writing tests, it is often useful to enable a specific logging configuration for a given test case. In this article, I will show how to use Apache Log4j and JUnit to achieve this goal.

Log4j is the granddaddy of logging framework and yes, I am partial to it as one of its developers and member of its Project Management Committee. I find it far superior, as we are about to release version 2.4, to all of its cousins. JUnit and the best practice of Test Driven Development should need no introduction; if you use neither, then I welcome you to the 21st century and its ‘extreme’ or ‘agile’ development practices.

When you are running a larger build with thousands of tests or even a small build with dozens, you do not want to set the logging level to debug or trace for the whole build. A small subsystem can end up generating a large amount of log events, leaving you with the task equivalent to finding a needle in a haystack in order to locate the few logging events relevant to your development session.

What I want, is to tell JUnit that a given test case should run with a specific Log4j configuration and then forget this configuration when the test finishes. To accomplish this, we will use a JUnit Rule and Log4j’s LoggerContextRule (This class is located in Log4j’s test jar. Log4j 2.3 named this class InitialContextRule).

Here is our test boilerplate:

public class MyTest {

  @Rule
  public LoggerContextRule ctx = 
    new LoggerContextRule("resources/MyConfig.xml");

  @Test
  public void testSomething() throws Exception {
    // testing…
  }

}

In JUnit, a @Rule annotates a field that points to a rule or a method that returns a rule. In our case, we store our rule in an instance variable, which must be public and non-static. A rule will run methods annotated with @Before, then @Test methods, and finally @After methods. JUnit throws an exception when any of these fail. The code for the rule itself runs before each @Before method. This means we get a fresh logging configuration for each @Test method.

The LoggerContextRule argument is a URI or path to a configuration file on your classpath.

If you want the rule to run only once for the whole test class, then use a class rule instead:

@ClassRule
public static LoggerContextRule ctx = 
  new LoggerContextRule("resources/MyConfig.xml");

Note that the variable is now static (and still public).

Whether you should use a rule or a class rule depends on how much isolation you want between test methods. In the case of logging, it should not matter and using a class rule should be preferred since your build be faster.

You can find the Log4j rule class in the Core’s test jar here.

Happy Coding,
Gary Gregory

Understanding JUnit method order execution (updated for version 4.11)

a-zI’m revisiting my post Understanding JUnit method order execution to discuss how to fix tests that depend on method order by using JUnit 4.11.

Let’s start with a rule: You should not create test methods that depend on the order in which they are executed.

If you do have such tests and they are failing randomly or rarely, this is why: JUnit relies on Java’s reflection API to get which test methods to execute. The problem is that the API does not define the order of the methods it returns. Your tests may work for a long time and then fail, apparently randomly. The things is, you’ve just been lucky all along and relying on the Java run-time giving you a consistent answer when it makes no such guarantee.

You may even see some very confusing behavior like a superclass’ test methods being mixed in with its subclass. I’ve seen this in Eclipse and Ant for example.

JUnit 4.11 provides a workarounds for this behavior with a new class-level annotation: FixMethodOrder(MethodSorters.NAME_ASCENDING). For example:

 @FixMethodOrder(MethodSorters.NAME_ASCENDING)
 public class MyTestCase { ... }
 

The annotation FixMethodOrder sorts the test methods in lexicographic method name order, and uses Method#toString() to break ties.

The recipe is: (1) Use @FixMethodOrder, (2) Fix your tests, (3) Remove @FixMethodOrder.

JUnit is available from GitHub.

JUnit 4.9 to the rescue with @ClassRule

In a previous article titled “JUnit @Rule not initialized before @BeforeClass“, I bemoaned the fact that an @Rule could not be set up to run before an @BeforeClass.

It appears my wish has been answered with the release of JUnit 4.9 and the new @ClassRule annotation.

The only change required to make our example work is to replace @Rule with @ClassRule:

package test;

import java.io.File;
import java.io.IOException;

import junit.framework.Assert;

import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;

public class TemporaryFolderRuleBeforeClass {
    @ClassRule
    public static TemporaryFolder testFolder = new TemporaryFolder();

    @BeforeClass
    public static void testInTempFolder() throws IOException {
        File tempFile = testFolder.newFile("file.txt");
        File tempFolder = testFolder.newFolder("folder");
        System.out.println("Test folder: " + testFolder.getRoot());
        // test
        Assert.assertNotNull(testFolder.getRoot());
    }

    @Test
    public void test() {
        // ...
    }
}

Voila! It just works, the @ClassRule is set up correctly before the @BeforeClass method is called. Thank you JUnit 4.9!

Java multi-threaded unit testing

With Java 5 and JUnit 4, writing multi-threaded unit tests has never been easier.

Let’s take a brain dead ID generator as our domain object example:

    /**
     * Generates sequential unique IDs starting with 1, 2, 3, and so on.
     * <p>
     * This class is NOT thread-safe.
     * </p>
     */
    static class BrokenUniqueIdGenerator {
        private long counter = 0;

        public long nextId() {
            return ++counter;
        }
    }

This is how you test this class with different threads loads of 1, 2, 4, 8, 16, and 32 threads. We use the Java java.util.concurrent API to manage threads to use our domain object concurrently. We use JUnit @Test methods to run the test and verify results.

    @Test
    public void test01() throws InterruptedException, ExecutionException {
        test(1);
    }

    @Test
    public void test02() throws InterruptedException, ExecutionException {
        test(2);
    }

    @Test
    public void test04() throws InterruptedException, ExecutionException {
        test(4);
    }

    @Test
    public void test08() throws InterruptedException, ExecutionException {
        test(8);
    }

    @Test
    public void test16() throws InterruptedException, ExecutionException {
        test(16);
    }

    @Test
    public void test32() throws InterruptedException, ExecutionException {
        test(32);
    }

    private void test(final int threadCount) throws InterruptedException, ExecutionException {
        final BrokenUniqueIdGenerator domainObject = new BrokenUniqueIdGenerator();
        Callable<Long> task = new Callable<Long>() {
            @Override
            public Long call() {
                return domainObject.nextId();
            }
        };
        List<Callable<Long>> tasks = Collections.nCopies(threadCount, task);
        ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
        List<Future<Long>> futures = executorService.invokeAll(tasks);
        List<Long> resultList = new ArrayList<Long>(futures.size());
        // Check for exceptions
        for (Future<Long> future : futures) {
            // Throws an exception if an exception was thrown by the task.
            resultList.add(future.get());
        }
        // Validate the IDs
        Assert.assertEquals(threadCount, futures.size());
        List<Long> expectedList = new ArrayList<Long>(threadCount);
        for (long i = 1; i <= threadCount; i++) {
            expectedList.add(i);
        }
        Collections.sort(resultList);
        Assert.assertEquals(expectedList, resultList);
    }

Let’s walk through the test(int threadCount) method. We start by creating our domain object:

final BrokenUniqueIdGenerator domainObject = new BrokenUniqueIdGenerator();

This class has one method, nextId, which we wrap into a task, an instance of Callable:

        Callable<Long> task = new Callable<Long>() {
            @Override
            public Long call() {
                return domainObject.nextId();
            }
        };

This is just a generic way to fit our API call in the Java concurrency API.

We then make copies of this task, one for each thread:

List<Callable<Long>> tasks = Collections.nCopies(threadCount, task);

Next, we create a thread pool, sized at least as big as the number of threads we want to test, in this case we use the exact given value threadCount.

ExecutorService executorService = Executors.newFixedThreadPool(threadCount);

And ask Java to run all the tasks concurrently using threads from the pool:

List<Future<Long>> futures = executorService.invokeAll(tasks);

The call to invokeAll blocks until all the threads are done. Each task is run on a thread, which invokes the tasks’ call method, which in turn calls our domain object API, nextId().

When you run this test case, it will sometimes pass and sometimes fail.

That’s multithreaded testing with Java 5 and JUnit 4. Voila!

BTW, the proper implementation is:

    /**
     * Generates sequential unique IDs starting with 1, 2, 3, and so on.
     * <p>
     * This class is thread-safe.
     * </p>
     */
    static class UniqueIdGenerator {
        private final AtomicLong counter = new AtomicLong();

        public long nextId() {
            return counter.incrementAndGet();
        }
    }

The full listing is:

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.atomic.AtomicLong;

import org.junit.Assert;
import org.junit.Test;

public class MultiThreadedTestCase {

    /**
     * Generates sequential unique IDs starting with 1, 2, 3, and so on.
     * <p>
     * This class is NOT thread-safe.
     * </p>
     */
    static class BrokenUniqueIdGenerator {
        private long counter = 0;

        public long nextId() {
            return ++counter;
        }
    }

    /**
     * Generates sequential unique IDs starting with 1, 2, 3, and so on.
     * <p>
     * This class is thread-safe.
     * </p>
     */
    static class UniqueIdGenerator {
        private final AtomicLong counter = new AtomicLong();

        public long nextId() {
            return counter.incrementAndGet();
        }
    }

    private void test(final int threadCount) throws InterruptedException, ExecutionException {
        final UniqueIdGenerator domainObject = new UniqueIdGenerator();
        Callable<Long> task = new Callable<Long>() {
            @Override
            public Long call() {
                return domainObject.nextId();
            }
        };
        List<Callable<Long>> tasks = Collections.nCopies(threadCount, task);
        ExecutorService executorService = Executors.newFixedThreadPool(threadCount);
        List<Future<Long>> futures = executorService.invokeAll(tasks);
        List<Long> resultList = new ArrayList<Long>(futures.size());
        // Check for exceptions
        for (Future<Long> future : futures) {
            // Throws an exception if an exception was thrown by the task.
            resultList.add(future.get());
        }
        // Validate the IDs
        Assert.assertEquals(threadCount, futures.size());
        List<Long> expectedList = new ArrayList<Long>(threadCount);
        for (long i = 1; i <= threadCount; i++) {
            expectedList.add(i);
        }
        Collections.sort(resultList);
        Assert.assertEquals(expectedList, resultList);
    }

    @Test
    public void test01() throws InterruptedException, ExecutionException {
        test(1);
    }

    @Test
    public void test02() throws InterruptedException, ExecutionException {
        test(2);
    }

    @Test
    public void test04() throws InterruptedException, ExecutionException {
        test(4);
    }

    @Test
    public void test08() throws InterruptedException, ExecutionException {
        test(8);
    }

    @Test
    public void test16() throws InterruptedException, ExecutionException {
        test(16);
    }

    @Test
    public void test32() throws InterruptedException, ExecutionException {
        test(32);
    }
}

Note: I used Oracle Java 1.6.0_24 (64-bit) on Windows 7 (64-bit).

Conditionally ignoring tests in JUnit

JUnit conveniently lets you annotate a test method with @Ignore.

@Test
@Ignore
public void testAdd() {
   ...
}

This causes JUnit to completely ignore the method.

What if you want to only ignore the test under a condition determined at runtime? This would be nice:

@Test
@Ignore(someCondition())
public void testAdd() {
   ...
}

But Java and JUnit do not work like that. Instead, you need to use the Assume class.

@Test
public void testAdd() {
   Assume.assumeTrue(someCondition());
   ...
}

Voila!