Understanding JUnit method order execution

[Note this additional post on this topic]

With the addition of the ClassRule annotation in JUnit 4.9, I thought I’d come back and revisit JUnit test order execution.

@ClassRule fills a gap in the JUnit API, by providing class level rules like @BeforeClass and @AfterClass provide class-wide set up and tear down.

This article explains how to control code execution of unit tests and where @ClassRule fits in.

The terminology I use:

  • A test implements @Test
  • A test case in a class with @Test methods

Ordering test methods

The simplest test case you can write in JUnit is to annotate methods with @Test:

package test;

import org.junit.Test;

public class OrderTest1 {

    @Test
    public void test1() {
        println("@Test test1()");
    }

    @Test
    public void test2() {
        println("@Test test2()");
    }

    private void println(String string) {
        OrderTestUtils.println(OrderTest1.class, string);
    }
}

In OrderTest1, the execution order is:

OrderTest1 @Test test1()
OrderTest1 @Test test2()

Managing test fixtures

If you need to initialize the same data for each test, you put that data in instance variables and initialize them in a @Before setUp method. The setUp method is called before each @Test method.

One test invocation becomes the following call sequence:

  1. Call @Before setUp
  2. Call one @Test method

If that data needs to be cleaned up, implement an @After tearDown method. The tearDown method is called after each @Test method.

One test invocation becomes the following call sequence:

  1. Call @Before setUp
  2. Call one @Test method
  3. Call @After tearDown

The call sequence for a class with two test methods is:

  1. Call @Before setUp
  2. Call @Test method test1
  3. Call @After tearDown
  4. Call @Before setUp
  5. Call @Test method test2
  6. Call @After tearDown

For example:

package test;

import java.io.Closeable;
import java.io.IOException;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

public class OrderTest2 {

    static class ManagedResource implements Closeable {
        @Override
        public void close() throws IOException {
        }
    }

    private ManagedResource managedResource;

    private void println(String string) {
        OrderTestUtils.println(OrderTest2.class, string);
    }

    @Before
    public void setUp() {
        this.println("@Before setUp");
        this.managedResource = new ManagedResource();
    }

    @After
    public void tearDown() throws IOException {
        this.println("@After tearDown");
        this.managedResource.close();
        this.managedResource = null;
    }

    @Test
    public void test1() {
        this.println("@Test test1()");
    }

    @Test
    public void test2() {
        this.println("@Test test2()");
    }
}

In OrderTest2, the execution order is:

OrderTest2 @Before setUp
OrderTest2 @Test test1()
OrderTest2 @After tearDown
OrderTest2 @Before setUp
OrderTest2 @Test test2()
OrderTest2 @After tearDown

Managing expensive test fixtures

When a resource is expensive to manage like a connection to a server, a database, or even managing an embedded server, it’s best to only initialize that resource once for the whole test case. You want to avoid starting and stopping a server for each @Test method. Instead, initialize the server once for all the tests in the class.

To do so, we use @BeforeClass and @AfterClass methods, instead of @Before and @After to get the following call sequence:

  1. Call @BeforeClass setUpClass
  2. Call @Test method test1
  3. Call @Test method test2
  4. Call @AfterClass tearDownClass

Let’s add class-level set up and tear down to our example:

package test;

import java.io.Closeable;
import java.io.IOException;

import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;

public class OrderTest3 {

    static class ExpensiveManagedResource implements Closeable {
        @Override
        public void close() throws IOException {
        }
    }

    static class ManagedResource implements Closeable {
        @Override
        public void close() throws IOException {
        }
    }

    @BeforeClass
    public static void setUpClass() {
        OrderTestUtils.println(OrderTest3.class, "@BeforeClass setUpClass");
        MyExpensiveManagedResource = new ExpensiveManagedResource();
    }

    @AfterClass
    public static void tearDownClass() throws IOException {
        OrderTestUtils.println(OrderTest3.class, "@AfterClass tearDownClass");
        MyExpensiveManagedResource.close();
        MyExpensiveManagedResource = null;
    }

    private ManagedResource myManagedResource;
    private static ExpensiveManagedResource MyExpensiveManagedResource;

    private void println(String string) {
        OrderTestUtils.println(OrderTest3.class, string);
    }

    @Before
    public void setUp() {
        this.println("@Before setUp");
        this.myManagedResource = new ManagedResource();
    }

    @After
    public void tearDown() throws IOException {
        this.println("@After setUp");
        this.myManagedResource.close();
        this.myManagedResource = null;
    }

    @Test
    public void test1() {
        this.println("@Test test1()");
    }

    @Test
    public void test2() {
        this.println("@Test test2()");
    }
}

In OrderTest3, the execution order is:

OrderTestAll @BeforeClass setUpClass
OrderTestAll @Before setUp
OrderTestAll @Test test1()
OrderTestAll @After setUp
OrderTestAll @Before setUp
OrderTestAll @Test test2()
OrderTestAll @After setUp
OrderTestAll @AfterClass tearDownClass

You can see that the setUpClass and tearDownClass wrap the execution of this test case.

Managing resources with rules

Instead of duplicating resource management code in each class, you can reuse your code, but instead of putting this common code in a superclass for all your tests. you can abstract external resource management with a JUnit Rule.

JUnit  rules are subclasses of the ExternalResource class.

Let’s do it both ways and compare.

package test;

import java.io.Closeable;
import java.io.IOException;

import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExternalResource;

public class OrderTest4 {

    static class ExpensiveExternalResource extends ExternalResource {

        ExpensiveExternalResource() {
            OrderTestUtils.println(ExpensiveExternalResource.class, "constructor");
        }

        @Override
        protected void after() {
            OrderTestUtils.println(ExpensiveExternalResource.class, "after");
        };

        @Override
        protected void before() throws Throwable {
            OrderTestUtils.println(ExpensiveExternalResource.class, "before");
        };
    };

    static class ExpensiveManagedResource implements Closeable {
        @Override
        public void close() throws IOException {
        }
    }

    static class ManagedResource implements Closeable {
        @Override
        public void close() throws IOException {
        }
    }

    @BeforeClass
    public static void setUpClass() {
        OrderTestUtils.println(OrderTest4.class, "@BeforeClass setUpClass");
        MyExpensiveManagedResource = new ExpensiveManagedResource();
    }

    @AfterClass
    public static void tearDownClass() throws IOException {
        OrderTestUtils.println(OrderTest4.class, "@AfterClass tearDownClass");
        MyExpensiveManagedResource.close();
        MyExpensiveManagedResource = null;
    }

    @Rule
    public ExternalResource resource = new ExpensiveExternalResource();

    private ManagedResource myManagedResource;

    private static ExpensiveManagedResource MyExpensiveManagedResource;

    private void println(String string) {
        OrderTestUtils.println(OrderTest4.class, string);
    }

    @Before
    public void setUp() {
        this.println("@Before setUp");
        this.myManagedResource = new ManagedResource();
    }

    @After
    public void tearDown() throws IOException {
        this.println("@After tearDown()");
        this.myManagedResource.close();
        this.myManagedResource = null;
    }

    @Test
    public void test1() {
        this.println("@Test test1()");
    }

    @Test
    public void test2() {
        this.println("@Test test2()");
    }
}

The methods are run in the following order:

OrderTest4 @BeforeClass setUpClass
ExpensiveExternalResource constructor
ExpensiveExternalResource before
OrderTest4 @Before setUp
OrderTest4 @Test test1()
OrderTest4 @After setUp
ExpensiveExternalResource after
ExpensiveExternalResource constructor
ExpensiveExternalResource before
OrderTest4 @Before setUp
OrderTest4 @Test test2()
OrderTest4 @After setUp
ExpensiveExternalResource after
OrderTest4 @AfterClass tearDownClass

The ExternalResource before method is called just before the @Before setUp method.

The ExternalResource after method is called just before the @After setUp method.

An ExternalResource is a nice way to abstract out and resuse your resource management code. You can now compose use of your resources instead of  subclassing.

Managing expensive resources with rules

Just like you can wrap a test case with @BeforeClass and @AfterClass, there was no way to do this with rules until JUnit 4.9 and the ClassRule annotation.

The following examples shows a ClassRule at work. Deltas from the previous example are highlighted.

package test;

import java.io.Closeable;
import java.io.IOException;

import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExternalResource;

public class OrderTest5 {

    static class ExpensiveExternalResource extends ExternalResource {

        private final String info;

        ExpensiveExternalResource(String info) {
            this.info = info;
            OrderTestUtils.println(ExpensiveExternalResource.class, "constructor " + info);
        }

        @Override
        protected void after() {
            OrderTestUtils.println(ExpensiveExternalResource.class, "after " + this.info);
        };

        @Override
        protected void before() throws Throwable {
            OrderTestUtils.println(ExpensiveExternalResource.class, "before " + this.info);
        };
    };

    static class ExpensiveManagedResource implements Closeable {
        @Override
        public void close() throws IOException {
        }
    }

    static class ManagedResource implements Closeable {
        @Override
        public void close() throws IOException {
        }
    }

    @BeforeClass
    public static void setUpClass() {
        OrderTestUtils.println(OrderTest5.class, "@BeforeClass setUpClass");
        MyExpensiveManagedResource = new ExpensiveManagedResource();
    }

    @AfterClass
    public static void tearDownClass() throws IOException {
        OrderTestUtils.println(OrderTest5.class, "@AfterClass tearDownClass");
        MyExpensiveManagedResource.close();
        MyExpensiveManagedResource = null;
    }

    @Rule
    public ExternalResource resource = new ExpensiveExternalResource("instance");

    @ClassRule
    public static ExternalResource StaticResource = new ExpensiveExternalResource("static");

    private ManagedResource myManagedResource;

    private static ExpensiveManagedResource MyExpensiveManagedResource;

    private void println(String string) {
        OrderTestUtils.println(OrderTest5.class, string);
    }

    @Before
    public void setUp() {
        this.println("@Before setUp");
        this.myManagedResource = new ManagedResource();
    }

    @After
    public void tearDown() throws IOException {
        this.println("@After tearDown()");
        this.myManagedResource.close();
        this.myManagedResource = null;
    }

    @Test
    public void test1() {
        this.println("@Test test1()");
    }

    @Test
    public void test2() {
        this.println("@Test test2()");
    }
}

The test run shows the ClassRule kick in at the start and end of the test case:

ExpensiveExternalResource constructor static
ExpensiveExternalResource before static
OrderTest5 @BeforeClass setUpClass
ExpensiveExternalResource constructor instance
ExpensiveExternalResource before instance
OrderTest5 @Before setUp
OrderTest5 @Test test1()
OrderTest5 @After tearDown()
ExpensiveExternalResource after instance
ExpensiveExternalResource constructor instance
ExpensiveExternalResource before instance
OrderTest5 @Before setUp
OrderTest5 @Test test2()
OrderTest5 @After tearDown()
ExpensiveExternalResource after instance
OrderTest5 @AfterClass tearDownClass
ExpensiveExternalResource after static

Subclassing test cases

What happens when you subclass a test case? For example:

package test;

import org.junit.Test;

public class SubOrderTest1 extends OrderTest1 {

    private void println(String string) {
        OrderTestUtils.println(SubOrderTest1.class, string);
    }

    @Test
    public void testSub1() {
        this.println("@Test testSub1()");
    }

    @Test
    public void testSub2() {
        this.println("@Test testSub1()");
    }
}

JUnit runs the given test case and then the super class test case:

SubOrderTest1 @Test testSub1()
SubOrderTest1 @Test testSub1()
OrderTest1 @Test test1()
OrderTest1 @Test test2()

If you add a third level with a SubSubOrderTest1 class extending SubOrderTest1, you get:

SubSubOrderTest1 @Test testSubSub1()
SubSubOrderTest1 @Test testSubSub1()
SubOrderTest1 @Test testSub1()
SubOrderTest1 @Test testSub1()
OrderTest1 @Test test1()
OrderTest1 @Test test2()

Method execution order starts with the given test case and goes up the super class chain until you get to the top of the hierarchy.

Subclassing test cases with managed resources

Here, things get trickier.

The order of @Before methods goes from the top to the bottom of the hierarchy: parent, child, child of child.

The order of @After methods goes from the bottom to the top of the hierarchy: child of child, child, parent.

For example:

package test;

import java.io.Closeable;
import java.io.IOException;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

public class SubOrderTest2 extends OrderTest2 {

    static class ManagedResource implements Closeable {
        @Override
        public void close() throws IOException {
        }
    }

    private ManagedResource managedResource;

    private void println(String string) {
        OrderTestUtils.println(SubOrderTest2.class, string);
    }

    @Before
    public void setUpSub() {
        this.println("@Before setUpSub");
        this.managedResource = new ManagedResource();
    }

    @After
    public void tearDownSub() throws IOException {
        this.println("@After tearDownSub");
        this.managedResource.close();
        this.managedResource = null;
    }

    @Test
    public void testSub1() {
        this.println("@Test testSub1()");
    }

    @Test
    public void testSub2() {
        this.println("@Test testSub2()");
    }
}

Here, JUnit runs the given test case and then the super class test case:

OrderTest2 @Before setUp
SubOrderTest2 @Before setUpSub
SubOrderTest2 @Test testSub1()
SubOrderTest2 @After tearDownSub
OrderTest2 @After tearDown
OrderTest2 @Before setUp
SubOrderTest2 @Before setUpSub
SubOrderTest2 @Test testSub2()
SubOrderTest2 @After tearDownSub
OrderTest2 @After tearDown
OrderTest2 @Before setUp
SubOrderTest2 @Before setUpSub
OrderTest2 @Test test1()
SubOrderTest2 @After tearDownSub
OrderTest2 @After tearDown
OrderTest2 @Before setUp
SubOrderTest2 @Before setUpSub
OrderTest2 @Test test2()
SubOrderTest2 @After tearDownSub
OrderTest2 @After tearDown

If you add a third level with a SubSubOrderTest2 class extending SubOrderTest2, you get:

OrderTest2 @Before setUp
SubOrderTest2 @Before setUpSub
SubSubOrderTest2 @Before setUpSubSub
SubSubOrderTest2 @Test testSubSub1()
SubSubOrderTest2 @After tearDownSubSub
SubOrderTest2 @After tearDownSub
OrderTest2 @After tearDown
OrderTest2 @Before setUp
SubOrderTest2 @Before setUpSub
SubSubOrderTest2 @Before setUpSubSub
SubSubOrderTest2 @Test testSubSub2()
SubSubOrderTest2 @After tearDownSubSub
SubOrderTest2 @After tearDownSub
OrderTest2 @After tearDown
OrderTest2 @Before setUp
SubOrderTest2 @Before setUpSub
SubSubOrderTest2 @Before setUpSubSub
SubOrderTest2 @Test testSub1()
SubSubOrderTest2 @After tearDownSubSub
SubOrderTest2 @After tearDownSub
OrderTest2 @After tearDown
OrderTest2 @Before setUp
SubOrderTest2 @Before setUpSub
SubSubOrderTest2 @Before setUpSubSub
SubOrderTest2 @Test testSub2()
SubSubOrderTest2 @After tearDownSubSub
SubOrderTest2 @After tearDownSub
OrderTest2 @After tearDown
OrderTest2 @Before setUp
SubOrderTest2 @Before setUpSub
SubSubOrderTest2 @Before setUpSubSub
OrderTest2 @Test test1()
SubSubOrderTest2 @After tearDownSubSub
SubOrderTest2 @After tearDownSub
OrderTest2 @After tearDown
OrderTest2 @Before setUp
SubOrderTest2 @Before setUpSub
SubSubOrderTest2 @Before setUpSubSub
OrderTest2 @Test test2()
SubSubOrderTest2 @After tearDownSubSub
SubOrderTest2 @After tearDownSub
OrderTest2 @After tearDown

I hope you found this post helpful. I was a nice reminder for me to lay it out and remember the power of JUnit rules. With the new ClassRule in JUnit 4.9, rules can finally be used full strength.

[Note this additional post on this topic.]

19 thoughts on “Understanding JUnit method order execution

  1. josefB

    Wonderful post! One little nitpick. You did not mention how @BeforeClass behaves when you are using subclassed tests. Or I’m dense and missed it.

    This gives me more ideas for a design problem I have. How to structure JUnit tests that are running tests against different browser types. So far, the most tractable way is to write the tests in one class, then subclass for each browser type; that way the @BeforeClass can be used to create the browser specific webdriver.

    Reply
  2. test2

    Woah this blog is fantastic i really like studying your posts. Keep up the good paintings! You know, a lot of individuals are looking around for this info, you could help them greatly.

    Reply
  3. Nicolas Barbulesco

    “In OrderTest1, the execution order is:
    1 OrderTest1 @Test test1()
    2 OrderTest1 @Test test2()”

    No. The execution order of JUnit tests in undefined, and it varies at each launch of the tests. “It’s not a bug, it’s a feature.” But it’s annoying. I would happily give up the added value of order variation, and have my list of test results in Eclipse in the order of the tests.

    Reply
      1. Nicolas Barbulesco

        Now JUnit has added an annotation which we can use to have the tests run in alphabetical order. Well, that’s an idea. But that’s not what I want. I want to have the tests run in their order of appearance in the class. Is there a way to do that ?

        Thank you.

  4. Andrew Muraco

    Do you know when the various methods in the org.junit.runner.notification.RunListener interface would be fired, particularly when dealing with failures, and @after/@afterclass?

    Reply
  5. Krishna

    Did something change in junit after 4.9 with respect to test fixture ordering. For me the super classes fixture is not run automatically unless I call them explicitly from my test’s fixtures.

    Reply
  6. Gasan Gus

    Nice post, in section “Managing resources with rules” on second code snippet there is an error: “OrderTest4 @After setUp” should be changed to “OrderTest4 @After tearDown” twice.

    Reply
  7. Maja

    Thank you for this article! I just started writing some junit and dbunit tests and I didn’t want to couple my design by using a super class for common initialization code. I didn’t know about @ClassRule and @Rule.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s