[Note this additional post on this topic]
With the addition of the ClassRule annotation in JUnit 4.9, I thought I’d come back and revisit JUnit test order execution.
@ClassRule fills a gap in the JUnit API, by providing class level rules like @BeforeClass and @AfterClass provide class-wide set up and tear down.
This article explains how to control code execution of unit tests and where @ClassRule fits in.
The terminology I use:
- A test implements @Test
- A test case in a class with @Test methods
Ordering test methods
The simplest test case you can write in JUnit is to annotate methods with @Test:
package test; import org.junit.Test; public class OrderTest1 { @Test public void test1() { println("@Test test1()"); } @Test public void test2() { println("@Test test2()"); } private void println(String string) { OrderTestUtils.println(OrderTest1.class, string); } }
In OrderTest1, the execution order is:
OrderTest1 @Test test1() OrderTest1 @Test test2()
Managing test fixtures
If you need to initialize the same data for each test, you put that data in instance variables and initialize them in a @Before setUp method. The setUp method is called before each @Test method.
One test invocation becomes the following call sequence:
- Call @Before setUp
- Call one @Test method
If that data needs to be cleaned up, implement an @After tearDown method. The tearDown method is called after each @Test method.
One test invocation becomes the following call sequence:
- Call @Before setUp
- Call one @Test method
- Call @After tearDown
The call sequence for a class with two test methods is:
- Call @Before setUp
- Call @Test method test1
- Call @After tearDown
- Call @Before setUp
- Call @Test method test2
- Call @After tearDown
For example:
package test; import java.io.Closeable; import java.io.IOException; import org.junit.After; import org.junit.Before; import org.junit.Test; public class OrderTest2 { static class ManagedResource implements Closeable { @Override public void close() throws IOException { } } private ManagedResource managedResource; private void println(String string) { OrderTestUtils.println(OrderTest2.class, string); } @Before public void setUp() { this.println("@Before setUp"); this.managedResource = new ManagedResource(); } @After public void tearDown() throws IOException { this.println("@After tearDown"); this.managedResource.close(); this.managedResource = null; } @Test public void test1() { this.println("@Test test1()"); } @Test public void test2() { this.println("@Test test2()"); } }
In OrderTest2, the execution order is:
OrderTest2 @Before setUp OrderTest2 @Test test1() OrderTest2 @After tearDown OrderTest2 @Before setUp OrderTest2 @Test test2() OrderTest2 @After tearDown
Managing expensive test fixtures
When a resource is expensive to manage like a connection to a server, a database, or even managing an embedded server, it’s best to only initialize that resource once for the whole test case. You want to avoid starting and stopping a server for each @Test method. Instead, initialize the server once for all the tests in the class.
To do so, we use @BeforeClass and @AfterClass methods, instead of @Before and @After to get the following call sequence:
- Call @BeforeClass setUpClass
- Call @Test method test1
- Call @Test method test2
- Call @AfterClass tearDownClass
Let’s add class-level set up and tear down to our example:
package test; import java.io.Closeable; import java.io.IOException; import org.junit.After; import org.junit.AfterClass; import org.junit.Before; import org.junit.BeforeClass; import org.junit.Test; public class OrderTest3 { static class ExpensiveManagedResource implements Closeable { @Override public void close() throws IOException { } } static class ManagedResource implements Closeable { @Override public void close() throws IOException { } } @BeforeClass public static void setUpClass() { OrderTestUtils.println(OrderTest3.class, "@BeforeClass setUpClass"); MyExpensiveManagedResource = new ExpensiveManagedResource(); } @AfterClass public static void tearDownClass() throws IOException { OrderTestUtils.println(OrderTest3.class, "@AfterClass tearDownClass"); MyExpensiveManagedResource.close(); MyExpensiveManagedResource = null; } private ManagedResource myManagedResource; private static ExpensiveManagedResource MyExpensiveManagedResource; private void println(String string) { OrderTestUtils.println(OrderTest3.class, string); } @Before public void setUp() { this.println("@Before setUp"); this.myManagedResource = new ManagedResource(); } @After public void tearDown() throws IOException { this.println("@After setUp"); this.myManagedResource.close(); this.myManagedResource = null; } @Test public void test1() { this.println("@Test test1()"); } @Test public void test2() { this.println("@Test test2()"); } }
In OrderTest3, the execution order is:
OrderTestAll @BeforeClass setUpClass OrderTestAll @Before setUp OrderTestAll @Test test1() OrderTestAll @After setUp OrderTestAll @Before setUp OrderTestAll @Test test2() OrderTestAll @After setUp OrderTestAll @AfterClass tearDownClass
You can see that the setUpClass and tearDownClass wrap the execution of this test case.
Managing resources with rules
Instead of duplicating resource management code in each class, you can reuse your code, but instead of putting this common code in a superclass for all your tests. you can abstract external resource management with a JUnit Rule.
JUnit rules are subclasses of the ExternalResource class.
Let’s do it both ways and compare.
package test; import java.io.Closeable; import java.io.IOException; import org.junit.After; import org.junit.AfterClass; import org.junit.Before; import org.junit.BeforeClass; import org.junit.Rule; import org.junit.Test; import org.junit.rules.ExternalResource; public class OrderTest4 { static class ExpensiveExternalResource extends ExternalResource { ExpensiveExternalResource() { OrderTestUtils.println(ExpensiveExternalResource.class, "constructor"); } @Override protected void after() { OrderTestUtils.println(ExpensiveExternalResource.class, "after"); }; @Override protected void before() throws Throwable { OrderTestUtils.println(ExpensiveExternalResource.class, "before"); }; }; static class ExpensiveManagedResource implements Closeable { @Override public void close() throws IOException { } } static class ManagedResource implements Closeable { @Override public void close() throws IOException { } } @BeforeClass public static void setUpClass() { OrderTestUtils.println(OrderTest4.class, "@BeforeClass setUpClass"); MyExpensiveManagedResource = new ExpensiveManagedResource(); } @AfterClass public static void tearDownClass() throws IOException { OrderTestUtils.println(OrderTest4.class, "@AfterClass tearDownClass"); MyExpensiveManagedResource.close(); MyExpensiveManagedResource = null; } @Rule public ExternalResource resource = new ExpensiveExternalResource(); private ManagedResource myManagedResource; private static ExpensiveManagedResource MyExpensiveManagedResource; private void println(String string) { OrderTestUtils.println(OrderTest4.class, string); } @Before public void setUp() { this.println("@Before setUp"); this.myManagedResource = new ManagedResource(); } @After public void tearDown() throws IOException { this.println("@After tearDown()"); this.myManagedResource.close(); this.myManagedResource = null; } @Test public void test1() { this.println("@Test test1()"); } @Test public void test2() { this.println("@Test test2()"); } }
The methods are run in the following order:
OrderTest4 @BeforeClass setUpClass ExpensiveExternalResource constructor ExpensiveExternalResource before OrderTest4 @Before setUp OrderTest4 @Test test1() OrderTest4 @After setUp ExpensiveExternalResource after ExpensiveExternalResource constructor ExpensiveExternalResource before OrderTest4 @Before setUp OrderTest4 @Test test2() OrderTest4 @After setUp ExpensiveExternalResource after OrderTest4 @AfterClass tearDownClass
The ExternalResource before method is called just before the @Before setUp method.
The ExternalResource after method is called just before the @After setUp method.
An ExternalResource is a nice way to abstract out and resuse your resource management code. You can now compose use of your resources instead of subclassing.
Managing expensive resources with rules
Just like you can wrap a test case with @BeforeClass and @AfterClass, there was no way to do this with rules until JUnit 4.9 and the ClassRule annotation.
The following examples shows a ClassRule at work. Deltas from the previous example are highlighted.
package test; import java.io.Closeable; import java.io.IOException; import org.junit.After; import org.junit.AfterClass; import org.junit.Before; import org.junit.BeforeClass; import org.junit.ClassRule; import org.junit.Rule; import org.junit.Test; import org.junit.rules.ExternalResource; public class OrderTest5 { static class ExpensiveExternalResource extends ExternalResource { private final String info; ExpensiveExternalResource(String info) { this.info = info; OrderTestUtils.println(ExpensiveExternalResource.class, "constructor " + info); } @Override protected void after() { OrderTestUtils.println(ExpensiveExternalResource.class, "after " + this.info); }; @Override protected void before() throws Throwable { OrderTestUtils.println(ExpensiveExternalResource.class, "before " + this.info); }; }; static class ExpensiveManagedResource implements Closeable { @Override public void close() throws IOException { } } static class ManagedResource implements Closeable { @Override public void close() throws IOException { } } @BeforeClass public static void setUpClass() { OrderTestUtils.println(OrderTest5.class, "@BeforeClass setUpClass"); MyExpensiveManagedResource = new ExpensiveManagedResource(); } @AfterClass public static void tearDownClass() throws IOException { OrderTestUtils.println(OrderTest5.class, "@AfterClass tearDownClass"); MyExpensiveManagedResource.close(); MyExpensiveManagedResource = null; } @Rule public ExternalResource resource = new ExpensiveExternalResource("instance"); @ClassRule public static ExternalResource StaticResource = new ExpensiveExternalResource("static"); private ManagedResource myManagedResource; private static ExpensiveManagedResource MyExpensiveManagedResource; private void println(String string) { OrderTestUtils.println(OrderTest5.class, string); } @Before public void setUp() { this.println("@Before setUp"); this.myManagedResource = new ManagedResource(); } @After public void tearDown() throws IOException { this.println("@After tearDown()"); this.myManagedResource.close(); this.myManagedResource = null; } @Test public void test1() { this.println("@Test test1()"); } @Test public void test2() { this.println("@Test test2()"); } }
The test run shows the ClassRule kick in at the start and end of the test case:
ExpensiveExternalResource constructor static ExpensiveExternalResource before static OrderTest5 @BeforeClass setUpClass ExpensiveExternalResource constructor instance ExpensiveExternalResource before instance OrderTest5 @Before setUp OrderTest5 @Test test1() OrderTest5 @After tearDown() ExpensiveExternalResource after instance ExpensiveExternalResource constructor instance ExpensiveExternalResource before instance OrderTest5 @Before setUp OrderTest5 @Test test2() OrderTest5 @After tearDown() ExpensiveExternalResource after instance OrderTest5 @AfterClass tearDownClass ExpensiveExternalResource after static
Subclassing test cases
What happens when you subclass a test case? For example:
package test; import org.junit.Test; public class SubOrderTest1 extends OrderTest1 { private void println(String string) { OrderTestUtils.println(SubOrderTest1.class, string); } @Test public void testSub1() { this.println("@Test testSub1()"); } @Test public void testSub2() { this.println("@Test testSub1()"); } }
JUnit runs the given test case and then the super class test case:
SubOrderTest1 @Test testSub1() SubOrderTest1 @Test testSub1() OrderTest1 @Test test1() OrderTest1 @Test test2()
If you add a third level with a SubSubOrderTest1 class extending SubOrderTest1, you get:
SubSubOrderTest1 @Test testSubSub1() SubSubOrderTest1 @Test testSubSub1() SubOrderTest1 @Test testSub1() SubOrderTest1 @Test testSub1() OrderTest1 @Test test1() OrderTest1 @Test test2()
Method execution order starts with the given test case and goes up the super class chain until you get to the top of the hierarchy.
Subclassing test cases with managed resources
Here, things get trickier.
The order of @Before methods goes from the top to the bottom of the hierarchy: parent, child, child of child.
The order of @After methods goes from the bottom to the top of the hierarchy: child of child, child, parent.
For example:
package test; import java.io.Closeable; import java.io.IOException; import org.junit.After; import org.junit.Before; import org.junit.Test; public class SubOrderTest2 extends OrderTest2 { static class ManagedResource implements Closeable { @Override public void close() throws IOException { } } private ManagedResource managedResource; private void println(String string) { OrderTestUtils.println(SubOrderTest2.class, string); } @Before public void setUpSub() { this.println("@Before setUpSub"); this.managedResource = new ManagedResource(); } @After public void tearDownSub() throws IOException { this.println("@After tearDownSub"); this.managedResource.close(); this.managedResource = null; } @Test public void testSub1() { this.println("@Test testSub1()"); } @Test public void testSub2() { this.println("@Test testSub2()"); } }
Here, JUnit runs the given test case and then the super class test case:
OrderTest2 @Before setUp SubOrderTest2 @Before setUpSub SubOrderTest2 @Test testSub1() SubOrderTest2 @After tearDownSub OrderTest2 @After tearDown OrderTest2 @Before setUp SubOrderTest2 @Before setUpSub SubOrderTest2 @Test testSub2() SubOrderTest2 @After tearDownSub OrderTest2 @After tearDown OrderTest2 @Before setUp SubOrderTest2 @Before setUpSub OrderTest2 @Test test1() SubOrderTest2 @After tearDownSub OrderTest2 @After tearDown OrderTest2 @Before setUp SubOrderTest2 @Before setUpSub OrderTest2 @Test test2() SubOrderTest2 @After tearDownSub OrderTest2 @After tearDown
If you add a third level with a SubSubOrderTest2 class extending SubOrderTest2, you get:
OrderTest2 @Before setUp SubOrderTest2 @Before setUpSub SubSubOrderTest2 @Before setUpSubSub SubSubOrderTest2 @Test testSubSub1() SubSubOrderTest2 @After tearDownSubSub SubOrderTest2 @After tearDownSub OrderTest2 @After tearDown OrderTest2 @Before setUp SubOrderTest2 @Before setUpSub SubSubOrderTest2 @Before setUpSubSub SubSubOrderTest2 @Test testSubSub2() SubSubOrderTest2 @After tearDownSubSub SubOrderTest2 @After tearDownSub OrderTest2 @After tearDown OrderTest2 @Before setUp SubOrderTest2 @Before setUpSub SubSubOrderTest2 @Before setUpSubSub SubOrderTest2 @Test testSub1() SubSubOrderTest2 @After tearDownSubSub SubOrderTest2 @After tearDownSub OrderTest2 @After tearDown OrderTest2 @Before setUp SubOrderTest2 @Before setUpSub SubSubOrderTest2 @Before setUpSubSub SubOrderTest2 @Test testSub2() SubSubOrderTest2 @After tearDownSubSub SubOrderTest2 @After tearDownSub OrderTest2 @After tearDown OrderTest2 @Before setUp SubOrderTest2 @Before setUpSub SubSubOrderTest2 @Before setUpSubSub OrderTest2 @Test test1() SubSubOrderTest2 @After tearDownSubSub SubOrderTest2 @After tearDownSub OrderTest2 @After tearDown OrderTest2 @Before setUp SubOrderTest2 @Before setUpSub SubSubOrderTest2 @Before setUpSubSub OrderTest2 @Test test2() SubSubOrderTest2 @After tearDownSubSub SubOrderTest2 @After tearDownSub OrderTest2 @After tearDown
I hope you found this post helpful. I was a nice reminder for me to lay it out and remember the power of JUnit rules. With the new ClassRule in JUnit 4.9, rules can finally be used full strength.
[Note this additional post on this topic.]
Wonderful post! One little nitpick. You did not mention how @BeforeClass behaves when you are using subclassed tests. Or I’m dense and missed it.
This gives me more ideas for a design problem I have. How to structure JUnit tests that are running tests against different browser types. So far, the most tractable way is to write the tests in one class, then subclass for each browser type; that way the @BeforeClass can be used to create the browser specific webdriver.
LikeLike
Hi Josef:
The @BeforeClass methods in super classes are run before those the current class.
But… I could not get that to work with JUnit 4.10.
LikeLike
Good post and thanks.
LikeLike
Good post!.
But you have to consider that the order or the Test themselves is not guaranteed by the framework.
http://junit.sourceforge.net/doc/faq/faq.htm#tests_2
http://stackoverflow.com/a/9529014/435693
Thanks!
LikeLike
Woah this blog is fantastic i really like studying your posts. Keep up the good paintings! You know, a lot of individuals are looking around for this info, you could help them greatly.
LikeLike
If you are interesting, you can read my solution about how to user test execution order: just as in your source code:
http://intellijava.blogspot.com/2012/05/junit-and-java-7.html
But remember that if your tests need some special ordering, it means that something goes wrong =)
LikeLike
“In OrderTest1, the execution order is:
1 OrderTest1 @Test test1()
2 OrderTest1 @Test test2()”
No. The execution order of JUnit tests in undefined, and it varies at each launch of the tests. “It’s not a bug, it’s a feature.” But it’s annoying. I would happily give up the added value of order variation, and have my list of test results in Eclipse in the order of the tests.
LikeLike
Right, JUnit 4.11 had made this more painful, see my update on the subject here: https://garygregory.wordpress.com/2013/01/23/understanding-junit-method-order-execution-updated-for-version-4-11/
LikeLike
Now JUnit has added an annotation which we can use to have the tests run in alphabetical order. Well, that’s an idea. But that’s not what I want. I want to have the tests run in their order of appearance in the class. Is there a way to do that ?
Thank you.
LikeLike
In my comment you can find link with some hack, and run your tests in order of appearance in the class.
LikeLike
OrderTestUtils which class does it denotes.
LikeLike
Do you know when the various methods in the org.junit.runner.notification.RunListener interface would be fired, particularly when dealing with failures, and @after/@afterclass?
LikeLike
Did something change in junit after 4.9 with respect to test fixture ordering. For me the super classes fixture is not run automatically unless I call them explicitly from my test’s fixtures.
LikeLike
Nice post, in section “Managing resources with rules” on second code snippet there is an error: “OrderTest4 @After setUp” should be changed to “OrderTest4 @After tearDown” twice.
LikeLike
Can I use @AfterClass to get all tests status?
LikeLike
thnx
LikeLike
Thank you for this article! I just started writing some junit and dbunit tests and I didn’t want to couple my design by using a super class for common initialization code. I didn’t know about @ClassRule and @Rule.
LikeLike
Reblogged this on TechCentral and commented:
Great post on junit method order execution
LikeLike
Excellent article that taught me how to use the Before and After annotations. Thank you.
LikeLike
Pingback: How do you assert that a certain exception is thrown in JUnit 4 tests? - QuestionFocus
Pingback: How Do You Assert That A Certain Exception Is Thrown In JUnit 4 Tests? - Code Utility
Pingback: How do you assert that a certain exception is thrown in JUnit tests?