Understanding JUnit method order execution (updated for version 4.11)

a-zI’m revisiting my post Understanding JUnit method order execution to discuss how to fix tests that depend on method order by using JUnit 4.11.

Let’s start with a rule: You should not create test methods that depend on the order in which they are executed.

If you do have such tests and they are failing randomly or rarely, this is why: JUnit relies on Java’s reflection API to get which test methods to execute. The problem is that the API does not define the order of the methods it returns. Your tests may work for a long time and then fail, apparently randomly. The things is, you’ve just been lucky all along and relying on the Java run-time giving you a consistent answer when it makes no such guarantee.

You may even see some very confusing behavior like a superclass’ test methods being mixed in with its subclass. I’ve seen this in Eclipse and Ant for example.

JUnit 4.11 provides a workarounds for this behavior with a new class-level annotation: FixMethodOrder(MethodSorters.NAME_ASCENDING). For example:

 @FixMethodOrder(MethodSorters.NAME_ASCENDING)
 public class MyTestCase { ... }
 

The annotation FixMethodOrder sorts the test methods in lexicographic method name order, and uses Method#toString() to break ties.

The recipe is: (1) Use @FixMethodOrder, (2) Fix your tests, (3) Remove @FixMethodOrder.

JUnit is available from GitHub.

4 thoughts on “Understanding JUnit method order execution (updated for version 4.11)

  1. Nicolas Barbulesco

    Now JUnit has added an annotation which we can use to have the tests run in alphabetical order. Well, that’s an idea. But that’s not what I want. I want to have the tests run in their order of appearance in the class. Is there a way to do that ?

    Thank you.

    Reply
  2. Gary Gregory Post author

    Not according to org.junit.runners.MethodSorters. You get to pick one of:

    /**
    * Sorts the test methods by the method name, in lexicographic order,
    * with {@link Method#toString()} used as a tiebreaker
    */
    NAME_ASCENDING(MethodSorter.NAME_ASCENDING),

    /**
    * Leaves the test methods in the order returned by the JVM.
    * Note that the order from the JVM may vary from run to run
    */
    JVM(null),

    /**
    * Sorts the test methods in a deterministic, but not predictable, order
    */
    DEFAULT(MethodSorter.DEFAULT);

    See also http://bugs.sun.com/view_bug.do?bug_id=7023180 which according to Oracle is “Not an issue”.

    Reply
  3. Pingback: Understanding JUnit method order execution | Gary Gregory

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s