I’m revisiting my post Understanding JUnit method order execution to discuss how to fix tests that depend on method order by using JUnit 4.11.
Let’s start with a rule: You should not create test methods that depend on the order in which they are executed.
If you do have such tests and they are failing randomly or rarely, this is why: JUnit relies on Java’s reflection API to get which test methods to execute. The problem is that the API does not define the order of the methods it returns. Your tests may work for a long time and then fail, apparently randomly. The things is, you’ve just been lucky all along and relying on the Java run-time giving you a consistent answer when it makes no such guarantee.
You may even see some very confusing behavior like a superclass’ test methods being mixed in with its subclass. I’ve seen this in Eclipse and Ant for example.
JUnit 4.11 provides a workarounds for this behavior with a new class-level annotation: FixMethodOrder(MethodSorters.NAME_ASCENDING). For example:
@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class MyTestCase { ... }
The annotation FixMethodOrder sorts the test methods in lexicographic method name order, and uses Method#toString() to break ties.
The recipe is: (1) Use @FixMethodOrder, (2) Fix your tests, (3) Remove @FixMethodOrder.
JUnit is available from GitHub.
Gary Gregory 
Now JUnit has added an annotation which we can use to have the tests run in alphabetical order. Well, that’s an idea. But that’s not what I want. I want to have the tests run in their order of appearance in the class. Is there a way to do that ?
Thank you.
LikeLike
Not according to org.junit.runners.MethodSorters. You get to pick one of:
/**
* Sorts the test methods by the method name, in lexicographic order,
* with {@link Method#toString()} used as a tiebreaker
*/
NAME_ASCENDING(MethodSorter.NAME_ASCENDING),
/**
* Leaves the test methods in the order returned by the JVM.
* Note that the order from the JVM may vary from run to run
*/
JVM(null),
/**
* Sorts the test methods in a deterministic, but not predictable, order
*/
DEFAULT(MethodSorter.DEFAULT);
See also http://bugs.sun.com/view_bug.do?bug_id=7023180 which according to Oracle is “Not an issue”.
LikeLike
A guy went ahead and wrote some classes to get the nethods ordered as in the file : http://intellijava.blogspot.gr/2012/05/junit-and-java-7.html
LikeLike
Pingback: Understanding JUnit method order execution | Gary Gregory
Once you introduce @FixMethodOrder your problems begin.
Dependent test cases lack isolation!
The isolation of tests is a self-correctiong mechanism to maintainable source code. If you have a problem with isolation of tests you may have isolation problems in your production code and it will be less maintanable.
On the other hand you will achieve isolation in your production code. Developers have to decouple things to make them isolated testable.
Isolation supports expressive tests that shows the problems at the point the test adresses. If you have dependent tests you may not find the cause for the problem at the point where the assertion fails.
That are really big issues and they will be larger the more source code you have.
So keep your tests independent from each other! And I would not rely on the presence of @FixMethodOrder in next major releases of JUnit.
LikeLike
There is more than meets the eye on issues like this.
In one case, you may be (as I at work) relying on third party libraries that contain bugs, various side effects, and oddities that arise when your code use their APIs. In another, you may be working in a larger team and another team’s code may contain side effects but the fact that they show up in your test case may not be a prime concern for them because your use case is not important. Maybe it will be later, but not today.
Regardless of how you get here, there is also the cost of debugging and rewriting tests, or worse, debugging and rewriting production code to handle executing test methods in any order. In any project, you need to prioritize issues like this against new features, bug fixes and general maintenance.
So yes, if I were to write new code with no dependencies, I would account for executing unit tests without the fixed order directory, of course! Even then, that would not guarantee that the JVM would run my tests in random order and that they would pass. Even without fixed order execution, a given JVM executes the test methods in the same order until you change the some part code.
LikeLike
Your unit tests should only be testing your code, in isolation. You should be mocking 3rd party libraries with frameworks such as Mockito so that you can guarantee specific behavior from dependencies.
LikeLike