New feature: controlling the order of test execution

New instructors using the JavaTddPlugin sometimes expect their reference tests to be executed in the order they appear in their source files, and are surprised when this doesn't happen.  However, JUnit does not actually guarantee any specific order for executing test methods.  JUnit determines the list of methods using Java's reflection API, and the reflection API does not guarantee that it returns the list of methods declared in a class in any specific order--in particular, the reflection API does not guarantee to return the list of methods in declaration order.  So what's an instructor to do?

Until now, we've lived with JUnit's "unspecified order" behavior.  However, this has become a more pressing issue in some situations.  As a result, I have modified the custom JUnit runner we are using so that it uses a deterministic execution order for all test methods in a test class by explicitly sorting the methods:

  1. First, test methods are sorted by declaring class, such that superclass methods come before subclass methods. This won't matter for many users, but if test classes inherit from other test classes, superclass tests get executed before subclass tests.

  2. Within a single test class, test methods are sorted in declaration order, meaning they will be executed in the order in which they are declared in the corresponding source file. This is actually what many instructors expect, but it isn't what JUnit provides. Now the order will match expectations, guaranteed.

We're testing this out on our production server right now, and it will be included in the next release of the JavaTddPlugin for everyone. If you can think of any negative consequences or you have any questions, add them here!

Groups: