Conditionally Disabling and Filtering Tests in JUnit 5

Attention: The following article was published over 6 years ago, and the information provided may be aged or outdated. Please keep that in mind as you read the post.

This article is part of my blog series on automated testing promoting my new Pluralsight course Effective Automated Testing with Spring.

I’m in the middle of several talks on JUnit 5, so it’s safe to say that JUnit has been on my mind lately. In the last article in this series, we covered how to use JUnit test interfaces to encourage good behavior.

In this article, we look at the improvements the JUnit team has made to filtering and conditionally disabling tests in JUnit 5.

Conditionally Disabling Tests in JUnit 5

In JUnit 4 there was the @Ignore annotation which told a JUnit runner to skip the execution of that test. In JUnit 5, the @Ignore annotation has been replaced with @Disabled. But, far from just changing the name, the JUnit team has also given us hooks into the test execution functionality allowing for easy customization on when a test should be executed.

Let’s take a look at creating a custom @Disabled tag that will disable a test based on the operating system of the host machine.

The first step would be creating a class that implements the ExecutionCondition interface. Implementing evaluateExecutionCondition gives access to ExtensionContext which provides extensive information on the context of the test being executed; tags, display name, method, etc. I won’t be using it in this example, but it can provide useful information when determining if a test should be disabled.

In this example, we will simply look up the name of the host machine’s operation system and then disable the test if the os.name matches “Mac OS X”. In the disabled method, the passed-in string is the reason the test is disabled. This will show up in test reports.

public class DisableOnMacCondition implements ExecutionCondition {
   @Override
   public ConditionEvaluationResult evaluateExecutionCondition(ExtensionContext context) {
      String osName = System.getProperty("os.name");
      if(osName.equalsIgnoreCase("Mac OS X")) {
         return ConditionEvaluationResult.disabled("Test disabled on mac");
      } else {
         return ConditionEvaluationResult.enabled("Test enabled");
      }
   }
}

After implementing the ExecutionCondition, we can create our own custom annotation, which I named DisabledOnMac.

@Target({ ElementType.TYPE, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@ExtendWith(DisableOnMacCondition.class)
public @interface DisabledOnMac {
}

With the custom annotation created, I can annotate any test that might be problematic when it’s executed on a Mac with @DisabledOnMac.

@Test
@DisabledOnMac
public void testFindByInvalidRoomType() {
   RoomRepo repo = mock(RoomRepo.class);
   RoomServiceImpl service = new RoomServiceImpl(repo, roomTypes);
   verify(repo, times(0)).findRoomsByRoomType(any());
   expectedException.expect(RoomServiceException.class);
   expectedException.expectMessage("Room type: NOT FOUND not found!");
   service.findRoomsByType("NOT FOUND");
}

Being able to conditionally execute tests can be very helpful. I think a lot of the benefit with this new functionality will come with integration tests or tests that reference system resources which might depend heavily upon local system configurations. Requiring developers to configure their systems to properly execute a test might be difficult to enforce. So, having the ability to conditionally skip a test when a machine isn’t configured properly to run it is much better than doing a blanket ignore or, having to deal with a failing test, which is problematic in its own right.

Tagging in JUnit 5

Most of my experience writing automated tests have been with JUnit 4; I have comparatively little experience TestNG. That said, one feature that I did like in TestNG was the relative ease of being able to place tests in groups and execute tests by group. JUnit 4 had @Category, however its implementation was fairly clunky and, in my experience, rarely used.

Note: Here’s a good resource to learn more about TestNG.

With the tagging feature, JUnit 5 takes some notes from TestNG, but also greatly improves upon it as well.

Tagging in JUnit5 is easy and straightforward enough. Simply add the @Tag annotation and provide a name to accomplish the goal.

@Test
@Tag("integration")
public void testFindByValidRoomType() {
   RoomRepo repo = mock(RoomRepo.class);
   RoomServiceImpl service = new RoomServiceImpl(repo, roomTypes);
   when(repo.findRoomsByRoomType("Single")).thenReturn(Arrays.asList(//
      new Room(1L, "100", "Single", new BigDecimal(145.99))));
   List rooms = service.findRoomsByType("Single");

  assertEquals(1, rooms.size());
}

Filtering by tag is also very easy to do in Eclipse 4.7.1a+.

As you notice in the gif, when you filter tests by tag (either by inclusion or exclusion), the filtered tests will not even show up in the test report. This is different from disabled, where disabled tests still show up as skipped. Just something to keep in mind if you do use tagging to filter tests in your automated test suite.

@Tag is a repeatable annotation as well, so if you want a test to have multiple tags, it would look like the below:

@Test
@Tag("integration")
@Tag("fast")
public void testFindByValidRoomType() {
...
}

A major goal of JUnit was extensibility and it shows its benefits with tagging. Adding @Test and @Tag to a test case isn’t too difficult, but what if those two tags could be merged into one? That’s easy to do in JUnit 5 by creating your own annotation:

@Target({ TYPE, METHOD })
@Retention(RetentionPolicy.RUNTIME)
@Tag("integration")
@Test
public @interface IntegrationTest {

}

Beyond simplifying the process of tagging a test, this extensibility can improve the readability of test classes. It can help guard against issues like typos, or developers using similar, but different, tags (e.g. using “Int” instead of “Integration”).

@IntegrationTest
public void testFindByValidRoomType() {
	RoomRepo repo = mock(RoomRepo.class);
	RoomServiceImpl service = new RoomServiceImpl(repo, roomTypes);
	when(repo.findRoomsByRoomType("Single")).thenReturn(Arrays.asList(//
			new Room(1L, "100", "Single", new BigDecimal(145.99))));
	List<Room> rooms = service.findRoomsByType("Single");

	assertEquals(1, rooms.size());
}

Filtering Tests in Maven

Configuring Maven to filter tests by tag is pretty easy as well. While tags can be passed in as arguments, in the below example, it would be done by mvn -DexcludeTags={tag names}.

The best way to handle filtering by tags in an enterprise setting would be with Maven profiles. In the below example, I have a check-in-build profile. As the name suggests, it would be executed on every check-in and a nightly-build profile. The check-in-build is configured to ignore integration tests which can take longer to execute and might be more prone to transient failures (if, for example, a remote resource is down or misconfigured). The nightly-build would be configured to execute all tests.

(Note: the version of surefire plugin I am using has a bug that causes it to fail if no value provided for the excludeTags/includeTags parameter; this issue has be resolved in later versions).

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	...
	<profiles>
		<profile>
			<id>check-in-build</id>
			<properties>
				<excludeTags>integration</excludeTags>
			</properties>
		</profile>
		<profile>
			<id>nightly-build</id>
			<properties>
				<excludeTags>none</excludeTags>
			</properties>
		</profile>
	</profiles>

	<build>
		<plugins>
			...
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-surefire-plugin</artifactId>
				<version>2.19.1</version>
				<configuration>
					<properties>
						<excludeTags>${excludeTags}</excludeTags>
					</properties>
				</configuration>
				<dependencies>
					<dependency>
						<groupId>org.junit.platform</groupId>
						<artifactId>junit-platform-surefire-provider</artifactId>
						<version>1.1.0-RC1</version>
					</dependency>
					<dependency>
						<groupId>org.junit.jupiter</groupId>
						<artifactId>junit-jupiter-engine</artifactId>
						<version>5.1.0-RC1</version>
					</dependency>
				</dependencies>
			</plugin>
		</plugins>
	</build>

	...

</project>

Conclusion

The code for the examples shown in this article can be found here: https://github.com/wkorando/WelcomeToJunit5.

JUnit 5 offers a number of improvements over JUnit 4.

While it is ideal to run every test in your automated test suite every time, this might not always be practical. Some tests might require extensive local configuration or resources that a machine does not have. Other tests might be costly to run, or take a long time to execute, so it would be best not to execute them with every code commit. JUnit 5 improving the experience of filtering and conditionally executing tests is a welcome and much needed enhancement to the popular testing framework.

JUnit 5.1 will be dropping soon. Among other enhancements coming in JUnit 5.1 is further improvements to filtering and conditionally executing tests. I will be doing a write up detailing what is coming in JUnit 5.1 soon.