Test Automation with JUnit 5

Created

Test Automation with JUnit 5

Test Automation

To use Software to design, generate, execute, and determine the outcome (i.e., compare actual vs. expected output) of test cases and/or control such or related activities

Revenue tasks vs. Excise tasks

  • Some tasks need human wisdom, intelligence, judgement, domain knowledge etc.
  • Some tasks can be automated following a well-defined process/algorithm/standard.

Software Testability

Testability is how testable the software is. It can be gauged by:

  • observability: how easy it is to observe the behavior of a software in terms of its outputs and/or effects on the environment.
  • controllability: how easy it is to provide a program with the needed inputs, in terms of values, operations, and behaviors.

Test case values are input values necessary to complete the execution of the test case Prefix values: inputs necessary to establish the appropriate state so that the test case can be properly run. Postfix values: inputs to be sent to the software after the test case has run Expected results: the corrected (i.e. expected) output for a given test case.

A test case is composed of test case values, prefix and postfix values, and expected results.

  • It can also contain test case ID ,test case summary, test case author, last status (pass/fail), actual result, etc.
  • At the very least a test case must include the test caste value and expected results.

Test Set (aka Test Suite): a set of test cases Test Framework (aka Test Tool): software tool that facilitates testing activities

JUnit

JUnit is the most popular/well-known test framework created by Kent Beck and Eric Gamma.

  • Used for unit testing and integration testing.
  • Can be launched from the command-line or IDEs.
  • xUnit Frameworks include: CUnit, CppUnit, JSUnit, PyUnit, NUnit, EUnit, RUnit, and more.

Use one or more test classes where each test class contains:

  • test cases written as test methods
  • methods to setup program state before each test method is run (i.e., prefix values)
  • methods to update state after each test method is run (i.e., postfix values)

A variety of assert methods available to check the actual result produced by a test case against expected output.

For example,

class MathUtilsTest { // test class contains test methods (i.e., test cases) 
	ArrayList<Integer> ray;

	@BeforeEach
	public static void setUp() {
		ray = new AraryList<Integer>();
	}

	@AfterEach
	public static void tearDown() {
		ray.clear();
	}
	
	@Test
	public void testAdd() { // a test method corresponding to a single test case 
		MathUtils mu = new MathUtils();  
		assertEquals(8, mu.add(2, 6));
	}
}

Test Instance Lifecycle: In order to allow individual test methods to be executed in isolation and to avoid unexpected side effects due to mutable test instance states, JUnit creates a new instance of each test class before executing each test method.

  • Override this by @TestInstance(Lifecycle.PER_CLASS)

Exceptions

In order to make sure that an error is thrown there are two ways:

@Test
public void testCountOfRayNull() { 
	ray = null;  
	try {
		ArrayUtils.countOf(ray, 2); }
	catch (NullPointerException e) { 
		return;
	}
	fail(“null must fail”); 
}

@Test
public void testCountOfRayNull() {  
	ray = null; 
	assertThrows(NullPointerException.class, () -> {
		ArrayUtils.countOf(ray, 2);
	});
} //assertThrows is only supported in JUnit 5

Parameterized Tests

Parameterize Testing is a common unit testing pattern used to test a single method with different input vlaues.

  • One way is to write a separate test method for each test case, but this creates lots of duplicated code

Example of the "add" method:

@DisplayName("MathUtils.add Tests")  
@ParameterizedTest  
@CsvSource({"0, 1, 1", "2, -4, -2", "-3, 3, 0", “-2, -2, -4"}) 
public void testAdd(int a, int b, int expected) {
	assertEquals(expected, MathUtils.add(a, b)); 
}

Example of the "countOf" method for few inputs that we expect to return the value of 1:

@ParameterizedTest  
@MethodSource("createListAndKey")  
public void testCountOfExpectOne(ArrayList<Integer> ray, int key) {
	assertEquals(1, ArrayUtils.countOf(ray, key));
}

private static Stream<Arguments> createListAndKey() {
	rayList<Integer> ray0 = new ArrayList<Integer>(); ray0.add(2);  
	ray0.add(5);
	ray0.add(-1);
	return Stream.of(
		Arguments.of(ray0, 2), 
		Arguments.of(ray0, 5), 
		Arguments.of(ray0, -1)
	);
}

@Nested

Use @Nested to have nested test classes. This is useful to group test cases.

@DisplayName(“Tree Test") 
class TreeTest {
	
	// ...
	
	@Nested
	@DisplayName(“Add new node")
	class AddNode {
		// ...
	}
	
	// ...
	
	@Nested
	@DisplayName(“Remove a node")
	class RemoveNode {
		// ...
	}
	
	// ...

}

Tags and Filtering

Tags are used to filter which tests are executed for a given test plan. For example, a development team may tag tests with values such as "fast", "slow", etc. and then supply a list of tags to be used for the current test plan

  • Can give test methods and/or test classes one or more tag names
@Tag(“all")  
class ClassATest {
	@Test  
	@Tag("development") @Tag("production")  
	void testCaseA(TestInfo testInfo) { }
}

Repeated Test

To run a test more than once,

@RepeatedTest(10)  
public void testRepeat() {
	// ...
}

Order of Execution

By default in JUnit, the test methods could run in any order

  • When unit testing, test cases must be independent of each other

Possible test case execution orderings:

  • Alphanumeric: sorts test methods alphanumerically based on their names and formal parameter lists.
  • OrderAnnotation: sorts test methods numerically based on values specified via the @Order annotation.
  • Random: orders test methods pseudo-randomly and supports configuration of a custom seed.

OrderAnnotation example:

@TestMethodOrder(OrderAnnotation.class) class OrderOfExecutionDemo {
	
	@Test  
	@Order(1)  
	void nullValues() {
		// perform assertions against null values
	}
	
	@Test  
	@Order(2)  
	void emptyValues() {
		// perform assertions against empty values
	}
	
	@Test  
	@Order(3)  
	void validValues() {
		// perform assertions against valid values
	}  
}

Alphanumeric order example:

@TestMethodOrder(Alphanumeric.class) class OrderOfExecutionDemo {

	@Test
	void testA() {  
		// run this first
	}
	
	@Test
	
	void testB() {  
		// run this next
	}
	
	@Test
	
	void testC() {  
		// and run this last
	}
}

Timeout

@Timeout allows a test to fail if the execution time exceeds the specified limit.

class TimeoutDemo {
	
	@BeforeEach
	@Timeout(5)
	void setUp() {
		// fails if execution time exceeds 5 seconds
	}
	
	@Test  
	@Timeout(value = 100, unit = TimeUnit.MILLISECONDS) void failsIfExecutionTimeExceeds100Milliseconds() {
		// fails if execution time > 100ms
	} 
}

Assumptions

Used to validate assumptions before/after test execution. This can customize which tests to run given certain scenarios.

@Test
public void assumeThatFileSeparatorTest(){  
	// only execute the test if OS is Windows
	assumeTrue(File.separatorChar == '\\'); // TEST
}

Disable

@Disable is used to disable a test case:

  • useful to ignore test cases
class DisabledTestsDemo {

	@Disabled("Disabled until fault #42 has been resolved") 
	@Test  
	void testWillBeSkipped() {  
	}
	
	@Disabled("Disabled until feature #23 is implemented") 
	@Test  
	void testWillBeExecutedLater() {  
	}

// ...

}

Assert

Very common to use Assert libraries in tandem with Junit:

Linked from Software Testing and Debugging