property based testing vs example-based testing

Example-based Testing

import java.util.*;  
import org.assertj.core.api.Assertions; import org.junit.jupiter.api.Test;

class ListReverseTests { 
	@Test
	void reverseList() {  
		List<Integer> aList = Arrays.asList(1, 2, 3);
		Collections.reverse(aList);
		Assertions.assertThat(aList).containsExactly(3, 2, 1);
	}
}

But how can I be confident that reverse also works with different values, different number of elements, empty, list, elements of different types, etc.

These problems are solved with input space paritiioning, boundary value analysis, error guessing, structural coverage (i.e., whitebox testing), etc.

Property-based testing

Abbreviated as PBT

We can approach the question of correctness from a different angle: under what preconditions and constraints should the functionality under test lead to which postconditions (results from computation)? Which invariants should never be violated in the course?

A property is a combination of preconditions and qualities that are expected to be present in the code under test.

Using the "reverse" function as an example:

• for any given list of elements, applying reverse twice should result in the original list.
• for any given list of elements, applying reverse must result in a list of the same size
• for any given list of elements, reverse of the list contains the same exact set of elements

Another example for isPrime(int n):

• n must be a postiive value if prime
• n must be an odd number if it is prime unless it is 2
• if n is negative, then it is not prime.

The two main steps in PBT:

1. finding general and desired properties for functions components and whole programs.
2. finding concrete inputs to falsify those properties by the randomized generation of test data

Ways to implement PBT: QuickCheck (Haskell), Jqwik (Java)

How to find properties?

• business rule:
  • E.g., for all customers with a yearly turnaroudn grerater than X $we give an additional discount of Y percent, if the invoice amount is larger than Z$.
• Inverese functions: applying the function first and the inverse function second should return the original.
  • E.g., adding and subtracting a fixed amount must not change a numeric value.
• Idempotent functions: multiple application of an idempotent function should not change results
  • E.g., ordering a list multiple times, shouldn't change it after the first time.
• Invariant functions: some properties of code must be true at all times.
  • E.g., sorting/maping should never change the size of a collection.
• Fuzzing: code should never explode, even if you feed it with lots of diverse and unforeseen input data.
  • No exceptions occur, at least no expected ones, there are no 5xx return codes for HTTP requests; maybe you even require 2xx status all the time, all return values are valid, runtime is under an aceptable threshold

Pros:

• gives us a new perspective to look at testing
• relieves us from (some of) the burden of finding concrete inputs, corner cases, and error guessing

cons:

• non exactly testing the actual output
• brings a flair of indeterminism that random generation brings into the game.

For All Annotation

@Forall

• adding parameters and annotating them with @ForAll tells jqwik that you want the framework to generate concrete random input tests for you.
• add various annotations along with @Forall like @StringLength (min = 1, max = 10) String string 1, @StringLength String string 1, @IntRange(int min=0, int max=Integer.MAX_VALUE) int anInt, etc.

Arbitrary

Arbitrary<T> is an interface that can be used to randomly choose a value from among a set of possibilities.

Provide

For parameter provisioning

@Property  
boolean concatenatingStringWithInt(@ForAll("shortStrings") String aShortString, @ForAll("10 to 99") int aNumber ) { 
	String concatenated = aShortString + aNumber;
	return concatenated.length() > 2 && concatenated.length() < 11; 
}

@Provide
Arbitrary<String> shortStrings() {  
	return Arbitraries.strings().withCharRange('a', 'z’).ofMinLength(1).ofMaxLength(8);
}

@Provide("10 to 99") Arbitrary<Integer> numbers() {
	return Arbitraries.integers().between(10, 99); 
}

Edge cases

@Example

void printEdgeCases() {  
	System.out.println(Arbitraries.integers().edgeCases()); 
	System.out.println(Arbitraries.strings().withCharRange('a', 'z').edgeCases()); 
	System.out.println(Arbitraries.floats().list().edgeCases());
}

EdgeCases[-2, -1, 0, 2, 1, -2147483648, 2147483647]  
EdgeCases["a", "z", ""]  
EdgeCases[[], [0.0], [1.0], [-1.0], [0.01], [-0.01], [-3.4028235E38], [3.4028235E38]]

Assumptions

@Property
boolean absOfNegativeNumbers(@ForAll @IntRange int a) {
	Assume.that(a>0);
	return a ==StringNIntUtil.abs(a);
}

Finding Properties

Business rules

• E.g., for all customers with a yearly turnaround greater than x, we give an additional discount of y percent if the invoice amount is larger than z.

Inverse function: applying the function first and the inverse function second should return the original

• E.g., adding and subtracting a fixed amount must not change a numeric value

Idempotent functions: some properties of code must be true at all times:

• E.g., sorting/mapping should never change the size of a collection.

Invariant functions: some properties of code must be true at all times.

• E.g., sorting/mapping should never change the size of a collection

Fuzzing: code should never explode, even if youf eed it with lots of diverse or unforeseen input data

• no (unexpected) exceptions should occur
• No 5xx return code for HTTP requests
• all return values are valid
• runtime is under an acceptable threshold

Summary

Property base testing pros:

• give us a new perspective to look at testing
• relieces us from (some of) the burden of finding concrete inputs, corner cases, and error guessing

Cons:

• not exactly testing the actual output
• brings a flair of indeterminism that random generation brings into the game.