Blackbox Testing

• Intro to Software Testing (ch1, ch2)
• The Art of Software Testing (ch1, ch2, ch4)

Blackbox vs Whitebox Testing

Blackbox testing views the software as a black box.

• Concentrate on "software specifications". Known as data-driven, io-driven, or specificaiton-based testing
• Draws soley from the specifications (e.g., formal specs, API docs, user manual, etc.)
• Exhaustive Blackbox testing is to try all possible inputs

Whitebox testing is concerned with the degree towhich test cases cover the source code of the software

• Glassbox or logic-driven testing
• Test cases drawn by looking at (and manipulating) sourcecode
• Exhaustive Whitebox testing is to try all execution paths

Greybox testing is used when there is noly partial access/understanding of the internal software fothe software under test (SUT).

• You know the algorithm, but the exact implementation
• You know the design or structure of the code, but not the exact implementation.
• Not exampled in this class.

Terminology

Equivalence Class

A subset of the form $\{x \in X: x R a\}$ where $a$ is an element of $X$ and the notation "x R y" is used to mean there is an equivalence relationship between $x$ and $y$.

• In other works, an equivalence class (or equivalence block) is the name that we give to the subset of $S$ which includes all elements that are equivalent to each other.
  • "Equivalent" is dependent on a specified relationship (i.e., characteristic), called an equivalence relation or characteristic.
  • If there exists an equivalence relation between any two element they're called equivalent.

Some examples of equivalence classes:

• $X$ is the set of all cars. ~ is the equivalence relation "has the same color as".
  • The equivalence classes consists of cars of different colors. E.g., the set of all red cars, the set of all blue cars, etc.
• $I$ is the set of all integer values. ~ is the equivalence relation "has the same sign as".
  • The equivalence classes consists of 1) the set of all negative integers, 2) zero, 3) the set of all positive integers.

To identify equivalence classes,

• Typically produced from specifications. Take the sentences/phrases about the input/output, to identify the characteristic(s) based on the specifed condition(s) and apply partitioning to produce the equivalence classees. Always produce both valid and invalid equivalence clases
  • Example: "the count should be range from 1-999 inclusive." Then, one valid equivalence class is: $1 \leq count \leq 999$ and two invalid equivalence classes are: $count > 999$ and $count < 1$
• If an input specifies a "must-be" situation, produce one valid and one invlaid equivalence class.
  • Example: "the first character of the string must be a digit." One valid equivalence class: string starts with a digit. One invalid equivalence class: stirng does not start with a digit.

Relation

Also known as characteristic.

Each partition is built based on a characteristic $C$. Some examples include:

• Object 'a' is null.
  • Two classes namely null and non-null
• Input devicetype
  • Multiple clases namely DVD, CD, VCR,...
• Shirt size
  • Multiple size classes namely xs, s, m, ...

Example:

• Input: text file $f$
• Characteristic: order of file $f$:
  • $b_1$: sorted in ascendng order
  • $b_2$: sorted in descending order
  • $b_3$: not sorted in any specific order

Blackbox Testing Techniques

Partitioning

Equivalence Partitioning

Partitioning domain into equivalence classes

1. Partition the input/output doman into a set of equivalence classes
2. Produce a representative concrete test case for each equivalence class.

Equivalence Partitioning is dividing into partitions of equivalent data. The idea is that:

• If a test case from an equivalence class detects an error/failure, so does all the other test cases of the same equivalence class
• Conversely, if a test case from an equivalence class does not detect an error/failure no other test cases of the same equivalence class does.

Given characteristic (i.e., relation) $C$, the partition $q$ defines a set of blocks (i.e., equivalence classes) over domain $D$: $q = b_1, b_2, ..., b_Q$

When selecting equivalence classes correctly, there are two important properties:

1. Disjointedness: blocks (i.e., classes) must be pairwise disjoint; that is no two blocks overlap: $b_i \cap b_j = \Phi, \forall i \neq j, b_i, b_j \in B_q$
2. Completeness: together the blocks cover entirety of the domain $D$: $\cup^{}_{b \in B_q}b=D$

Input Space Partitioning

Step for input space partitioning:

1. Design the characteristics to create partition(s) over the input/output domain

   • It is possible to design characteristics based on output!

2. Decide on the blocks (i.e., equivalence classes) for each partitioning characteristics.

3. Derive representative values for each block

Triangle Example

/**  
* decides the type of the triangle given the lengths of the three sides  
* @param a first length  
* @param b second length  
* @param c third length  
* @return an int indicating the type of the triangle: 0 is invalid,
          1 is scalene, 2 is isosceles, and 3 is equilateral */  
public static int triangleType(int a, int b, int c)

Let's partition over the output domain using the characteristic "Geometric Classification". The four classes are: scalene, isosceles but not equilateral, equilateral, invalid (technically, an equilateral is isosceles so you must be clear!

Boundary Analysis

Test conditions on bounds between equivalence classes.

Rationale:

• likely source of programming errors ($<$ vs $\leq$)
• software specifications may be fuzzy/vague about behavir on boundaries
• uncovers internal hidden limits on code.

Example:

• arrays must be sized no less than 1 and no longer than 10.
  • Three equivalence blocks: $size < 1$, $1 \leq size \leq 10$, $size > 10$
  • Try array sizes $0$, $1$, $10$, and $11$. Also try: MAX_INT, MAX_INT+1, MIN_INT, MIN_INT-1
• input condition specify that valid domain of an input is between $4-1$ and $1.0$.
  • Write test cases with input values: $-1.0$, $1.0$, $-1.001$, and $1.001$
• Specifies an input fiel can contain 1-255 records.
  • Write test cases for files with 0, 1, 255, 256 records
• Output condition specifes payroll software computes the monthly FICA deduction of minimum $0.00 and maximum of$1,165.25.
  • Try to write/invent test cases that might cause a negative deduction/deduction > $1,165.25

Error Guessing

No systematic way. Use intuition and experience to try to cause errors and failures in the system

• Try error-prone situations like zero for int values, null for objects, invalid inputs, out of bound inputs, empty sets/lists, sets/lists with one entry, inputs based on holes in specifications, negative inputs where they are not relevant

• Complementary to other testing techniques