Testing Principles, Oracle Problem, Test Automation

Testing Levels

We use the V Model for testing in different phases https://en.wikipedia.org/wiki/V-Model_(software_development)

The different testing levels are: Acceptance Testing: assess software with respect to user’s needs

• Alpha & Beta Testing System Testing: assess software with respect to architectural design and overall expected behavior Integration Testing: assess software with respect to sub-system design Unit Testing: assess software with respect to implementation of isolated modules/units

The cost per fault increases the further we are in the develpment.

RIPR

This is a fault/failure model.

The model includes a few stages: Reachability: The location or locations in the program that contain the fault must be reached Infection: The state of the program must be incorrect Propagation: The infected state must cause some output or final state of the program to be incorrect Reveal: The tester must observe part of the incorrect portion of the program state

Test Automation

There can be manual and automation testing.

Test Automation is the automation of testing-related activities.

• Generation: generate test cases automatically
• Execution: run tests on the software under test (SUT)
• Evaluation: evaluate test results i.e., does the test case pass or fail

Types of testing include:

• Unit testing
• System testing (e.g., testing google search engine with a query and check the set of results it should return)
• Security testing (e.g., a test case that simulates a SQL injection attack, how and what oracle would you write, what is the expected behavior)
• Usability testing (e.g., test gui for friendliness, was it user friendly enough)

Test Evaluation

Test Oracle: a mechanism for determining whether a test has passed or failed. An oracle can be

• Expected output value
• A program
• Documentation that gives specific correct outputs for specific given inputs
• A (human) domain expert who can tell whether test output is correct or not
• Any other way or combination of the above that can tell that output is correct or not

For example, If we use unit testing to check something like

assertEquals(4, sum(2,2))

• 4 is the hard coded oracle.

A “complete” oracle would be based on the entire final state (including internal) after running a test class. This is impractical/impossible.

• Internal meaning think of states are “wrong” but don’t affect the output.

Weak (“partial”) oracles are usually good enough in practice. These include

• checking expected output
• Checking for software crashes

The Oracle Problem: Making good oracles is one of the most challenging problems in software testing. It is not straightforward what the expected output is/should be. It requires knowledge of domain, UI, and psychology.

Testing Principles

A test case means the input and outputs together. This leads the Testing Principle 1: A necessary part of any test case is a definition of the expected output/behavior

Testing Principle 2: a test case must not have logic in it (no calculations). It must merely set things up, make calls, and verify the results.

Testing Principle 3: ideally, a programmer should avoid attempting to test her own program.

• Leave debugging to the original developer!

Testing Prinicple 4: faults are not uniformly distributed

• the probability of the existence of more errors In a section of a program is proportional to the number of errors already found in that section.

Testing principle 5: examining a program to see if it does what it is supposed to do is half the battle. The other half is seeing whether the program does not do what it is not supposed to do.

Some other testing principles include:

• test early
• Pesticide paradox
• Absence of error fallacy
• Testing is context dependent.

Side Note: a fundamental problem to equivalence relations in OO languages is that there is no way to extend an instantiable class and add a value component while preserving the equals contract.

• A reasonable workaround is to use composition in place of inheritance.