Intro, Goal and Psychology of Software Testing

terminology

Fault: static defect in the software

• analogy: the doctor tries to diagnose the root cause, the ailment
• Example: i is initialized to the wrong value in the code

Error: an incorrect internal state. This can be caused by the fault.

• analogy: the doctor looks for anomalous internal conditions (like high blood pressure, irregular heartbeat, bacteria in blood stream)
• Example: i is 1 instead of 0

Failure: the external incorrect behavior with respect to the requirements or expected behavior.

• Analogy: a patient gives a doctor a list of symptoms

Bug: an informal term used in place of any of the above

• comes from Grace Hopper finding a moth on the relays of a computer she was working

Debugging: given a failure, find or fix the fault

why do we test

Finding bugs is a crucial activity because it can have costly losses

• mars climate orbiter resulted in loss of millions of dollars
• Thread-25 radiation therapy machine killed 3 patients
• Arian 5 rocket exploded because a 64-bit floating-point number relating to the horizontal velocity of the rocket with respect to the platform was converted to a 16-bit signed integer causing a data overflow. Cost $7 billion.
• Northeast blackout of 2003. Alarm system didn’t work. Caused 11 deaths and $6 billion.
• Toyota Prius ABS brake software recall
• Korean education software fault miscalculated grades effecting government investigation
• NIST published a report saying that software defects cost the US economy $60 billion a year.

The cost of fixing a problem increases drastically the later in the development phase.

psychology of testing

The goal of software testing: to produce high quality software by revealing and repairing as many faults as possible.

• We want to show the presence of bugs, not it’s absence.
• Showing correctness is impossible because there is a potentially infinite number of possible test input values. There is also a psychological issue that we are inclined to write tests that work fine rather than ones that produce failure. There is also a question of what is “correct.”

Exhaustive path testing is the execution of all possible paths of control flow graph. This is not the right answer because the program can still be loaded with faults even if we cover all possible execution paths. This can include missing paths or data sensitivity.

Testing is a destructive (even sadistic) process.

• we increase the confidence about our software by diligently searching for faults.

difficulties

Pesticide paradox: every strategy you use to find bugs leaves a residue of subtler bugs against which those strategies are ineffectual.

Testing is context-dependent

Absence-of-errors fallacy: testing software thoroughly but for wrong requirements

verification vs validation

Verification: technical, does not reveal flaws in requirement/specification Validation: depends on domain knowledge, exposes flaws in specification and requirement, deficiencies and discrepancies.