Debugging

Created Apr 3, 2023

Debugging

Testing looks for inputs that cause failures (coverage criteria, test generation, test oracle), but what happens when we do find a program failure? We need to debug.

failures are typically discovered by tests and real users.

Debugging steps:

Reproduce the error, understand
Isolate and minimize (shrink) simplification
eyeball the code, where/what could it be? reason backwards
devise and run an experiment to test your hypothesis
repeat 3 and 4 until you understand what is wrong
fix the bug and verify the fix
create a regression test.

Debugging

Debugging is a search in space (a given state of the program) and time (all the states that the program goes through) to find and resolve the faults.

Each single program state may involve a large numberof variables
A program may pass through millions of states before failure occurs.
This may seem like searching for a needle in endless rows of haystack.

A failure comes to be in three stages

the programmer makes a fault by creating a defect in the code
the defect causes an infection (i.e., incorrect state or error)
the infection causes a failure, an externally visible error.

Note that not every defect results in an infection, and not every infect results in a failure.

Automation

Automate debugging with a variety of tools and techniques:

program slicing
observing and watching state (logging, using debugger tools)
asserting invariants
detecting anomaies
isolating cause-effect chains

Debugging initiation when an automated test case causes a failure or a user experiences a failure and submits a "bug report"

Debugging by Observation

Debugging by Observation: determine facts based on what has happened in a concrete run

know what to observe and when to observe ina systematic way
techniques include: logging, interactive debugging, postmortem debugging.

Some observation principles:

proceed systematically: dont observe values at random, search scientifically -> develop hypothesis
know what to observe and when to observe: a program run is a long sucession of huge program states (i.e., large number of variables), so it is impossible/impractical to observe everything all the time.
do not interfere: whatever you observe should be the effect of the original pogram run rather than an effect of yur observation.

How to observe software state: log the execution

general idea: insert output statements at specific places in the program.
called "prinln debugging"
issues: cluttered code (serves no purpose other than debugging), cluttered output (large amount of output gets interleaved with ordinary output, so it's recommended to designate a separate channel for logging), slowdown (huge amount of logging can slow down the program), loss of data (for performance reasons, outputs are buffered before being outputted, if the program crashes, data is loss, do not buffer or buffer less frequetly -> slowdown).

To better log:

make a dedicated function to log the debugging-related messages
- e.g., write a function called dprintln(String) and call that when logging for debugging.
Turn the function off when releasing/in production
- calculating arguments and calling an empty function is still costly (so dont just make the body empty)
- in langauges that support macros like C/C++, this is not an issue.
Better yet, use "logging libraries"
- first and foremost advantage of any logging API over plain System.out.println is its ability to disable certain log statements while allowing others to print unhindered.
- tools available are Log4j, Log4net, Log4c, etc.

Tools

Apache Log4j 2 is a full-fledged logging framework that offers more functionality than java.util.loggign. Many open-source applications use log4j.

It has log levels: All < Trace < Debug < Info < Warn < Error < Fatal < Off

SLF4J is the Simple Logging Facade for Java. It acts as a facade for different logging frameworks like jav.util.logging or Log4J 2.

The underlying logging framework can be plugged in at runtime

Aspect

Aspect Oriented (AO) Programming is a programming paradigm that aims to increase modularity by allowing the separation of cross-cutting concerns from the core concern.

Separates concerns into individual syntactic entities (aspects)
Aspect code (advice) is woven into the program code at specific places (join points)
the same aspect code can be woven into multiple places (pointcuts).
- e.g., log all function calls when the function's name begins with "set".

Logging is a cross-cutting concern as it does not have anything to do with the logic of the program (logging is a separate concern).

Impelment "logging" as a separate aspect that cuts through points of interest in the ode and logs events/activities/variable values/etc. of interest Other cross-cutting concerns (i.e,. aspects) might be security, data validation, authentication system, synchrnization, optimizations, etc.

Debuggers

Observation Tools (Debuggers) allow

Getting started fast- without altering the program code at hand.
Flexible observation of arbitrary events.
Transient sessions-no code it written

This is compared to typical logging- which requires writing and integrating extra code into the program.

Debuggersan external observer tool that hooks itself into the execution of the program and obesrves (and possibly manipulates) the state of the program. Its functionalities include:

executing the program and making it stop under specific conditions
observe the state of the stopped program
change the state of the program

Before you start a debugging session using a debugger:

develop a hypothesis(es), or explanations for what might be wrong
make note of parts of the code and variables that are involved (what should be investigated) based on your hypothesis(es).
Decide on particular points of interest in the program where you like to stop and check things out.
- A breakpoint is the location at where the program reaches and will stop (i.e.,l hands over the control to the debugger), giving you the chance to check things out.

Simplification

Simplifying: once one has reproduced a problem, one must find out what's relevant

Example questions to find relevancy
- Does the problem really depend on 10,000 lines of input?
- Does the failure really requier this exact schedule
- Do we need this sequence of calls?
E.g., if an airplane crashes, find out if it still crashes if you remove the passenger seats, or remove the coffee machine, or remove the engines.

For every circumstance of the problem, check whether it is relevant for the problem to occur. If it is not, remove it from the problem report or the test case in question. Any aspect that may influence a problem is a circumstance.

aspects of the problem environment
individual steps of the problem history

Strategy to simplify: Simplifying by experimentation

by experimentation, one finds out whether a circumstnace is relevant or not
omit the circumstance and try to reproduce the problem
the circumstance is relvant iff the problem no longer occurs.

Strategy to simplify: binary search

proceed by binary search - throw away half the input and see if the output is still wrong
if not, go back to the previous state and discard the other half of the input
benefits of this strategy:
- ease of communiation ("this tag causes a crash")
- easier debugging (focus on the piece of code that renders that tag)
- identify duplicate (check to see if other cases are tag related too)

Simplification is a slow and boring task, so we can set up an automatic test to figure out if the occurs or not. Process of the automated test for checking if the browser Mozilla crashes or not:

launch mozilla
replay (previously recorded) steps from the problem erport
wait to see whether
- mozilla crashes (test fails)
- mozilla still runs (test passes)
- niether (unresolved)

What happens if both halves pass?

increase the granularity, i.e., break the input into smaller pieces.

Example: public static int checkSum(int[] a) is supposed to compute the checksum of an integer array. However, it gives the wrong results whenever "a" contains two identical consecutive numbers (but we don't know that yet).

We have a failed test case from the protocol transmission {1, 3, 5, 3, 9, 17, 44, 3, 6, 1, 1, 0, 44, 1, 44, 0}
You can use binary search to narrow down the indicies of the array to find out what part is causing the error. You'll then land no {1,1}

ddmin algorithm

Let c be a failing input configuration (sequence of individual inputs)

test(c) runs a test on c with possible outcome PASS or FAIL.
n is the number of chunks to split c into (initially n=2). We will remove one chunk at a time and test the remaining input.

ddMin(c,n):

if |c| = 1 return c otherwise systematically remove one chunk c_i at a time. Test the reamining input c \ c_i
if there exists some c_i such that test (c \ c_i) = FAIL return ddMin (c \ c_i, max(n-1, 2))
else, if n < |C| return ddMin(c, min(2n, |C|))
else: (can't split into smaller chunks) return c.

This technique is an instance of delta debugging, which is an approach to isolate failures caused by narrowing down differences (delta) between runs. It can be applied to various types of input such as

failure-inducing program input, e.g., HTML page
failure-inducing user interactions e.g., the key/mouse strokes that make a program crash
failure-inducing changes to the program code, e.g., after a failing regression test
etc.

(Delta Debugging gives an alternate mindset to the debugging problem. Normally, when something does not work as expected (i.e., you have a “failure”), you would naturally think: “hmmm, what’s wrong here?”. Delta Debugging takes an alternate approach: “what could NOT be wrong here?” In other words, “what is irrelevant here?” so that I can exclude those parts and put them aside to simplify things)

Linked from Software Testing and Debugging