Debugging

Kernel Oops

Sometimes a running Linux kernel detects an error during runtime that can’t be corrected. This usually generates a stack trace that points to the location in the code that caused the oops.

Linux kernels in Gluon (and OpenWrt) are stripped. That means they do not contain any debug symbols. On one hand this leads to a smaller binary and faster loading times on the target. On the other hand this means that in a case of a stack trace the unwinder can only print memory locations and no further debugging information.

Gluon stores a compressed kernel with debug symbols for every target in the directory output/debug/. These kernels should be kept along with the images as long as the images are in use. This allows the developer to analyse a stack trace later.

Decoding Stacktraces

The tooling is contained in the kernel source tree in the file decode_stacktrace.sh. This file and the needed source tree are available in the directory:

openwrt/build_dir/target-<architecture>/linux-<architecture>/linux-<version>/

Note

Make sure to use a kernel tree that matches the version and patches that was used to build the kernel. If in doubt just re-build the images for the target.

Some more information on how to use this tool can be found at LWN.

Obtaining Stacktraces

On many targets stack traces can be read from the following location after reboot:

/sys/kernel/debug/crashlog