Using watchdogs is an excellent practice. It makes the system more stable by forcing it to reboot in case of a hang. But why did the hang actually happen in the first place? Watchdog itself doesn’t answer this question. Its job is simple: when there is no Keep Alive signal for some period of time, it resets the HW. If you would install the sysstat package, you will probably see the latest entry in its report. However, it will probably not be so helpful because that entry was logged earlier than the watchdog reset, and the system status has changed since then. So, what could be done to get more info?
Modern watchdog drivers support a two-stage mode. In this mode, they trigger a kernel panic (using an interrupt) after the first timeout and then reset after the second one. When the kernel panics, we can get some information about the state of the running kernel using the KDUMP kernel feature. To use this mode, see the driver code, but usually, it is just about passing an additional kernel option that will look like that: wd_name.action=1.
Another issue that can be addressed with KDUMP is accessing kernel panic information after reboot. When it happens, we can see the output on the terminal (usually serial, in embedded devices), but after some time, it goes away and leaves no trace. The panic mechanism is quite simple: the kernel cannot be sure that, for example, the filesystem or MMC works. So it will not dump any kernel panic information to log files. This situation is inconvenient and requires a solution. One of them is using KDUMP.
KDUMP is a feature that makes it possible to collect some information about the kernel state after it panics (kernel dump). These kernel dumps can be inspected later with special tools.
First of all, we need to configure our kernel to support this feature by enabling some configuration options in the kernel config. It can vary depending on the kernel version, but usually, you need to enable at least the following ones: CONFIG_KEXEC, CONFIG_SYSFS, CONFIG_CRASH_DUMP, CONFIG_PROC_VMCORE, CONFIG_PROC_KCORE.
Using KDUMP also requires reserving some additional memory for crashkernel. Crashkernel is a second instance of the kernel (it can be the same kernel image as the primary one) that runs when panic happens and collects the dump. The amount of memory varies from system to system. Usually, it’s not too big (in most cases, 128M will be enough). To reserve this memory, use crashkernel=128M as an additional kernel parameter. Be aware that this memory cannot be used from a primary kernel anymore.
Also, we need to add some additional packages to our image: kexec, kdump. Kexec is the main utility that loads a kernel image to the memory that was reserved for the crashkernel, and kdump is a service that manages this whole thing and makes it automatic. Also, kernel dumps could be huge, and we need something that will make them smaller using compression or stripping them from some information we don’t really need (like empty memory pages). Luckily, there is a makedumpfile utility that was designed to do so.
We also have the /etc/kdump.conf configuration file that usually comes with the kdump package. There we can tune what kernel image and command line parameters to use for crashkernel. For example, by tweaking KDUMP_CMDLINE, we can blacklist some drivers we don’t actually need for a crashkernel (modprobe.blacklist=module_name). Also, there we can change the level of details going to be logged by changing MAKEDUMPFILE_ARGS.
After setting everything up, we can check if KDUMP is working by using Magic IRQ:
echo c > /proc/sysrq-trigger
Then you should see how crashkernel is starting and collecting the dump.
To check if crashkernel starts by watchdog stage 1, we can kill the watchdog as follows:
kill -s 9 `pidof watchdog`
This command does not stop the hardware watchdog, just a feeding process. So, we emulate a freeze. Then wait till the timeout happens.
The dump will be saved in KDUMP_VMCORE_PATH(/etc/kdump.conf). Usually, it is /var/crash. For additional tuning, you may need to see the kdump initscript (/etc/init.d/kdump).
To debug kernel dumps, we need the vmlinux.bin file for the corresponding kernel and crash utility. Also, it does not support cross-debugging, so when we want to work with the dump the host should be at least of the same architecture and endianness.
crash /path/to/vmlinux /path/to/vmcore
For additional information, please check:
Documentation/kdump/kdump.txt (Linux Kernel Sources, depends on kernel version)
Happy dumping!
