kernel, linux

Debugging the Linux Kernel

This is a little compilation of some useful configuration options and basic methods to debug a Linux kernel. There is good documentation in kernel.org and elsewhere but back when I was learning these things I felt that there are few places where you can get a high level overview. We will go through more advanced methods of profiling and tracing in another post.

It is assumed that we already know how to compile a kernel, use menuconfig and what kernel options are.

Basic kernel configuration

 

First, there are a few options that we want to activate in order to have them available in our running kernel. Some of them will have a performance toll and won’t be present in a production system. Others will just increase kernel size. You can get an overview of what is available in Kconfig.debug, and also you might find this kernel symbol reference handy.

DEBUG_KERNEL

This option exists specifically to unhide debugging options in menuconfig.

DEBUG_INFO
The resulting kernel image will include debugging info resulting in a larger kernel image. This adds debug symbols to the kernel and modules (gcc -g), and is needed if you intend to use kernel crashdump or binary object tools like crash, kgdb, LKCD, gdb, etc on the kernel.
DEBUG_BUGVERBOSE
When a program crashes due to an exception, or the kernel detects an internal error, the kernel can print a not so brief message explaining what the problem was. This debugging information is useful to developers and kernel hackers when tracking down problems, but mostly meaningless to other people. This is always helpful for debugging but serves no purpose on a production system.
BUG() panics will output the file name and line number of the BUG call as well as the EIP and oops trace. This aids debugging but costs about 70-100K of memory.
CONFIG_FRAME_POINTER

This option inserts code to into the compiled executable which saves the frame information in registers or on the stack at different points which allows a debugger such as gdb to more accurately construct stack back traces while debugging the kernel.

CONFIG_KALLSYMS

This will give us more information in stack traces from kernel oopses. The kernel will print out symbolic crash information and symbolic stack backtraces. This increases the size of the kernel somewhat, as all symbols have to be loaded into the kernel image.

If a kernel crashes, this option will allow us to get an output that includes function names and not only addresses so it will be much easier to discover what is going on. This is an example taken from the Ubuntu wiki

We can manually check the symbol to address map in /proc/kallsyms. This output is in nm format.

Also, it is easy to access this information from kernel code. We can just use  kallsyms_lookup_name()  from kallsyms.c.

CONFIG_KALLSYMS_ALL

Normally kallsyms only contains the symbols of functions for nicer OOPS messages and backtraces (i.e., symbols from the text and inittext sections). This is sufficient for most cases. And only in very rare cases (e.g., when a debugger is used) all symbols are required (e.g., names of variables from the data sections, etc).

This option makes sure that all symbols are loaded into the kernel image (i.e., symbols from all sections) in cost of increased kernel size (depending on the kernel configuration, it may be 300KiB or something like this).

CONFIG_IKCONFIG

This option enables the complete Linux kernel .config file contents to be saved in the kernel. It provides documentation of which kernel options are used in a running kernel or in an on-disk kernel. This information can be extracted from the kernel image file with the script scripts/extract-ikconfig and used as input to rebuild the current kernel or to build another kernel.

CONFIG_IKCONFIG_PROC

Enable access to .config through /proc/config.gz in a running kernel.

cmdline

We can check what parameters were used to invoke our running kernel in /proc/cmdline

These are normally provided by the bootloader (GRUB or uboot), but sometimes those are locked so we can use CONFIG_CMDLINE and CONFIG_CMDLINE_OVERRIDE to work around this when we are building a kernel.

  • CONFIG_CMDLINE. Enter arguments here that should be compiled into the kernel image and used at boot time. If the boot loader provides a command line at boot time, it is appended to this string to form the full kernel command line, when the system boots.
  • CONFIG_CMDLINE_OVERRIDE. have the kernel ignore the boot loader command line, and use ONLY the built-in command line. This is used to work around broken boot loaders.

Analyzing running modules

printk

Probably the most basic means of debugging is reading printk output. There are several logging levels, and not all of them are active by default. Values go from emergency which is level 0 to debug which is level 8, with the default being warning which is level 4.

We can activate maximum verbose logging with

We can check the current logging level with

We can only see debug messages ( level 8 ), if we compile our kernel with CONFIG_DEBUG. Note that if we are generating too much logging we can severely affect system performance, specially if we are sending kernel logs through slow UART. For this situations, it is a good idea to use printk_ratelimited()  or printk_once() if we are trying to introduce printk statements in places where they can get called too often, like the network packet stack.

For instance,

will result in many iterations being suppresed with a message like

an example of printk_once()  could be

See printk.h and this article for more details.

Dynamic debug

Using pr_debug() globally will cause a tremendous amount of logging so it is not very practical. To make the debug level more manageable, dynamic debugging was introduced.

We can activate it with CONFIG_DYNAMIC_DEBUG. We can see in printk.h that dynamic_pr_debug()  will take over in this case.

In order to control what we want to print, a file will appear in dynamic_debug/control. We now have absolute control about what printk lines we want to be logged. We can hide ( -p ) or show ( +p ) log messages at the file level, function level or even line level.

Examples

We can also configure this at module load time with the dyndbg option, for example

CONFIG_DEBUG_FS

If we have this option selected, then we can mount debugfs like so

debugfs is a virtual filesystem where kernel modules can offer debug information and sometimes even receive specific configuration options through the hooks defined in debugfs.h.

For instance, if we are compiling Zswap with DEBUG_FS we can check different status parameters such as

kgdb

This is really powerful. We can use good old gdb to analyze our kernel. We can set breakpoints, see memory contents and all the rest. We will need gdb or some cross-gdb if we are working with a different architecture, for example an embedded ARM board.

In our kernel we will want to activate

  • CONFIG_KGDB. Include in-kernel hooks for kgdb, the Linux kernel source level debugger
  • CONFIG_KGDB_SERIAL_CONSOLE. Share a serial console with kgdb. Sysrq-g must be used to break in initially.
  • CONFIG_MAGIC_SYSRQ . This will activate the magic SysRq Key.

First we configure kgdboc ( kgdb over console ) with the tty and baud rate of our UART.

Next, we tell our kernel to enter in debug mode. This will freeze waiting for a gdb connection.

We will probably also want to activate kgdbcon in order to send printk statements to the gdb session.

Finally, we connect our (cross)gdb instance to the QEMU instance or real board. First, we run the debugger passing it the kernel that we are running, compiled with debug symbols (CONFIG_DEBUG_INFO). Then, we configure the serial port and attach with the target remote  command, just like when we use gdbserver .

We can also have the kernel wait for the gdb connection at early boot, but then we have to configure kgdboc in the kernel command line, and use the kgdbwait option. In this example we also are deactivating Kernel Address Space Layout Randomization so the memory locations are not messed up.

The kernel will be frozen until we connect with gdb. This will allow us to debug from early in the boot process if we need to.

kdb

kdb is a shell interface that lets us connect through a keyboard or serial console and inspect memory, registers and logs. It isn’t as powerful as kgdb in terms of inspecting execution following the source code. More details can be found in the kernel docs, but I recommend just sticking to kgdb.

CONFIG_PROC_KCORE

Kcore provides a virtual ELF core file of the live kernel accessible at /proc/kcore. This can be read with gdb and other ELF tools. No modifications can be made of the kernel memory using this mechanism.

This is very cool, because we can analyze a picture of the memory at the moment of invoking gdb. We need a copy of the running kernel compiled with debug symbols (CONFIG_DEBUG_INFO).

Unlike with kgdb, this is a limited simulation of the memory contents, and we cannot do fancy stuff such as control execution. Still we can access memory contents. For instance, if your kernel with symbols is vmlinux we would invoke gdb just like we would run it to analyze a core dump from any executable, but our core dump will be /proc/kcore.

We can refresh memory contents with

You can read a more detailed example in this article.

Author: nachoparker

Humbly sharing things that I find useful [ github dockerhub ]

One Commnet on “Debugging the Linux Kernel

  1. I’m working on bringing up a custom embedded linux board which uses the NXP i.mx6UL processor. I’ve got u-boot running, I have an image on my LCD (800×480 parallel LCD), and have built the kernel and device tree, and the kernel seems to load, but I’m getting stuck at the same spot no matter what I do: I’m monitoring via the serial port console output and after the kernel loads various drivers it stops at “Key type dns_resolver registered”. I’m running off of an SD card, and from everything I can tell, the sd card works fine (again, u-boot is running, and I was able to read and write to and from the card in my host system no problem).

    I really have no idea whether the kernel is hung or waiting on something or maybe the OS has fully come up but simply isn’t displaying on my LCD for some reason along with maybe not using the serial port any longer as the proper console port to show the shell prompt. I’m at a loss. Two thoughts I had: 1.) can I modify the kernel source to do something like toggle a gpio continuously just to show that the kernel isn’t hunh; 2.) add string output to the console to tell me what it’s currently working on; or 3.) use one or more of the debug methods you mention in this presentation, and if so which would you recommend first?

    Thanks in advance for your help.

Leave a Reply

Your email address will not be published. Required fields are marked *