netconsole

Yet another mechanism to get the kernel debug messages is netconsole which sends the messages over Ethernet. You would not be able to get those early printks or crashes but it is useful nonetheless.

Setting up netconsole is fairly easy. You can either pass the netconsole parameters at the boot time or enable it after the machine has booted using modprobe.

Setting netconsole using boot parameters

Add the following parameter to the kernel boot string

netconsole=[s-port]@[s-ip]/[dev],[t-port]@[t-ip]/[t-mac]

s-port = source port, default is 6665

s-ip = source IP address

dev = device (eth0, eth1 ...)

t-port = target port, default is 6666

t-ip = target  IP address

t-mac = target MAC address

For example:

netconsole=1234@10.1.1.152/eth0, 1235@10.1.1.153/11:22:33:44:55

Setting netconsole using modprobe

It is preferred to use modprobe instead of insmod

$ sudo modprobe netconsole netconsole="@/,@t-ip/"

For example:

$ sudo modprobe netconsole netconsole="@/,@10.1.1.153/"

Target (receiver) side 

Use netcat to get the messages

$ nc -l -u [t-port]

The full documentation on using the netconsole can be found in the "Documentation/networking/netconsole.txt" file in the kernel folder.

Online resources -  1, 2, 3 (some of these might be dated ... )

Kernel debugging using KGDB

KGDB is full integrated with the latest kernel. These instructions are for linux-2.6.31.12. Not sure if these will work for any previous or later versions.

1. make menuconfig and select the following
1. Select Kernel Hacking → KGDB: kernel debugging with remote gdb
1. Select KGDB: use kgdb over the serial console
2. Build the kernel
    make bzImage; make modules; sudo make modules_install
3. Install the kernel. Copy bzImage, System.map and .config into /boot
4. On another machine (say remote) where the serial null modem cable is connected, copy the entire kernel folder that you compiled on the target machine. 
5. Restart the target machine.  At the grub menu, select your kernel, edit it and add the following at the end of the kernel line:
    kgdboc=ttyS0,115200 kgdbwait
   This is assuming that you have a serial port and it is connected on ttyS0. Boot into your kernel. It will wait for gdb to connect.
6. On the remote machine, go in the kernel folder you copied over and debug vmlinx by starting gdb
1. gdb ./vmliux
2. Set the baud rate - (gdb) set remotebaud 115200
3. Connect to the target - (gdb) target remote /dev/ttyS0
4. Run the kernel - (gdb) continue

For more details, refer to the docbook on KGDB in the kernel documentation folder. Just do a make htmldocs to convert into a readable format. You would have to install xmlto first. sudo aptitude install xmlto should help.

Serial console debugging

Setting up the serial console

1. First find which devices are attached to the system

dmesg | grep "tty"

2. For the device found above (say ttyS0), create the /etc/event.d/ttyS0 file with the following contents

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from
# the point the system is started until it is
# shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty -L 115200 ttyS0 vt102

3. Edit /etc/securetty and add ttyS0

4. During reboot edit the kernel in grub menu (press 'e'). At the end of the kernel line add console=ttyS0,115200n8 tty1

5. Boot into your kernel

6. Once you log in, if you want all the dmesg output to go on the serial console do the following:
sudo tail -f /var/log/kern.log > /dev/ttyS0

Setting up the client

1. Open minicom and set the port as the one you are talking on, it could be ttySn or ttyUSBn depending on if you are using a serial port or a USB to serial converter. (e.g. ttyS0, ttyUSB0)

2. Set the setting as 115200 baud, 8 bits, 1 stop bit, no parity, no flow control

Update: There is another program called gtkterm if you don't fancy the command line applications. Also I would recommend using konsole to run minicom within as it supports unlimited buffer, which can be very useful if you are trying to look at the /var/log/kern.log of the target machine, for example.

Update (Apr 20, 2010): Only if you want to have a serial console (like a bash on the serial port) should you do the getty for ttyS0. If you just want all the kernel message sent to the terminals just do the following:

 # echo 8 > /proc/sys/kernel/printk

Read this article for more details on kernel oops.

Threading and parallelism

Another good series of articles found on www.embedded.com on threading and parallelism

Part 1: Parallelism and threading primer
Part 2: The threading development cycle
Part 3: Debuging and tuning multi-threaded code

Kernel debugging tools

While working inside the kernel you are prone to crashes, oops and complete kernel freeze. Some of these tools\methods help

1. lockdep
http://www.mjmwired.net/kernel/Documentation/lockdep-design.txt

2. netconsole
http://www.mjmwired.net/kernel/Documentation/networking/netconsole.txt

Using netconsole inside Ubuntu
https://wiki.ubuntu.com/KernelTeam/Netconsole

Another netconsole tutorial
http://www.cyberciti.biz/tips/linux-netconsole-log-management-tutorial.html

3. Debugging kernel Oops
https://wiki.ubuntu.com/DebuggingKernelOops

4. Using the built-in kernel debugger
http://oss.sgi.com/projects/kdb/
IBM tutorial (http://www.ibm.com/developerworks/linux/library/l-kdbug/) .. looks dated .. not sure if this works with 2.6.24+ kernels

5. Nice article on Kernel_Debugging_Tips

6. Another brilliant article on Kernel oops

Update: I recently posted a piece on how to use KGDB over the serial null modem cable

oprofile

oprofile is very helpful in getting a trace out of the kernel especially to know the %age usage of methods from the kernel and the application. Helps in debugging some weird problems.

First, enable oprofile in the kernel, it is good to build it part of the kernel rather than a kernel module, so make sure in the menuconfig option it is a [*] rather than [m].

You can install the oprofile client once you boot back into your file system.
sudo apt-get install oprofile oprofile-gui

Before starting your application, initialize
$ sudo opcontrol --init

The following two options can be done once in the beginning

Tell oprofile the path to your compiled and unstripped vmlinux file so that it can pick the symbols from there
$ sudo opcontrol --vmlinux=/path/to/kernel/image

Set the call stack depth
# sudo opcontrol --callgraph=#depth

Now check the status,
$ sudo opcontrol --status

Reset the earlier dump,
$ sudo opcontrol --reset

Start the profiler
$ sudo opcontrol --start

Run your application
$ ./mystupidapp

All done, stop the profiler
$ sudo opcontrol --stop

Get the dump
$ sudo opcontrol --dump

Now, get the report
$ sudo opreport -l (and various options if you want)

Clear it,
$ sudo opcontrol --reset

Deinit,
$ sudo opcontrol --deinit

and start again if you want to.

I use alias(es) in the .bashrc file. Really helps!
alias opc='sudo opcontrol'
alias opcdeinit='opc --deinit'
alias opcdump='opc --dump'
alias opcinit='opc --init'
alias opcreset='opc --reset'
alias opcstart='opc --start'
alias opcstatus='opc --status'
alias opcstop='opc --stop'
alias opr='sudo opreport -l'
alias oprcall='opr -c'