Kernel debugging using KGDB

KGDB is full integrated with the latest kernel. These instructions are for linux-2.6.31.12. Not sure if these will work for any previous or later versions.

1. make menuconfig and select the following
1. Select Kernel Hacking → KGDB: kernel debugging with remote gdb
1. Select KGDB: use kgdb over the serial console
2. Build the kernel
    make bzImage; make modules; sudo make modules_install
3. Install the kernel. Copy bzImage, System.map and .config into /boot
4. On another machine (say remote) where the serial null modem cable is connected, copy the entire kernel folder that you compiled on the target machine. 
5. Restart the target machine.  At the grub menu, select your kernel, edit it and add the following at the end of the kernel line:
    kgdboc=ttyS0,115200 kgdbwait
   This is assuming that you have a serial port and it is connected on ttyS0. Boot into your kernel. It will wait for gdb to connect.
6. On the remote machine, go in the kernel folder you copied over and debug vmlinx by starting gdb
1. gdb ./vmliux
2. Set the baud rate - (gdb) set remotebaud 115200
3. Connect to the target - (gdb) target remote /dev/ttyS0
4. Run the kernel - (gdb) continue

For more details, refer to the docbook on KGDB in the kernel documentation folder. Just do a make htmldocs to convert into a readable format. You would have to install xmlto first. sudo aptitude install xmlto should help.

Force fsck on reboot

If you want to force an fsck check on your drives on reboot do the following:

sudo touch /forcefsck
sudo shutdown -r now

Serial console debugging

Setting up the serial console

1. First find which devices are attached to the system

dmesg | grep "tty"

2. For the device found above (say ttyS0), create the /etc/event.d/ttyS0 file with the following contents

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from
# the point the system is started until it is
# shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty -L 115200 ttyS0 vt102

3. Edit /etc/securetty and add ttyS0

4. During reboot edit the kernel in grub menu (press 'e'). At the end of the kernel line add console=ttyS0,115200n8 tty1

5. Boot into your kernel

6. Once you log in, if you want all the dmesg output to go on the serial console do the following:
sudo tail -f /var/log/kern.log > /dev/ttyS0

Setting up the client

1. Open minicom and set the port as the one you are talking on, it could be ttySn or ttyUSBn depending on if you are using a serial port or a USB to serial converter. (e.g. ttyS0, ttyUSB0)

2. Set the setting as 115200 baud, 8 bits, 1 stop bit, no parity, no flow control

Update: There is another program called gtkterm if you don't fancy the command line applications. Also I would recommend using konsole to run minicom within as it supports unlimited buffer, which can be very useful if you are trying to look at the /var/log/kern.log of the target machine, for example.

Update (Apr 20, 2010): Only if you want to have a serial console (like a bash on the serial port) should you do the getty for ttyS0. If you just want all the kernel message sent to the terminals just do the following:

 # echo 8 > /proc/sys/kernel/printk

Read this article for more details on kernel oops.