netconsole

Yet another mechanism to get the kernel debug messages is netconsole which sends the messages over Ethernet. You would not be able to get those early printks or crashes but it is useful nonetheless.

Setting up netconsole is fairly easy. You can either pass the netconsole parameters at the boot time or enable it after the machine has booted using modprobe.

Setting netconsole using boot parameters

Add the following parameter to the kernel boot string

netconsole=[s-port]@[s-ip]/[dev],[t-port]@[t-ip]/[t-mac]

s-port = source port, default is 6665

s-ip = source IP address

dev = device (eth0, eth1 ...)

t-port = target port, default is 6666

t-ip = target  IP address

t-mac = target MAC address

For example:

netconsole=1234@10.1.1.152/eth0, 1235@10.1.1.153/11:22:33:44:55

Setting netconsole using modprobe

It is preferred to use modprobe instead of insmod

$ sudo modprobe netconsole netconsole="@/,@t-ip/"

For example:

$ sudo modprobe netconsole netconsole="@/,@10.1.1.153/"

Target (receiver) side 

Use netcat to get the messages

$ nc -l -u [t-port]

The full documentation on using the netconsole can be found in the "Documentation/networking/netconsole.txt" file in the kernel folder.

Online resources -  1, 2, 3 (some of these might be dated ... )

Kernel debugging using KGDB

KGDB is full integrated with the latest kernel. These instructions are for linux-2.6.31.12. Not sure if these will work for any previous or later versions.

1. make menuconfig and select the following
1. Select Kernel Hacking → KGDB: kernel debugging with remote gdb
1. Select KGDB: use kgdb over the serial console
2. Build the kernel
    make bzImage; make modules; sudo make modules_install
3. Install the kernel. Copy bzImage, System.map and .config into /boot
4. On another machine (say remote) where the serial null modem cable is connected, copy the entire kernel folder that you compiled on the target machine. 
5. Restart the target machine.  At the grub menu, select your kernel, edit it and add the following at the end of the kernel line:
    kgdboc=ttyS0,115200 kgdbwait
   This is assuming that you have a serial port and it is connected on ttyS0. Boot into your kernel. It will wait for gdb to connect.
6. On the remote machine, go in the kernel folder you copied over and debug vmlinx by starting gdb
1. gdb ./vmliux
2. Set the baud rate - (gdb) set remotebaud 115200
3. Connect to the target - (gdb) target remote /dev/ttyS0
4. Run the kernel - (gdb) continue

For more details, refer to the docbook on KGDB in the kernel documentation folder. Just do a make htmldocs to convert into a readable format. You would have to install xmlto first. sudo aptitude install xmlto should help.

Serial console debugging

Setting up the serial console

1. First find which devices are attached to the system

dmesg | grep "tty"

2. For the device found above (say ttyS0), create the /etc/event.d/ttyS0 file with the following contents

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from
# the point the system is started until it is
# shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty -L 115200 ttyS0 vt102

3. Edit /etc/securetty and add ttyS0

4. During reboot edit the kernel in grub menu (press 'e'). At the end of the kernel line add console=ttyS0,115200n8 tty1

5. Boot into your kernel

6. Once you log in, if you want all the dmesg output to go on the serial console do the following:
sudo tail -f /var/log/kern.log > /dev/ttyS0

Setting up the client

1. Open minicom and set the port as the one you are talking on, it could be ttySn or ttyUSBn depending on if you are using a serial port or a USB to serial converter. (e.g. ttyS0, ttyUSB0)

2. Set the setting as 115200 baud, 8 bits, 1 stop bit, no parity, no flow control

Update: There is another program called gtkterm if you don't fancy the command line applications. Also I would recommend using konsole to run minicom within as it supports unlimited buffer, which can be very useful if you are trying to look at the /var/log/kern.log of the target machine, for example.

Update (Apr 20, 2010): Only if you want to have a serial console (like a bash on the serial port) should you do the getty for ttyS0. If you just want all the kernel message sent to the terminals just do the following:

 # echo 8 > /proc/sys/kernel/printk

Read this article for more details on kernel oops.

Deadline scheduling for Linux kernel

A new scheduler class has been proposed for including deadline based scheduling algorithms like EDF

• The SCHED_DEADLINE details
• News on OS-News
• The email archived on LKML

The public git repository is available on gitorious

git clone git://gitorious.org/sched_deadline/linux-deadline.git

Generating patches without cleaning code

Instead of generating a patch from two pristine copies of the code (one with your changes and both not compiled), it is useful to create a exclude-list of all the files that you want to ignore.

 .*
 *.lds
 *.o
 *.o.*
 *.a
 *.s
 *.ko
 *.so
 *.so.dbg
 *.mod.c
 *.i
 *.lst
 *.symtypes
 *.order
 *.elf
 *.bin
 *.gz
 *.lzma
 *.patch
 *.gcno
 tags
 TAGS
 bzImage
 vmlinux
 System.map
 Module.markers
 Module.symvers
 !.gitignore
 !.mailmap
 patches-*
 patches
 series
 exclude-list
 filefortag
 filelist
 cscope.*
 ncscope.*
 GPATH
 GRTAGS
 GSYMS
 GTAGS
 *.orig
 *~

Then create the patch

diff --exclude-from /path/to/exclude-list -urNd linux-2.6.24-hunterwala linux-2.6.24 > patch_file

Each line in exclude-list is a pattern which diff would match and ignore the files. I am having trouble excluding following directories.

include/asm
include/asm-*/asm-offsets.h
include/config
include/linux/autoconf.h
include/linux/compile.h
include/linux/version.h
include/linux/utsrelease.h
include/linux/bounds.h
include/generated

So, the -N passed to diff would treat these as new files and include them in the patch. :(

TODO: Update post once I find a better way to handle directories.

[The exclude list above has been taken from .gitignore]

Kernel debugging tools

While working inside the kernel you are prone to crashes, oops and complete kernel freeze. Some of these tools\methods help

1. lockdep
http://www.mjmwired.net/kernel/Documentation/lockdep-design.txt

2. netconsole
http://www.mjmwired.net/kernel/Documentation/networking/netconsole.txt

Using netconsole inside Ubuntu
https://wiki.ubuntu.com/KernelTeam/Netconsole

Another netconsole tutorial
http://www.cyberciti.biz/tips/linux-netconsole-log-management-tutorial.html

3. Debugging kernel Oops
https://wiki.ubuntu.com/DebuggingKernelOops

4. Using the built-in kernel debugger
http://oss.sgi.com/projects/kdb/
IBM tutorial (http://www.ibm.com/developerworks/linux/library/l-kdbug/) .. looks dated .. not sure if this works with 2.6.24+ kernels

5. Nice article on Kernel_Debugging_Tips

6. Another brilliant article on Kernel oops

Update: I recently posted a piece on how to use KGDB over the serial null modem cable

Script to install a new kernel

#!/bin/sh
#
# (c) Hunterwala

if [ $(/usr/bin/id -u) -ne 0 ]; then
     echo "run this script with sudo"
     exit 2
fi

if [ -z $1 ]; then
     echo "kernel version not found"
     echo "usage: ./install_kernel [kernel_version]"
     exit 2
else
     VERSION=$1
     echo "using $VERSION"
fi

if [ ! -d /lib/modules/$VERSION ]; then
     echo "kernel modules not installed"
     echo "run make install_modules from the kernel directory"
     exit 2
fi

BOOTDIR=/boot

echo "Installing kernel ........................"
cp $PWD/.config $BOOTDIR/config-$VERSION
cp $PWD/System.map $BOOTDIR/System.map-$VERSION
cp $PWD/arch/i386/boot/bzImage $BOOTDIR/vmlinuz-$VERSION
echo "done"

echo "Creating initrd image ...................."
update-initramfs -c -k $VERSION
echo "done"

echo "Updating grub ............................"
update-grub
echo "done"

echo "Linux kernel $VERSION successfully installed"
echo "Please restart and select $VERSION in grub menu"

Kernel module programming

Found a nice article on TLDP on Linux kernel module programming in the 2.6 kernel. (link)

64 bit division

If you are trying to do this inside the kernel:
unsigned long long arg1, arg2;
arg1 = arg1/arg2;

and you get this error:
Unknown symbol __udivdi3

You ought to do this instead:
unsigned long long arg1, arg2;
unsigned long temp = arg1;
do_div(temp, arg2);
arg1 = temp;


Linux Kernel design patterns

Linux kernel design patterns, part 1, part 2 and part 3.

oprofile

oprofile is very helpful in getting a trace out of the kernel especially to know the %age usage of methods from the kernel and the application. Helps in debugging some weird problems.

First, enable oprofile in the kernel, it is good to build it part of the kernel rather than a kernel module, so make sure in the menuconfig option it is a [*] rather than [m].

You can install the oprofile client once you boot back into your file system.
sudo apt-get install oprofile oprofile-gui

Before starting your application, initialize
$ sudo opcontrol --init

The following two options can be done once in the beginning

Tell oprofile the path to your compiled and unstripped vmlinux file so that it can pick the symbols from there
$ sudo opcontrol --vmlinux=/path/to/kernel/image

Set the call stack depth
# sudo opcontrol --callgraph=#depth

Now check the status,
$ sudo opcontrol --status

Reset the earlier dump,
$ sudo opcontrol --reset

Start the profiler
$ sudo opcontrol --start

Run your application
$ ./mystupidapp

All done, stop the profiler
$ sudo opcontrol --stop

Get the dump
$ sudo opcontrol --dump

Now, get the report
$ sudo opreport -l (and various options if you want)

Clear it,
$ sudo opcontrol --reset

Deinit,
$ sudo opcontrol --deinit

and start again if you want to.

I use alias(es) in the .bashrc file. Really helps!
alias opc='sudo opcontrol'
alias opcdeinit='opc --deinit'
alias opcdump='opc --dump'
alias opcinit='opc --init'
alias opcreset='opc --reset'
alias opcstart='opc --start'
alias opcstatus='opc --status'
alias opcstop='opc --stop'
alias opr='sudo opreport -l'
alias oprcall='opr -c'

initird

Nice (but a little old) article on initrd internals. Click here.

Viewing the initrd image

initrd is a RAM disk file image of the kernel used at the time of boot. On Ubuntu it is created using mkinitramfs or update-initramfs commands when a new kernel is installed. This image is a gunzip\cpio archive.

To see the contents do the following

1. Copy the initrd image which you want to open into a directory somewhere /home/node/temp. Let's say it is initrd-2.6.24-generic

2. Do the following now:

cat initrd-2.6.24-generic | gunzip | cpio -ivdm

OR,

bzcat initrd-2.6.24-generic | cpio -ivdm

your own system call in 5 easy steps

You *might* want to write your system call for various reasons

Assuming the path to your kernel source is "L". Create a new folder L/mysyscall. Inside the folder create the source file mysyscall.c and a Makefile

Step 1. Changing the System Table
L/arch/x86/kernel/syscall_table_32.S

Add your system call at the end of the file.

.long sys_new_system_call

Step 2. Changing the unistd.h
L/linux/include/asm-x86/unistd_32.h

Add your system call at the end of the existing list and append the next number

#define __NR_new_system_call XXX

Where XXX is the existing system call number plus 1. Also update the total system calls (as you just added another)
#define __NR_syscalls XXY
Where XXY is XXX+1

Step 3: Changing syscalls.h
L/include/linux/syscalls.h

Add the declaration of your system call at the end.
asmlinkage long new_system_call (whatever params you want to pass)

Step 4: Changing the kernel Makefile
Add the new folder to the kernel compile

core-y += /kernel /blah /blah /blah /mysyscall

Step 5: Write your system call

Write whatever crap you want to write inside the mysyscall.c file

asmlinkage long new_system_call (whatever params you want to pass)
{
  // whatever you want to do
}

Change the makefile as well and add the following line

obj-y := mysyscall.o

Compile your kernel and test the system call from a user level program. You can create a header file that the user space program can use.

/* header.h */
#include < linux/unistd.h >
#define __NR_new_system_call XXX

/* if you system call returns int and takes no parameter
* use this macro
*/
_syscall0(int,new_system_call)

/* Otherwise, depending on the number of parameters
* being passed use the _syscallN macro, N being the no
* of params, like
_syscall1(int, new_system_call, int)
*/

Last thing to do is to test the code:

/* test client */
#include "header.h"

int main (void)
{
  printf ("System call returned %d \n", new_system_call());
  return 1;
}

NOTE
Starting around kernel 2.6.18, the _syscallXX macros were removed from header files supplied to user space. Instead we need to use syscall() function.

printf ("System call returned %d \n", syscall (__NR_new_system_call, params_if_any));

or, make the following changes in the header.h

/* header.h */
#include < linux/unistd.h >
#include < sys/syscall.h >
#define __NR_new_system_call XXX

long new_system_call (params_if_any)
{
  return syscall (__NR_new_system_call, params_if_any);
}

Kernel compile notes

NOTE: The upstream (git) kernel *sometimes* does not compile with gcc-4.x. Install gcc-3.4 and change /usr/bin/gcc (which links to gcc-4.x) to point to gcc-3.4 instead.

Compile using make-kpkg on Debian. Kernel compiles as a .deb (initrd image + modules) and can be directly installed using dpkg -i. (Steps, UbuntuWiki). The problem with make-kpkg is that every time you make a change to the kernel code it recompiles everything from scratch, which is a pain in the neck! To avoid that, it is better to compile the kernel from scratch and not use jazzy distro scripts


The old fashioned way

1. Go to the kernel directory (say, kernel_dir) and edit the Makefile to specify the EXTRAVERSION. (This is what --append-to-version does in make-kpkg). It is good to do this as it helps to identify your kernel from the rest.

2. Configure the kernel
make menuconfig

If you do not know where to start for, use
make defconfig

This creates a default configuration for i386. You can then confirm the configuration with
make menuconfig

[OLD]It might be useful to run (not required)
make oldconfig

3. Compile the kernel and modules
make bzImage
make modules

In the 2.6+ kernel, you can just do the following instead

make -j N, where N is the number of parallel compilation tasks you want to kick in for a faster compile. (N = 2,4,...)

TIP: If you have already compiled and installed the kernel once and later made changes ONLY to the kernel, doing a make would build the modules too. To avoid that do

make bzImage

TIP: To reduce the compilation noise you can forward the output of make to /dev/null.

4. Install the modules in /lib/modules

sudo make modules_install

Confirm that /lib/modules/ has the modules corresponding to your kernel version.

5. Copy the kernel image binaries to /boot

Lets say that the kernel version (along with the EXTRAVERSION) is 2.6.24-custom.010209

cd kernel_dir
sudo mv arch/i386/boot/bzImage /boot/vmlinuz-2.6.24-custom.010209
sudo mv .config /boot/config-2.6.24-custom.010209
sudo mv System.map /boot/System.map-2.6.24-custom.010209

6. Create the RAM Disk image (initrd).

In Ubuntu the mkinitrd is no longer supported. Instead mkinitramfs is used.

update-initramfs -c -k 2.6.24-custom.010209 # your kernel version

Note that update-initramfs looks at /lib/module for the kernel version. Make sure that you have installed the modules in step 4.

The initrd file initrd-2.6.24-custom.010209 would be created in /boot

NOTE: Usually the following are selected by default at the time of kernel configuration. If these are not set, you might get a kernel panic while booting.
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_CRAMFS=y

7. Last step is to update grub

sudo update-grub

Check in /boot/grub/menu.lst and confirm.

8. Reboot, and select your kernel

Linux system calls

1. How to implement my own system call in kernel 2.6 -> here
2. IBM developerWorks "Kernel command using Linux system calls" -> here
3. Linux system call table -> here
4. How system call works on Linux/i386 -> here
5. syscalls() man page -> here

Build a custom kernel

Compiling the ubuntu kernel