Using diff

Remember: when using diff, use the following flags

diff -Naur


-N = Treat absent files as empty
-a = Treat all files as text
-u = Output 3 lines of unified content
-r = Recursively compare any subdirectories found

netconsole

Yet another mechanism to get the kernel debug messages is netconsole which sends the messages over Ethernet. You would not be able to get those early printks or crashes but it is useful nonetheless.

Setting up netconsole is fairly easy. You can either pass the netconsole parameters at the boot time or enable it after the machine has booted using modprobe.

Setting netconsole using boot parameters

Add the following parameter to the kernel boot string
netconsole=[s-port]@[s-ip]/[dev],[t-port]@[t-ip]/[t-mac]

s-port = source port, default is 6665
s-ip = source IP address
dev = device (eth0, eth1 ...)
t-port = target port, default is 6666
t-ip = target  IP address
t-mac = target MAC address

For example:
netconsole=1234@10.1.1.152/eth0, 1235@10.1.1.153/11:22:33:44:55

Setting netconsole using modprobe

It is preferred to use modprobe instead of insmod
$ sudo modprobe netconsole netconsole="@/,@t-ip/"

For example:
$ sudo modprobe netconsole netconsole="@/,@10.1.1.153/"

Target (receiver) side 

Use netcat to get the messages

$ nc -l -u [t-port]

The full documentation on using the netconsole can be found in the "Documentation/networking/netconsole.txt" file in the kernel folder.

Online resources -  1, 2, 3 (some of these might be dated ... )

Kernel debugging using KGDB

KGDB is full integrated with the latest kernel. These instructions are for linux-2.6.31.12. Not sure if these will work for any previous or later versions.
  1. make menuconfig and select the following
    1. Select Kernel Hacking → KGDB: kernel debugging with remote gdb
      1. Select KGDB: use kgdb over the serial console
  2. Build the kernel
     make bzImage; make modules; sudo make modules_install
  3. Install the kernel. Copy bzImage, System.map and .config into /boot
  4. On another machine (say remote) where the serial null modem cable is connected, copy the entire kernel folder that you compiled on the target machine. 
  5. Restart the target machine.  At the grub menu, select your kernel, edit it and add the following at the end of the kernel line:
     kgdboc=ttyS0,115200 kgdbwait
    This is assuming that you have a serial port and it is connected on ttyS0. Boot into your kernel. It will wait for gdb to connect.
  6. On the remote machine, go in the kernel folder you copied over and debug vmlinx by starting gdb
    1. gdb ./vmliux
    2. Set the baud rate - (gdb) set remotebaud 115200
    3. Connect to the target - (gdb) target remote /dev/ttyS0
    4. Run the kernel - (gdb) continue
For more details, refer to the docbook on KGDB in the kernel documentation folder. Just do a make htmldocs to convert into a readable format. You would have to install xmlto first. sudo aptitude install xmlto should help.

    Force fsck on reboot

    If you want to force an fsck check on your drives on reboot do the following:

    sudo touch /forcefsck
    sudo shutdown -r now

    Serial console debugging

    Setting up the serial console

    1. First find which devices are attached to the system

    dmesg | grep "tty"

    2. For the device found above (say ttyS0), create the /etc/event.d/ttyS0 file with the following contents

    # ttyS0 - getty
    #
    # This service maintains a getty on ttyS0 from
    # the point the system is started until it is
    # shut down again.

    start on runlevel 2
    start on runlevel 3
    start on runlevel 4
    start on runlevel 5

    stop on runlevel 0
    stop on runlevel 1
    stop on runlevel 6

    respawn
    exec /sbin/getty -L 115200 ttyS0 vt102


    3. Edit /etc/securetty and add ttyS0

    4. During reboot edit the kernel in grub menu (press 'e'). At the end of the kernel line add console=ttyS0,115200n8 tty1

    5. Boot into your kernel

    6. Once you log in, if you want all the dmesg output to go on the serial console do the following:
    sudo tail -f /var/log/kern.log > /dev/ttyS0

    Setting up the client

    1. Open minicom and set the port as the one you are talking on, it could be ttySn or ttyUSBn depending on if you are using a serial port or a USB to serial converter. (e.g. ttyS0, ttyUSB0)

    2. Set the setting as 115200 baud, 8 bits, 1 stop bit, no parity, no flow control

    Update: There is another program called gtkterm if you don't fancy the command line applications. Also I would recommend using konsole to run minicom within as it supports unlimited buffer, which can be very useful if you are trying to look at the /var/log/kern.log of the target machine, for example.

    Update (Apr 20, 2010): Only if you want to have a serial console (like a bash on the serial port) should you do the getty for ttyS0. If you just want all the kernel message sent to the terminals just do the following:

     # echo 8 > /proc/sys/kernel/printk

    Read this article for more details on kernel oops.

    Texmaker

    Texmaker is yet another GUI centric tool for LaTeX and a pretty good one. The latest edition (1.9.9) has been spruced up and looks pretty neat. One of the best features in Texmaker that is not available in other tools like Kile, LyX,  is the online spell-check. For a person like me who makes a billion spelling mistakes, the online spell-check is a savior. Texmaker is available across platforms for free. Check the download page for more details.

    For Ubuntu users

    $ sudo aptitude install texmaker

    HDR Photography open source tool

    Luminance HDR is an open-source tool for creating HDR images (for example). The idea is to take multiple shots with varying exposure across the dynamic range and then merge all of these together to create one beautiful HDR image.

    The package name is called qtpfsqui and it is available via Ubuntu's package manager. I have not installed it yet, would be doing it later. I am waiting to get a tripod first. :)

    Graphical SVN clients on Ubuntu

    Two ways of doing it.

    Kdesvn: (http://kdesvn.alwins-world.de/) - A client that works well with GNOME as well.

    But, the winner is:

    RabbitVCS: (http://rabbitvcs.org/). RabbitVCS integrates with Nautilus and works like glue. People used to TortoiseSVN on Windows won't miss a thing. Easy to configure and use.

    On the same note, if you are looking for free hosting service for your project look at Unfuddle (for a free single user private account) or Google project hosting if you are looking for open-source.