oprofile

oprofile is very helpful in getting a trace out of the kernel especially to know the %age usage of methods from the kernel and the application. Helps in debugging some weird problems.

First, enable oprofile in the kernel, it is good to build it part of the kernel rather than a kernel module, so make sure in the menuconfig option it is a [*] rather than [m].

You can install the oprofile client once you boot back into your file system.
sudo apt-get install oprofile oprofile-gui

Before starting your application, initialize
$ sudo opcontrol --init

The following two options can be done once in the beginning

Tell oprofile the path to your compiled and unstripped vmlinux file so that it can pick the symbols from there
$ sudo opcontrol --vmlinux=/path/to/kernel/image

Set the call stack depth
# sudo opcontrol --callgraph=#depth

Now check the status,
$ sudo opcontrol --status

Reset the earlier dump,
$ sudo opcontrol --reset

Start the profiler
$ sudo opcontrol --start

Run your application
$ ./mystupidapp

All done, stop the profiler
$ sudo opcontrol --stop

Get the dump
$ sudo opcontrol --dump

Now, get the report
$ sudo opreport -l (and various options if you want)

Clear it,
$ sudo opcontrol --reset

Deinit,
$ sudo opcontrol --deinit

and start again if you want to.

I use alias(es) in the .bashrc file. Really helps!
alias opc='sudo opcontrol'
alias opcdeinit='opc --deinit'
alias opcdump='opc --dump'
alias opcinit='opc --init'
alias opcreset='opc --reset'
alias opcstart='opc --start'
alias opcstatus='opc --status'
alias opcstop='opc --stop'
alias opr='sudo opreport -l'
alias oprcall='opr -c'