pthread

A very good one-shot tutorial to the POSIX thread libraries (pthread). Talks all about threading, synchronization and scheduling with example code.

Linux performance analysis for embedded system

Nice series of articles found on www.embedded.com

Part 1: Available tools
Part 2: Profiling/analysis methods and techniques

Script to install a new kernel

#!/bin/sh
#
# (c) Hunterwala

if [ $(/usr/bin/id -u) -ne 0 ]; then
     echo "run this script with sudo"
     exit 2
fi

if [ -z $1 ]; then
     echo "kernel version not found"
     echo "usage: ./install_kernel [kernel_version]"
     exit 2
else
     VERSION=$1
     echo "using $VERSION"
fi

if [ ! -d /lib/modules/$VERSION ]; then
     echo "kernel modules not installed"
     echo "run make install_modules from the kernel directory"
     exit 2
fi

BOOTDIR=/boot

echo "Installing kernel ........................"
cp $PWD/.config $BOOTDIR/config-$VERSION
cp $PWD/System.map $BOOTDIR/System.map-$VERSION
cp $PWD/arch/i386/boot/bzImage $BOOTDIR/vmlinuz-$VERSION
echo "done"

echo "Creating initrd image ...................."
update-initramfs -c -k $VERSION
echo "done"

echo "Updating grub ............................"
update-grub
echo "done"

echo "Linux kernel $VERSION successfully installed"
echo "Please restart and select $VERSION in grub menu"

Battery monitoring from console

If you prefer to work on a console based machine (without gdm\kdm), the following script is useful if you are running your laptop on battery. It shows the current battery charge and can be used for monitoring the battery discharge.

#!/bin/sh
# While working in console mode (without gdm) this script
# tells you the remaining charge of your laptop battery
#
# Run it along with watch on a separate tty
# $ watch -n 10 ./batmon
#
# (c) Hunterwala, 2009
#
# TODO: check for error levels and buzz using pcspkr mod
# when the charge level decreases beyond that level.

PROC_PATH="/proc/acpi/battery/BAT0"

org_cap=`cat $PROC_PATH/info | grep "last full capacity" | sed "s/ //g" | awk -F: '{print($2)}' | sed "s/mAh//g"`
rem_cap=`cat $PROC_PATH/state | grep "remaining capacity" | sed "s/ //g" | awk -F: '{print($2)}'| sed "s/mAh//g"`
message=`cat $PROC_PATH/state | grep "charging state" | sed "s/ //g" | awk -F: '{print($2)}'| sed "s/mAh//g"`

diff=$(echo "($org_cap-$rem_cap)" | bc)
total_per=$(echo "100-($diff*100/$org_cap)" | bc)

echo "Battery is" $message ":" $total_per"%"

Save this script as batmon.sh

Change the permissions chmod a+x batmon.sh

Run it on a separate tty watch -n 10 ./batmon.sh

new system

Just installed a new system with a fresh Ubuntu install? Install all the packages in one shot,

~$ sudo aptitude install build-essential ddd insight kdbg kdevelop eclipse eclipse-cdt eclipse-jdt kompare kate bless kile lyx cscope global octave kmplot vim emacs virtualbox-ose pdfedit scribus inkscape checkgmail kompozer manpages manpages-posix easytag gtkpod vlc libncurses5 libncurses5-dev dialog curl sun-java6-jre sun-java6-jdk wireshark qt4-dev-tools qt4-assistant qt4-designer qdevelop monodevelop umbrello git git-core gitk git-gui graphviz doxygen doxygen-gui

Apart from that, install the individual .deb for

1. Google Chrome (dev channel)
2. Skype for 32 bit (64-bit still shaky)
3. Picasa for Linux

Before installing new packages, do
~$ sudo apt-get update; sudo apt-get upgrade

Linux Kernel design patterns

Linux kernel design patterns, part 1, part 2 and part 3.

Git

Started using git.. this page would get more info as I learn more.

1. A very good presentation\video - here
2. Git documentation guide\book - here
3. Understanding Git, another tutorial - here

oprofile

oprofile is very helpful in getting a trace out of the kernel especially to know the %age usage of methods from the kernel and the application. Helps in debugging some weird problems.

First, enable oprofile in the kernel, it is good to build it part of the kernel rather than a kernel module, so make sure in the menuconfig option it is a [*] rather than [m].

You can install the oprofile client once you boot back into your file system.
sudo apt-get install oprofile oprofile-gui

Before starting your application, initialize
$ sudo opcontrol --init

The following two options can be done once in the beginning

Tell oprofile the path to your compiled and unstripped vmlinux file so that it can pick the symbols from there
$ sudo opcontrol --vmlinux=/path/to/kernel/image

Set the call stack depth
# sudo opcontrol --callgraph=#depth

Now check the status,
$ sudo opcontrol --status

Reset the earlier dump,
$ sudo opcontrol --reset

Start the profiler
$ sudo opcontrol --start

Run your application
$ ./mystupidapp

All done, stop the profiler
$ sudo opcontrol --stop

Get the dump
$ sudo opcontrol --dump

Now, get the report
$ sudo opreport -l (and various options if you want)

Clear it,
$ sudo opcontrol --reset

Deinit,
$ sudo opcontrol --deinit

and start again if you want to.

I use alias(es) in the .bashrc file. Really helps!
alias opc='sudo opcontrol'
alias opcdeinit='opc --deinit'
alias opcdump='opc --dump'
alias opcinit='opc --init'
alias opcreset='opc --reset'
alias opcstart='opc --start'
alias opcstatus='opc --status'
alias opcstop='opc --stop'
alias opr='sudo opreport -l'
alias oprcall='opr -c'

Linux embedded Systems

Nice tutorial that concludes a series of tutorials on embedded Linux system development by noted ARM Linux kernel hackers Vincent Sanders and Daniel Silverstone, here

ping sweep

You have a machine on the subnet for which you know the MAC address but you don't know what IP address it got after a restart, use nmap and do a ping sweep:

sudo nmap -n -sP 192.168.1.0/24 | grep "00:11:22:33:44"

-n tells nmap not to never do reverse DNS resolution on the active IP address it finds. Makes scanning faster
-sP Only perform a ping scan (host discovery) and prints the results, go no further.

HTML editor

I am working on creating the website for my lab and I had to find a good open source alternative to Dreamweaver.

Bluefish Editor
It is a nice editor but it is not a WYSIWYG. One needs to go back to learning HTML and family and it becomes irritating after sometime.





Quanta Plus
Like Bluefish but more crappy and archaic. I would suggest installing software using aptitude rather than apt-get because it is easy to purge and clean the dependency chain that it creates.




KompoZer
This is best editor. Powered by Mozilla (even looks like Firefox) and is a WYSIWYG with powerful features like tag view, source view and preview. It provides easy layering (div) and properties. It took me 10 minutes to create the same page that took 1 hour in Bluefish. Winner!!!

[Book] Linux in a nutshell

This O'Reilly book is a good starter for people new to Linux. Here are the list of useful commands from the book.

[Image courtsey http://www.oreillynet.com]

Using cut and uniq

For some weird reason I had to do this, cut and uniq turned out to be quite handy and can be used along with sed

cat /proc/cpuinfo | grep "model name" | uniq | cut -d: -f2

So, instead of this
model name : Intel(R) Core(TM)2 Duo CPU E8500 @ 3.16GHz
model name : Intel(R) Core(TM)2 Duo CPU E8500 @ 3.16GHz

I get this
Intel(R) Core(TM)2 Duo CPU E8500 @ 3.16GHz

OK, another approach is to use awk
cat /proc/cpuinfo | grep "model name" | uniq | awk -F: '{print($2)}'

Emacs command list

Sometimes I tend to forget some of these :D :P

C = Control
M = Meta = Alt|Esc

Basics
C-x C-f "find" file i.e. open/create a file in buffer
C-x C-s save the file
C-x C-w write the text to an alternate name
C-x C-v find alternate file
C-x i insert file at cursor position
C-x b create/switch buffers
C-x C-b show buffer list
C-x k kill buffer
C-z suspend emacs
C-X C-c close down emacs

Basic movement
C-f forward char
C-b backward char
C-p previous line
C-n next line
M-f forward one word
M-b backward one word
C-a beginning of line
C-e end of line
C-v one page up
M-v scroll down one page
M-< beginning of text
M-> end of text

Editing
M-n repeat the following command n times
C-u repeat the following command 4 times
C-u n repeat n times
C-d delete a char
M-d delete word
M-Del delete word backwards
C-k kill line

C-Space Set beginning mark (for region marking for example)
C-W "kill" (delete) the marked region region
M-W copy the marked region
C-y "yank" (paste) the copied/killed region/line
M-y yank earlier text (cycle through kill buffer)
C-x C-x exchange cursor and mark

C-t transpose two chars
M-t transpose two words
C-x C-t transpose lines
M-u make letters uppercase in word from cursor position to end
M-c simply make first letter in word uppercase
M-l opposite to M-u

Important
C-g quit the running/entered command
C-x u undo previous action
M-x revert-buffer RETURN (insert like this) undo all changes since last save
M-x recover-file RETURN Recover text from an autosave-file
M-x recover-session RETURN if you edited several files

Online-Help
C-h c which command does this keystroke invoke
C-h k which command does this keystroke invoke and what does it do?
C-h l what were my last 100 typed keys
C-h w what key-combo does this command have?
C-h f what does this function do
C-h v what's this variable and what is it's value
C-h b show all keycommands for this buffer
C-h t start the emacs tutorial
C-h i start the info reader
C-h C-k start up info reader and go to a certain key-combo point
C-h F show the emacs FAQ
C-h p show infos about the Elisp package on this machine

Search/Replace
C-s Search forward
C-r search backward
C-g return to where search started (if you are still in search mode)
M-% query replace

Space or y replace this occurrence
Del or n don't replace
. only replace this and exit (replace)
, replace and pause (resume with Space or y)
! replace all following occurrences
^ back to previous match
RETURN or q quit replace


Search/Replace with regular expressions
Characters to use in regular expressions:
^ beginning of line
$ end of line
. single char
.* group or null of chars
\< beginning of a word
\> end of a word
[] every char inside the backets (for example [a-z] means every small letter)

M C-s RETURN search for regular expression forward
M C-r RETURN search for regular expression backward
M C-s incremental search
C-s repeat incremental search
M C-r incremental search backwards
C-r repeat backwards
M-x query-replace-regexp search and replace

Window-Commands
C-x 2 split window vertically
C-x o change to other window
C-x 0 delete window
C-x 1 close all windows except the one the cursors in
C-x ^ enlarge window
M-x shrink-window command says it ;-)
M C-v scroll other window
C-x 4 f find file in other window
C-x 4 o change to other window
C-x 4 0 kill buffer and window
C-x 5 2 make new frame
C-x 5 f find file in other frame
C-x 5 o change to other frame
C-x 5 0 close this frame

Bookmark commands
C-x r m set a bookmark at current cursor pos
C-x r b jump to bookmark
M-x bookmark-rename says it
M-x bookmark-delete "
M-x bookmark-save "
C-x r l list bookmarks

d mark bookmark for deletion
r rename bookmark
s save all listed bookmarks
f show bookmark the cursor is over
m mark bookmarks to be shown in multiple window
v show marked bookmarks (or the one the cursor is over)
t toggle listing of the corresponding paths
w " path to this file
x delete marked bookmarks
Del ?
q quit bookmark list

M-x bookmark-write write all bookmarks in given file
M-x bookmark-load load bookmark from given file

Shell
M-x shell starts shell modus
C-c C-c same as C-c under unix (stop running job)
C-d delete char forward
C-c C-d Send EOF
C-c C-z suspend job (C-z under unix)
M-p show previous commands

DIRectory EDitor (dired)
C-x d start up dired
C (large C) copy
d mark for erase
D delete right away
e or f open file or directory
g reread directory structure from file
G change group permissions (chgrp)
k delete line from listing on screen (don't actually delete)
m mark with *
n move to next line
o open file in other window and go there
C-o open file in other window but don't change there
P print file
q quit dired
Q do query-replace in marked files
R rename file
u remove mark
v view file content
x delete files marked with D
z compress file
M-Del remove all marks (whatever kind)
~ mark backup files (name~ files) for deletion
# mark auto-save files (#name#) for deletion
*/ mark directory with * (C-u * removes that mark again)
= compare this file with marked file
M-= compare this file with it's backup file
! apply shell command to this file
M-} change to the next file marked with * od D
M-{ " previous "
% d mark files described through regular expression for deletion
% m " (with *)
+ create directory
> changed to next dir
< change to previous dir
s toggle between sorting by name or date
M-x speedbar starts up a separate window with a directory view

Telnet
M-x telnet starts up telnet-modus
C-d either delete char or send EOF
C-c C-c stop running job (similar to C-c under unix)
C-c C-d send EOF
C-c C-o clear output of last command
C-c C-z suspend execution of command
C-c C-u kill line backwards
M-p recall previous command

Text
Works only in text mode
M-s center line
M-S center paragraph
M-x center-region name says

Macro-commands
C-x ( start macro definition
C-x ) end of macro definition
C-x e execute last definied macro
M-n C-x e execute last defined macro n times
M-x name-last-kbd-macro give name to macro (for saving)
M-x insert-keyboard-macro save named macro into file
M-x load-file load macro
M-x macroname execute macroname

Programming
M C-\ indent region between cursor and mark
M-m move to first (non-space) char in this line
M-^ attach this line to previous
M-; formatize and indent comment
C, C++ and Java Modes
M-a beginning of statement
M-e end of statement
M C-a beginning of function
M C-e end of function
C-c RETURN Set cursor to beginning of function and mark at the end
C-c C-q indent the whole function according to indention style
C-c C-a toggle modus in which after electric signs (like {}:';./*) emacs does the indention
C-c C-d toggle auto hungry mode in which emacs deletes groups of spaces with one del-press
C-c C-u go to beginning of this preprocessor statement
C-c C-c comment out marked area
M-x outline-minor-mode collapses function definitions in a file to a mere {...}
M-x show-subtree If you are in one of the collapsed functions, this un-collapses it

In order to achive some of the feats coming up now you have to run etags *.c *.h *.cpp (or what ever ending you source files have) in the source directory

M-. (Thats Meta dot) If you are in a function call, this will take you to it's definition
M-x tags-search ENTER Searches through all you etaged
M-, (Meta comma) jumps to the next occurence for tags-search
M-x tags-query-replace yum. This lets you replace some text in all the tagged files

GDB (Debugger)
M-x gdb starts up gdm in an extra window

Version Control
C-x v d show all registered files in this dir
C-x v = show diff between versions
C-x v u remove all changes since last checkin
C-x v ~ show certain version in different window
C-x v l print log
C-x v i mark file for version control add
C-x v h insert version control header into file
C-x v r check out named snapshot
C-x v s create named snapshot
C-x v a create changelog file in gnu-style

(Reference: http://lpn.rnbhq.org/tools/xemacs/emacs_ref.html)

Mounting an ISO

sudo mount -t iso9660 -o loop /path/to/the/iso /folder/to/mount/into

In case you are getting errors like ISOFS: Unable to identify CD-ROM format., do a file /path/to/the/iso and confirm that the file that you think is an ISO, actually is.

Using ssh, scp, sftp

SSH
The server node should be running the SSH server

sudo aptitude install openssh-server

Connect to the server

ssh username@ipaddress:port

Connect to the server with X enabled

ssh username@ipaddress:port

Note: specify port only if the default 22 is not being used.

SCP
scp can be used to transfer the files between the nodes (works on top of SSH)

From remote node to local node

cd local_folder
scp username@ipaddress:port:/home/username/path/to/folder_file .

From local node to remote node

scp /path/to/local/folder_file username@ipaddress:port:/home/username/path

SFTP
If you have a luxury of a GUI, you can copy using sftp.

Open nautilus (if using Gnome) and type in the location bar the address to the server.

nautilus --no-dektop --browser
Location Bar: sftp://username@ipaddress:port/home/username

Although you can go to the root folder as well, but is not a good practice to screw around in that area

Using VNC over SSH

Okay, now I need to connect to this Linux box in the department which is running Matlab and access it on the wire.

Options:

1. SSH + X11 forwarding [slow]

% ssh -X username@hostname
% matlab &

In order to have SSH forward X11 by default change the following in /etc/ssh/ssh_config and also remove the #

ForwardAgent yes (default no)
ForwardX11 yes (default is no)
StrictHostKeyChecking no (default is ask)

With this you can now use

ssh username@hostname

2. VNC without SSH. [fast]
% vncviewer hostname:display

3. VNC tunneling over SSH [faster]
% ssh -C -L 5901/localhost/5901 username@hostname
% vncviewer localhost:5901

Ensure that vncserver is running on the remote node and also that you have set your vncpasswd. The "display" is the number that you get after starting the vncserver

NOTE:
If you are running compiz\beryl (any composite window manager), Matlab's pre-bundled java does not support it and hence it would not render it properly with option (1). You would have to point to the latest version of the JVM before starting matlab

% export MATLAB_JAVA=/usr/lib/java/blah-blah/jre
% matlab &

Update:
Got a local copy of Matlab and realized that it has the same problem and is solved by pointing to the latest JVM. Create a launcher that takes care of setting up the environment before the launch.

env MATLAB_JAVA="/usr/lib/jvm/java-6-sun/jre" xterm "/home/piyush/matlab-2007b/bin/matlab"

your own system call in 5 easy steps

You *might* want to write your system call for various reasons

Assuming the path to your kernel source is "L". Create a new folder L/mysyscall. Inside the folder create the source file mysyscall.c and a Makefile

Step 1. Changing the System Table
L/arch/x86/kernel/syscall_table_32.S

Add your system call at the end of the file.

.long sys_new_system_call

Step 2. Changing the unistd.h
L/linux/include/asm-x86/unistd_32.h

Add your system call at the end of the existing list and append the next number

#define __NR_new_system_call XXX

Where XXX is the existing system call number plus 1. Also update the total system calls (as you just added another)
#define __NR_syscalls XXY
Where XXY is XXX+1

Step 3: Changing syscalls.h
L/include/linux/syscalls.h

Add the declaration of your system call at the end.
asmlinkage long new_system_call (whatever params you want to pass)

Step 4: Changing the kernel Makefile
Add the new folder to the kernel compile

core-y += /kernel /blah /blah /blah /mysyscall

Step 5: Write your system call

Write whatever crap you want to write inside the mysyscall.c file

asmlinkage long new_system_call (whatever params you want to pass)
{
  // whatever you want to do
}

Change the makefile as well and add the following line

obj-y := mysyscall.o

Compile your kernel and test the system call from a user level program. You can create a header file that the user space program can use.

/* header.h */
#include < linux/unistd.h >
#define __NR_new_system_call XXX

/* if you system call returns int and takes no parameter
* use this macro
*/
_syscall0(int,new_system_call)

/* Otherwise, depending on the number of parameters
* being passed use the _syscallN macro, N being the no
* of params, like
_syscall1(int, new_system_call, int)
*/

Last thing to do is to test the code:

/* test client */
#include "header.h"

int main (void)
{
  printf ("System call returned %d \n", new_system_call());
  return 1;
}

NOTE
Starting around kernel 2.6.18, the _syscallXX macros were removed from header files supplied to user space. Instead we need to use syscall() function.

printf ("System call returned %d \n", syscall (__NR_new_system_call, params_if_any));

or, make the following changes in the header.h

/* header.h */
#include < linux/unistd.h >
#include < sys/syscall.h >
#define __NR_new_system_call XXX

long new_system_call (params_if_any)
{
  return syscall (__NR_new_system_call, params_if_any);
}

Kernel compile notes

NOTE: The upstream (git) kernel *sometimes* does not compile with gcc-4.x. Install gcc-3.4 and change /usr/bin/gcc (which links to gcc-4.x) to point to gcc-3.4 instead.

Compile using make-kpkg on Debian. Kernel compiles as a .deb (initrd image + modules) and can be directly installed using dpkg -i. (Steps, UbuntuWiki). The problem with make-kpkg is that every time you make a change to the kernel code it recompiles everything from scratch, which is a pain in the neck! To avoid that, it is better to compile the kernel from scratch and not use jazzy distro scripts


The old fashioned way

1. Go to the kernel directory (say, kernel_dir) and edit the Makefile to specify the EXTRAVERSION. (This is what --append-to-version does in make-kpkg). It is good to do this as it helps to identify your kernel from the rest.

2. Configure the kernel
make menuconfig

If you do not know where to start for, use
make defconfig

This creates a default configuration for i386. You can then confirm the configuration with
make menuconfig

[OLD]It might be useful to run (not required)
make oldconfig

3. Compile the kernel and modules
make bzImage
make modules

In the 2.6+ kernel, you can just do the following instead

make -j N, where N is the number of parallel compilation tasks you want to kick in for a faster compile. (N = 2,4,...)

TIP: If you have already compiled and installed the kernel once and later made changes ONLY to the kernel, doing a make would build the modules too. To avoid that do

make bzImage

TIP: To reduce the compilation noise you can forward the output of make to /dev/null.

4. Install the modules in /lib/modules

sudo make modules_install

Confirm that /lib/modules/ has the modules corresponding to your kernel version.

5. Copy the kernel image binaries to /boot

Lets say that the kernel version (along with the EXTRAVERSION) is 2.6.24-custom.010209

cd kernel_dir
sudo mv arch/i386/boot/bzImage /boot/vmlinuz-2.6.24-custom.010209
sudo mv .config /boot/config-2.6.24-custom.010209
sudo mv System.map /boot/System.map-2.6.24-custom.010209

6. Create the RAM Disk image (initrd).

In Ubuntu the mkinitrd is no longer supported. Instead mkinitramfs is used.

update-initramfs -c -k 2.6.24-custom.010209 # your kernel version

Note that update-initramfs looks at /lib/module for the kernel version. Make sure that you have installed the modules in step 4.

The initrd file initrd-2.6.24-custom.010209 would be created in /boot

NOTE: Usually the following are selected by default at the time of kernel configuration. If these are not set, you might get a kernel panic while booting.
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_CRAMFS=y

7. Last step is to update grub

sudo update-grub

Check in /boot/grub/menu.lst and confirm.

8. Reboot, and select your kernel

Linux system calls

1. How to implement my own system call in kernel 2.6 -> here
2. IBM developerWorks "Kernel command using Linux system calls" -> here
3. Linux system call table -> here
4. How system call works on Linux/i386 -> here
5. syscalls() man page -> here