udev 150

I've successfully updated udev for our target from 142 to 150 versions. I've added three patches to fix the compile errors. Those patches are introduced because the kernel header used in our toolchain is older than the latest version, like the inotify_init1 system call is not available in our version of toolchain. I have to use the inotify_init system call instead.

Anyway, it works in my first attempt. The old 142 version was ported by an intern, with too much modification to the makefiles/configure scripts. Those patches are 90% of junk, I would say. Anyway, the newer version is more maintainable now.

udev

I've got udev working on our platform. It's initially ported by an intern, but it seemed not working before he left. The problem was due to the kernel configuration. Once I sorted out the proper kernel configuration, udevd started properly and it works. The ota also worked with the udev. The previous port of udev was udev 142 version, which is rather old. I am porting the 150 version to our system.

Toolchain study digest

Some information I learned recently about GNU GCC toolchain and related topic from the book "Professional Linux Programming". Although I am familiar with most of the content, I'd just record here for a better memory.

* -fPIC (position independent code), used to generate code can be executed anywhere in the kernel, usually used to generate shared library.

* -shared -o libfoo.so, generate shared library directly, convenience provided by GCC.

* LD_LIBRARY_PATH, specify the additional path to shared library, other than regular /lib, /usr/lib ... directories.

GCC options:

* General Options:
-c (compilation only)
-S (generate assembly code)

* Language Options:
-ansi (ansi C90 standard)
-std=c89, gnu89, c99, gnu89 (use various ANSI C standards, or GNU C standards)
-fno-builtin (do not use the built-in version of printf/memcpy/etc in gcc, use external library version)

* Warning Levels:
-pedantic (strictly follow C standard, give warnings, or -pedantic-errors)
-Wformat (one of many warnings to check the code)
-Wall (a wide range of warnings)

* Debugging:
-g

* Optimization:
-O0, -O3
-Os

* Hardware options:
-march
-msoft-float
-mbig-endian, -mlittle-endian
-mabi

GNU Binutils

Assembler:

as -o foo.o foo.s

Linker:

ELF: Executable and Linking Format
crtbegin.o, crtend.o
Linking script: /usr/lib/ldscripts

Objcpy and objdump:
objdump -x -d -S hello_world

Kernel and toolchain:
* inline assembly
__asm__ __volatile__
* __attribute__, section, alignment, alias
* custom linker script, arch//kernel/vmlinux.lds

Useful Mutt documents and tips

I have started to use mutt as my main email client for a couple of months. It is powerful and convenient, especially for me as I am a lover of terminal programs.

I'd collect mutt relevant document here.

Training your mutt. http://www.linux.com/archive/articles/58760

Vim tricks

I've read another blog today about Efficient Editing with Vim. I learned some new tricks as the follows.

* g + h,j,k,l, to move screen line instead of real line. This can be real useful when you are editing an email with long lines.
* fx, Fx, go forward/backword to the next occurrence of x.
* (, ), move to the next sentence or the previous sentence
* xG, goto the xth line, the same as :x, just faster
* H, M, L, move to the top, middle, and end of the screen

kernel upgrade to 2.6.27 for mips/arm platform

I have encountered other interesting issue when I was upgrading the kernel from 2.6.23 to 2.6.27 for our MIPS32 platform.

1. the timer, as in 2.6.24 or later, the default MIPS timer has been separated from the old timer code. For our architecture used the default cp0 comparison based timer, we need to enable the r4k timer to get the timer working. I spend couple of days to understand this change. The start_kernel function was able to proceed after the proper is enabled. However, we do have a timer block in our chip, and we can set the timer to a certain frequency as an individual timer source. In my later debugging process, I have enabled the timer block and use our own timer source. It also works.

2. the cache should be disabled when kernel started and re-enable later on. In 2.6.23, the cache was enabled by default. However, in 2.6.27 or some version later than 2.6.23, the cache was configurable by a kernel option "cca=" and it is disabled by default. This change really hurts me. As there are so many changes from 23 to 27 kernel, it is almost impossible for me to notice this change at first. What I have observed at first was the slowness of the system. The BogoMips dropped from about 273 to 3, which is unbelievable. I was doubting the correctness of the timer function at the beginning. I scrutinized the code and well studied the new timer implementation. I even implemented our own timer by using the timer block in our chip. Those doesn't help either. The system was able to boot to busybox but it is really slow. I accidentally tried to use our performance counter program to measure the performance. The performance counter reported the cache hit is 0, which means that there is no cache enabled. I checked our private i/d cache register and they seems enabled. However, I forget to check the setting of the cp0 register of MIPS. There is another setting to enable/disable cache policy. I used a very stupid and old method to pinpoint the problem. I added NOP test to both 23 and 27 kernels. In 23 kernel, when the cache is enabled, the NOP test gives much lower CPI (clock per instruction), otherwise the CPI is high. In 27 kernel, the CPI doesn't change. I tried to figure out the exact point where the CPI drop 23 kernel and check the corresponding code in 27 kernel. I finally found that the default cache policy was disable in 27 kernel, while it is enabled in 23 kernel. By adding the "cca=3" kernel command line option, everything backs to normal, BogoMips, kernel boots properly.


3. Export symbol and export symbol gpl'ed. If your driver, kernel module used the latter symbols, your driver/kernel module must be gpl'ed. This can cause problem for us as we don't want to open source all our kernel modules, especially wlan driver. We deliver binary kernel module for our wlan drivers.

kernel upgrade to 2.6.27

I had just fixed a network driver bug when I upgrade the kernel from 2.6.23 to 2.6.27 for our MIPS platform. It takes couple of weeks. The original problem appears when the NAPI was used in network driver and I changed the net_poll function accordingly. Then, I got kernel panic with memory access failure. After long time debugging, I found that there is some problem when the driver tries to figure out the address of skb out of the skb->data structure. This is weird because the same code was used in both 2.6.23 kernel and 2.6.27 kernel. The original author of the network driver gave some hints that he had experienced similar problem when he was creating a network driver for our next generation chip, based on the old driver. He mentioned that the original driver was confused about the physical/virtual address when accessing the dma'ed memory. This is quite helpful. I spend a whole day dig into this issue, and studied the new driver for next generation chip. After replacing the memory allocation function for the skb buffer, I finally got the proper method to access the memory. I've learned about the physical/virtual/bus address when accessing memory in kernel. The principle was simple as stated by Linus, "use virtual address when accessing memory in kernel, and use bus address when the memory was given to device". In some architecture, the bus address is identical to physical address. Never use physical address directly. The functions: phys_to_virt, virt_to_bus, bus_to_virt, virt_to_phys, are all helper functions.

It seems that we still had a lot of bugs in our network driver. Apparently, our engineers haven't had enough knowledge creating drivers in Linux. Most of their experience was in VxWorks, with flat memory model.

kernel upgrade

I am recently working on kernel upgrade from 2.6.23.17 to 2.6.27-rc5 for our VoIP SoCs.

The following are lessons I learned this time.

* use git to help migrate the patches, as we manage patches in our system, not by git. So, we may use git to help migrate the patches.
* get a kernel git tree (git fetch)
* check out a branch for your current kernel version (git branch, git checkout)
* patch the git tree with your patches (patch, git commit)
* rebase the git tree to the newer kernel (git rebase)
* resolve conflict, for each resolved conflict, remember your changes (git rerere)
* once resolved all conflicts, generate new patches for the new kernel (git format-patch)


* to get your new kernel built for the platform at first
* use make oldconfig to migrate the kernel configurations
* keep all the old configuration as much as possible

* turn on most of the kernel debug option, like early_printk, printk_with_time, spinlock, etc.

* If it doesn't boot, there may be a lot of reasons
* check kernel dump, especially the dumped code, disassemble the whole kernel to figure out where the dump happened
* there are usually a big change in each major version of kernel, such as timer change in mips architecture, path and inclusion changes in arm architecture, sd/mmc infrastructure changes, be very careful.

vmlinux.lds

The vmlinux.lds is the linker script for the linux kernel. It is generated/stored in arch/mips/boot/compressed directory for mips architecture. For x86 architecture, it is stored in arch/x86/boot/compressed directory.

check the version of gcc in your program

use #if __GNUC__ < 4

#if __GNUC_MINOR__ < 3

Linker script .lds

A script xxx.lds is used to instruct the linker, how to generate and link the final elf target. I recently solved a bootloader problem, by moving a part of (reginfo) into Data section. The bug in binutil overwrite my target elf file, with the reginfo, which shouldn't be loaded.

Some links about the linker.

http://www.gnu.org/software/binutils/manual/ld-2.9.1/html_mono/ld.html
http://www.embedded.com/2000/0002/0002feat2.htm

CGI internal server error

Haven't worked on CGI/perl for a while, recently I am reworking on my perl scripts for star5.ca website. The following items should be checked when you got the 'Internal Server Error" in Perl. GoDaddy has no server log enabled by default, so you have to be very careful in doing CGI/Perl programming.

1. File permission (your perl program should have 755 mode)
2. DOS ending (your perl program should be using unix line ending, instead of dos line ending. A lot system will complain of "file not found", because of the ending issue)
3. Path to perl (usual !/usr/bin/perl)
4. Library path (use "path to your local library"), to support local perl modules.

gcc

I am working on toolchain upgrade recently, from gcc 3.4.6 to gcc 4.1.1.

Thing I learned during the upgrade.

1. libgcc contains compiler specific library, usually used for floating point computation. For example, 64 bit floating point has no support on the host system, the libgcc has to convert the computation into some native instructions.

You can use "gcc -v" to figure out the default library path "-L xxx" used by gcc, besides the path passed by your Makefile. "gcc -v" can also give you the exact command called by gcc, as gcc itself is an umbrella program, which calls "cc1", "collect2" etc.

2. -ffreestanding, flag may be used for kernel compilation. It implies that standard library may not exist and the program startup may not necessarily be at "main". To use gcc 4.1.1 to compile linux kernel 2.6.17, we need to add -ffreestanding in our makerules.

3. Optimization, gcc optimize the code differently in each version. Some functions were optimized away in kernel, but called in bootloader. I have to create a dummy function in bootloader to avoid linking problem.

4. -std=gnu99. I've met several preprocessing error when I was building gcc in buildroot. cpp was complaining about the unknown labels in assembly code. It turns out the the cpp using gnu standard is not 100% compatible with the iso c standard. By removing the -std=gnu99 flag, I can get gcc compiled.

5. -sysroot. This option will enable you to use a different set of "/include", "/lib" directories in a different root. I think that it may be useful in cross-compiling environment.

Kernel network tune-up

I closed two open tickets related to networking in Linux Kernel. So, I'd write it down in order not to forget.

* broadcast ping problem. After Linux kernel 2.6.14+, the kernel wont' respond to broadcast ping by default. You have to "echo 0 > /proc/sys/net/ipv4/ignore_broadcast_icmp" to enable response to broadcast ping. The default behavior has been changed.

* loopback ping > 32K payload. For small system with <= 16M memory, we use loopback UDP for IPC. But we can't receive packets with 32K+ payload. I wrote a test program to verify the problem. After one day's hack, I found that it is due to a limitation in internal buffer size of skbuff (the internal data structure in Linux network stack). "/proc/sys/net/core/rmem_max", "/proc/sys/net/core/rmem_default". We have to increase the limit on those two entries to allow loopback ping with 32K+ payload. There won't any problem with network ping, as the MTU limitation will eventually split the skbuff and you won't need a big skbuff. The loopback has no MTU limitation, thus it matters.

Kernel programming and driver debugging

Don't forget to kill the klogd and syslogd daemons before you start to debug your driver or kernel codes. Otherwise, your printk will be buffered and you won't know the exact point of crash or hang.

"
killall klogd
killall syslogd
"

For interrupt handler, the first thing in your ISR is to disable or mask out the interrupt, otherwise, the interrupt may keep firing and your system may be locked up.
At the end of your ISR, you can re-enable and enable the mask of the interrupt.

To get a list of interrupt and interrupt handler,
"
cat /proc/interrupts
"

godaddy programming tip

to use php5, your script extension should be .php5, instead of php

my .vimrc

" LEO's customization

" version 4.0

set nocompatible

" Broadcom coding style
set shiftwidth=3
set tabstop=3
set expandtab

" Nice search
set incsearch
set ignorecase
set smartcase

set ai
set backspace=2


"set backupdir=~/.backup
"set nobackup

" My shortcut key mapping
map gs :%s/

map <c-j> <c-w>j
map <c-k> <c-w>k
map <c-h> <c-w>h
map <c-l> <c-w>l

map <f2> @q
map <f3> @a
map <f10> :set paste<cr>
map <f11> <c-w>-
map <f12> <c-w>+

map :Q :qa!

" For nicer scroll, who knows
set showcmd
set sm
set ss=1
set siso=9
set so=3

" highlight search result
set hls

" syntax highlight
syntax on
colorscheme darkblue

"copy and paste betweeen different vim sessions
nmap <c-y> :!echo ""> ~/.vi_tmp<cr><cr>:w! ~/.vi_tmp<cr>
vmap <c-y> :w! ~/.vi_tmp<cr>
nmap <c-p> :r ~/.vi_tmp<cr>
vmap <c-p> c<esc>:r ~/.vi_tmp<cr>
nmap :Q :qa

"expand the directory with pwd of file under editing
nmap ,e :e <c-r>=expand("%:p:h") . "/" <cr>
nmap ,n :new <c-r>=expand("%:p:h") . "/" <cr>

" Remember the last edit position
set viminfo='10,\"100,:20,%,n~/.viminfo
au BufReadPost * if line("'\"") > 0|if line("'\"") <= line("$")|exe("norm '\"")|else|exe "norm $"|endif|endif

" Command abbreviation for spell checking
cab aspe :w<cr>:!aspell -e -x -c %<cr>:e<cr><cr>

highlight RedundantSpaces term=standout ctermbg=red guibg=red
match RedundantSpaces /\s\+$\| \+\ze\t/

set ruler
set autoindent
set smartindent

"set spell

"set statusline=%F%m%r%h%w\ [FORMAT=%{&ff}]\ [TYPE=%Y]\ [ASCII=\%03.3b]\ [HEX=\%02.2B]\ [POS=%04l,%04v][%p%%]\ [LEN=%L]
"set statusline=%F%m%r%h%w\ [[TYPE=%Y]\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ [POS=%04l,%04v][%p%%]\ [LEN=%L]
"set laststatus=2


let perl_extended_vars=1
filetype plugin on " load filetype plugins

set visualbell t_vb=
let loaded_matchparen=1

Working on my first driver

I have been working on the touchscreen driver for the past two weeks. In last Friday, it seems that my work had made a milestone. The user land touchscreen calibration and test programs works just fine using tslib.

The touchscreen driver I created conforms to the input layer and event interface of Linux kernel. I have to use "input_register_driver" and "input_register_device" to register my device and the driver. The "probe" function then called.

I have to work out the proc entries and cleanup the code, and write documentation in next week, which should be done without much trouble.

let's discuss Linux kernel programming here

As told by Dave today, the MIPS kernel stack dump doesn't dump the calling stack in order. It just dump all the seemly kernel functions in the stack. They could be functions called before the real BUG happened.

For example, the following call trace dump is not the real calling stack. The bug is in set_mctrl, which called spin_lock_irqsave, but the same lock was acquired by the caller of set_mctrl function already.

------------------
BUG: spinlock recursion on CPU#0, swapper/1
lock: 80302a84, .magic: dead4ead, .owner: swapper/1, .owner_cpu: 0
Call Trace:
[<801f03c8>] _raw_spin_lock+0x4c/0x154
[<802c2ef4>] _spin_lock_irqsave+0x40/0x58
[<80212bfc>] bcm1103serial_set_mctrl+0x70/0xb8
[<8012e934>] printk+0x1c/0x28
[<802123ac>] uart_add_one_port+0x290/0x354
[<80184320>] exact_match+0x0/0x8
[<80184328>] exact_lock+0x0/0x28
[<801ff9d4>] alloc_tty_driver+0x24/0x64
[<80212088>] uart_register_driver+0x194/0x1cc
[<80350000>] keypad_init+0x48/0x14c
[<80351020>] bcm1103serial_init+0x44/0x68
[<80100558>] init+0xc4/0x29c
[<80100558>] init+0xc4/0x29c
[<80109cd4>] kernel_thread_helper+0x10/0x18
[<80109cc4>] kernel_thread_helper+0x0/0x18
-------------------

Remember, the call trace dump of MIPS is not the exact calling trace, you have to find clue in the functions dumped.

Sort complex dictionary in Python

Use lambda function and sorted function to sort a complex dictionary in Python

>>> a={"a":[1,"a"], "b":[2,"b"], "c":[0,"A"], "d":[-2, "z"]}

>>> a.items()
[('a', [1, 'a']), ('c', [0, 'A']), ('b', [2, 'b']), ('d', [-2, 'z'])]

>>> sorted(a.items(), lambda x, y : cmp(x[1][0], y[1][0]))
[('d', [-2, 'z']), ('c', [0, 'A']), ('a', [1, 'a']), ('b', [2, 'b'])]

>>>sorted(a.items(), lambda x, y : cmp(x[1][1], y[1][1]))
[('c', [0, 'A']), ('a', [1, 'a']), ('b', [2, 'b']), ('d', [-2, 'z'])]

It could be useful.