Linux Kernel Submission

I think it is time to summarize my experience in the submission of bcmring architecture into the linux kernel.

You may find my contribution to the kernel 2.6.32 in the following page. Noted Broadcom ranked No. 4 by lines of changes. Mostly by me. :-)
Linux Kernel Development Statistics

1) subscribe to the arm-linux-kernel mailing list
2) send out your first email asking about other people's attitude about submit a driver/architecture
3) prepare your own git tree, better start with the latest git tree
4) port your core code into the git tree, put only the core code and the minimal amount of drivers to start, like core, dma, serial.
5) change the kernel makefile and Kconfig file to support your architecture
6) prepare a arch_defconfig configuration file, this is used to regression test the build of your architecture
7) split your patches into small chunk
8) run Lindent to make sure there is no errors nor warnings. A lot of kernel developers are picky, you'd prepared to reformat your code, rename the variables as much as possible
9) try a submission, with detailed explanation and message
10) be patient
11) get feedback and prepare another set of submission
12) ask for feedback once a week and push to get feedback, until nobody complain about your code
13) repeat until your code is merged

u-boot again

I am still working the u-boot project. After I got the ARM realview ice debugger, my life is easier, but still tough. I traced the assembly code of Linux kernel and compare the execution of the same kernel booted by both boot2 and u-boot. As the boot2 can successfully boot the kernel image, I just need to make sure u-boot can get to the same state that u-boot created for the kernel.

I figured out the memory starting address configuration is not set to proper value in u-boot today, but checking the kernel parameters. However, after I modified the u-boot code, I still can't get through the same point in kernel image. Anyway, there is a problem to be fixed. I'd keep working on that tomorrow, by checking the memory set by boot2/uboot, before the kernel booted.

The Linux kernel needs some settings in ATAG. Those settings will be read by kernel during the boot stage. There are several ATAG parameters, like the memory starting address, size, command line, arch id, etc.

kernel upgrade to 2.6.27 for mips/arm platform

I have encountered other interesting issue when I was upgrading the kernel from 2.6.23 to 2.6.27 for our MIPS32 platform.

1. the timer, as in 2.6.24 or later, the default MIPS timer has been separated from the old timer code. For our architecture used the default cp0 comparison based timer, we need to enable the r4k timer to get the timer working. I spend couple of days to understand this change. The start_kernel function was able to proceed after the proper is enabled. However, we do have a timer block in our chip, and we can set the timer to a certain frequency as an individual timer source. In my later debugging process, I have enabled the timer block and use our own timer source. It also works.

2. the cache should be disabled when kernel started and re-enable later on. In 2.6.23, the cache was enabled by default. However, in 2.6.27 or some version later than 2.6.23, the cache was configurable by a kernel option "cca=" and it is disabled by default. This change really hurts me. As there are so many changes from 23 to 27 kernel, it is almost impossible for me to notice this change at first. What I have observed at first was the slowness of the system. The BogoMips dropped from about 273 to 3, which is unbelievable. I was doubting the correctness of the timer function at the beginning. I scrutinized the code and well studied the new timer implementation. I even implemented our own timer by using the timer block in our chip. Those doesn't help either. The system was able to boot to busybox but it is really slow. I accidentally tried to use our performance counter program to measure the performance. The performance counter reported the cache hit is 0, which means that there is no cache enabled. I checked our private i/d cache register and they seems enabled. However, I forget to check the setting of the cp0 register of MIPS. There is another setting to enable/disable cache policy. I used a very stupid and old method to pinpoint the problem. I added NOP test to both 23 and 27 kernels. In 23 kernel, when the cache is enabled, the NOP test gives much lower CPI (clock per instruction), otherwise the CPI is high. In 27 kernel, the CPI doesn't change. I tried to figure out the exact point where the CPI drop 23 kernel and check the corresponding code in 27 kernel. I finally found that the default cache policy was disable in 27 kernel, while it is enabled in 23 kernel. By adding the "cca=3" kernel command line option, everything backs to normal, BogoMips, kernel boots properly.


3. Export symbol and export symbol gpl'ed. If your driver, kernel module used the latter symbols, your driver/kernel module must be gpl'ed. This can cause problem for us as we don't want to open source all our kernel modules, especially wlan driver. We deliver binary kernel module for our wlan drivers.

kernel upgrade to 2.6.27

I had just fixed a network driver bug when I upgrade the kernel from 2.6.23 to 2.6.27 for our MIPS platform. It takes couple of weeks. The original problem appears when the NAPI was used in network driver and I changed the net_poll function accordingly. Then, I got kernel panic with memory access failure. After long time debugging, I found that there is some problem when the driver tries to figure out the address of skb out of the skb->data structure. This is weird because the same code was used in both 2.6.23 kernel and 2.6.27 kernel. The original author of the network driver gave some hints that he had experienced similar problem when he was creating a network driver for our next generation chip, based on the old driver. He mentioned that the original driver was confused about the physical/virtual address when accessing the dma'ed memory. This is quite helpful. I spend a whole day dig into this issue, and studied the new driver for next generation chip. After replacing the memory allocation function for the skb buffer, I finally got the proper method to access the memory. I've learned about the physical/virtual/bus address when accessing memory in kernel. The principle was simple as stated by Linus, "use virtual address when accessing memory in kernel, and use bus address when the memory was given to device". In some architecture, the bus address is identical to physical address. Never use physical address directly. The functions: phys_to_virt, virt_to_bus, bus_to_virt, virt_to_phys, are all helper functions.

It seems that we still had a lot of bugs in our network driver. Apparently, our engineers haven't had enough knowledge creating drivers in Linux. Most of their experience was in VxWorks, with flat memory model.

kernel upgrade

I am recently working on kernel upgrade from 2.6.23.17 to 2.6.27-rc5 for our VoIP SoCs.

The following are lessons I learned this time.

* use git to help migrate the patches, as we manage patches in our system, not by git. So, we may use git to help migrate the patches.
* get a kernel git tree (git fetch)
* check out a branch for your current kernel version (git branch, git checkout)
* patch the git tree with your patches (patch, git commit)
* rebase the git tree to the newer kernel (git rebase)
* resolve conflict, for each resolved conflict, remember your changes (git rerere)
* once resolved all conflicts, generate new patches for the new kernel (git format-patch)


* to get your new kernel built for the platform at first
* use make oldconfig to migrate the kernel configurations
* keep all the old configuration as much as possible

* turn on most of the kernel debug option, like early_printk, printk_with_time, spinlock, etc.

* If it doesn't boot, there may be a lot of reasons
* check kernel dump, especially the dumped code, disassemble the whole kernel to figure out where the dump happened
* there are usually a big change in each major version of kernel, such as timer change in mips architecture, path and inclusion changes in arm architecture, sd/mmc infrastructure changes, be very careful.

let's discuss Linux kernel programming here

As told by Dave today, the MIPS kernel stack dump doesn't dump the calling stack in order. It just dump all the seemly kernel functions in the stack. They could be functions called before the real BUG happened.

For example, the following call trace dump is not the real calling stack. The bug is in set_mctrl, which called spin_lock_irqsave, but the same lock was acquired by the caller of set_mctrl function already.

------------------
BUG: spinlock recursion on CPU#0, swapper/1
lock: 80302a84, .magic: dead4ead, .owner: swapper/1, .owner_cpu: 0
Call Trace:
[<801f03c8>] _raw_spin_lock+0x4c/0x154
[<802c2ef4>] _spin_lock_irqsave+0x40/0x58
[<80212bfc>] bcm1103serial_set_mctrl+0x70/0xb8
[<8012e934>] printk+0x1c/0x28
[<802123ac>] uart_add_one_port+0x290/0x354
[<80184320>] exact_match+0x0/0x8
[<80184328>] exact_lock+0x0/0x28
[<801ff9d4>] alloc_tty_driver+0x24/0x64
[<80212088>] uart_register_driver+0x194/0x1cc
[<80350000>] keypad_init+0x48/0x14c
[<80351020>] bcm1103serial_init+0x44/0x68
[<80100558>] init+0xc4/0x29c
[<80100558>] init+0xc4/0x29c
[<80109cd4>] kernel_thread_helper+0x10/0x18
[<80109cc4>] kernel_thread_helper+0x0/0x18
-------------------

Remember, the call trace dump of MIPS is not the exact calling trace, you have to find clue in the functions dumped.