caritas' space

X application startup script with wnck

2010-03-29T04:56:00.000-07:00

It is common that several X applications are involved for one task. For example, web browser is used for searching and help document, Emacs is used to edit, and X termainal is used for testing. Here, script can be used to startup several X programs with one command typing or click.

But simple shell script can only startup applications. You must arrange the screen layout by hand. In fact the screen layout can be program with script too. Python + wnck can be used for that.

The script as follow is an example. It startup pidgin and gnome-terminal, minimize pidgin and maximize gnome-terminal vertically. With wnck, you can arrage the layout of your X applications arbitrarily.

#!/usr/bin/python

import gobject
import gtk
import wnck
import os
import time

def gtk_wait():
    gobject.idle_add(gtk.main_quit)
    gtk.main()

def get_wnck_apps_from_wins(wins):
    apps = set()
    for win in wins:
        app = win.get_application()
        apps.add(app)
    return list(apps)

def screen_get_wnck_apps(s):
    wins = s.get_windows()
    return get_wnck_apps_from_wins(wins)

def screen_find_windows_by_pid(s, pid):
    wins = []
    all_wins = s.get_windows()
    for win in all_wins:
        app = win.get_application()
        if app.get_pid() == pid:
            wins.append(win)
    return wins

def gtk_wait_timeout():
    gtk_wait()
    time.sleep(0.1)

def apply_timeout(func, args, timeout = 10, wait = gtk_wait_timeout):
    start = time.time()
    res = None
    while not res:
        res = apply(func, args)
        now = time.time()
        if now - start > timeout:
            break
        wait()
    return res

def wait_child_proc():
    try:
        os.waitpid(-1, os.WNOHANG)
    except OSError, err:
        if err.errno == 10: # No child processes
            pass

def pid_is_valid(pid):
    return os.path.exists('/proc/%d' % (pid,))

def screen_find_windows_by_pid_timeout(s, pid):
    while True:
        wins = apply_timeout(screen_find_windows_by_pid, (s, pid), 0.5)
        if wins:
            break
        wait_child_proc()
        if not pid_is_valid(pid):
            break
    return wins

def init():
    global screen
    screen = wnck.screen_get_default()
    gtk_wait()

class app(object):
    def __init__(self, pid):
        self.pid = pid
        self.update()
    def update(self):
        self.wnck_wins = screen_find_windows_by_pid_timeout(screen, self.pid)
        self.wnck_apps = get_wnck_apps_from_wins(self.wnck_wins)
    def minimize(self):
        for win in self.wnck_wins:
            win.minimize()
    def maximize(self):
        for win in self.wnck_wins:
            wt = win.get_window_type()
            if wt == wnck.WINDOW_NORMAL:
                win.maximize()
    def maximize_vertically(self):
        for win in self.wnck_wins:
            wt = win.get_window_type()
            if wt == wnck.WINDOW_NORMAL:
                win.maximize_vertically()
    def close(self):
        for win in self.wnck_wins:
            win.close(0)

class app_info(object):
    def __init__(self, start_argv, app_name):
        object.__init__(self)
        self.start_argv = start_argv
        self.app_name = app_name
    def get_instances(self):
        pids = []
        wnck_apps = screen_get_wnck_apps(screen)
        for wnck_app in wnck_apps:
            if wnck_app.get_name() == self.app_name:
                pids.append(wnck_app.get_pid())
        if pids:
            return [app(pid) for pid in pids]
        else:
            return [self.new_instance()]
    def new_instance(self):
        argv = self.start_argv[:]
        argv.insert(0, 'nohup')
        pid = os.spawnvp(os.P_NOWAIT, 'nohup', argv)
        return app(pid)

def start_apps():
    pidgin_info = app_info(['pidgin'], 'Pidgin')
    terminal_info = app_info(['gnome-terminal'], 'Terminal')

    os.system('rm -f nohup.out ~/nohup.out')

    apps = pidgin_info.get_instances()
    apps = pidgin_info.get_instances()
    for app in apps:
        app.close()

    apps = terminal_info.get_instances()
    for app in apps:
        app.maximize_vertically()

init()
start_apps()

Disk reading performance with hdparm

2010-03-29T04:50:00.000-07:00

hdparm can be used to get disk reading performance with following command line:

hdparm -T <dev>. Get cached disk reading performance. In fact, the page cache performance is tested.
hdparm -t <dev>. Get raw disk reading performance.

Implementation

hdparm -T

ioctl(devfd, BLKFLSBUF, NULL);
read(devfd, buf, READING_SIZE)
do {
 seek(devfd, 0, SEEK_SET);
 read(devfd, buf, READING_SIZE);
} while (/* time is not over */);

Because the data to be read is in page cache always (if READING_SIZE << memory size), the result is page cache performance.

hdparm -t

ioctl(devfd, BLKFLSBUF, NULL);
do {
 read(devfd, buf, READING_SIZE);
} while (/* time is not over */);

Because the data to be read is not in page cache, and the bottleneck lies in raw disk reading speed, the result is raw disk reading performance.

reading prepare *         *         *
memory copy          -         -
raw disk read  ++++++++  ++++++++

As in above figure, reading reparation and memory copying in testing is neglectable, especially when readahead is considered.

Reading notes: Linux kernel perf event source

2010-03-29T04:46:00.000-07:00

For Linux kernel 2.6.33

perf event enumeration

Kernel has no hardware perf event enumeration interface.

tools command line: perf list
Available abstract hardware perf events are hard-coded in perf user space tool. Raw hardware perf events are not enumerated by the tool, they can be specified via a 64-bit number in the tool command line. Abstract hardware perf events are implemented with the raw hardware perf events. Whether the abstract hardware perf events are available can be queried via syscall perf_event_open.
Software trace point are exported in debugfs (/debugfs/tracing/events)

perf event management

There is at most one perf event context for each task and each CPU. This is used to manage all perf events inside one context (task or CPU).
All perf events in a context are linked as list in context->event_list.
Perf events in one context are organized into groups. All group leaders in a context is linked as list in context->group_list. All perf events in a group is linked as list in group_leader->sibling_list.
It seems that perf events in the group share some attributes, such as enable/disable, cpu, inherit, etc. In "perf record", for task events, one group is created for perf events for one CPU. In __perf_event_sched_in, it is assumed perf events in one group is for one CPU. But in "perf_record", group support can not be turned on now, that is, group is not used at all now.
perf events inherited in child tasks are linked as list in parent_perf_event->child_list.
In "perf record", for task profile, for one perf event type, one perf event is created for each task and each CPU. For perf event operation, it appears that one perf event for each task is sufficient, because one task can only run on one CPU at any time. But perf events may be inherited by children perf events in children tasks (forked tasks). The children perf events use the original perf events for sample output. To make sample ring buffer a per-CPU data structure, perf events are created for each CPU too. So that, when parent and child tasks run on different CPU simultaneously, they all use perf events in parent task for output, but they use different perf events for corresponding CPU.

perf event state track

task schedule in/out (perf_event_task_sched_in/out). Record/disable counter before schedule out, restore/enable counter after schedule in. The schedule in/out event is recorded too.
task fork
- perf_event_fork: record fork event.
- perf_event_init_task: perf_event->attr.inherit control whether to inherit. If inheriting, child perf event will be created for child task. Children events information is accumulated in some statistics. The sample collected via children events are sent to corresponding parent events for same CPU.
task exit
- perf_event_exit_task: feed back event values to parent events (child_total_time_running, etc). Free all resources for children events.
fd closed, perf_release: free all resources. perf event in original task is managed by corresponding fd. Children perf events in children tasks are freed in perf_event_exit_task.

Dataflow

For perf event samples:

hardware counter ---> perf event mmap data page -> user space tool (perf)
                  |
soft trace point -/

For ftrace and tracing point:

 software trace point -> tracing ring buffer -> user space

User space interface

File based interface. File descriptor is obtained via a new syscall perf_event_open. Because one file describptor is needed for each perf event. Perf event may for a specific task, cpu, etc.
memory map. Sample data is passed from kernel to user space mainly via memory map. Used to implemend shared memory based ring buffer. Details are in ring buffer.
ring buffer
- Work in both per-CPU and not-per-CPU mode
- write side in kernel, read side in user space. shared memory communication between kernel and user space. It has better performance than memory copy.
- lock-less write side, similar as Steven Rostedt's unified tracing buffer, perf_mmap_data.head is used to reserve space, while perf_mmap_data.user_page->data_head is used for committed data.

Code generation for software tracing point

A set of macro is defined for software tracing point related code generation, such as binary format description, trace point function generation, etc.
Some macros are un-defined and re-defined again and again, and some files are included again and again for different set of macro definitions. This is tricky but really powerful. Source code is in include/trace/define_trace.h and include/trace/ftrace.h.

Tracing and perf event

Lock-less trace ring buffer is not used in perf event, another simple ring buffer implementation for mmap data pages is used instead.
For software tracing point, the tracing point sample collecting code is shared between tracing and perf event.

Random thought

ring buffer in perf event has some special features.
- work in both per-CPU and not-per-CPU mode
- shared memory communication between kernel and user space, with write side in kernel and read side in user space.
In the code generation for software tracing point
- macro is un-defined and re-defined again and again, and some files are included again and again for different set of macro definitions.

SWRAP: Session WRAPper

2010-03-04T04:56:00.000-08:00

Why we need this?

Even when working on one project, usually multiple software are needed. For example, in programming, emacs is used to edit source code, one bash is used for testing, another is used for compiling. It is tedious to startup them, open latest file, type latest command etc. screen and session management of the software can be used to facilitate this. Multiple software can be started by screen automatically if you specify them in ~/.screenrc, Latest opened files can be saved by emacs desktop mechanism, latest typed command can be remember by bash history.

But it is common to work on multiple projects simultaneously. Even if you work on just one project, you may use your computer both for work and leisure. The configuration and saved state for different projects may conflict. For example, two project may use different coding style, so that different emacs configuration are needed. The bash history from one project may flush that from another project. swrap provides a way to manage mutiple configuration and saved state for each software using a simple directory based scheme. A group of software for one project can be managed with a directory hierarchy. While multiple projects are managed with multiple directory hierarchy.

Introduction

swrap stands for Session WRAPper, it provides a directory based session management for some software. The session here incude configuration files, saved state files etc. All configuration files, saved state files and other related files for each session are put into one directory. So that you can have multiple configuration, saved state at the same time. For example, you can have multiple bash histories, one for each session. This way, if you work on multiple projects simultaneously, bash history uses in one project will not flush the bash history used in another project.

Usage

swrap <command> <session directory> [<arg1> <arg2> ...]

Where

<command> is software that is wrapped.
<session directory> is the directory to store session related file, which can be an absolute, relative path name or a directory under ~/sessions. The base name of the <session directory> will be used as the session name at least for screen.
<arg1> <arg2> ... are arguments for <command>.

Supported software

Bash

Configuration

.bashrc in session directory instead of the global one will be used if provided.

Saved state

A dedicated .bash_history in session directory will be used instead of the global one.

Emacs

Configuration

emacs_init.el in session directory will be used as initialization file in addition to the default one.

Saved state

Emacs desktop is used to save state, and saved state (.emacs.desktop) will be put into session directory instead of emacs startup directory or home directory.

Screen

Configuration

screenrc in session directory will be used as initialization file instead of default one, but the default one can be loaded in this file if desired.
screen is used to manage other software, so screen session can be seen as a compound session, while software run inside screen can be seen as sub-sesssion. So it is natural to make session directories for software run inside screen the sub-directories of screen session directory. These sub-directories can be specified as session directories for software run in screen in screenrc.

Saved state

No saved state is supported

Example

Directory "example" shows a project session directory hierarchy managed by swrap. There are 3 software in project: screen, emacs and bash. "example" is session directory for screen, "example/screenrc" is screen configuration file. "example/0" is session directory for emacs, which is in window 0 of screen. "example/1" is session directory for bash, which is in window 1 of screen. "example/screenrc" is as follow:

source ~/.screenrc

chdir ~/projects/mce/apei/firmwarekit/linuxfirmwarekit
screen -t build 1
stuff "exec swrap bash $SESSION_DIR/1""^M"
stuff "clear""^M"

screen -t emacs 0
stuff "swrap emacs $SESSION_DIR/0 --no-window-system""^M"

Where swrap is used to execute software in screen windows, and corresponding session directories are specified too.

When you execute:

  swrap screen <path to>/example

  cp -r <path to>/example ~/sessions
  sscreen example

You will return to your latest project state or a new project state will be created for you.

Download

swrap can be downloaded at here.

Reading notes: gnome desktop user guide

2010-03-04T04:52:00.000-08:00

Overall
- Shift and Alt is often used with mouse to change the default behavior.
Keyboard shortcut
- Ctrl+Alt+D: show desktop
- Ctrl+Alt+Tab: switch focus between desktop and panels
- Alt+F9: Minimize window
In Open/Save File Dialog
- Show Hidden Files can be toggle with right click
- Bookmarks can be added/removed
When moving a panel object, alt and shift key can be used to change movement mode (switched, free, push).
In Nautilus spatial mode, the folder can be opened with parent folder closed with "shift" pressed. I don't like leaving too much folder windows open or closing them by hand when navigating in file hierarchy.
In Nautilus sptial mode, the browser mode can be entered with right click menu: "Browser Folder" on folder.
Search can be saved with "File -> Save Search" in Nautilus.
In Nautilus icon view, items can be sorted with "View -> Arrange Items -> ...".
In Nautilus, "glob" can be used to select items with "Edit -> Select Items Matching ...".
$HOME/Templates can be used to place arbitrary document templates, which can be used in "Create Document" menu in Nautilus.
Nautilus sripts are placed in "$HOME/.gnome2/nautilus-scripts". Some special environment variables are used to passes some infomation.
Custom keyboard shortcut can be added in "Keyboard Shortcuts" preference dialog directly, need not gconf-editor.

Revise muse-index

2010-01-30T03:37:00.000-08:00

Muse is a document writing/publishing tool for Emacs.

Muse-index can be used to show all document in a muse project. But now the file name is shown and sorted by it. I think it will be more useful if document title and create or modify time is shown and sorted by create or modify time. So I revise the muse to do that. The code can be found here.

File Synchronization between N78 and PC

2010-01-30T03:32:00.000-08:00

My Nokia N78 have no WIFI support and local GPRS/3G data connection fee is quite high, so I can not synchronize large amount of data between N78 and PC with cloud service. I use N78 disk mode USB link to synchronize, details are as follow. Tested in Debian (gnome version 2.28).

In fact, this is not N78 specific, and can be used to sync files with any devices mountable in Linux.

Setup gnome auto-mount VFAT options

Traditionally VFAT file system is case insensitive, but can preserve case optionally, on the other hand, Linux native file system is case sensitive. So VFAT should be mounted in case sensitive mode for synchronizing software to work properly.

Gnome auto-mount VFAT options can be setup as follow:

Start gconf-editor (System tools -> Configuration editor), check key: /system/storage/default_options/vfat/mount_options, make sure there is "shortname=mixed" instead of something like "shortname=lower" (default value for some distributions) in the value. And "iocharset=utf8" should be set in mount_options too.

Setup autorun upon USB connecting

In gnome, open nautilus-file-management-properties (System -> Preferences -> File management), select tab named "Media", for media handling, set "Open Autorun Prompt" for "Software" media. I have thought gnome-volume-manager will be responsible for autorun script execution, but that is turned off in Debian, and autorun mechanism in nautilus is used instead.

Create a file named ".autorun" in the root directory of the N78 micro-SD card. This is the synchronization script.

Unidirectional synchronization via rsync

Rsync can be used for unidirectional synchronization, such as backup some directories (photos, notes etc) from N78 to PC or push some directories (music, podcast etc) vice versa.

One issue here is that VFAT time resolution is not enough for rsync. This can be fixed via specifying "—modify-window=2" in command line.

Bidirectional synchronization

It seems that unison can be used for bidirectional synchronization. But I have not tried it. Unidirectional synchronization is enough for me.

My autorun script

.autorun in N78

#!/bin/sh

/usr/local/bin/autorun_sync N78

/usr/local/bin/autorun_sync

#!/bin/sh

usage()
{
 echo "Usage: $prog <media>"
}

prog=$(basename $0)

if [ $# -lt 1 ]; then
 usage
 echo -n "Press any key to continue:"
 read
 exit -1
fi

media=$1
mdir=/media/$media

ldir=$HOME/sync/$media
odir=$ldir/data_out
idir=$ldir/data_in

echo "----------------------------------------------"
echo -n "Rsyncing ..."
rsync -rutv --delete --modify-window=2 $odir/* $mdir/

if [ -f $ldir/dirs.in ]; then
 while read d p; do
  if [ -z "$d" ]; then
   continue;
  fi
  rsync -rutv --modify-window=2 $p $mdir/$d $idir/
 done < $ldir/dirs.in
fi
echo "Done!"
echo "----------------------------------------------"
echo -n "Syncing ..."
sync
echo "Done!"
echo "----------------------------------------------"
echo -n "Unmounting ..."
umount /media/$media
echo "Done!"
echo "----------------------------------------------"
echo -n "Done! press any key to close:"
read

$HOME/sync/N78/dirs.in

Sounds

Work with multiple devices

2010-01-30T03:29:00.000-08:00

One big issue of working with multiple devices is synchronizing necessary information among them. For example, synchronizing the document you are writing between multiple PC (notebook, desktop, netbook, handheld), so that you can write it in any time. Synchronizing the web site you are interested between multiple PC and your phone (has web accessing). So the information need to be synchronized should include:

PIM information such as calendar, task and contacts etc.
Mails
Files, including the document you are writing and reading. The bookmark information about your reading progress is necessary too, this can be a reading application specific meta-data file.
Web bookmarks, this can be used for URL you want to read but have no time now.

One of the most convenient method to synchronize information amongst multiple devices so far is "cloud" based synchronization. One of the biggest advantages of this method is that it needs not turn on and connect two or more machines simultaneously for synchronization. The disadvantage of this method is the possibility to leak your privacy information, so should use encryption for sensitive information.

I use following cloud services:

Google PIM (contacts, calendar, tasks?) for PIM information. On my N78, Google PIM Moblie Sync can be setup as in Nokia Setup Instructions for Google Mobile Sync. Contacts, calendar is supported, but tasks is not supported. On Linux PC, Google PIM Sync can be setup as in How to sync Evolution with Google's PIM apps. Contacts, calendar is supported, but tasks is not supported too.
Google GMail for mails. Google Mobile Sync support pushing mail, but my N78 does not support it. But I don't need that too. On Linux PC, GMail can be accessed with IMAP protocol.
Dropbox for file synchronization. Dropbox is not opensource software, but UbuntuOne does not work under Debian so far. I may switch to UbuntuOne after it support Debian. Dropbox does not support N78. I use rsync based script to synchronize some data between N78 and my primary PC.
Google bookmarks for web bookmarks. N78 can access google bookmarks via web browser, but it is not convenient. Firefox GMarks addon works good with Google bookmarks on PC.

Sony VAIO P

2010-01-30T03:14:00.000-08:00

Working on a Sony VAIO P for two days. Some thoughts are as follow.

The screen is very good. The backlight is better than my Lenovo Thinkpad X61T, even readable under sun light. But the viewing angle is much worse than X61T. The resolution is good too. The screen size is OK for most tasks, but sometimes I want a bigger screen.
It's very thin and light (about 700g?). I can hold it comfortable for hours. So it is very good to be used in holding style. X61T is a little heavy for me to hold (about 1.9kg).
Keyboard and trackpoint is good, although the keyboard is a little small for my big hand. I think I can get more comfortable with the keyboard size after more usage. But I hate the position of right shift key. Don't know how long will it take for me to fit in with it. I think I will not programming with it.
Terrible Linux support, mainly the graphics drivers. Why does Intel use such graphic card? Hope 2D open source driver ready soon.
The CPU of the model I used is only 800MHZ. So the performance is not very good. Not too bad, but not satisfactory too.
The battery life is too limited, only 2 to 3 hours.

All in all, I think Sony VAIO P is not the right choice for me. A netbook or CULV notebook with bigger keyboard and screen, longer battery life, better performance and acceptable weight (about 1kg) is better for me.

Linux Softwares for Desktop

2010-01-30T03:11:00.000-08:00

Multimedia

EBook management

I need an ebook management software, like rhythmbox for musics or f-spot for photos.

Photo management

F-spot is an exellent software for photo management. I love it! But it is not very stable, hope it can improve with the time.
Comix is good for comics, although I seldom use it.

Music management

I think rhythmbox is good. It seems that MP3 ID3 tags editing changes some internal database instead of file itself.
For convert the encoding of ID3 tag of MP3 file from GBK to UTF-8, you can use python-mutagen. Command line is: "mid3iconv -e gbk .mp3".

To edit ID3 tags of MP3 file, exfalso is good. It seems that easytag is better?

Backup

I think backup2l is good for me, because it is very easy to configure.
I run backup2l in my working machine for periodic backuping, use rsync to synchronize the backup files to my portable backup hard disk.
rsync is used to backup my N78 phone information. It will be run automatically upon plug via ".autorun" mechanism.

Import/sync N78 contacts into/with Evolution

2010-01-30T03:03:00.001-08:00

Backup N78 contacts via vcard

In address book of N78, contacts can be backupped from phone to memory card via menu. By default the backupped contacts is saved in Others/Contacts directory of micro-SD card. There is one vcard file (.vcf) for one contact.

Import vcard into Evolution

Vcard can be imported into Evolution, but only one vcard file once. To import vcard files generated by N78, the vcard files from N78 can be catenated into one vcard file, then imported into Evolution.

The vcard file generated by N78 is compatible with Evolution, that is, all information can be imported properly so far. But some information is not displayed in Evolution, and not available when accessed via python-evolution (should be not available in other libedata client too). After changing any field then change back to original value and save the "changes", evething goes OK. Appears a BUG of Evolution, hopes that can be fixed in the future.

Multple contacts in Evolution can exported into one vcard file too. Just select multiple contacts, then click "Save Contact as VCard" in menu, Or save all contacts in one address book with "Save Address Book as VCard".

Import vcard into Evolution via python

Because of the BUG of Evolution vcard importing, I try python-evolution and python-vobject to import N78 vcards into Evolution.

python-vobject is not completely compatible with the vcard 2.1 version, which is used by N78. This can be fixed with simple python scripting or via importing vcards into Evolution then exporting them because Evolution will export vcard 3.0 version file.

This method works good for me. But I encountered one issue too, the address can not be imported because EContactAddress is not wrapped properly in python-evolution yet. Hope that will be resolved in the future. Fortunately, there is only one not so important contact in my N78 address book has address field set.

Import vcard into Evolution via opensync

msynctool --addgroup evo2-file
msynctool --addmember evo2-file evo2-sync
msynctool --addmember evo2-file file-sync
msynctool --configure evo2-file 1
#
# You can keep it as default or use the address book URL in <address_path>
#
msynctool --configure evo2-file 2
#
# Set directory hold the vcard files from N78 in <path>
#

There is issue for this method too. Some information such as phone number is not displayed in imported Evolution contacts, similar with importing vcard with Evolution itself, and can be fixed in same way too. But situation is much better here, that is, more information can be displayed, at least one phone number for each contact.

Bidirectional synchronization with Evolution via opensync

After testing, I found N78 can be bidirectionally synchronized with Evolution via N78 syncml over USB. But again, there is some issues for method too. Like file-Evolution sync via opensync, some information can not be displayed correctly. But among supported object type, contact has the best support, then calendar and task, but notes must be turned off, or opensync will hang.

Firstly, setup USB device file permission for opensync user space driver. Add following lines into udev rules file or put into a rule file under rules.d directory of udev.

BUS=="usb", SYSFS{idProduct}=="007b", SYSFS{idVendor}=="0421", MODE="660", GROUP="plugdev"

Command line as follow can be used to test the USB link.

syncml-obex-client -u
# Found 2 USB OBEX interfaces
# Interface 0:
#         Manufacturer: Nokia
#         Product: N78
#         Interface description: SYNCML-SYNC
# Interface 1:
#         Manufacturer: Nokia
#         Product: N78
#         Interface description: PC Suite Services
syncml-obex-client -u 0 --identifier "PC Suite" --slow-sync text/x-vcard Contacts --wbxml
syncml-obex-client -u 0 --identifier "PC Suite" --slow-sync text/x-vcalendar Calendar --wbxml
syncml-obex-client -u 0 --identifier "PC Suite" --slow-sync text/plain Notes --wbxml

Command line for setup is as follow.

msynctool --addgroup n78-evo2
msynctool --addmember n78-evo2 syncml-obex-client
msynctool --addmember n78-evo2 evo2-sync
msynctool --configure n78-evo2 1
msynctool --configure n78-evo2 2
#
# You can keep it as default or use the address book URL in <address_path>
#

syncml-obex-client configuration is as follow. The notes database name should be left as empty, because notes synchronization for Evolution is not supported, opensync will hang if you set notes database name.

<?xml version="1.0"?>
<config>
  <!-- (Only for bluetooth) The bluetooth address if the bluetooth mode is selected -->
  <bluetooth_address></bluetooth_address>

  <!-- (Only for bluetooth) The bluetooth channel to use. `sdptool browse $MAC` to search for the correct channel -->
  <bluetooth_channel></bluetooth_channel>

  <!-- (Only for USB) The usb interface number of the SYNCML-SYNC target. use syncml-obex-client -u (you will need access to the USB raw device) to find it. -->
  <interface>0</interface>

  <!-- The string that the plugin will use to identify itself. Some devices need a special string here. -->
  <identifier>PC Suite</identifier>

  <!-- The syncml version to use: 0 for 1.0, 1 for 1.1 and 2 for 1.2 -->
  <version>1</version>

  <!-- if the plugin should use wbxml -->
  <wbxml>1</wbxml>

  <!-- The username to use. Leave empty to not require a username -->
  <username></username>

  <!-- the password for the username -->
  <password></password>

  <!-- sets the connection type to use. 5 means obex over usb, 2 means obex over bluetooth -->
  <type>5</type>

  <!-- If wbxml is enabled, defines wether the wbxml should use string tables -->
  <usestringtable>0</usestringtable>

  <!-- Never send ADD command, but send REPLACE (not needed normally) -->
  <onlyreplace>0</onlyreplace>

  <!-- Workaround around for mobile phones which only use local timestamps and _no_ UTC timestamps! -->
  <onlyLocaltime>0</onlyLocaltime>

  <!-- Sets the maximum allowed size in bytes of incoming messages (some device need this option set). Example: 10000 -->
  <recvLimit>0</recvLimit>

  <maxObjSize>0</maxObjSize>

  <!-- The name of the contacts db. Must be the same as the phones sends -->
  <contact_db>Contacts</contact_db>

  <!-- The name of the calendar db. Must be the same as the phones sends -->
  <calendar_db>Calendar</calendar_db>

  <!-- The name of the note db. Must be the same as the phones sends -->
  <note_db></note_db>
</config>

Google Mobile Sync

The last and the best so far. I use it now. The best features of this method are as follow:

Works good so far, although some information is not displayed inEvolution, similar with other Evolution importing method.
Synchronizing automatically, periodically.
Wireless, Evolution side uses WIFI, and N78 side uses GPRS (because my N78 has no WIFI). Some methods above can be done in wireless way too, such as with bluetooth, with some additional work.
You need not keep both PC and N78 on for synchronization, because you have the third side now, the Google, in other word, the cloud.

To setup N78 side, you can refer to: Nokia Setup Instructions for Google Mobile Sync.

To setup Evolution side, you can refer to: How to sync Evolution with Google's PIM apps.

Embolden font for PDF rendering

2010-01-30T02:12:00.000-08:00

On E-ink devices, PDF seems to be rendered a little light, the readability will improve if it can be rendered a little darker. I found a way to do that, with the "FT_Outline_Embolden", the patch for poppler can be downloaded from here.

Screenshots are as follow:

Hope useful for E-ink PDF viewer or PDF to image converter.

Cellwriter translucence and docking

2010-01-30T02:08:00.000-08:00

Cellwriter is my primary on screen virtual keyboard now. But the docking or translucence does not work for me until recently. Now I finally find a way to make it work.

Translucence

Translucence is not the cellwriter's business in fact. It is done by a composite window manager such as compiz. To configure it in compiz, using compizconfig setting manager:

Enable "Regex Matching" under "Utility" section.
Enable "Opacity, Brightness and Saturation" under "Accessibility" section. Configure this item:
- In "Opacity" tab page, add "Window specific settings", "class=Cellwriter" for "Windows", "50" for "Window values".

Docking

Cellwriter has full docking support in fact. But it does not work properly in metacity on x86_64. Seems there is a bug of cellwriter and an issue of metacity of gtk+.

Enable "window docking" in cellwriter setup.
Invoking cellwriter with command line option "—window-struts".
Apply the patches to cellwriter (1.3.4).

At first, I found _NET_WM_STRUT_PARTIAL has no effect. After comparing the window strut setting of cellwriter and gnome-panel, I found the main difference lies in data type of struts structure passed into XChangeProperty. Cellwriter uses int, while gnome-panel uses gulong. So I try gulong in cellwriter, and it works like a charm!

Although the format of struts structure passed into XChangeProperty is 32, in "man XChangeProperty", I found following words:

"If the specified format is 32, the property data must be a long array."

So format 32 should be 64-bit on 64 bit machine in fact, and struts should be long instead of int.

Switch network configuration in Debian

2010-01-29T20:07:00.000-08:00

Home and office have different network configuration, most lies in proxy setup. A simple script as follow is used to switch configuration. Hope it is useful for someone else too.

#!/bin/bash

prog=$(basename $0)

usage()
{
 echo "Usage: $prog <home|work>"
}

if [ $# -ne 1 ]; then
 usage
 exit -1
fi

tnet=$1
if [ $tnet != "home" -a $tnet != "work" ]; then
 usage
 exit -1
fi

if [ "$tnet" == "work" ]; then
 sudo cp /etc/apt/apt.conf.avail/99proxy /etc/apt/apt.conf.d
 gconftool-2 -s -t string /system/proxy/mode auto
else
 sudo rm -f /etc/apt/apt.conf.d/99proxy
 gconftool-2 -s -t string /system/proxy/mode none
fi

Gnome notify in command line

2010-01-29T20:05:00.000-08:00

Method is copied from somebody's blog, can't remember who.

#!/usr/bin/python

import pynotify
import sys

def notify(title="Notify", message=""):
    n = pynotify.Notification(title, message)
    n.show()


if __name__ == '__main__':
    pynotify.init("cli notify")
    if len(sys.argv) > 1:
        title = sys.argv[1]
        msg = " ".join(sys.argv[2:])
        notify(title, msg)
    else:
        notify()

Switch distribution version in Debian

2010-01-29T20:04:00.000-08:00

Use testing version of Debian as my primary working environment. But some softwares are available only in sid version, and bugs are fixed in sid firstly. So switching between testing and sid is routine. A simple script as follow is used to simplify/accelerate the operation.

#!/bin/sh

prog=$(basename $0)

usage()
{
        echo "Usage: $prog <testing|sid>"
}

current_dist()
{
        if grep sid /etc/apt/sources.list > /dev/null; then
                echo sid
        else
                echo testing
        fi
}

if [ $# -ne 1 ]; then
        usage
        exit -1
fi

tdist=$1
if [ $tdist != "sid" -a $tdist != "testing" ]; then
        usage
        exit -1
fi

cdist=$(current_dist)
if [ "$tdist" != "$cdist" ]; then
        cp /etc/apt/sources.list.$tdist /etc/apt/sources.list
        mv /var/lib/apt/lists /var/lib/apt/lists.$cdist
        if [ -d /var/lib/apt/lists.$tdist ]; then
                mv /var/lib/apt/lists.$tdist /var/lib/apt/lists
        fi
fi

apt-get update

Debug bash function interactively

2009-06-23T06:04:00.000-07:00


#!/bin/bash

# processing command line parameters
if [ $# -eq 2 ]; then
    p1=$1
    p2=$2
fi

# setup some environment
export A=a
export B=b

tmpfile=/tmp/myshell-$$-$RANDOM

trap "rm $tmpfile" EXIT

cat <<EOF > $tmpfile
if [ -f /etc/bash.bashrc ]; then
    source /etc/bash.bashrc
fi

if [ -f $HOME/.bashrc ]; then
    source $HOME/.bashrc
fi

source <shell functions in this file>

export PS1="My Shell$ "
EOF

/bin/bash --rcfile $tmpfile

Shell script parameters - a simpler way

2009-06-23T06:01:00.000-07:00


#!/bin/sh

for param in "$@"; do
    eval "$param"
done

param1=${param1:-param1}
param2=${param2:-param2}
param3=${param3:-param3}

echo $param1 $param2 $param3

Usage:


param.sh param1=<param1> param2=<param2> param3=<param3>

GNOME desktop layout for small screen

2009-06-19T06:45:00.000-07:00

Default GNOME desktop layout is for typical workstation screen (> XGA). For smaller screen such as netbook (typically 1024x600), too much vertical space is been occupied by two desktop panel and window title bar. This is worse for wide screen, which has relatively small vertical space.

Desktop screenshot before optimizing

To optimize vertical space utility, the following method can be used:

Remove bottom panel (or top panel)
Replace expanded menubar in panel with collapsed one.
Compile and install window-picker-applet, which provides a compact task icon list and focus task title.
It can downloaded from here. I write a small patch to add window maximizing toggling support to it, which can be downloaded from here.
Remove window title bar for maximized window. For metacity this can be done via theme customization, that is, change the "frame_geometry" with name "normal_maximized" as follow:

 <frame_geometry name="normal_maximized" parent="normal"  rounded_top_left="false"
rounded_top_right="false"  rounded_bottom_left="false" rounded_bottom_right="false"  has_title="false">
 <!-- strip frame spacing off the normal geometry when maximized -->
 <distance name="left_width" value="0"/>
 <distance name="right_width" value="0"/>
 <distance name="bottom_height" value="0"/>
 <distance name="left_titlebar_edge" value="0"/>
 <distance name="right_titlebar_edge" value="0"/>
 <distance name="title_vertical_pad" value="0"/>
 <border name="title_border" left="0" right="0" top="0" bottom="0"/>
</frame_geometry>

Desktop screenshot after optimizing

Reading notes: optimizing Linux performance

2009-06-13T20:44:00.001-07:00

Chapter 3

vmstat can show swap in/out pages. A simultaneously high swap-in and swap-out rate could indicate that the system does not have enough memory to handle all the running processes.
slabtop is "top" for slab.

Chapter 4

strace can show system call performance statistics too (strace -c).
ltrace: performance statistics for library (like strace, but library function instead of system call).
ld.so: dynamic load/link statistics for libraries. Example: env LD_DEBUG=statistics LD_DEBUG_OUTPUT=lddebug gcalctool

Chapter 6

iostate: show disk IO statistics

Chapter 7

netstat -c: period update
netstat -t: choose tcp protocol
netstat -l, -a: listening sockets or listening + conntected.

Chapter 8

script: record what you have done and the output, seems useful.
gcc -g3: provide more debug information then gcc -g (default to
- g2), such as the macro definitions present in the source.

Chapter 11

Use gdb to check call chain:
- Set a breakpoint at check-point
- gdb can execute a given set of commands when it hits a breakpoint. By using the command command, we can tell gdb to execute bt; cont every time it hits the breakpoint.
- Conditional breakpoint can be implemented via gdb script (commands) too. For example, we can enable breakpoint 2, when we reach breakpoint 1.
- gdb output can be logged to a text file via: set logging on
oprofile reveals which function is hot, but do not show how many times it is called. ltrace or some other traces shows which function (including subfunctions) consumes most CPU time. This is from two perspective, and they can be combined to inspect the performance property.
For Fedora and Enterprise Linux, Red Hat provides a set of debuginfo rpms that contain all the symbol information and sources that were generated by the compiler when the application was complied. Each binary package or library has a corresponding debuginfo rpm that contains the debugging information. This allows Red Hat to ship the binaries without the disk-space–consuming debugging information. However, it allows developers, or those investigating performance problems, to download the appropriate debuginfo packages and use them. In this case, Red Hat's version of oprofile will also recognize the debuginfo packages and pick up the symbols when profiling both an application, such a nautilus, and a library, such as gtk.
- Debian has similar binary debug symbol packages: -dbg

Chapter 12

strace to find bottleneck of IO intensive application.
- Oprofile can be used only for CPU intensive application. Because IO is performed in kernel space, strace is the right tool to reveal which kind of IO application is performing.

Chapter 13

This call-tree tool would be useful even if it dramatically slowed down application performance as it runs. A common way of using this would be to run oprofile to figure out which functions in an application are "hot," and then run the call-tree program to figure out why the application called them.

Reading notes of internals of the RT patch

2009-06-13T20:42:00.000-07:00

Real-time is something about predictability instead of performance, throughput or latency. That is, real-time system may have worse average throughput or latency, but it will have better largest latency or lowest throughput. Linux kernel have great average performance but the worst one is not so good.
The drawback of Robert Love's kernel preemption patch is: A high priority process must still wait on a lower priority process while it is in a critical section, even if that same high priority process did not need to access that section.
The RT patch is all about determinism and being able to have full control of the kernel. But, like being root, the more power you give the user the more likely they will destroy themselves with it. Such as, a user process can have higher priority than IRQ handler, if the process holding the CPU, system will hang.

Why NMI is not used widely in Linux kernel

2009-06-13T20:41:00.001-07:00

"There are some systems where NMIs are broken (e.g. a lot of ThinkPads explode when they get NMI inside SMI). This was one of the reasons the NMI watchdog is not enabled by default anymore." From Andi Kleen.

force_sig_info

2009-06-13T20:39:00.000-07:00

Behavior of force_sig_info

If SIGXXX is blocked or ignored: unblock SIGXXX, reset action to SIG_DFL

Force_sig_info in some special environment

kvm_arch_vcpu_ioctl_run(), SIGXXX is blocked
- SIGXXX is unblocked, but before returning from kernel:kvm_arch_vcpu_ioctl_run(), it is blocked again (restore host signal mask).
In sys_rt_sigtimedwait(), SIGXXX is blocked
- SIGXXX is unblocked, but before returning from sys_rt_sigtimedwait(), it is blocked again (restore to original signal mask).

Fast boot

2009-06-13T20:38:00.000-07:00

How to make Linux boot faster?

Keep CPU and disk I/O usage rate to or near 100%, that is, take full advantage of CPU speed and disk I/O throughput.
Reduce works needed in boot.

Previous, I think the method 2 is more important, such as substituting boot with resuming from hibernation. But I think Moblin team achieve a great result with mainly method 1.

Audio and real-time

2009-06-13T20:37:00.000-07:00

Only audio playback and audio recording do not need much real-time capability of system. Because you can just need to prepare a buffer big enough until next refill (or re-consume).

The real real-time demanding work is something like VOIP, a small buffer should be used to keep speak-to-hear lag low.

SNES9x for Zaurus Akita with rotate

2009-06-13T20:36:00.001-07:00

The resolution of zaurus akita is 480x640 (or 240x320) instead of 640x480 (or 320x240). While SNES9x uses resolution 640x480 or 320x240, so rotation should be implemented in some layer of the graphic stack. The possible graphic stacks are:

SNES9x + X Window + frame buffer
SNES9x + SDL + X Window + frame buffer
SNES9x + SDL + frame buffer

Rotation has been implemented in X window so for 1 and 2, there is no need to implement rotation in SDL or SNES9x. But the performance of 1 and 2 is worse than 3. So I implement the rotation in SNES9x. The patch can be downloaded here.

Some software for tablet PC

2009-06-13T20:35:00.001-07:00

Stroke (gesture recognition)
- I find easystroke is the best so far. To make it easier to enable/disable easytroke, (because several gesture recognition programs in system may conflict, for example, there is gesture recognition in firefox too.) I write a small patch to disable/enable easystroke with left button, which can be downloaded from here.
Note taking with pen
- I think xournal is very good. It can be used to add annotations to PDF files.

Firefox addons for tablet PC

2009-06-13T20:33:00.000-07:00

Grab and Drag
Enables Adobe Acrobat-style grab and drag scrolling in Mozilla applications.
All-in-One Gestures
This extension allows you to execute common commands using mouse (pen) gestures.

Several mouse gestures programs may conflict each other, so I write a simple AIOGControl (All-in-One Gestures) control addon to enable/disable All-in-One Gestures via a Firefox toolbar button. This is easier for pen to operate.

Finger friendly scrollbar in GTK+

2009-06-13T20:25:00.000-07:00

Standard GTK+ scrollbar is too small for finger, to make it bigger, you can put following code in your .gtkrc-2.0 or gtkrc file of theme.

style "big-scrolls" {
  GtkRange::stepper-size = 50
  GtkRange::slider-width = 50
}

class "GtkScrollbar" style "big-scrolls"

Please refer to http://www.roosmaa.net/index.php/2009/02/09/making-gtk-scrollbars-more-touchscreen-friendly/ too.

Comparison against overflow

2009-06-13T20:21:00.000-07:00

Time variables are subject to overflow. So, the comparison statement as follow will return incorrect result if t1 has overflowed and t2 has not overflowed.

unsigned long t1, t2; if (t1 > t2) printf("t1 > t2");

The code as follow can be used to deal with the above issue.

unsigned long t1, t2; if ((long)(t1 - t2) > 0) printf("t1 > t2");

Suppose sizeof(unsigned long) = 4, and t1 < t2, the result of t1 - t2 under the unsigned semantics is (0x100000000 + t1 - t2), because of borrow, if the result < 0x80000000, in signed semantics and complemental code, the result is positive.

Screen with title in status bar

2009-06-13T20:18:00.000-07:00

screen is a nice terminal tool.

cp example screenrc from doc to home directory
Add following lines to screenrc, window titles of current screen session will be shown in status bar with current window title hilighted.

hardstatus string "%h"
caption always "%{=r}%?%{b}%-Lw%?%{+bu r}%n*%f %t%{R}%{-bu}%?%{b}%+Lw%: %?%{+b B}%=%C:%s"

Mer: Comparison with desktop distributions

2009-06-13T20:17:00.000-07:00

Comes from a presentation of mer community.

Is Mer just yet another linux distribution? Many people would disagree since we break quite a lot of things (Maemo GTK is a big one!), and I'll tell you some of the ways we differ.

Like Maemo we focus on power saving, some things that make sense on desktop/laptop distributions simply doesn't make sense on mobile devices that's supposed to last on the same charge several days being alwaysconnected.

A tablet still comes with limited storage space, so there might be a need to shrink packages or dependancies (why do we need a weather library for NetworkManager?); but not hard as on an embedded system; there's miles difference between 256m flash and 64mb flash.

Yes, you can't run Crysis on a tablet, it has CPU, memory and bandwidth constraints and apps should be suited towards that. But, what it can do is to be more powerful because of the environment and services surrounding it.

Hard disk image in qemu

2009-06-13T20:15:00.000-07:00

create/update image

Install a mini linux system in </sub-linux>. It is very easy to install debian inside another Linux.

Update image quickly

quick_upimg.sh

 qemu-img create -b <hd.img> -f qcow2 <hd.qcow2>

Update image

upimg.sh

 sudo genext2fs -d </sub-linux> -b 344800 <hd.img> chown <yourself>. <hd.img> quick_upimg.sh

If you use <hd.qcow2> in QEMU, <hd.img> will not changed for ordinary usage.

commit changes

You can explicitely commit changes to <hd.img> by qemu monitor command "commit".

access qcow2 image

 $ qemu-nbd [-p <port>] hd.qcow2 $ sudo modprobe nbd nbds_max=<2> $ sudo nbd-client localhost <port> /dev/nbd0 $ sudo mount /dev/nbd0 /mnt

after using:

 $ sudo umount /mnt $ sudo nbd-client -d /dev/nbd0

Blog with muse

2009-06-13T20:13:00.000-07:00

Recently I decide to use muse as my primary composing tool. Because:

Easy to use.
Mark based instead of WYSIWYG. Full control of output and document structure.
Can convert to many other format, including docbook and latex. So if formating capability of muse is not enough, we can use other format instead later.

To use muse to write blog (mainly for blogspot):

Instal muse-el
Add "(require 'muse-blosxom)" into your .emacs
Write your blog in emacs with muse mode.
"C-c C-t" to generate output in blosxom-html format.
Open generated blosxom-html, "C-x h" to mark whole buffer, "M-!" to run a external shell script on your current buffer to do some post-processing. The shell script can be as follow.

 sed -e '1d' -e '/^#postdate/d' | tr '\r\n' '  '

ACPI dump

2009-06-13T20:11:00.000-07:00

acpidump > <dump_file>
acpixtract -a <dump_file>
iasl -d <TAB>.dat

New process group in shell script

2009-06-13T20:08:00.000-07:00

To create a new process group for a command (or another shell script) in bash script, the following method can be used.

 b=$(bash -i -c "<command> & echo \$!")

This can be put in a shell script. String after -c will be executed by the (sub-)bash, while -i means the (sub-)bash will behave like a interactive bash. So "<command> &" in string will execute <command> in a new process group, just like "<command> &" is typed in a login shell. The "echo \$!" in string will print the process ID, that is, the new process group ID, so you can operate on the new created process group, such as send signal to process group via "kill -SIGNAL -$b".

A new method to distinguish between ring buffer full and empty

2009-05-16T18:58:00.000-07:00

One design choice for ring buffer implementation is to distinguish
between full buffer and empty buffer. Traditional methods including
waste one entry for full (head + 1 == tail), or use
another flag for full. A new method is implemented as follow:

#define RING_LEN   16
#define RING_LIMIT (2 * RING_LEN - 1)

struct ring_entry
{
  ...
};

struct ring
{
  int head;
  int tail;
  struct ring_entry entries[RING_LEN];
};

int ring_index(int n)
{
  if (n > RING_LEN)
      return n - RING_LEN;
  else
      return n;
};

int ring_get(struct ring *ring, struct ring_entry *entry)
{
  /* empty */
  if (ring->head == ring->tail)
      return 0;

  *entry = ring->entries[ring_index(ring->tail)];

  ring->tail++;
  if (ring->tail > RING_LIMIT)
      ring->tail = 0;
  return 1;
}

int ring_put(struct ring *ring, struct ring_entry *entry)
{
  int ihead, itail;

  ihead = ring_index(ring->head);
  itail = ring_index(ring->tail);

  /* full */
  if (ihead == itail && ring->head != ring->tail)
      return 0;

  ring->entries[ihead] = *entry;

  ring->head++;
  if (ring->head > RING_LIMIT)
      ring->head = 0;
  return 1;
}

Buffers in Linux kernel

2009-05-16T17:54:00.000-07:00

For 2.6.30

Oprofile Buffer

One cpu_buffer for each CPU
- NMI or IRQ handler add record to cpu_buffer
- Timer based per-cpu work queue remove record from cpu_buffer
- One reader and one writer, so it is easy for mutual exclusion and synchronization
- fix size, may overflow if write too much or not removed quickly enough
One event_buffer
- Timer based per-cpu work queue add record to event_buffer from that of cpu_buffer
- User space program remove record from event_buffer. User space program is waken up based on threshold.
- One reader and multiple writer, mutex is used on write side, because writers are in process context (work queue).
- fix size, may overflow if write too much or not read out quickly enough
Synchronization between cpu_buffer, event_buffer and use space program
- Used for profile, so record flow bandwidth is predictable
  - Maximum bandwidth for cpu_buffer: CPU number * (cpu_buffer size / timer interval)

Unified trace ring buffer

One ring_buffer_per_cpu for each CPU
- In fact two levels of records, the first level is struct buffer_page, the second level is struct ring_buffer_event.
- Preempt disabling is used to provide mutual exclusion between process contexts.
- Inside one struct buffer_page, atomic operation (local_add_return) is used to provide mutual exclusion for struct ring_buffer_event. Because process context can only be preempted by IRQ and NMI, and process context can not continue until IRQ and NMI finishes. The length of struct buffer_page is known (local_add_return may exceed buffer_page length).
- In writer side, "write" indicates allocated records, while "commit" indicates completed records.
- spinlock (ring_buffer_per_cpu->lock) and IRQ disable is used to provide mutual exclusion for struct buffer_page (reader and writer). In NMI environment, if try_spin_lock failed, the record is discarded, this acceptable for tracing.
- At least 3 pages are needed for each CPU, 2 buffer_page, 1 reader page.
- ring_buffer_per_cpu->reader_lock is used to provide mutual exclusion between multiple readers, while ring_buffer_per_cpu->lock is used to provide mutual exclusion between reader and writer.
- fix size, may overflow if write too much or not consumed quickly enough
- Iterator (iter) can be used on reader side

Mcelog buffer

One global buffer with fixed record length, fixed size. May overflow.
Writer side lock-less, implemented using cmpxchg() + finished flag. Records are added in NMI/timer context.
Memory order should be considered explicitly because of lock-less
Reader side is protected by mutex, because normally there is only one reader. Records are removed in process context.
Multiple writers, one reader, throughput bottleneck lies in reader side.

SMP preemption -> UP preemption

2009-05-16T16:59:00.000-07:00

Sometimes, a SMP environment can turn into a UP environment via per-CPU data structure. Such as unified trace ring buffer is a per-CPU data structure, so the mutual exclusion can be considered in UP model for each per-CPU buffer.

Preemption in UP

There are 3 kinds of contexts in UP environment.

Process context
IRQ context
NMI context

Where 1 can be preempted by 1, 2 or 3; 2 can be preempted by 2 or 3; 3 can not be preempted.

If preempt is off in 1, 1 can only be preempted by 2 or 3. And there is no interleave between contexts, that is, if 1 is preempted by 2, 1 will not be resumed until 2 is finished, so do 2 and 3. This can be used to develop more efficient synchronization algorithm.

If IRQ is off in 1 and 2, 1 and 2 can only be preempted by 3, without interleave.

Reading notes of on submit kennel patches (OLS)

2009-05-09T19:51:00.000-07:00

!!! Important kernel design point !!!

Very generic notifiers and hooks tend to have a large maintenance impact because they have the potential to alter the code flow in unexpected ways, lead to lock order problems, bugs, and unexpected behavior, and generally making code harder to maintain and debug later.

Linux kernel is a lock-based complex parallel system, the main method to prevent ABBA style dead-lock is enforcing lock order. But notifiers or hooks makes it hard to audit the order of locks.

Notifiers or hooks in lower layer to call functions on higher layer should be avoided. One method to call higher layer function in lower layer is via work queue like asynchronous mechanism.

Notifiers or hooks in higher layer to call functions on lower layer is OK. Such as struct file_operations is collections of function pointers (a kind of hooks?), it is only used as some kind of abstraction for lower layer, make adding new lower layer component more convenient.

Maintenance

For a project which is very complex like Linux kernel, (seems that most projects become more and more complex as time goes by), maintenance is something in the core of Linux kernel development.

To make code maintainable, firstly, code should be made readable. The code is clean-upped and redesigned again and again to make it cleaner, thus more readable.

It is much easier to add a component under the current design. Such as add a new device driver. If you need something to be changed in subsystem core, you should consider the design, maybe you need clean-up or redesign some part of the subsystem core to make code maintainable.

A patch submitter should maintain his code at least for a short while after merging.

Interface designs for user space programs is very important too. It should be stable for imaginable future.

Changing a published interface later is much harder and often special backwards compatiblility code is required.

Why MID?

2009-05-08T06:31:00.001-07:00

What I need for a Mobile device

Fast from picking up to using (Fast boot)
light, easy for taking and hand holding
Applications
- Web browsering
- Electric book
- Online video and offline video
- PIM
- Game
- GPS?

MID vs Notebook

pros:

Fast from picking up to using: fast boot time or always suspend to RAM instead of power off.
lighter, easy for taking and hand holding

cons:

Screen is too small for some applications, especially most web page, PDF, DJVU ebooks
Notebook has better Linux support, some application has best usability on Notebook.

MID vs Smartphone

pros:

Screen is bigger, although smaller than Notebook, much better than smartphone's.
OS is compatible with Desktop ones (Linux or Windows), most desktop applications can run on MID. Although Linux based smartphone can run most desktop applications too, the resource of most smartphone is not enough for desktop application.
Computing resource of MID (CPU, RAM, storage) is much better than smartphone.

cons:

Smartphone is all-in-one solution, that is, you can bring just smartphone or smartphone AND MID.
Smartphone has GPRS support in China, while most MID has not.
Some smartphone application is good for small screen, such as ucweb for web browsing. Some desktop application is not good for small screen, even bigger screen of MID.

Comparison

	Notebook(X61T)	MID (no yet)	Smartphone(N78)	PDA (Zaurus)
Fast usage	1	4.5	5	4.5
Weight	1	4.5	5	5
Browsing	5	4 ~ 4.5	3 (ucweb)	2
EBook	5	3 ~ 3.5	2.5 (screen)	3
Online Video	5	4.5	0 (no WIFI)	0 (too slow)
Offline Video	5	4.5	4	3.5
PIM	4	4	4	3
Game	5	5	2 (keypad)	4
GPS	0	5 (maybe)	5	0

Conclusion

Although MID should be better than smartphone, there is no good choice until now. Aigo or Vilivs is too expensive. Smart Q5 is not better than Zaurus (No stable software too). Most functions of MID can be accomplished with smartphone.
Notebook + smartphone is good for now, waiting for better MID.

On screen keyboard

2009-04-26T02:18:00.000-07:00

On screen keyboard is necessary for a tablet PC. Have tried several on screen keyboard implementation under X window, no one is perfect. Currently using one is cellwriter. Hope the one developed for Moblin will be better.

Requirement for on screen keyboard:

Focus: the on screen keyboard should not be focused, and the faked key should be sent to the focused window.
Window placement: the on screen keyboard window should not be covered by other windows (for example to be the top-most window), and it should not cover other windows completely (semi-transparent or docked at the desktop side, but it seems docking is not supported well in my metacity 2.24.0).
Compatible with XIM. I am Chinese, I need use XIM to input Chinese.
Configurable layout (to keep only the needed keys).
Show/Hide can be controlled easily (via a panel button is preferable).
Support screen rotation. On screen keyboard should not outside the screen or bigger than screen after rotation.

Tried on screen keyboard software:

xvkbd: does not handle focus perfectly, the xvkbd itself seem be focused too (titlebar highlighted), so need two click for one key; need to specify which widget faked keys go, not very conveniently.
matchbox-keyboard: good support for rotation, configurable laout, XIM, focus. Docking support for matchbox-wm is good. Original version does not support docking for metacity. I have a hack to support docking for metacity, but the hack doesn't support metacity 2.24.0. Show/Hide control can be done with daemon mode (-d) + matchbox-keyboard-toggler utility.
gok (gnome on screen keyboard): recommended by many guys, but not good for me. Appears not works with XIM, and not works well without pointer in/out event.
onboard: not used too much, I think it is good and simple.
cellwriter: hand writing recoganizing program, but it is a good and simple on screen keyboard too. To use it as on screen keyboard add "--keyboard-only" to command line. Do no support configurable layout. Seems not support docking good.

Lenovo Thinkpad X61 Tablet

2009-04-26T02:09:00.000-07:00

Bought a Lenovo Thinkpad X61 Tablet recently. Will post some usage tips. Most useful links:

http://www.thinkwiki.org/wiki/ThinkWiki

Evtouch configuration

2009-04-26T01:49:00.001-07:00

This article is mainly for Tablet Classmate PC.

Evtouch is a Xorg (X window system) input driver for touchscreen device with Linux kernel input/event driver. For example some USB touchscreen device with USBHID class.

To use evtouch Xorg driver, xorg should be setup as follow:


--------------------------------------------------------------------------------
Section "InputDevice"
    Identifier    "TOUCH"
    Driver        "evtouch"
    Option        "Device" "/dev/input/by-path/pci-0000:00:1d.3-usb-0:1:1.0-event-"

    Option        "DeviceName" "touchscreen"
    Option        "ReportingMode" "Raw"
    Option        "MoveLimit" "10"
    Option        "SendCoreEvents" "On"
#       Option        "Calibrate" "1"
    Option        "MinX"      "1"
    Option        "MinY"      "1"
    Option        "MaxX"      "4090"
    Option        "MaxY"      "4090"
    Option        "x0"        "-6"
    Option        "y0"        "8"
    Option        "x1"        "1"
    Option        "y1"        "2"
    Option        "x2"        "12"
    Option        "y2"        "2"
    Option        "x3"        "5"
    Option        "y3"        "3"
    Option        "x4"        "-6"
    Option        "y4"        "3"
    Option        "x5"        "-4"
    Option        "y5"        "3"
    Option        "x6"        "0"
    Option        "y6"        "5"
    Option        "x7"        "1"
    Option        "y7"        "5"
    Option        "x8"        "1"
    Option        "y8"        "5"
EndSection

Section "ServerLayout"
    ......
    InputDevice     "TOUCH"
EndSection
--------------------------------------------------------------------------------

Calibrate by hand

Evtouch comes with a calibration utility. It works, but I find the result is not very accurate. In fact it can be calibrated further by editing xorg.conf directly.

If you find when you touch the margin of the screen, the cursor is not at margin, you can change the "MinX", "MinY", "MaxX", "MaxY" in xorg.conf to calibrate.

The x0-y0 to x9-y9 corresponds to 9 regions on screen, with the x0-y0 at the left-bottom corner, x2-y0 at the right-bottom corner, x9-y9 at the right-top corner. To calibrate the touch screen in the region, just the values of corresponding options.

Rotate screen

To rotate screen:
xrandr --output LVDS --rotate left

evtouch should have supported rotation automatically. But the evtouch-0.8.7 in my debian does not work. I write a small patch 08__fix_rotate.patch to fix this issue.

Suspend and Resume

My touchscreen device is a usb device. This means it will be removed by Linux kernel before suspend and re-plugged after resume. Evtouch xorg driver has suspend and resume support mechansim, but it does not work sometimes. From the error message in /var/log/Xorg.0.log, it seems that the underlaying usb kernel device is not ready when evtouch xorg driver is being initialized. Fortunatally, it works sometimes. So if it your touchscreen does not work after resume, just go suspend again and hope it works after next resume.

Right Button

I am not successful in configuring right button. Maybe a kernel input driver issue, usbhid can not report BTN_TOUCHED for my device. But it seems that the following configuration should work for correct kernel input drivers.


--------------------------------------------------------------------------------
Section "InputDevice"
    Identifier      "TOUCH"
    Driver          "evtouch"
    ......
    Option          "longtouched_action" "down"
    Option          "longtouched_button" "3"
EndSection
--------------------------------------------------------------------------------

IBSuite

2009-04-02T05:22:00.000-07:00

To read book on my PRS505 e-book reader, I write some tools. Now I collect these tools into IBSuite (Image Book Suite), and put that on sourceforge. You can download the source code from the git of sourceforge project. Hope that can be useful for somebody.

http://sourceforge.net/projects/ibsuite/

IBSUITE

ibsuite stands for image book suite. It contains a set of tools to convert ebook in various format (pdf, chm, html) into a set of images, reformat the images (crop, embold, divide, etc), and assemble the result images into a new ebook.

COMPONENTS

ibhtml2img: convert html to image with xulrunner
ibhtml2pdf: convert html to pdf with xulrunner
iblineparser: parser input image, extract line information
ibpdfinfo: get some meta-information from pdf file, such as title, author, table of contents etc.
ibpy: python module, which is the driver of the whole system, it uses above programs to convert input file to image, extract line information from image, dilate image, and re-assemble lines into a new image, generate output e-book.
ibtools: A set of utilities and tools, some of them are used internal by ibsuite, others are user command provided by ibsuite.

USAGE

The most important command of ibsuite is ibreformat, the basic usage is as follow:

ibreformat [options] <input file>

In most cases, something like following:

ibreformat -o <output file> --iprof=<iprof> --oprof=<oprof>
--pprof=<pprof> <input file>

Where <input file> is input file name, <output file> is output file name, <iprof> is input profile, <pprof> is output profile, is <pprof> processing profile.

Available input/output/processing profiles are as follow:

input profiles:
img: for scanned book

output profiles:
prs505p: for Sony PRS505 in portrait mode
prs505l: for Sony PRS505 in landscape mode

processing profiles:
divide2: divide one line into two line (a kind of simple reflow)
resize: Resize and dilate pages
repage: Re-page input book, without much other processing such as dilate.

For other command line options, please refer to "ibreformat -h".

Some useful command line option combinations:

For scanned book on PRS505:

ibsuite -o -iprof=img -pprof=resize -oprof=prs505l <input file>

For chm file on PRS505:

ibchm2imb .chm

When it finishs, .imb will be generated, then

ibsuite -o -pprof=repage --oprof=prs505l .imb

INSTALL

Currently, only Linux is supported, but I think it may work on some unix enviroment (including cygwin on Windows) after some work. Currently only install from source code is supported.

Pre-requirement:

gcc, g++, bash, make
libfontconfig-dev, libnetpbm-dev, libgtk-dev
python, python-imaging
imagemagick
for HTML/CHM support: python-chm, xulrunner
for scanned book: unpaper

Build:

./configure [--prefix=]
make [PREFIX=] [NO_XUL=1]

Install:

# become root
make install [PREFIX=]

Google notebook

2008-11-24T18:17:00.000-08:00

I find google notebook is a convenient tool. You can take note quite freely during surfing. And you can share your notebook quite easy too. I think it is even easier than Blog. Some of my exported notebook:

Programming

Smart phone

An algorithm to render PDF on small devices

2008-07-18T23:31:00.000-07:00

I am interested in ebook reader for quite a while. But after trying with a 6-inch e-ink reader (Hanlin V3), I found it is almost useless to read normal PDF files (A4 size) on these devices. The font size is too small, while the page size is too wide. So, a method to render PDF for these small devices is thought about and prototyped. The details are as follow:

Convert pdf to image. I use pdftoppm of xpdf. Such as: pdftoppm -r 180 -f 245 -l 245 -gray -aa yes a.pdf a
Analyze the generated images. Break page into lines.
Divide each line long enough to two segments.
Rearrange the segments into a new page, with half of the width.

A full functional PDF re-flow algorithm is very difficult to implemented. But a more simplified version as above is much easier.

No state updating: functional language

2008-07-18T23:04:00.000-07:00

The programming model of command language is state machine. Such as in C, every variable is a state, one of the most common statements are assigning new value to a variable, that is, state updating. But in functional programming language, there is no state updating, how to program without state updating? Let's begin with a simple example of bank account management system.


struct account
{
    char *name;
    int remain;
    struct account *next;
};

#define OP_NOP         0
#define OP_WITHDRAW    1
#define OP_DEPOSIT     2
#define OP_PRINT       3
#define OP_EXIT        4

void deposit(struct account *account, int val)
{
    account->remain += val;
}

void withdraw(struct account *account, int val)
{
    account->remain -= val;
}

int main(int argc, char *argv[])
{
    struct account *accounts = NULL;
    char *name;
    int op = OP_NOP, val;

    while (op != OP_EXIT) {
        op = read_operation();
        case OP_WITHDRAW:
        case OP_DEPOSIT:
            name = read_name();
            account = list_find(accounts, name);
            val = read_val();
            if (op == OP_WITHDRAW)
                withdraw(account, val);
            else
                deposit(account, val);
            break;
        case OP_NEW:
            name = read_name();
            account = malloc(sizeof(struct account));
            account->name = name;
            account->remain = 0;
            list_add(&accounts, account);
            break;
        case OP_PRINT:
            /* a macro to iterate every element of list */
            list_for_each(accounts, account) {
                printf("Name: %s\tRemain: %d\n", account->name, account->remain);
            }
            break;
        case OP_EXIT:
            break;
        }
    }
    return 0;
}

In erlang, a functional programming language:


-module(bank).
-export([service/1]).

deposit(Account, Val) ->
    {Name, Remain} = Account,
    {Name, Remain + Val}.

withdraw(Account, Val) ->
    {Name, Remain} = Account,
    {Name, Remain - Val}.

print_account({Name, Remain}) ->
    io:format("Name: ~s Remain: ~B~n", [Name, Remain]).

service(Accounts) ->
    {ok, [Operation|_]} = io:fread("Operation: ", "~a"),
    if
    Operation == deposit; Operation == withdraw ->
        {ok, [Name|_]} = io:fread("Name: ", "~s"),
        {ok, [Val|_]} = io:fread("Val: ", "~d"),
        case lists:keysearch(Name, 1, Accounts) of
        false ->
            io:format("Error: no such account.~n"),
            NewAccounts = Accounts;
        {value, Account} ->
            case Operation of
            deposit ->
                NewAccount = deposit(Account, Val);
            withdraw ->
                NewAccount = withdraw(Account, Val)
            end,
            NewAccounts = lists:keyreplace(Name, 1, Accounts,
                           NewAccount)
        end,
        service(NewAccounts);
    Operation == new ->
        {ok, [Name|_]} = io:fread("Name: ", "~s"),
        case lists:keymember(Name, 1, Accounts) of
        true ->
            NewAccounts = Accounts,
            io:format("Error: account already exist.~n");
        false ->
            NewAccounts = [{Name, 0} | Accounts]
        end,
        service(NewAccounts);
    Operation == print ->
        lists:foreach(fun print_account/1, Accounts),
        service(Accounts);
    Operation == exit ->
        ok;
    true ->
        io:format("Error: unknown operation~n"),
        service(Accounts)
    end.

Notes:

In functional programming language, instead of updating object (value) in-place, every time a new object (value) is constructed. In example above, in C, deposit() change account->remain directly, while in erlang, a new account is constructed and returned.
If data of program is organized in a hierarchical way, to change the data in leaf, in command language, only the leaf is changed in place; in functional language, all data structure from that leaf to root will be reconstructed. In example above, in C, during deposit operation, only the corresponding account object is changed in-place, while in erlang, both corresponding account object and the whole account list will be re-constructed. With the help of carefully designed implementation, althought the performance of that in functional language is wrose, it is not so bad as it appears.
In functional programming language, instead of updating the binding of variable name to new value directly, a new scope is constructed and in the new scope, same variable name is binded to a new value. In example above, during "new" operation, accounts may be bind to a new value, in erlang, a new scope is constructed by "recursing", that is, the new constructed account list is bind to Accounts in a new scope. Thanks to the "tail recursing" optimization, the performance is not so bad.

Questions:

How to shared objects in functional programming language? Such as in a database system, the cached database table should be shared between service threads.

OS as parallel system

2007-04-28T00:25:00.000-07:00

There are subsystems as following in Linux kernel
- process management (virtual memory system, kernel entry point)
- physical memory management system
- file system
- device driver, device management
- network system
The subsystems in Linux kernel like the "server" in micro-kernel operating system. They can be seen as multi-thread server too. Although the Linux kernel provides multi-process environment for user space, the kernel itself should be seen as a multi-thread program, because everything is shared.
Most resources are not shared between processes in process management system, i.e., the virtual memory system is not shared between processes, while they are shared between threads.
Process/thread is used as one of the parallel roles, the other parallel roles include IRQ handler, softirq, tasklet, work queue, timer operations. The subsystems in Linux kernel can be seen as multi-thread server system.
If multiple threads access different objects, such as the different file, there will be no conflict.
Copy, cache, transaction technologies in parallel system can be used in Linux kernel too, such as the per CPU cache in slab subsystem.

Scheduler for all "threads"

2007-04-28T00:03:00.000-07:00

In operating system, not only the traditional process and thread are schedule-able entities, but also the IRQ handler, software IRQ, tasklet, timer operations and work queue should be the schedule-able entities too, conceptually.

Some threads can have higher priority than some tasklets, so, it can preempt these tasklets. It can even have higher priority than some IRQ handler, so that the IRQ is not serviced during thread executing. I think this is a useful concept in real-time system. I think the thread-style IRQ handing in FreeBSD maybe follow the same concept.

Video pipeline

2007-04-28T00:02:00.001-07:00

Video capture

Analog video

Tuner
ADC
Analog video decoder (PAL, NTSC)

Digital video

Digital tuner
Demodulating
Demux
Decoder

Video processing

Video clip
Video scaling
Video scan type/scan rate conversion

Frame/field rate conversion
Interleaved -> progressive, progressive -> interleaved

Color space conversion

RGB -> YUV
YUV -> RGB
Brightness, contrast, Hue/Tint, color temperature conversion

Video enhancement

Histogram equilization
Flesh tone correction
Noise reduction

Video blending

Chroma keying
video mixer

Video display

Gamma correction
Converting to panel signal format (Panel timing control)

Cancelable operations

2007-04-28T00:02:00.000-07:00

There are many cancelable operations in software system with user interaction, such as cancel auto tunning in TV. The implementation mechanisms of cancelable operations are as follow:

message
The "cancel" message is sent to thread which do long-time job if necessary. While the function which do long-time job must check whether there are "cancel" message received, if so, the function must stop working and return.

Cons:
- It isn't native programming model in Linux/Unix, so third party software will not use it.
- Can't cancel work in kernel (for example the sleep or block read can not be interrupted)
signal / real time signal
The "cancel" signal is sent to thread which do long-time job if neccesary. Upon the signal is received, the blocked operation (such as sleep, blocked read, etc) will be interrupted and a signal action function will be called which set a flag to indicate the canel signal is received by the thread. The function which do long-time job must check the flag to see whether there are "cancel" signal received, if so, the function must stop working and return.

Pros:
- Native programming model in Linux/Unix, easy to cooperate with third party software.
- Can cancel work in kernel.

Problems:

race conditions !!!

Message mechanism implementation in POSIX system

2007-04-27T23:40:00.000-07:00

The message mechanism is widely used for inter-thread (process) communication. Some implementation mechanisms of message are as follow:

pipe (anonymous or named)
- Pros:
  - Can be used for inter-process and inter-thread message
  - Can suspend on get message (desired)
  - Support non-block operation
- Cons:
  - Can not support message priority
Socket (stream or datagram)
Used by X Window system message system. Almost same as pipe, with following additional pros and cons
- Pros:
  - Can be used in network environment (not important now)
- Cons:
  - Need kernel networking support.
POSIX message queue
- Pros:
  - Can be used for inter-process and inter-thread message
  - Can suspend on get message (desired)
  - Support time out
  - Support message priority
  - Support non-block operation
- Cons:
  - Need new kernel (> 2.6.6)
SysV message queue
- Pros:
  - Can be used for inter-process and inter-thread message
  - Can suspend on get message (desired)
  - Support message priority
  - Support non-block operation
- Cons:

Conclusion:

The function of mechanisms above is almost same, the POSIX message queue or SysV message queue is preferred if available because they are more similar to message then need less implementation effort.

Simple component system for embedded system

2007-04-27T20:38:00.000-07:00

1. Objective

Separate interface and implementation
- User depends on only interface not implementation, the implementation can evolve without breaking user.
- User can use several implementation of one interface simultaneously.
Modular
- Component is a special module, more modular.
- Manage the dependency relationship of software modules.
Testable
- The component is the test unit.
Position transparent (single-thread/multi-thread transparent)
- local/remote, same thread/different thread transparent

2. Interface

Describe in IDL formally: language neutral
Binary standard: vtable

Implementation in C: function pointer structure

typedef struct
{
    void (*dvFooInit)(dvFooIntf intf);
    void (*dvFooDestroy)(dvFooIntf intf,
        FOO_DESCRIPTION sFooDesc);
    void (*dvFooBarEnable)(dvFooIntf intf, BOOL bEnable);
    ...
} dvFooOps;

typedef struct
{
    dvFooOps *ops;
} dvFoo;

typedef dvFoo *dvFooIntf;

Implementation in C++: class with only pure virtual method

class dvFoo
{
public:
    virtual void dvFooInit(void) = 0;
    virtual int32 dvFooDesctroy(FOO_DESCRIPTION desc) = 0;
    virtual void dvFooBarEnable(BOOL bEnable) = 0;
    ...
};

typedef dvFoo *dvFooIntf;

A tool can be developed to generate C++ and C implementation of interface from IDL automatically.

3. Component (object)

Properties:
- Software module that implement one or several interfaces. The interfaces is the API of component and should be stable.
- The source code, document, configuration information, test code of component is organized in individual folder.
- Every component has its own version number and can evolve individually.
- Can be selected and configured via a configuration tool.

Implementation

Implementation in C: fill function pointer structure

dvFooOps opsfoo = {
    dvLisaFooInit,
    dvLisaFooDesctory,
    dvLisaFooBarEnable,
    ...
};

typedef struct
{
    dvFooOps *ops;
    /* data members for Lisa */
} dvLisaFoo;

dvLisaFoo foo = {
    &opsfoo,
    /* data members for Lisa */
};

For driver implementation, instance number can be used as the data member.

Implementation in C++: inherit implemented interfaces

class dvLisaFoo : public dvFoo
{
    /* data members for Lisa */
public:
    void dvFooInit(void);
    int32 dvFooDestroy(FOO_DESCRIPTION desc);
    void dvFooBarEnable(BOOL bEnable);
    ...
};

For driver implementation, instance number can be used as the data member.

4. Usage

After getting the pointer to the interface of the component, user of component can use the component via direct function call without need to know the implementation of component. (C user can use C++ implementation).

#define dvFooInit(intf) (intf)->ops->dvFooInit((intf))
#define dvFooDestroy(intf, sFooDesc) 
    (intf)->ops->dvFooDesctroy((intf), (sFooDesc))

dvFooIntf intf;
/* Get interface */
dvFooInit(intf);
dvFooDestroy(intf, sFooDesc);

5. Inter-thread (inter-process) component

If the the user and implementer of component is resided in different thread (or process), it is not sufficient to use direct function call simply. To use component across thread (process), the following mechanism is provided.

A proxy is provided in user thread (process), which is the representative of the component which implemented in different thread (process). The user can use the proxy in same way of component in single thread environment. The proxy implement the interface with inter-thread (inter-process) communication (such as message sending and receiving).
A stub is provided in implementation thread (process), which is the representative of the user which is in different thread (process). The stub use the component in the same way of user in single thread environment. The stub simulate the user with inter-thread (inter-process) communication (such as message receiving and sending).
A simple implementation of inter-thread inter-process provides a simple mutual exclusion resolution like crude force serialization between all functions of one component.
In implementation, the implementing thread id can be used in proxy as the data member.
A tool can be developed to generate proxy and stub implementation of interface from IDL automatically.

6. Component manager (component directory/component factory)

In order to get the interface of component, a component manager should be provided. The simplest component manager is as follow:

Create all components before using components. Such as, all device driver components are created using the source code generated by configuration tool. All middleware components are created using the source code generated by configuration tool too.
Component managers have lists of all components of system. Such as the device driver manager has the list of all device driver components.
Component manager provide a interface to get the interfaces of components. Such as all the interfaces of device driver components can be gotten from device driver manager.
For inter-thread (inter-process) components, the proxy and stub is created before using as that of the component. The proxy instead of component itself is in the list of components in component manager. The user of component will get the proxy, it is desirable.

Auto Shanghai 2007

2007-04-27T01:27:00.001-07:00

Photos of Auto Shanghai 2007: http://picasaweb.google.com/huang.ying.caritas/AutoShanghai2007

virtualization

2007-04-15T01:47:00.001-07:00

1. Virtualization level

a. Timing-accurate virtualization

The timing logic is simulated by software. It is mainly used for chip
verification. The performance is lowest.

b. Full virtualization

It is instruction-accurate CPU virtualization and necessary peripheral
virtualization. The "unmodified" guest operating system can be run on
it.

Examples: Bochs, Qemu, Skyeye, Xcopilot.

c. Para-virtualization

It is instruction-accurate CPU virtualization and necessary peripheral
virtualization but with some modification to real hardware
architecture, so that the guest operating system must be "modified" to
run on it.

Examples: Xen, UML.

d. Operating system level virtulization

Some instruction (most non-privilege instruction) of CPU and the
operating system interface (mostly system call interface) is
virtualized. The target application can be run on it. Same or
different operating system interface as that of host operating system
may be virtualized.

Examples with same OS interface as that of host OS: Linux vserver,
FreeBSD jail, Qemu user space emulation.

Examples with different OS interface as that of host OS: Linux binary
compatibility support on FreeBSD, Wine.

2. Implementation

a. Hardware based or OS based implementation

Some virtulization systems work as a user space application (perhaps
with some kernel space components) on a host OS, which are called OS
based implementation, while others work on bare hardware directly,
which are called hardware based implementation.

b. Interpretation based implementation

It is mainly used for full virtualization. The core of virtualization
system usually is a loop to fetch an instruction, interpret the
instruction, execute the instruction and advance the PC, perhaps with
some pipeline handling.

Interpretation based virtualization system is usually good on
portability, if it is programmed in portable language.

Example: Bochs, Skyeye, Xcopilot

c. Dynamic translation based implementation

In dynamic translation based virtualization system, the target binary
code is translated into host binary code first, then the translated
code is executed on host CPU directly.

In general, its performance is better than interpretation based
implementation with good portability.

It can be used for full virtualization, in which, both the kernel
space code and user space code is translated before execution. It can
also be used for operating system level virtualization, in which, only
the user space code is translated and the OS interface (system call)
is implemented through calling the host system OS interface directly.

Examples: Qemu, Qemu user space virtualization.

d. Monitor based implementation

In monitor based virtualization system, the target CPU must be same as
host CPU. The target code is executed on host CPU directly but usually
with lower priority. Some privileged instruction or some resource
access will trap into monitor, which usually is executed with highest
priority, then, the monitor do some check and complete the privileged
operation or resource access for target code and return to guest code.

Its performance can reach nearly native, but its main constraint is
that the guest CPU must be same as the host CPU.

Examples: KQemu.

e. Para-virtualization implementation

On para-virtualization system, the "unmodified" guest code can not be
run directly. Instead, usually the guest OS must be "modified" to be
implemented based on para-virtualization system instead of real
hardware. Some privileged operation or resource access are done
through calling the para-virtualization system instead of direct
hardware programming.

Examples: Xen, UML, coLinux.

f. Implement the API of another OS in user space

This can be used for operating system level virtualization.

Examples: wine.

g. Implement the system call of another OS in kernel space

This can be used for operating system level virtualization.

Examples: Linux binary compatibility support on FreeBSD.

h. Partition resources between processes

Some resources such as file system, CPU time, network address and
memory is partitioned between processes. Such that, some processes
works on the restricted resources only.

Examples: Linux vserver, FreeBSD jail.

GTK+ Map Viewer

2007-01-30T07:23:00.000-08:00

Recently, I developed a simple map viewer for my Zaurus with GTK+.

Because most map image file is very big, from several MB to tens MB, it is very inconvenient to view map with ordinary image viewer, especially for embedded system like PDA and mobile phone.

This program is developed to facilitate map viewing, the basic idea behind it is:

The big map image file is splitted into many small image files, loaded on demand to reduce memory requirement.
The big map image file is pre-scaled too, because scaling a big image file is also a memory/CPU hungry task.

Some other useful feature for a map viewer is also provided. For example, personal marks can be added to a map. A map making tools is also provided to facilitate splitting and scaling the big map image file.

The source code of the program can be download from:
http://linux-mcr700.sourceforge.net/gviewmap-0.1.tar.gz

Stack-less multi-task system (2) -- implementation

2007-01-29T06:31:00.000-08:00

The basic idea behind implementation:

Replace implicit C stack with explict link stack.
Use label to record PC.
Store task state in heap instead of stack.
Only functions will sleep directly or indirectly need to deal with stack explictly, other functions aren't affected.

Performance comparison:

Task stack space is saved, only one stack is needed.
Task switching time may be a little longer because of task swithing executing path will go through all functions invoked before sleep. But, if the task is small enough and function invoking depth is short, the task switching time will be affordable.

Sample Code:


#include <stdlib.h>
#include <assert.h>
#include <string.h>

struct stack
{
 short   stk_label;
 void    *stk_data;
 struct stack *stk_prev;
 struct stack *stk_next;
};

#define TASK_STATE_RUN         0
#define TASK_STATE_SLEEP       1

struct task
{
 short    tsk_state;
 void     (*tsk_entry)(void);
 struct stack tsk_stk;
 struct stack *tsk_stk_curr;
};

extern struct task tasks[];
struct task *task_curr;

void sleep(void)
{
 task_curr->tsk_state = TASK_STATE_SLEEP;
}

void wakeup(int task_id)
{
 tasks[task_id].tsk_state = TASK_STATE_RUN;
}

/* return code */
#define RC_SUCCESS    0
#define RC_FAILED     1
#define RC_PENDING    2

void push_stack()
{
 if (!task_curr->tsk_stk_curr->stk_next) {
  struct stack *stk;

  stk = malloc(sizeof(struct stack));
  assert(stk);
  memset(stk, 0, sizeof(*stk));
  stk->stk_prev = task_curr->tsk_stk_curr;
  task_curr->tsk_stk_curr->stk_next = stk;
 }
 task_curr->tsk_stk_curr = task_curr->tsk_stk_curr->stk_next;
}

void pop_stack()
{
 struct stack *stk;

 stk = task_curr->tsk_stk_curr->stk_prev;
 assert(stk->stk_next->stk_next == NULL);
 free(stk->stk_next);
 stk->stk_next = NULL;
 task_curr->tsk_stk_curr = stk;
}

#define TEST_TASK_LABEL1    0
#define TEST_TASK_LABEL2    1
#define TEST_TASK_LABEL3    2

#define TEST_TASK_SUB_LABEL1   0
#define TEST_TASK_SUB_LABEL2   1

void test_task_label1(void)
{
 /* everything as normal, without sleep */
}

int test_task_label2(void)
{
 int rc = RC_SUCCESS;

 push_stack();

 switch (task_curr->tsk_stk_curr->stk_label) {
 case TEST_TASK_SUB_LABEL1:
  task_curr->tsk_stk_curr->stk_label = TEST_TASK_SUB_LABEL2;
  rc = RC_PENDING;
  sleep();
  break;
 case TEST_TASK_SUB_LABEL2:
  break;
 }
 if (rc != RC_PENDING)
  pop_stack();
 return rc;
}

void test_task(void)
{
 switch (task_curr->tsk_stk.stk_label) {
 case TEST_TASK_LABEL1:
  test_task_label1();
  task_curr->tsk_stk.stk_label = TEST_TASK_LABEL2;
  task_curr->tsk_stk.stk_data = malloc(10);
  sleep();
  break;
 case TEST_TASK_LABEL2:
  if (test_task_label2() == RC_PENDING) {
   break;
  }
  task_curr->tsk_stk.stk_label = TEST_TASK_LABEL3;
 case TEST_TASK_LABEL3:
  break;
 default:
  assert(0);
  break;
 }
}

void test_task2(void)
{
 wakeup(0);
}

struct task tasks[2] =
{
 {
  TASK_STATE_RUN,
  test_task,
  { 0, NULL, NULL, NULL },
  NULL,
 },
 {
  TASK_STATE_RUN,
  test_task2,
  { 0, NULL, NULL, NULL},
  NULL,
 },
};

void schedule(void)
{
 int i;

 for (i = 0; i < 2; i++) {
  if (tasks[i].tsk_state == TASK_STATE_RUN) {
   task_curr = &tasks[i];
   task_curr->tsk_stk_curr = &task_curr->tsk_stk;
   tasks[i].tsk_entry();
  }
 }
}

int main(int argc, char *argv[])
{
 for (;;)
  schedule();
 return 0;
}

Stack-less multi-task system

2007-01-14T05:41:00.000-08:00

In traditional multi-task operating system, there is one stack for one task. The stack is mainly used for storing call chain, function parameters and local variables, that is, the context of the task. When task is switched, the stack of task is switched too.

The problem of stack based multi-task system is the size of stack should be the maximum possible size that the stack maybe grow to. In Linux system, the size of kernel task stack is 4kB or 8kB. But, in 99% condition, only small part of whole stack is used. The efficiency of stack memory usage is low.

Another solution is that all tasks share the same stack. When task is executed, the stack is used for storing call chain, function parameters and local variables as that of task based multi-task. When task is about to be switched out of CPU, it saves all state (context) needed to continue the work elsewhere (such as global variable, malloced memory) and hands the stack to the task to be switched into CPU. The new task will ruin all the contents in the stack and use it exclusively until the next switching.

The problem of stack-less multi-task system is that when switching point is deeply nested, it is hard to storing all state needed in a modular way. For example, when a "page fault" occurs in a memory mapped page, the handler will go through memory subsystem and file system and maybe be switched out of CPU for waiting for disk reading. How can the task save the state of memory subsystem and file system in a modular way? So, the stack-less multi-task system is used mainly in the kernel of micro-kernel operating system, because there are only few simple subsystems, the state saving can be done in a modular or manageable way.

A resolution is adaptive stack scheme. If task is working for a system call that have very small stack usage (such as the wait in POSIX), all the state needed is stored elsewhere and the stack is freed, otherwise the stack is preserved. But this resolution is not very effective if most system calls use more stack.

I think another possible resolution is split task into more small tasks. Because every task is so small, its state can be stored without refer to other subsystem, that is in a modular way. And because the overhead of task is much less than before, the total overhead is not too much. Perhaps one small task corresponds one subsystem in traditional system. For example, the memory system, file system can be tasks too. In this system, create a process turns to create several tasks.

Gadmei UTV 330 on Linux

2006-12-05T05:35:00.000-08:00

Recently, I bought a Gadmei UTV 330 USB TV Box. Because I want to know more about video capturing while it is the cheapest USB TV Box I can find.

The basic hardware information is as follow (gotten from V4L wiki: http://www.linuxtv.org/v4lwiki/index.php/Em2820/gadmei_utv_310 and driver log information (dmesg)):

Tuner: TNF 5323
Decoder: Philip SAA7113
USB bridge: Empia em2860

Thanks the great work of V4L guys, the card almost works out of box. It can be recognized correctly under Linux and accept NTSC video input. But the TV broadcast standard used in my country is PAL, so I added the PAL standard to card configuration in driver, and it works for PAL video immediately. The data driven design makes adding new card support so easy.

The sound and IR doesn't work out of box, and Gadmei corporation doesn't release hardware specification, so the work method to add sound and IR support is to do USB snooping under Windows.

After reviewing the USB standard and studying the source code of existing driver, adding sound and IR support is a fairly easy work.

The sound control is through register 0x08 of em2860, the effect of different value is as follow:

0xfc: switch to line in (composite video or svideo)
0xfd: switch to tuner
0xfe: turn off sound

The IR is turn on through writing 0x07 to register 0x0f. The key press and release are gotten through pulling register 0x45 periodically. The only remaining thing is a correct key map.

The maintainer of em28xx driver is Markus Rechberger. It is very kind of him to review and revise my patch.

Many USB TV Box is based on Empia em28xx series chip. The em28xx driver information can be found on V4L wiki: http://www.linuxtv.org/v4lwiki/index.php/Em2880.