Setup CLFSWM

This article is based on my previous post "Setup Common Lisp Environment". Please read that first if you have not.
For me, one way to learn a new language is to use it frequently. To learn common lisp, I try to use clfswm (Common Lisp FullScreen Window Manager: http://common-lisp.net/project/clfswm/) in one of my machine. It is written/configured with common lisp.
I use quick lisp to install clfswm as follow:

* (qlsetup)
* (ql:quickload "clfswm")

Gnome is my primary desktop now, to continue use gnome applications in clfswm, I add a .xsession file as follow in my home directory:

#!/bin/bash

# log for debugging, quite useful if you make some mistake in your configuration
exec &> $HOME/.clfswm.log

gnome-settings-daemon &
eval `gnome-keyring-daemon`
nm-applet &
# or some other panel application, I mainly use it for systray support
fbpanel &
# some applications I used everyday, so I start up them along with login
gnome-terminal &
emacs &

# to setup background, you can select other applications for it too
feh --bg-center /usr/share/wallpapers/joy/contents/images/1366x768.png
exec sbcl --noinform --no-userinit --no-sysinit --eval "(require :asdf)" \
    --eval "(require :clfswm)" --eval "(clfswm:main)" --quit

To adapt clfswm to my habit (mostly based on gnome), I customized clfswm a little via .clfswmrc. I changed fbpanel a little for clfswm too. Details will be in following posts.
In fact, I still have several issues with my clfswm, this prevents me to use it as my primary window manager in my laptop.

• Sometimes modal dialog will make a frame "frozen" after covered by other window, that is, no reaction for hot key. I need to click another frame and back to the frame to switch to the modal dialog.
• I do not find real full screen support. Some software need real full screen to display a full screen "edge menu" (such as virtualbox). Or maybe software thinks it is in full screen mode, I just can not activate the "edge menu".

Setup Common Lisp Environment

Debian is my main Linux distribution, so usually it is an easy task to setup some programming language development and usage environment. At first, it looks like before for common lisp. I installed sbcl deb package according to the recommendation from cliki (http://www.cliki.net/).

 # apt-get install sbcl sbcl-doc 

Then I installed slime and several common lisp deb packages. But clfswm deb package does not work. And I found quicklisp provide much more common lisp systems than debian and it is actively maintained. So I decide to use quicklisp for all common lisp systems, only use sbcl deb package.
It is very easy to install quicklisp (from http://www.quicklisp.org/beta/).

 $ curl -O http://beta.quicklisp.org/quicklisp.lisp

 $ sbcl --load quicklisp.lisp

 * (quicklisp-quickstart:install)

 * (ql:add-to-init-file) 

Then you can use (ql:system-apropos) to search systems and install them with (ql:quickload "xxx"). But quicklisp take too long to initialize, so I put quicklisp initialize code in $HOME/.sbclrc into a function (qlsetup) and call it only when I want to install something with quicklisp. To make systems installed with quicklisp avaiable without quicklisp initialization. I add quicklisp software directory into asdf search path. Create a file named $HOME/.config/common-lisp/source-registry.conf.d/33-quicklisp.conf, its contents is:

 (:tree (:home #p"quicklisp/dists/quicklisp/software")) 

Then install and configure slime (from https://github.com/quicklisp/quicklisp-slime-helper),

 * (qlsetup) * (ql:quickload "quicklisp-slime-helper") 

Add the following to $HOME/.emacs

 (load (expand-file-name "~/quicklisp/slime-helper.el"))

 (setq inferior-lisp-program "sbcl") 

Then install hyperspec

 # apt-get install hyperspec 

Try to use Lisp As My Second Language

Linux kernel programming is my job, so C is my first language. But C is not high level enough for some work such as generating a HTML report from a git repository. So I use python as my second language.
Now I think about using lisp as my second language. Because

• It is high level enough.
• It has some interesting features like macro. It is said I can build my own mini-language with it.
• I want to learn meta-programming and compiler. Lisp macro can be a start point of them.
• I love Emacs, I need lisp to customize my Emacs.

The two main lisp dialects now are scheme and common lisp. I have learn something about basic scheme (without macro) and common lisp macro (through "On Lisp"). I will start with common lisp, because from google, it seems to be more practical than scheme, which is considered somewhat academic style.

Install Debian testing or Wheezy (7.0) into Acer S7

Install Debian testing or Wheezy (7.0) into Acer S7.  Dual boot with Windows 8

The steps is not what I have done exactly.  What I have done is more complicated with similar solution.  Use it at your own risk.

1. In windows, delete data partitions and shrink windows partitions
2. Download Debian testing or Wheezy (7.0) netinst iso, dd it to the USB key
3. Goto BIOS (press F2 during boot), disable secure boot, set boot from USB
4. Boot with Debian install USB key, begin install
5. Before partition, go to console 2 (Alt + F2), activate rescue console
6. Run following command to backup

# dd if=/dev/sda bs=1M | gzip -c > <a usb hard disk>/acer-s7-sda.gz
# dd if=/dev/sdb bs=1M | gzip -c > <a usb hard disk>/acer-s7-sdb.gz

They are not too big, about 10G for each.
7. After backup, continue to install
7.1 When create partition, choose "Manual", create partitions on RAID disk, NOT sda and sdb.
7.2 grub install may failed, that is not a big issue. Just continue without bootloader until install finished.
8. Boot with Debian install USB key, get the rescue console
9. chroot into the partition you installed Debian into
10. in chroot shell, do

# mount -t proc none /proc
# mount -t sysfs sysfs /sys

11. install dmraid, kpartx

# apt-get install dmraid kpartx

12. Add i915 into initramfs, this will give you a opportunity to debug during boot

# echo i915 >> /usr/share/initramfs-tools/modules

13. Add kpartx into initramfs, becuase dmraid can not recoganize gpt partition table.  Download kpartx-initramfs-tools.tar.gz, extract it, then

# cp -r kpartx-initramfs-tools/* /usr/share/initramfs-tools

14. update-initramfs

# update-initramfs -u

15. install grub-efi

# apt-get install grub-efi-amd64

16. update grub

# update-grub

17. mount efi partition?

# mkdir /boot/efi
# mount /dev/disk/by-id/dm-uuid-part2-DMRAID-isw_xxx_HDD0 /boot/efi

18. install grub

# grub-install

18. All done, reboot, and bless

X application startup script with wnck

It is common that several X applications are involved for one task. For example, web browser is used for searching and help document, Emacs is used to edit, and X termainal is used for testing. Here, script can be used to startup several X programs with one command typing or click.

But simple shell script can only startup applications. You must arrange the screen layout by hand. In fact the screen layout can be program with script too. Python + wnck can be used for that.

The script as follow is an example. It startup pidgin and gnome-terminal, minimize pidgin and maximize gnome-terminal vertically. With wnck, you can arrage the layout of your X applications arbitrarily.

#!/usr/bin/python

import gobject
import gtk
import wnck
import os
import time

def gtk_wait():
    gobject.idle_add(gtk.main_quit)
    gtk.main()

def get_wnck_apps_from_wins(wins):
    apps = set()
    for win in wins:
        app = win.get_application()
        apps.add(app)
    return list(apps)

def screen_get_wnck_apps(s):
    wins = s.get_windows()
    return get_wnck_apps_from_wins(wins)

def screen_find_windows_by_pid(s, pid):
    wins = []
    all_wins = s.get_windows()
    for win in all_wins:
        app = win.get_application()
        if app.get_pid() == pid:
            wins.append(win)
    return wins

def gtk_wait_timeout():
    gtk_wait()
    time.sleep(0.1)

def apply_timeout(func, args, timeout = 10, wait = gtk_wait_timeout):
    start = time.time()
    res = None
    while not res:
        res = apply(func, args)
        now = time.time()
        if now - start > timeout:
            break
        wait()
    return res

def wait_child_proc():
    try:
        os.waitpid(-1, os.WNOHANG)
    except OSError, err:
        if err.errno == 10: # No child processes
            pass

def pid_is_valid(pid):
    return os.path.exists('/proc/%d' % (pid,))

def screen_find_windows_by_pid_timeout(s, pid):
    while True:
        wins = apply_timeout(screen_find_windows_by_pid, (s, pid), 0.5)
        if wins:
            break
        wait_child_proc()
        if not pid_is_valid(pid):
            break
    return wins

def init():
    global screen
    screen = wnck.screen_get_default()
    gtk_wait()

class app(object):
    def __init__(self, pid):
        self.pid = pid
        self.update()
    def update(self):
        self.wnck_wins = screen_find_windows_by_pid_timeout(screen, self.pid)
        self.wnck_apps = get_wnck_apps_from_wins(self.wnck_wins)
    def minimize(self):
        for win in self.wnck_wins:
            win.minimize()
    def maximize(self):
        for win in self.wnck_wins:
            wt = win.get_window_type()
            if wt == wnck.WINDOW_NORMAL:
                win.maximize()
    def maximize_vertically(self):
        for win in self.wnck_wins:
            wt = win.get_window_type()
            if wt == wnck.WINDOW_NORMAL:
                win.maximize_vertically()
    def close(self):
        for win in self.wnck_wins:
            win.close(0)

class app_info(object):
    def __init__(self, start_argv, app_name):
        object.__init__(self)
        self.start_argv = start_argv
        self.app_name = app_name
    def get_instances(self):
        pids = []
        wnck_apps = screen_get_wnck_apps(screen)
        for wnck_app in wnck_apps:
            if wnck_app.get_name() == self.app_name:
                pids.append(wnck_app.get_pid())
        if pids:
            return [app(pid) for pid in pids]
        else:
            return [self.new_instance()]
    def new_instance(self):
        argv = self.start_argv[:]
        argv.insert(0, 'nohup')
        pid = os.spawnvp(os.P_NOWAIT, 'nohup', argv)
        return app(pid)

def start_apps():
    pidgin_info = app_info(['pidgin'], 'Pidgin')
    terminal_info = app_info(['gnome-terminal'], 'Terminal')

    os.system('rm -f nohup.out ~/nohup.out')

    apps = pidgin_info.get_instances()
    apps = pidgin_info.get_instances()
    for app in apps:
        app.close()

    apps = terminal_info.get_instances()
    for app in apps:
        app.maximize_vertically()

init()
start_apps()

Disk reading performance with hdparm

hdparm can be used to get disk reading performance with following command line:

• hdparm -T <dev>. Get cached disk reading performance. In fact, the page cache performance is tested.
• hdparm -t <dev>. Get raw disk reading performance.

Implementation

hdparm -T

ioctl(devfd, BLKFLSBUF, NULL);
read(devfd, buf, READING_SIZE)
do {
 seek(devfd, 0, SEEK_SET);
 read(devfd, buf, READING_SIZE);
} while (/* time is not over */);

Because the data to be read is in page cache always (if READING_SIZE << memory size), the result is page cache performance.

hdparm -t

ioctl(devfd, BLKFLSBUF, NULL);
do {
 read(devfd, buf, READING_SIZE);
} while (/* time is not over */);

Because the data to be read is not in page cache, and the bottleneck lies in raw disk reading speed, the result is raw disk reading performance.

reading prepare *         *         *
memory copy          -         -
raw disk read  ++++++++  ++++++++

As in above figure, reading reparation and memory copying in testing is neglectable, especially when readahead is considered.

Reading notes: Linux kernel perf event source

For Linux kernel 2.6.33

perf event enumeration

Kernel has no hardware perf event enumeration interface.

• tools command line: perf list
• Available abstract hardware perf events are hard-coded in perf user space tool. Raw hardware perf events are not enumerated by the tool, they can be specified via a 64-bit number in the tool command line. Abstract hardware perf events are implemented with the raw hardware perf events. Whether the abstract hardware perf events are available can be queried via syscall perf_event_open.
• Software trace point are exported in debugfs (/debugfs/tracing/events)

perf event management

• There is at most one perf event context for each task and each CPU. This is used to manage all perf events inside one context (task or CPU).
• All perf events in a context are linked as list in context->event_list.
• Perf events in one context are organized into groups. All group leaders in a context is linked as list in context->group_list. All perf events in a group is linked as list in group_leader->sibling_list.
• It seems that perf events in the group share some attributes, such as enable/disable, cpu, inherit, etc. In "perf record", for task events, one group is created for perf events for one CPU. In __perf_event_sched_in, it is assumed perf events in one group is for one CPU. But in "perf_record", group support can not be turned on now, that is, group is not used at all now.
• perf events inherited in child tasks are linked as list in parent_perf_event->child_list.
• In "perf record", for task profile, for one perf event type, one perf event is created for each task and each CPU. For perf event operation, it appears that one perf event for each task is sufficient, because one task can only run on one CPU at any time. But perf events may be inherited by children perf events in children tasks (forked tasks). The children perf events use the original perf events for sample output. To make sample ring buffer a per-CPU data structure, perf events are created for each CPU too. So that, when parent and child tasks run on different CPU simultaneously, they all use perf events in parent task for output, but they use different perf events for corresponding CPU.

perf event state track

• task schedule in/out (perf_event_task_sched_in/out). Record/disable counter before schedule out, restore/enable counter after schedule in. The schedule in/out event is recorded too.
• task fork
  • perf_event_fork: record fork event.
  • perf_event_init_task: perf_event->attr.inherit control whether to inherit. If inheriting, child perf event will be created for child task. Children events information is accumulated in some statistics. The sample collected via children events are sent to corresponding parent events for same CPU.
• task exit
  • perf_event_exit_task: feed back event values to parent events (child_total_time_running, etc). Free all resources for children events.
• fd closed, perf_release: free all resources. perf event in original task is managed by corresponding fd. Children perf events in children tasks are freed in perf_event_exit_task.

Dataflow

For perf event samples:

hardware counter ---> perf event mmap data page -> user space tool (perf)
                  |
soft trace point -/

For ftrace and tracing point:

 software trace point -> tracing ring buffer -> user space 

User space interface

• File based interface. File descriptor is obtained via a new syscall perf_event_open. Because one file describptor is needed for each perf event. Perf event may for a specific task, cpu, etc.
• memory map. Sample data is passed from kernel to user space mainly via memory map. Used to implemend shared memory based ring buffer. Details are in ring buffer.
• ring buffer
  • Work in both per-CPU and not-per-CPU mode
  • write side in kernel, read side in user space. shared memory communication between kernel and user space. It has better performance than memory copy.
  • lock-less write side, similar as Steven Rostedt's unified tracing buffer, perf_mmap_data.head is used to reserve space, while perf_mmap_data.user_page->data_head is used for committed data.

Code generation for software tracing point

• A set of macro is defined for software tracing point related code generation, such as binary format description, trace point function generation, etc.
• Some macros are un-defined and re-defined again and again, and some files are included again and again for different set of macro definitions. This is tricky but really powerful. Source code is in include/trace/define_trace.h and include/trace/ftrace.h.

Tracing and perf event

• Lock-less trace ring buffer is not used in perf event, another simple ring buffer implementation for mmap data pages is used instead.
• For software tracing point, the tracing point sample collecting code is shared between tracing and perf event.

Random thought

• ring buffer in perf event has some special features.
  • work in both per-CPU and not-per-CPU mode
  • shared memory communication between kernel and user space, with write side in kernel and read side in user space.
• In the code generation for software tracing point
  • macro is un-defined and re-defined again and again, and some files are included again and again for different set of macro definitions.