Reading notes: optimizing Linux performance

Chapter 3

1. vmstat can show swap in/out pages. A simultaneously high swap-in and swap-out rate could indicate that the system does not have enough memory to handle all the running processes.
2. slabtop is "top" for slab.

Chapter 4

1. strace can show system call performance statistics too (strace -c).
2. ltrace: performance statistics for library (like strace, but library function instead of system call).
3. ld.so: dynamic load/link statistics for libraries. Example: env LD_DEBUG=statistics LD_DEBUG_OUTPUT=lddebug gcalctool

Chapter 6

1. iostate: show disk IO statistics

Chapter 7

1. netstat -c: period update
2. netstat -t: choose tcp protocol
3. netstat -l, -a: listening sockets or listening + conntected.

Chapter 8

1. script: record what you have done and the output, seems useful.
2. gcc -g3: provide more debug information then gcc -g (default to
   • g2), such as the macro definitions present in the source.

Chapter 11

1. Use gdb to check call chain:
   • Set a breakpoint at check-point
   • gdb can execute a given set of commands when it hits a breakpoint. By using the command command, we can tell gdb to execute bt; cont every time it hits the breakpoint.
   • Conditional breakpoint can be implemented via gdb script (commands) too. For example, we can enable breakpoint 2, when we reach breakpoint 1.
   • gdb output can be logged to a text file via: set logging on
2. oprofile reveals which function is hot, but do not show how many times it is called. ltrace or some other traces shows which function (including subfunctions) consumes most CPU time. This is from two perspective, and they can be combined to inspect the performance property.
3. For Fedora and Enterprise Linux, Red Hat provides a set of debuginfo rpms that contain all the symbol information and sources that were generated by the compiler when the application was complied. Each binary package or library has a corresponding debuginfo rpm that contains the debugging information. This allows Red Hat to ship the binaries without the disk-space–consuming debugging information. However, it allows developers, or those investigating performance problems, to download the appropriate debuginfo packages and use them. In this case, Red Hat's version of oprofile will also recognize the debuginfo packages and pick up the symbols when profiling both an application, such a nautilus, and a library, such as gtk.
   • Debian has similar binary debug symbol packages: -dbg

Chapter 12

1. strace to find bottleneck of IO intensive application.
   • Oprofile can be used only for CPU intensive application. Because IO is performed in kernel space, strace is the right tool to reveal which kind of IO application is performing.

Chapter 13

1. This call-tree tool would be useful even if it dramatically slowed down application performance as it runs. A common way of using this would be to run oprofile to figure out which functions in an application are "hot," and then run the call-tree program to figure out why the application called them.

Reading notes of internals of the RT patch

1. Real-time is something about predictability instead of performance, throughput or latency. That is, real-time system may have worse average throughput or latency, but it will have better largest latency or lowest throughput. Linux kernel have great average performance but the worst one is not so good.
2. The drawback of Robert Love's kernel preemption patch is: A high priority process must still wait on a lower priority process while it is in a critical section, even if that same high priority process did not need to access that section.
3. The RT patch is all about determinism and being able to have full control of the kernel. But, like being root, the more power you give the user the more likely they will destroy themselves with it. Such as, a user process can have higher priority than IRQ handler, if the process holding the CPU, system will hang.

Reading notes of on submit kennel patches (OLS)

!!! Important kernel design point !!!

Very generic notifiers and hooks tend to have a large maintenance impact because they have the potential to alter the code flow in unexpected ways, lead to lock order problems, bugs, and unexpected behavior, and generally making code harder to maintain and debug later.

Linux kernel is a lock-based complex parallel system, the main method to prevent ABBA style dead-lock is enforcing lock order. But notifiers or hooks makes it hard to audit the order of locks.

Notifiers or hooks in lower layer to call functions on higher layer should be avoided. One method to call higher layer function in lower layer is via work queue like asynchronous mechanism.

Notifiers or hooks in higher layer to call functions on lower layer is OK. Such as struct file_operations is collections of function pointers (a kind of hooks?), it is only used as some kind of abstraction for lower layer, make adding new lower layer component more convenient.

Maintenance

For a project which is very complex like Linux kernel, (seems that most projects become more and more complex as time goes by), maintenance is something in the core of Linux kernel development.

To make code maintainable, firstly, code should be made readable. The code is clean-upped and redesigned again and again to make it cleaner, thus more readable.

It is much easier to add a component under the current design. Such as add a new device driver. If you need something to be changed in subsystem core, you should consider the design, maybe you need clean-up or redesign some part of the subsystem core to make code maintainable.

A patch submitter should maintain his code at least for a short while after merging.

Interface designs for user space programs is very important too. It should be stable for imaginable future.

Changing a published interface later is much harder and often special backwards compatiblility code is required.