Realtime Preemption
Ingo Molnar’s awesome realtime preemption patches are available here.
Further reading:
After you read this page you might find the “Ramblings on linux audio/midi system setup” page interesting, too.
Patching the kernel source
Ok, i suppose i should say a little bit about patching the kernel source with the -rt patches. Let’s take a look at an example.
The name of the file carries the information about which source version to apply it to. In this case it’s 2.6.16, not 2.6.16.1 or any other -stable team release. So assuming you have linux-2.6.16.tar.bz2 and patch-2.6.16-rt20 it goes like this:
cd linux-2.6.16
patch –dry-run -p1 < /path/to/patch-2.6.16-rt20
if the previous patch command returned without error (the –dry-run option means it doesn’t apply the patch for real), you can run:
Now copy your previous kernel .config to the current dir. I.e. if you enabled /proc/config.gz support you can just take the one from the currently runnning kernel:
and run
which only asks about any newly added config values. Usually you also want to have a quick glance with
too, to see whether everything’s alright. Then just build your kernel as usual.
Choosing the preemption model
The realtime preemption patches introduce new preemption models that allow for reliable low latency operation. Have a look at the Section
Processor type and features
in the menu of make menuconfig in the kernel source. Here a new config option has appeared:
It offers four choices:
Voluntary Kernel Preemption (Desktop)
Preemptible Kernel (Low-Latency Desktop)
Complete Preemption (Real-Time)
Generally speaking each step down in above list should improve the kernels worst case latencies at the cost of some additional overhead. Make sure you read the help text for each option. The most interesting for low latency audio work on linux are the bottom two (”Preemptible Kernel (Low-Latency Desktop)” and “Complete Preemption (Real-Time)”:
Preemptible Kernel (Low-Latency Desktop)
This is a suitable setting for people that do not have a need for really low latencies. While i personally always run full preemption, this might still be useful for some people. When choosing this preemption model, make sure to also enable:
and
This setting basically makes irq execution subject to scheduler decisions. Irq’s are not nessecarily executed directly anymore, but rather deferred to a handler thread. This handler thread runs with realtime privileges. The priorities of these irq handler threads are around 50. So by assigning a higher realtime priority to jackd we can make sure irq’s do not disturb jack’s operation.
Also make sure
is disabled. This would basically destroy the low latency features of the kernel.
Build the rest of the kernel to your liking, boot into it and make your soundcard irq handler thread unthreaded. On my system i use a command like this:
This makes the soundcard irq the only interrupt which is handled immeaditly. Note that the filename on your system most likely differs. “CS46XX” is the module name of my soundcard driver (which is based on a cs46xx chip). All other IRQ’s are deferred to their respective handler threads. To find out which irq your soundcard uses, have a look at the output of
On my system it looks like this:
~$ cat /proc/interrupts
CPU0
0: 357694670 XT-PIC timer 0/94670
1: 192527 XT-PIC i8042 0/92527
2: 0 XT-PIC cascade 0/0
3: 3683355 XT-PIC CS46XX 0/83355 < --- my soundcard
5: 203367747 XT-PIC ICE1712 0/67747 <--- another one :)
8: 12983156 XT-PIC rtc 0/83156
10: 18757427 XT-PIC eth0 0/57427
11: 652848 XT-PIC nvidia 0/52848
12: 3109504 XT-PIC i8042 0/9504
14: 1693997 XT-PIC ide0 0/93997
15: 3617940 XT-PIC ide1 0/17940
NMI: 0
ERR: 600
It is somewhat important to have a single irq for each soundcard, because otherwise the other device on the irq gets the same priority treatment which might disturb the soundcard operation.
Also make sure to run jackd at realtime a prio of at least 60 (because the other threaded irq handlers have a prio of around 50 and we want jack to be higher).
Complete Preemption (Real-Time)
When choosing complete preemption, you don’t have to specifically enable the threaded irq handlers as they are mandatory in this preemption model.
The main difference is that because of this we cannot unthread the irq handler of our soundcard (because all IRQ handlers run threaded in complete preemption mode). This means that all irq’s are deferred to their respective handler threads. We have to give the soundcard’s irq handler thread a very high priority. You can use the chrt utility for this (it should usually be in a package called schedutils).
To give my CS46XX soundcard’s irq handler a priority of 99 (the highest possible) i would use a commandline similar to this:
The other irq handlers will run at priorities around 50, so make sure jackd’s RT prio is lower than the soundcard, but higher than all other irq’s. I use a priority of 70 with great success (
).
Also, because i’m a lazy bugger i have written two small scripts for myself: irq_check and irq_set:
irq_check:
chrt -p `pidof “IRQ $1″`
irq_set:
chrt -f -p $2 `pidof “IRQ $1″`
High Resolution Timers
I would recommend turning on high resolution timers if you use full preemption.
Hi. Your blog is very usefull to me.
I have a problem whith the kernel compilation with the realtime preemption patches.
Here is the error log:
In file included from kernel/rt.c:71:
kernel/rtmutex_common.h: En la función ‘task_has_pi_waiters’:
kernel/rtmutex_common.h:81: error: ’struct task_struct’ no tiene un miembro llamado ‘pi_waiters’
kernel/rtmutex_common.h: En la función ‘task_top_pi_waiter’:
kernel/rtmutex_common.h:87: error: ’struct task_struct’ no tiene un miembro llamado ‘pi_waiters’
make[2]: *** [kernel/rt.o] Error 1
The kernel version is 2.6.16 and the patch file is patch-2.6.16-rt20 on a Debian Etch with the Complete Preemption option selected. I have hours trying to resolve this, but maybe i need a little help. Thanks.
(Sorry my bad english :S)
Try a newer kernel :) There’s patches available for kernel versions up to 2.6.19-rcX.
Hi, i’m try use kernel 2.6.20 and patch-2.6.20-rt8 and see that don’t use realtime-lsm module and i don’t konw use the PAM and rt-limits! I try the set_rlimits but don’t work well, many “xruns” and jack messager say: “cannot write to jackstart sync pipe 4 (bad file descriptor)”
I try several kernels, 2.6.(11 12 14 14.3 and 20) and the RP-script don’t compile, always returns with erros!
Have You site or tips with main steps to jack run right?
Thank you
Benedito
you must not use jackstart on 2.6.x kernels. Simply run jackd by itself. If you have more questions, come to the channel #lad or #lau on irc.freenode.org
hi, i’ve a well configured kernel, rlimits and realtime-lsm well configured, tryed out jack from debian etch and sid, but everytime i try to start jack i cannot get realtime priorities and i got a lot of xruns, i’ve a macbookpro 2,16ghx with snd_hda_intel soundcard.
thank you in advance
you don\t need both realtime-lsm and rlimits. One of them is enough. It is impossible to debug your problem in this comment boxes. best come over to #lad on irc.freenode.org and we will help you there..
Dear Florian,
Maybe you might help me. I ran Gutsy with succes and low latency. But I made a fresh install of hardy, and now i see that my soundcard (RME) is on IRQ 19. How can I give it a higher priority? Or can I even give it it’s own IRQ?
Right now, I get xruns very regularly, even at 12 ms latency, and I am sure it has something to do with my Nvidia card having more priority, since the xruns are very severe when I manipulate windows while playing/recording.
These are the IRQś:
CPU0 CPU1
0: 551 0 IO-APIC-edge timer
1: 597 0 IO-APIC-edge i8042
3: 2 0 IO-APIC-edge
4: 2 0 IO-APIC-edge
6: 5 0 IO-APIC-edge floppy
7: 0 0 IO-APIC-edge parport0
8: 7 0 IO-APIC-edge rtc
9: 0 0 IO-APIC-fasteoi acpi
10: 0 0 IO-APIC-edge MPU401 UART
14: 28055 0 IO-APIC-edge libata
15: 38000 0 IO-APIC-edge libata
16: 249821 0 IO-APIC-fasteoi uhci_hcd:usb1, uhci_hcd:usb4, nvidia
17: 9 0 IO-APIC-fasteoi uhci_hcd:usb2
18: 234 0 IO-APIC-fasteoi uhci_hcd:usb3, libata
19: 632204 0 IO-APIC-fasteoi ehci_hcd:usb5, rme9652
20: 8087 0 IO-APIC-fasteoi eth0
21: 3094 0 IO-APIC-fasteoi Intel ICH5
NMI: 0 0 Non-maskable interrupts
LOC: 2729386 2730264 Local timer interrupts
RES: 1402649 1429321 Rescheduling interrupts
CAL: 1683 2174 function call interrupts
TLB: 8792 9501 TLB shootdowns
TRM: 0 0 Thermal event interrupts
SPU: 0 0 Spurious interrupts
ERR: 0
MIS: 0
Please bear in mind have been on Linux for just two months now, and I really love it.
Thanks for your help!
Alfons Verreijt
Hi Florian,
I try to apply the chrt-commands but it looks like I did not get the synthax right.
chrt -f -p 89 `pidof “IRQ 19″`
gives me the message:
failed to get pid 89’s policy
so where can I find out, how to use chrt on my given system?
best regards :-)
@zettberlin: try adding a minus sign in between IRQ and the number “IRQ-19″
(check out “ps ax | grep IRQ” and you’ll your syntax).
Cheers,
ERik
———————
share the experience
Hi Florian,
Thanks for this very useful tutorial, which I really enjoyed! Could you give a description of the effect of the “High Resolution Timers” (CONFIG_HIGH_RES_TIMERS) option in the case of full preemption? On http://rt.wiki.kernel.org/Frequently_Asked_Questions , it is merely mentioned that this is an “optional parameter for configuring realtime kernel, but it affects realtime performance”, and I am wondering in what way that articulates itself?
Thanks a bunch,
flo.H
That’s actually
http://rt.wiki.kernel.org/index.php/Frequently_Asked_Questions#Optional_parameters_for_configuring_realtime_kernel.2C_but_they_effect_realtime_performance.3F
Sorry for the typo in the post before.
best,
flo.H
Awesome! Your articles are superb for understanding real time audio on linux. Thumbs up!
Is it possible to set the priorities for soundcards etc in the boot scripts?
Regards, Rob
Yes, of course. Since you can do it from a terminal, you can also do it from a shell script. Search the linux audio developers’ email list for init scripts. I think Rui posted some