Juce plugins cause realtime kernel lockup

56 posts / 0 new
Last post
falkTX
falkTX's picture
Offline
Last seen: 10 hours 34 min ago
Joined: 4 Jun 2011 - 16:15
Juce plugins cause realtime kernel lockup

Hey Jules and all Juce friends,

I noticed when using Juce plugins in a realtime kernel, the system will eventually lockup (after around ~5secs of usage).
I experienced this a long time ago, in a 2.6.33-rt kernel, but since I don't run a realtime kernel for some time, I was unware if this issue was fixed or not.

Some days ago, 2 IRC users reported to me that they had experienced lockups right after opening a juce based plugin (happened during testing of my juce-lv2 wrapper code).
Recently a friend of mine also experienced this, and the lockup -> hard-reset was very problematic to him (happened with VST versions this time).

I'm not sure what's causing this, but I can already guess will probably be very problematic to debug (and test too, since it locks the system, so we need a hard-reboot to re-test).
If anyone has some idea of what is the cause of this, please say so. I'm afraid I can't publish any juce-lv2 plugins if this issue remains.

TheVinn
Offline
Last seen: 2 months 3 weeks ago
Joined: 29 Aug 2009 - 11:31
Re: Juce plugins cause realtime kernel lockup

Just a wild guess, could it be a signaling floating-point exception?

falkTX
falkTX's picture
Offline
Last seen: 10 hours 34 min ago
Joined: 4 Jun 2011 - 16:15
Re: Juce plugins cause realtime kernel lockup

TheVinn wrote:
Just a wild guess, could it be a signaling floating-point exception?

There's one thing I know for sure - it happens even outside the audio thread.
When we start generating lv2 *.ttl files, it needs to initialize the plugin to get it's info. The kernel will hang/lock when changing programs.
(as reported to me by an user, complaining that it was not possible to compile/generate the lv2 plugins in a rt kernel)

jpo
Offline
Last seen: 1 day 1 hour ago
Joined: 20 Mar 2008 - 13:45
Re: Juce plugins cause realtime kernel lockup

you could try to check all calls to pthread_setshedparam , if it is not a kernel bug then maybe it could be a realtime prio thread that spins forever ?

X-Ryl669
Offline
Last seen: 7 months 2 weeks ago
Joined: 24 Apr 2005 - 17:30
Re: Juce plugins cause realtime kernel lockup

+1, it definitively looks like one of your thread is using a realtime priority and not giving/waiting on any other.
Easy to test through, just run your software without root priviledge, you shouldn't be able to use RT or FIFO scheduler in that case.

X-Ryl669

falkTX
falkTX's picture
Offline
Last seen: 10 hours 34 min ago
Joined: 4 Jun 2011 - 16:15
Re: Juce plugins cause realtime kernel lockup

The problem is that a linux realtime kernel is a required setup for some users. The realtime kernel has been here for a long time, and this sort of issues are not very common (ie, system lockup just by initializing the plugin and chaning presets).
normal users *do* have access to RT and FIFO, and that's intentional. we want to be able to fully use our system resources, so I need to assume that plugins are safe to use.
In this case, they're are not, so currently juce plugins are just unusable on some linux systems.

I'm still not sure what causes this (I have an idea, but I want to make sure it's correct before telling about it).
I'll do more testing and report back once I got a definitive answer.

X-Ryl669
Offline
Last seen: 7 months 2 weeks ago
Joined: 24 Apr 2005 - 17:30
Re: Juce plugins cause realtime kernel lockup

Ok, another test then, try on a linux box with no RT kernel, change:
/proc/sys/kernel/sched_rt_runtime_us
to be the same as
/proc/sys/kernel/sched_rt_period_us
and run the software as root.

If it freeze, it's clearly a RT thread that's not giving up its timeslice.
You can figure out which one by running a shell with the highest priority and send a SIGSEGV signal to your frozen process, in order to get a core file.
Start gdb with that core file and get the backtrace.

X-Ryl669

falkTX
falkTX's picture
Offline
Last seen: 10 hours 34 min ago
Joined: 4 Jun 2011 - 16:15
Re: Juce plugins cause realtime kernel lockup

I did some more testing, and it seems to me that what is locking the system is not the audio stuff, but the GUI side of things.
The plugin runs nicely, but when I open the UI things started to change. Closing the UI and re-opening it is usually enough to cause the lockup.
When I run the standalone version, the application locks after showing and closing the device settings dialog.

I made a simple screencast/video running a clean VM (ArchLinux 32bit), with realtime on (linux-rt kernel and JACK in realtime mode).
http://kxstudio.sourceforge.net/tmp/juce-lockup.mkv

EDIT: The source code for this specific plugin is here:
http://distrho.git.sourceforge.net/git/gitweb.cgi?p=distrho/distrho;a=tree;f=plugins/3BandEQ/source;hb=HEAD

X-Ryl669
Offline
Last seen: 7 months 2 weeks ago
Joined: 24 Apr 2005 - 17:30
Re: Juce plugins cause realtime kernel lockup

Instead of (try to) killing the app (and freeze the kernel), trigger and attach a debugger to figure out where it is.
It's hard to guess what you're doing, and I don't want to study all your code.

X-Ryl669

falkTX
falkTX's picture
Offline
Last seen: 10 hours 34 min ago
Joined: 4 Jun 2011 - 16:15
Re: Juce plugins cause realtime kernel lockup

X-Ryl669 wrote:
Instead of (try to) killing the app (and freeze the kernel), trigger and attach a debugger to figure out where it is.

The kill is just a way to trigger the lockup faster. If we wait long enough (and click some buttons while at it), it will freeze the kernel.

X-Ryl669 wrote:
It's hard to guess what you're doing, and I don't want to study all your code.

You missed the point here, this is not about my plugin, it's about juce. The test plugin I put there was the simplest one (6 parameters, no presets, no special stuff there).

The lockup will happen with the official JuceDemo application (using untouched juce-1.53.zip code, just adjusted the makefile to add needed links '-ldl -lXext').
I even tried the loomer plugins (http://www.loomer.co.uk/), and they also lock the system eventually.
renoise is the exception here (lots of UI updates and no kernel lockup). I don't know how they do it...
EDIT: Pianoteq also doesn't lock the kernel.

X-Ryl669
Offline
Last seen: 7 months 2 weeks ago
Joined: 24 Apr 2005 - 17:30
Re: Juce plugins cause realtime kernel lockup

There is no point in convincing me Juce is wrong, I already trust you.
Please do what I ask, that is, once the application seems frozen, and you still have a hand on the terminal, fire up gdb, attach to the application, break it, and post a backtrace for all threads.
That's the only way we could figure out the issue in Juce, by understanding what thread does what (and what could be wrong).

X-Ryl669

Pages