Make JUCE audio play nicely with other iPhone apps

13 posts / 0 new
Last post
billythekid
Offline
Last seen: 3 months 1 week ago
Joined: 29 Sep 2010 - 16:33
Make JUCE audio play nicely with other iPhone apps

I am very new to iOS development and have been experimenting with a simple app over the past week that processes audio from the audio input in real time and then sends it to the output. I have made the app successfully process audio in the background by adding the following to the custom plist field in the introjucer. So far, so good.

<?xml version="1.0" encoding="UTF-8"?>

<plist>
  <dict>
    <key>UIBackgroundModes</key>
	     <array>
		      <string>audio</string>
	     </array>
  </dict>
</plist>

Now the methods I describe from here on might not be the recommended way to do things, so please correct me . . .

To process the audio, I initialise a device manager object and register an audioCallback with it. Apple says that we are not allowed to programatically close our application, but the user will not want the application to be processing audio indefinitely. Therefore, I have added a pause button inside the app, and to stop the CPU cooking away and draining the battery, I have made the pause button do a deviceManager.closeAudioDevice () when pushed. When the user wants to restart the processing, the pause button is pushed once again resulting in a deviceManager.restartLastAudioDevice ();

Now, this little setup works just fine when debugging the app on its own, but someone tried to call me in the middle of a testing session the other day causing an app crash and I then realised that more thought was needed.

I tested some simple game apps to see how they handle audio intrusions from other apps. The game that I tested paused itself when there was an incoming phone call or when siri was activated. How would I do similar using Juce? Also, I started some music playing and then opened the game and it mixed its audio with the music in the background. This is the functionality that I'd like. I left the music playing and then started my JUCE app which stopped the music and then crashed the juce app. I guess I'm not exactly doing this correctly.

How is everyone else dealing with this kind of thing? Can anyone nudge me in the right direction? Thanks in advance folks :)

billythekid
Offline
Last seen: 3 months 1 week ago
Joined: 29 Sep 2010 - 16:33
Re: Make JUCE audio play nicely with other iPhone apps

Perhaps a little more info and simplifying the question would spur someone into throwing me a bone . . . .

When my app is processing audio in the background and I open up the music app and begin playing a song, the following warning messages appear in the debugger output

2012-02-27 18:22:57.804 CrapApp[2159:707] 18:22:57.803 <Juce Message Thread> AURemoteIO::Initialize failed: -12985 (enable 3, outf< 2 ch,      0 Hz, Int16, inter> inf< 2 ch,      0 Hz, Int16, inter>)
2012-02-27 18:22:57.851 CrapApp[2159:707] 18:22:57.852 <Juce Message Thread> AURemoteIO::Initialize failed: -12985 (enable 3, outf< 2 ch,      0 Hz, Int16, inter> inf< 2 ch,      0 Hz, Int16, inter>)

If I instead run music in the background and then start my app, the music player gracefully stops playing and allows my app to have the audio IO. How do I respond to the operating system message telling my app that another app wants the audio IO like the built in music player? I'm sure that if I could gracefully close the device then I would not get any of the catastrophic failures that occur on restarting my app.

I'm guessing after reading around that it probably has something to do with responding to applicationWillResignActive (or similar?). I'll happily try and figure this out if I know where to begin, but I'm totally new to objective-C. If anyone could even suggest some places to look for inspiration in the JUCE codebase then that would get me off on the right foot.

Thanks for looking at my ramblings.

jfitzpat
Offline
Last seen: 2 months 3 weeks ago
Joined: 10 Jan 2012 - 05:29
Re: Make JUCE audio play nicely with other iPhone apps

Yes, once you change the plist to indicate that you are going to play music in the background you have a responsibility to respond to some additional 'delegates'.

Right now, Juce's iOS application delegate is pretty simple. Although it isn't precisely your problem, I did add a suspend and resume path for basic background messaging in a couple simple little apps recently. I needed to release network resources and close some network connections, etc., but it is along the lines of what you probably need.

The simplest app is here at a friend's repo:

https://github.com/LightInMotion/lPanel

The tweaked library is here:

https://github.com/LightInMotion/juce

It's merged with the tip from a day or so ago. The ONLY differences between it and master at this point are a suspend/resume mechanism for iOS and Android and some additional socket related classes and services. Jule's is great about getting fixes and enhancements in. Sockets and suspend/resume are just big subjects which should be given some thought and properly generalized (I pretty much hack for my immediate need).

I know it isn't exactly what you need, but I hope that it helps. I've done background VOIP on iOS, but not background music, but I'd be glad to answer whatever I can.

billythekid
Offline
Last seen: 3 months 1 week ago
Joined: 29 Sep 2010 - 16:33
Re: Make JUCE audio play nicely with other iPhone apps

:!: :!: :!: :!: :!:

Thanks jfitzpat! You are a very useful person to have around. I'll look at those right away. :mrgreen:

billythekid
Offline
Last seen: 3 months 1 week ago
Joined: 29 Sep 2010 - 16:33
Re: Make JUCE audio play nicely with other iPhone apps

Thanks again jfitz. I now have some nice debug console text telling me when my app is going to, and coming back from background mode. For anyone else who is interested, all of the useful edits to juce are in this commit . . . .

https://github.com/LightInMotion/juce/commit/8c6f85fe420d7498fedeb03871080d35bde09015

I also added a callback for applicationWillResignActive based on the code in the linked commit. This happens before the app suspends.

Sadly, these messages are of no use for finding when something else wants to take control of the background audio. However, I had overlooked the fact that Jules has already implemented a function in juce_ios_Audio.cpp..

void interruptionListener (const UInt32 interruptionType)

This function gets hit successfully whenever there is an incoming call, or I try to play music using the iPhone music library app. Unfortunately, this function calls other functions that crash the app frequently and irregularly. The debugger stops all over the place, sometimes when the interruption is kAudioSessionBeginInterruption, and sometimes when it is kAudioSessionEndInterruption. Even so, the fact that interruptionListener is getting hit makes me optimistic.

jfitzpat
Offline
Last seen: 2 months 3 weeks ago
Joined: 10 Jan 2012 - 05:29
Re: Make JUCE audio play nicely with other iPhone apps

FWIW, I only meant to convey that I had an example of how to get yourself access to some additional delegates. For background audio there are a number of things you will still need to do.

For example, when the user presses home twice, there are some transport controls for background audio. To get those, you will need to register for them and unregister for them (I think it is something like UIApplication beginReceivingRemoteControlEvents).

My recollection is that background audio requires the correct type of "audio session" (so the system knows to keep you playing during lock, etc.) and that interruptions are dispatched as audio session events. It seems like quite a few people are already doing Juce audio centric iOS apps, I'd hope that one of them would chime in. It certainly sounds like Jule's already handles the interruption case. The quick check I would do is Play a file in JuceDemo (you can record one there, or stick a Wav file in the bundle). With the app in the foreground on the iPad simulator, simulate a call while audio is playing. If the app behaves correctly in that case, then it is probably some specific session tweaking, etc. for true background execution. If that crashes, then there is something else going on.

Again, sorry I couldn't be of more help.

jfitzpat
Offline
Last seen: 2 months 3 weeks ago
Joined: 10 Jan 2012 - 05:29
Re: Make JUCE audio play nicely with other iPhone apps

Additional: From quick look at juce_ios_Audio.cpp, I would expect background audio to work (no proper icon or transport controls in the background, but work). The session property appears to be correct to me. Interruption of audio doesn't really require any special activity by the application. It's important enough that the OS takes care of the actual stopping for you. Basically, you just stop getting asked for audio buffer data. You can even pass in null for the interruption handler when you initialize the session.

It looks like Juce isn't doing anything when the interruption begins, only when it ends, then basically restarting the related Apple audio elements. Are you saying that one of those two calls is crashing?

I'd forgotten that the simulator doesn't really simulate a call, it just changes the status bar. If I have a few minutes tomorrow, I'll do a quick experiment with a physical iOS device.

billythekid
Offline
Last seen: 3 months 1 week ago
Joined: 29 Sep 2010 - 16:33
Re: Make JUCE audio play nicely with other iPhone apps

Quote:
I only meant to convey that I had an example of how to get yourself access to some additional delegates.

And for that, I am very very grateful.

Quote:
For example, when the user presses home twice, there are some transport controls for background audio.

I think this is only the case for iOS versions before the multitasking was added. The double-tap in the latest versions of iOS shows the background tasks. The app that I'm making needs to real-time process audio input and generate output in the background. I think that this is only possible in the multitasking incarnations of iOS.

Quote:
The quick check I would do is Play a file in JuceDemo (you can record one there, or stick a Wav file in the bundle)....

That is a superb idea. I will try it.

Quote:
It looks like Juce isn't doing anything when the interruption begins, only when it ends, then basically restarting the related Apple audio elements.

Whoops, yes. Very bad of me. I had poked around modifying that code block in an attempt to get things working last night and momentarily forgot that the original code only did stuff at the end of the interruption. I will revert JUCE back to the "factory default" before doing anything with the JUCE demo on the device.

Quote:
If I have a few minutes tomorrow, I'll do a quick experiment with a physical iOS device

You are too kind. I will try and come up with some answers using the juce demo idea quickly

billythekid
Offline
Last seen: 3 months 1 week ago
Joined: 29 Sep 2010 - 16:33
Re: Make JUCE audio play nicely with other iPhone apps

Just played with the JUCE demo on a 4S. With the default settings, I recorded some random sound and the played it back. The interrupts caused by pressing the home button were handled just fine. At the end of the interrupt, playback resumed.

However . . . . . when making a call to the phone the following error is sent to the debug console . .

2012-02-28 10:53:45.269 Juce Demo[2835:707] 10:53:45.264 <Juce Message Thread> AURemoteIO::Initialize failed: -12985 (enable 3, outf< 2 ch,      0 Hz, Int16, inter> inf< 2 ch,      0 Hz, Int16, inter>)
2012-02-28 10:53:45.274 Juce Demo[2835:707] 10:53:45.274 <Juce Message Thread> AURemoteIO::Initialize failed: -12985 (enable 3, outf< 2 ch,      0 Hz, Int16, inter> inf< 2 ch,      0 Hz, Int16, inter>)

After I hang up, the app occasionally completely dies bailing out here in juce_ios_Audio.cpp (I have put a comment next to the line where the access fault occurs) . . .

void interruptionListener (const UInt32 interruptionType)
    {
        /*if (interruptionType == kAudioSessionBeginInterruption)
        {
            isRunning = false;
            AudioOutputUnitStop (audioUnit);

            if (juce_iPhoneShowModalAlert ("Audio Interrupted",
                                           "This could have been interrupted by another application or by unplugging a headset",
                                           @"Resume",
                                           @"Cancel"))
            {
                isRunning = true;
                routingChanged (nullptr);
            }
        }*/

       
        if (interruptionType == kAudioSessionEndInterruption)
        {
            isRunning = true;
            AudioSessionSetActive (true);
            AudioOutputUnitStart (audioUnit); //Thread 1: Program received signal: "EXC_BAD_ACCESS"
        }

When I modified the plist (as shown in my original post) to make audio backgroundable, I began to see all of the same issues that I have with my app. Playback stops when I play a track using the Apple music player and the same message as above is sent to the debug console.

When I reenter the JUCE demo, I can get the audio to start working again by setting the device to "None" and then back to "iPhone Audio". However, if I instead generate an interruption by calling the phone, the audio stops with the same warning as above, but then the app completely dies bailing out in juce_ios_Audio.cpp at the same place again.

I see exactly the same sort of thing in my app. It is as if the incoming phone call is the more deadly interrupt variant.

jfitzpat
Offline
Last seen: 2 months 3 weeks ago
Joined: 10 Jan 2012 - 05:29
Re: Make JUCE audio play nicely with other iPhone apps

billythekid wrote:

I think this is only the case for iOS versions before the multitasking was added. The double-tap in the latest versions of iOS shows the background tasks. The app that I'm making needs to real-time process audio input and generate output in the background. I think that this is only possible in the multitasking incarnations of iOS.

No. If you drag the task bar (list of 'running' apps) right, the transport controls are still there, along with a shortcut icon of the app currently playing music. Also, if you double click the home button while the phone is 'locked' transport controls appear at the top of the lock screen.

As far as call interruption, my Juce iOS apps handle it just fine, but I'm not doing audio.

Interesting that you are having trouble with setting audio for a background task in your plist, a number of other users report success. I'd hoped that one of them would chime in. If time permits, I'll take a closer look at audio later today. One potential problem could be 'who owns the audioManager?' When you aren't set for a background task, 'multitasking' is really more like suspending. When you do run in the background, some resources can still be released. If, say (just for example), Jules associates some elements to UIViews, they could get released. If the audioManager belongs to a child component that is also released, that would be, uh, bad.

I can't promise (little busy at work), but if I have a few minutes I'll take a look.

billythekid
Offline
Last seen: 3 months 1 week ago
Joined: 29 Sep 2010 - 16:33
Re: Make JUCE audio play nicely with other iPhone apps

Quote:
One potential problem could be 'who owns the audioManager?' When you aren't set for a background task, 'multitasking' is really more like suspending. When you do run in the background, some resources can still be released.

Hmmm. This is something that I didn't think about. Perhaps I should put the device manager in a some global object and see what happens.

Pages