Javafxports How to Call Android Native Media Player

Audio performance with Javafx for Android (MediaPlayer and NativeAudioService)

I've changed a little bit the implementation you mentioned, given that you have a bunch of short audio files to play, and that you want a very short time to play them on demand. Basically I'll create the AssetFileDescriptor for all the files once, and also I'll use the same single MediaPlayer instance all the time.

The design follows the pattern of the Charm Down library, so you need to keep the package names below.

EDIT

After the OP's feedback, I've changed the implementation to have one MediaPlayer for each audio file, so you can play any of them at any time.

1. Source Packages/Java:

package: com.gluonhq.charm.down.plugins

AudioService interface

public interface AudioService {
    void addAudioName(String audioName);
    void play(String audioName, double volume);
    void stop(String audioName);
    void pause(String audioName);
    void resume(String audioName);
    void release();
}

AudioServiceFactory class

public class AudioServiceFactory extends DefaultServiceFactory<AudioService> {

    public AudioServiceFactory() {
        super(AudioService.class);
    }

}

2. Android/Java Packages

package: com.gluonhq.charm.down.plugins.android

AndroidAudioService class

public class AndroidAudioService implements AudioService {

    private final Map<String, MediaPlayer> playList;
    private final Map<String, Integer> positionList;

    public AndroidAudioService() {
        playList = new HashMap<>();
        positionList = new HashMap<>();
    }

    @Override
    public void addAudioName(String audioName) {
        MediaPlayer mediaPlayer = new MediaPlayer();
        mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
        mediaPlayer.setOnCompletionListener(m -> pause(audioName)); // don't call stop, allows reuse
        try {
            mediaPlayer.setDataSource(FXActivity.getInstance().getAssets().openFd(audioName));
            mediaPlayer.setOnPreparedListener(mp -> {
                System.out.println("Adding  audio resource " + audioName);
                playList.put(audioName, mp);
                positionList.put(audioName, 0);
            });
            mediaPlayer.prepareAsync();
        } catch (IOException ex) {
            System.out.println("Error retrieving audio resource " + audioName + " " + ex);
        }

    }

    @Override
    public void play(String audioName, double volume) {
        MediaPlayer mp = playList.get(audioName);
        if (mp != null) {
            if (positionList.get(audioName) > 0) {
                positionList.put(audioName, 0);
                mp.pause();
                mp.seekTo(0);
            }
            mp.start();
        }

    }

    @Override
    public void stop(String audioName) {
        MediaPlayer mp = playList.get(audioName);
        if (mp != null) {
            mp.stop();
        }
    }

    @Override
    public void pause(String audioName) {
        MediaPlayer mp = playList.get(audioName);
        if (mp != null) {
            mp.pause();
            positionList.put(audioName, mp.getCurrentPosition());
        }
    }

    @Override
    public void resume(String audioName) {
        MediaPlayer mp = playList.get(audioName);
        if (mp != null) {
            mp.start();
            mp.seekTo(positionList.get(audioName));
        }
    }

    @Override
    public void release() {
        for (MediaPlayer mp : playList.values()) {
            if (mp != null) {
                mp.stop();
                mp.release();
            }
        }

    }

}

3. Sample

I've added five short audio files (from here), and added five buttons to my main view:

@Override
public void start(Stage primaryStage) throws Exception {

    Button play1 = new Button("p1");
    Button play2 = new Button("p2");
    Button play3 = new Button("p3");
    Button play4 = new Button("p4");
    Button play5 = new Button("p5");
    HBox hBox = new HBox(10, play1, play2, play3, play4, play5);
    hBox.setAlignment(Pos.CENTER);

    Services.get(AudioService.class).ifPresent(audio -> {

        audio.addAudioName("beep28.mp3");
        audio.addAudioName("beep36.mp3");
        audio.addAudioName("beep37.mp3");
        audio.addAudioName("beep39.mp3");
        audio.addAudioName("beep50.mp3");

        play1.setOnAction(e -> audio.play("beep28.mp3", 5));
        play2.setOnAction(e -> audio.play("beep36.mp3", 5));
        play3.setOnAction(e -> audio.play("beep37.mp3", 5));
        play4.setOnAction(e -> audio.play("beep39.mp3", 5));
        play5.setOnAction(e -> audio.play("beep50.mp3", 5));
    });

    Scene scene = new Scene(new StackPane(hBox), Screen.getPrimary().getVisualBounds().getWidth(), 
                        Screen.getPrimary().getVisualBounds().getHeight());
    primaryStage.setScene(scene);
    primaryStage.show();
}

@Override
public void stop() throws Exception {
    Services.get(AudioService.class).ifPresent(AudioService::release);
}

The prepare step takes place when the app is launched and the service is instanced, so when playing later on any of the audio files, there won't be any delay.

I haven't checked if there could be any memory issues when adding several media players with big audio files, as that wasn't the initial scenario. Maybe a cache strategy will help in this case (see CacheService in Gluon Charm Down).

Gluon Mobile iOS Audio Player

While on Android we can easily add native API to the Android folders of a Gluon project (check this solution for using native Media on Android), the use of native code (Objetive-C) with the Media API on iOS folders is not enough, since it has to be compiled, and the compiled files have to be included into the ipa.

Currently, Charm Down is doing this for a bunch of services. If you have a look at the build.gradle script for iOS, it includes the xcodebuild task to compile and build the native library libCharm.a, that later should be included in any iOS project using any of those services.

My suggestion will be cloning Charm Down, and providing a new service: AudioService, with some basic methods like:

public interface AudioService {
    void play(String audioName);
    void pause();
    void resume();
    void stop(); 
}

This service will be added to the Platform class:

public abstract class Platform {
    ...
    public abstract AudioService getAudioService();
}

and you should provide implementations for Desktop (empty), Android (like the one here) and iOS.

IOS implementation - Java

You will have to add this to the IOSPlatform class:

public class IOSPlatform extends Platform {
    ...
    @Override
    public AudioService getAudioService() {
        if (audioService == null) {
            audioService = new IOSAudioService();
        }
        return audioService;
    }
}

and create the IOSAudioService class:

public class IOSAudioService implements AudioService {

    @Override
    public void play(String audioName) {
        CharmApplication.play(audioName);
    }

    @Override
    public void pause() {
        CharmApplication.pause();
    }

    @Override
    public void resume() {
        CharmApplication.resume();
    }

    @Override
    public void stop() {
        CharmApplication.stop();
    }
}

Finally, you will have to add the native calls in CharmApplication:

public class CharmApplication {
    static {
        System.loadLibrary("Charm");
        _initIDs();
    }
    ...
    public static native void play(String audioName);
    public static native void pause();
    public static native void resume();
    public static native void stop();
}

IOS implementation - Objective-C

Now, on the native folder, on CharmApplication.m, add the implementation of those calls:

#include "CharmApplication.h"
...
#include "Audio.h"

// Audio
Audio *_audio;

JNIEXPORT void JNICALL Java_com_gluonhq_charm_down_ios_CharmApplication_play
(JNIEnv *env, jclass jClass, jstring jTitle)
{
    NSLog(@"Play audio");
    const jchar *charsTitle = (*env)->GetStringChars(env, jTitle, NULL);
    NSString *name = [NSString stringWithCharacters:(UniChar *)charsTitle length:(*env)->GetStringLength(env, jTitle)];
    (*env)->ReleaseStringChars(env, jTitle, charsTitle);

    _audio = [[Audio alloc] init];
    [_audio playAudio:name];
    return;
}

JNIEXPORT void JNICALL Java_com_gluonhq_charm_down_ios_CharmApplication_pause
(JNIEnv *env, jclass jClass)
{
    if (_audio) 
    {
        [_audio pauseAudio];
    }
    return;   
}

JNIEXPORT void JNICALL Java_com_gluonhq_charm_down_ios_CharmApplication_resume
(JNIEnv *env, jclass jClass)
{
    if (_audio) 
    {
        [_audio resumeAudio];
    }
    return;   
}

JNIEXPORT void JNICALL Java_com_gluonhq_charm_down_ios_CharmApplication_stop
(JNIEnv *env, jclass jClass)
{
    if (_audio) 
    {
        [_audio stopAudio];
    }
    return;   
}

and create the header file Audio.h:

#import <AVFoundation/AVFoundation.h>
#import <UIKit/UIKit.h>

@interface Audio :UIViewController <AVAudioPlayerDelegate>
{
}
    - (void) playAudio: (NSString *) audioName;
    - (void) pauseAudio;
    - (void) resumeAudio;
    - (void) stopAudio;
@end

and the implementation Audio.m. This one is based on this tutorial:

#include "Audio.h"
#include "CharmApplication.h"

@implementation Audio 

AVAudioPlayer* player;

- (void)playAudio:(NSString *) audioName 
{
    NSString* fileName = [audioName stringByDeletingPathExtension];
    NSString* extension = [audioName pathExtension];

    NSURL* url = [[NSBundle mainBundle] URLForResource:[NSString stringWithFormat:@"%@",fileName] withExtension:[NSString stringWithFormat:@"%@",extension]];
    NSError* error = nil;

    if(player)
    {
        [player stop];
        player = nil;
    }

    player = [[AVAudioPlayer alloc] initWithContentsOfURL:url error:&error];
    if(!player)
    {
        NSLog(@"Error creating player: %@", error);
        return;
    }
    player.delegate = self;
    [player prepareToPlay];
    [player play];

}

- (void)pauseAudio
{
    if(!player)
    {
        return;
    }
    [player pause];
}

- (void)resumeAudio
{
    if(!player)
    {
        return;
    }
    [player play];
}

- (void)stopAudio
{
    if(!player)
    {
        return;
    }
    [player stop];
    player = nil;
}

- (void)audioPlayerDidFinishPlaying:(AVAudioPlayer *)player successfully:(BOOL)flag
{
    NSLog(@"%s successfully=%@", __PRETTY_FUNCTION__, flag ? @"YES"  : @"NO");
}

- (void)audioPlayerDecodeErrorDidOccur:(AVAudioPlayer *)player error:(NSError *)error
{
    NSLog(@"%s error=%@", __PRETTY_FUNCTION__, error);
}

@end

Build the native library

Edit the build.gradle for the iOS module, and add the Audio service to the xcodebuild task:

def nativeSources = ["$project.projectDir/src/main/native/CharmApplication.m",
                     ...,
                     "$project.projectDir/src/main/native/Audio.m"]

...
def compileOutputs = [
                "$project.buildDir/native/$arch/CharmApplication.o",
                "$project.buildDir/native/$arch/Charm.o",
                 ...,
                "$project.buildDir/native/$arch/Audio.o"]

Build the project

Save the project, and from the root of the Charm Down project, on command line run:

./gradlew clean build

If everything is fine, you should have:

• Common/build/libs/charm-down-common-2.1.0-SNAPSHOT.jar
• Desktop/build/libs/charm-down-desktop-2.1.0-SNAPSHOT.jar
• Android/build/libs/charm-down-android-2.1.0-SNAPSHOT.jar
• IOS/build/libs/charm-down-ios-2.1.0-SNAPSHOT.jar
• IOS/build/native/libCharm.a

Gluon Project

With those dependencies and the native library, you will be able to create a new Gluon Project, and add the jars as local dependencies (to the libs folder).

As for the native library, it should be added to this path: src/ios/jniLibs/libCharm.a.

Update the build.gradle script:

repositories {
    flatDir {
       dirs 'libs'
   }
    jcenter()
    ...
}

dependencies {
    compile 'com.gluonhq:charm-glisten:3.0.0'
    compile 'com.gluonhq:charm-down-common:2.1.0-SNAPSHOT'
    compile 'com.gluonhq:charm-glisten-connect-view:3.0.0'

    iosRuntime 'com.gluonhq:charm-glisten-ios:3.0.0'
    iosRuntime 'com.gluonhq:charm-down-ios:2.1.0-SNAPSHOT'
}

On your View, retrieve the service and provide some basic UI to call the play("audio.mp3"), pause(), resume() and stop() methods:

private boolean pause;

public BasicView(String name) {
    super(name);

    AudioService audioService = PlatformFactory.getPlatform().getAudioService();
    final HBox hBox = new HBox(10, 
            MaterialDesignIcon.PLAY_ARROW.button(e -> audioService.play("audio.mp3")),
            MaterialDesignIcon.PAUSE.button(e -> {
                if (!pause) {
                    audioService.pause();
                    pause = true;
                } else {
                    audioService.resume();
                    pause = false;
                }

            }),
            MaterialDesignIcon.STOP.button(e -> audioService.stop()));
    hBox.setAlignment(Pos.CENTER);
    setCenter(new StackPane(hBox));
}

Finally, place an audio file like audio.mp3, under src/ios/assets/audio.mp3, build and deploy to iOS.

Hopefully, this API will be provided by Charm Down any time soon. Also if you come up with a nice implementation, feel free to share it and create a Pull Request.

Can't set volume, volume control is not forwarded to the system

While I hate to constantly answer my own questions, I found a solution after a couple of hours playing with the Android API, digging through some documentations and so on.

My solution is partially based the answer, given by @josé-pereda on the topic "javafxports how to call android native Media Player".

I created an interface for the tasks "volumeUp", "volumeDown" and "mute":

public interface NativeVolumeService {
    void volumeUp();
    void volumeDown();
    void mute();
}

Then, based on the following answer on how to set the system volume on Android, I came up with the following implementation on Android:

import java.util.ArrayList;
import java.util.logging.Logger;

import android.content.Context;
import android.media.AudioManager;
import android.view.KeyEvent;
import android.view.View;
import android.view.ViewGroup;
import my.package.platform.NativeVolumeService;
import javafxports.android.FXActivity;

public class NativeVolumeServiceAndroid implements NativeVolumeService {

    private static final Logger LOG = Logger.getLogger(NativeVolumeServiceAndroid.class.getName());

    private final AudioManager audioManager;

    private final int maxVolume;
    private int preMuteVolume = 0;
    private int currentVolume = 0;

    public NativeVolumeServiceAndroid() {
        audioManager = (AudioManager) FXActivity.getInstance().getSystemService(Context.AUDIO_SERVICE);
        maxVolume = audioManager.getStreamMaxVolume(AudioManager.STREAM_MUSIC);
    }

    @Override
    public void volumeUp() {
        LOG.info("dispatch volume up event");
        KeyEvent event = new KeyEvent(KeyEvent.ACTION_DOWN, KeyEvent.KEYCODE_VOLUME_UP);
        dispatchEvent(event, true, false);
    }

    @Override
    public void volumeDown() {
        LOG.info("dispatch volume down event");
        KeyEvent event = new KeyEvent(KeyEvent.ACTION_DOWN, KeyEvent.KEYCODE_VOLUME_DOWN);
        dispatchEvent(event, false, false);
    }

    @Override
    public void mute() {
        LOG.info("dispatch volume mute event");
        KeyEvent event = new KeyEvent(KeyEvent.ACTION_DOWN, KeyEvent.KEYCODE_VOLUME_MUTE);
        dispatchEvent(event, false, true);
    }

    private void dispatchEvent(KeyEvent event, boolean up, boolean mute) {

        // hardware key events (amongst others) get caught by the JavaFXPorts engine (or better: the Dalvik impl from Oracle)
        // to circumvent this, we need to do the volume adjustment the hard way: via the AudioManager

        // see: https://developer.android.com/reference/android/media/AudioManager.html
        // see: https://stackoverflow.com/questions/9164347/setting-the-android-system-volume?rq=1

        // reason: 
        // FXActivity registers a FXDalvikEntity, which etends the surface view and passing a key processor 
        // called KeyEventProcessor - this one catches all key events and matches them to JavaFX representations.
        // Unfortunately, we cannot bypass this, so we need the AudioManager

        currentVolume = audioManager.getStreamVolume(AudioManager.STREAM_MUSIC);
        if (mute) {
            if (currentVolume > 0) {
                preMuteVolume = currentVolume;
                audioManager.adjustStreamVolume(AudioManager.STREAM_MUSIC, AudioManager.ADJUST_SAME,
                        AudioManager.FLAG_REMOVE_SOUND_AND_VIBRATE | AudioManager.FLAG_SHOW_UI);
            } else {
                preMuteVolume = 0;
                audioManager.adjustStreamVolume(AudioManager.STREAM_MUSIC, preMuteVolume, AudioManager.FLAG_SHOW_UI);
            }
        } else if (up) {
            audioManager.adjustStreamVolume(AudioManager.STREAM_MUSIC,
                    (currentVolume + 1) <= maxVolume ? AudioManager.ADJUST_RAISE : AudioManager.ADJUST_SAME, AudioManager.FLAG_SHOW_UI);
        } else if (!up) {
            audioManager.adjustStreamVolume(AudioManager.STREAM_MUSIC,
                    (currentVolume - 1) >= 0 ? AudioManager.ADJUST_LOWER : AudioManager.ADJUST_SAME, AudioManager.FLAG_SHOW_UI);
        }
    }
}

to integrate it, I created the appropriate instance in my main class (I need this globally - you will see, why)

private void instantiateNativeVolumeService() {
    String serviceName = NativeVolumeService.class.getName();
    if (Platform.isDesktop()) {
        serviceName += "Desktop";
    } else if (Platform.isAndroid()) {
        serviceName += "Android";
    }
    try {
        volumeService = (NativeVolumeService) Class.forName(serviceName).newInstance();
    } catch (ClassNotFoundException | InstantiationException | IllegalAccessException e) {
        LOG.log(Level.SEVERE, "Could not get an instance of NativeAudioService for platform " + Platform.getCurrent(), e);
    }
}

volumeService is a class variable.

Then I registered an event handler on my Stages Scene:

@Override
public void start(Stage stage) throws Exception {
    // initiate everything
    scene.addEventFilter(KeyEvent.ANY, this::handleGlobalKeyEvents);
    // do more stuff, if needed
}

And finally, the method handleGlobalKeyEvents looks like this:

private void handleGlobalKeyEvents(KeyEvent event) {
    // use a more specific key event type like
    // --> KeyEvent.KEY_RELEASED == event.getEventType()
    // --> KeyEvent.KEY_PRESSED == event.getEventType()
    // without it, we would react on both events, thus doing one operation too much
    if (event.getCode().equals(KeyCode.VOLUME_UP) && KeyEvent.KEY_PRESSED == event.getEventType()) {
        if (volumeService != null) {
            volumeService.volumeUp();
            event.consume();
        }
    }
    if (event.getCode().equals(KeyCode.VOLUME_DOWN) && KeyEvent.KEY_PRESSED == event.getEventType()) {
        if (volumeService != null) {
            volumeService.volumeDown();
            event.consume();
        }
    }
}

In the end, the solution is as clean as it gets and not too complicated. Only the way until it worked was a bit nasty.

@JoséPereda: If you want to integrate this solution as a charm down plugin or so, please feel free, but it would be nice to be mentioned and notified, if you do.

Regards,
Daniel

Javafxports on Android won't load image due to Improper call to JPEG library in state 200

The cause of the problem appears to be the file size of the image. The image showed up after reducing the file size.

---

Related Topics