IjkPlayer Series Player Creation Process

Today, I will introduce the player creation process of IjkPlayer. This article will officially begin the journey of reading the IjkPlayer source code, with the main content as follows:

Initialize so
Java layer player creation
IjkMediaPlayer structure
Native layer player creation
Call flowchart

Before reading, you can check the previous articles:

Initialize so#

View the initialization of IjkPlayer from the VideoActivity in the IjkPlayer source code, with the key code as follows:

// Load so library
IjkMediaPlayer.loadLibrariesOnce(null);
// android-ndk-profiler performance analysis
IjkMediaPlayer.native_profileBegin("libijkplayer.so");

View the source code of loadLibrariesOnce as follows:

private static volatile boolean mIsLibLoaded = false;
public static void loadLibrariesOnce(IjkLibLoader libLoader) {
    synchronized (IjkMediaPlayer.class) {
        // Ensure it is loaded only once
        if (!mIsLibLoaded) {
            if (libLoader == null)
                // sLocalLibLoader is the default IjkLibLoader
                libLoader = sLocalLibLoader;
            libLoader.loadLibrary("ijkffmpeg");
            libLoader.loadLibrary("ijksdl");
            libLoader.loadLibrary("ijkplayer");
            mIsLibLoaded = true;
        }
    }
}

loadLibrariesOnce loads the corresponding so libraries, while native_profileBegin is the function for performance analysis using the android-ndk-profiler tool, corresponding to native_profileEnd, which is defined as follows:

public static native void native_profileBegin(String libName);
public static native void native_profileEnd();

The above methods call the functions monstartup and moncleanup, which are called at the start and end of the program, respectively. There will be no further introduction to android-ndk-profiler here.

In the article IjkPlayer series on JNI basics and source directory introduction, the registration method of JNI is introduced, where IjkPlayer uses the dynamic registration method, which means that when calling System.loadLibrary to load the library, it will look for the JNI_OnLoad function, and then register in the callback of that function. Below is the implementation of JNI_OnLoad in ijkplayer:

JNIEXPORT jint JNI_OnLoad(JavaVM *vm, void *reserved)
{
    JNIEnv* env = NULL;
    // Save global JavaVM
    g_jvm = vm;
    if ((*vm)->GetEnv(vm, (void**) &env, JNI_VERSION_1_4) != JNI_OK) {
        return -1;
    }
    assert(env != NULL);
    // Initialize mutex
    pthread_mutex_init(&g_clazz.mutex, NULL );
    // Convert IjkMediaPlayer to a global reference
    IJK_FIND_JAVA_CLASS(env, g_clazz.clazz, JNI_CLASS_IJKPLAYER);
    // Register Native methods corresponding to Java methods
    (*env)->RegisterNatives(env, g_clazz.clazz, g_methods, NELEM(g_methods) );
    // Initialize: register codecs, demuxers, protocols
    ijkmp_global_init();
    // Set callback, corresponding to the Java layer's onNativeInvoke callback function
    ijkmp_global_set_inject_callback(inject_callback);
    // Register the relationship between av_base64_encode and FFmpegApi_av_base64_encode
    FFmpegApi_global_init(env);
    return JNI_VERSION_1_4;
}

The array g_methods defines the one-to-one correspondence between Native functions and Java methods as follows:

static JNINativeMethod g_methods[] = {
    {
        "_setDataSource",
        "(Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;)V",
        (void *) IjkMediaPlayer_setDataSourceAndHeaders
    },
    { "_setDataSourceFd",       "(I)V",     (void *) IjkMediaPlayer_setDataSourceFd },
    { "_setDataSource",         "(Ltv/danmaku/ijk/media/player/misc/IMediaDataSource;)V", (void *)IjkMediaPlayer_setDataSourceCallback },
    { "_setAndroidIOCallback",  "(Ltv/danmaku/ijk/media/player/misc/IAndroidIO;)V", (void *)IjkMediaPlayer_setAndroidIOCallback },

    { "_setVideoSurface",       "(Landroid/view/Surface;)V", (void *) IjkMediaPlayer_setVideoSurface },
    { "_prepareAsync",          "()V",      (void *) IjkMediaPlayer_prepareAsync },
    { "_start",                 "()V",      (void *) IjkMediaPlayer_start },
    { "_stop",                  "()V",      (void *) IjkMediaPlayer_stop },
    { "seekTo",                 "(J)V",     (void *) IjkMediaPlayer_seekTo },
    { "_pause",                 "()V",      (void *) IjkMediaPlayer_pause },
    { "isPlaying",              "()Z",      (void *) IjkMediaPlayer_isPlaying },
    { "getCurrentPosition",     "()J",      (void *) IjkMediaPlayer_getCurrentPosition },
    { "getDuration",            "()J",      (void *) IjkMediaPlayer_getDuration },
    { "_release",               "()V",      (void *) IjkMediaPlayer_release },
    { "_reset",                 "()V",      (void *) IjkMediaPlayer_reset },
    { "setVolume",              "(FF)V",    (void *) IjkMediaPlayer_setVolume },
    { "getAudioSessionId",      "()I",      (void *) IjkMediaPlayer_getAudioSessionId },
    { "native_init",            "()V",      (void *) IjkMediaPlayer_native_init },
    { "native_setup",           "(Ljava/lang/Object;)V", (void *) IjkMediaPlayer_native_setup },
    { "native_finalize",        "()V",      (void *) IjkMediaPlayer_native_finalize },

    { "_setOption",             "(ILjava/lang/String;Ljava/lang/String;)V", (void *) IjkMediaPlayer_setOption },
    { "_setOption",             "(ILjava/lang/String;J)V",                  (void *) IjkMediaPlayer_setOptionLong },

    { "_getColorFormatName",    "(I)Ljava/lang/String;",    (void *) IjkMediaPlayer_getColorFormatName },
    { "_getVideoCodecInfo",     "()Ljava/lang/String;",     (void *) IjkMediaPlayer_getVideoCodecInfo },
    { "_getAudioCodecInfo",     "()Ljava/lang/String;",     (void *) IjkMediaPlayer_getAudioCodecInfo },
    { "_getMediaMeta",          "()Landroid/os/Bundle;",    (void *) IjkMediaPlayer_getMediaMeta },
    { "_setLoopCount",          "(I)V",                     (void *) IjkMediaPlayer_setLoopCount },
    { "_getLoopCount",          "()I",                      (void *) IjkMediaPlayer_getLoopCount },
    { "_getPropertyFloat",      "(IF)F",                    (void *) ijkMediaPlayer_getPropertyFloat },
    { "_setPropertyFloat",      "(IF)V",                    (void *) ijkMediaPlayer_setPropertyFloat },
    { "_getPropertyLong",       "(IJ)J",                    (void *) ijkMediaPlayer_getPropertyLong },
    { "_setPropertyLong",       "(IJ)V",                    (void *) ijkMediaPlayer_setPropertyLong },
    { "_setStreamSelected",     "(IZ)V",                    (void *) ijkMediaPlayer_setStreamSelected },

    { "native_profileBegin",    "(Ljava/lang/String;)V",    (void *) IjkMediaPlayer_native_profileBegin },
    { "native_profileEnd",      "()V",                      (void *) IjkMediaPlayer_native_profileEnd },

    { "native_setLogLevel",     "(I)V",                     (void *) IjkMediaPlayer_native_setLogLevel },
    { "_setFrameAtTime",        "(Ljava/lang/String;JJII)V", (void *) IjkMediaPlayer_setFrameAtTime },
};

The above code mainly works as follows:

Register the correspondence between Native methods and Java methods.
Register codecs, demuxers, protocols, etc.
Set the onNativeInvoke callback for Native layer calls, mainly concerned with its return value, such as network reconnection, etc.

Java layer player creation#

Let's look at the creation of IjkMediaPlayer in the Java layer.

IjkMediaPlayer ijkMediaPlayer = new IjkMediaPlayer();

View the constructor as follows:

public IjkMediaPlayer() {
    // sLocalLibLoader is the default IjkLibLoader
    this(sLocalLibLoader);
}

public IjkMediaPlayer(IjkLibLoader libLoader) {
    initPlayer(libLoader);
}

private void initPlayer(IjkLibLoader libLoader) {
    // Try to load the library to avoid never loading so before
    loadLibrariesOnce(libLoader);
    // Corresponding to c layer IjkMediaPlayer_native_init method, temporarily empty implementation
    initNativeOnce();
    
    // Used to handle messages passed from the Ijk lower layer
    Looper looper;
    if ((looper = Looper.myLooper()) != null) {
        mEventHandler = new EventHandler(this, looper);
    } else if ((looper = Looper.getMainLooper()) != null) {
        mEventHandler = new EventHandler(this, looper);
    } else {
        mEventHandler = null;
    }

    // Wrap IjkMediaPlayer as a weak reference to pass to the native layer
    native_setup(new WeakReference<IjkMediaPlayer>(this));
}

It can be seen that during initialization, the ijkffmpeg, ijksdl, and ijkplayer libraries are loaded, and an mEventHandler is created to handle playback events thrown from Native, such as playback start, playback completion, playback error, etc. The specific events are sent by the Native layer calling the Java layer function postEventFromNative. Before reading the C layer code, let's first understand the IjkMediaPlayer structure.

IjkMediaPlayer structure#

IjkPlayer corresponds to the IjkMediaPlayer structure, and its initialization mainly involves initializing this structure. Its definition is as follows:

struct IjkMediaPlayer {
    /* The ref_count increases by one each time IjkMediaPlayer is created */
    volatile int ref_count;
    /* Lock to protect interface calls */
    pthread_mutex_t mutex;
    /* FFPlayer is the structure from the original ffplayer, which has been extended by ijk */
    FFPlayer *ffplayer;
    /* A message loop function for IjkPlayer to callback to the application layer */
    int (*msg_loop)(void*);
    /* Message thread */
    SDL_Thread *msg_thread;
    SDL_Thread _msg_thread;
    /* Player state */
    int mp_state;
    /* Playback address */
    char *data_source;
    /* Weak reference object corresponding to Java layer IjkMediaPlayer */
    void *weak_thiz;
    /* Whether to replay */
    int restart;
    /* Whether to restart from the beginning */
    int restart_from_beginning;
    /* Indicates whether the user has sought the progress bar */
    int seek_req;
    /* Seek milliseconds value */
    long seek_msec;
};

The parameters involved in the entire playback process are basically member variables of the IjkMediaPlayer structure. The subsequent creation of the Native layer IjkPlayer is all about initializing the member variables of the above structure.

Native layer player creation#

The creation of the Native layer player mainly involves the creation and initialization of the IjkMediaPlayer structure. Continuing from the previous text, let's look at the specific implementation of the native_setup method as follows:

static void
IjkMediaPlayer_native_setup(JNIEnv *env, jobject thiz, jobject weak_this)
{
    MPTRACE("%s\n", __func__);
    // Create the corresponding IjkMediaPlayer in the C layer
    IjkMediaPlayer *mp = ijkmp_android_create(message_loop);
    JNI_CHECK_GOTO(mp, env, "java/lang/OutOfMemoryError", "mpjni: native_setup: ijkmp_create() failed", LABEL_RETURN);
    // Initialize Java layer mNativeMediaPlayer
    jni_set_media_player(env, thiz, mp);
    // Initialize mp->weak_thiz
    ijkmp_set_weak_thiz(mp, (*env)->NewGlobalRef(env, weak_this));
    // Initialize ffp->inject_opaque, etc.
    ijkmp_set_inject_opaque(mp, ijkmp_get_weak_thiz(mp));
    // Initialize ffp->ijkio_inject_opaque, etc.
    ijkmp_set_ijkio_inject_opaque(mp, ijkmp_get_weak_thiz(mp));
    // Set decoder selection callback
    ijkmp_android_set_mediacodec_select_callback(mp, mediacodec_select_callback, ijkmp_get_weak_thiz(mp));

LABEL_RETURN:
    ijkmp_dec_ref_p(&mp);
}

The above code mainly starts creating the C layer IjkMediaPlayer, initializing part of its corresponding structure attributes and decoder callbacks. The mediacodec_select_callback will call the Java layer function onSelectCodec to obtain suitable decoder information, which can be adapted in IjkMediaCodecInfo.

Next, let's look at the ijkmp_android_create function as follows:

IjkMediaPlayer *ijkmp_android_create(int(*msg_loop)(void*))
{
    // Fill in the IjkMediaPlayer structure
    IjkMediaPlayer *mp = ijkmp_create(msg_loop);
    if (!mp)
        goto fail;
    // Initialize SDL_Vout, indicating the display context in IJK
    mp->ffplayer->vout = SDL_VoutAndroid_CreateForAndroidSurface();
    if (!mp->ffplayer->vout)
        goto fail;
    // Initialize IJKFF_Pipeline, decoder, audio output
    mp->ffplayer->pipeline = ffpipeline_create_from_android(mp->ffplayer);
    if (!mp->ffplayer->pipeline)
        goto fail;
    // Bind SDL_Vout to IJKFF_Pipeline_Opaque
    ffpipeline_set_vout(mp->ffplayer->pipeline, mp->ffplayer->vout);
    return mp;
fail:
    ijkmp_dec_ref_p(&mp);
    return NULL;
}

In which ijkmp_create mainly creates the FFPlayer and assigns the msg_loop. The msg_loop event loop related calls will be introduced in subsequent articles. Let's continue to look at the following functions:

SDL_VoutAndroid_CreateForAndroidSurface: Initializes the display context SDL_Vout of IjkPlayer.
ffpipeline_create_from_android: Initializes IJKFF_Pipeline, decoder, audio output.
ffpipeline_set_vout: Binds SDL_Vout to IJKFF_Pipeline_Opaque.

Call flowchart#

The main function call flow of IjkPlayer creation is as follows:

Mermaid Loading...

Other details will be introduced in subsequent articles. The next article will introduce the message loop mechanism in IjkPlayer.