IjkPlayer Series Message Loop Mechanism

PS: Recently I read a line from a poem by Goethe: Whatever you can do, or dream you can do, begin it. Boldness is genius, power, and magic.

The previous two articles introduced the basics of JNI and the creation process of the IjkPlayer:

The main content of this article is as follows:

AVMessage and MessageQueue
Message Queue Initialization
Starting the Message Loop
Message Loop Thread
Message Loop Function
Summary

AVMessage and MessageQueue#

Let's first look at the definitions of the AVMessage and MessageQueue structures:

// ff_ffmsg_queue.h
typedef struct AVMessage {
    int what;
    int arg1;
    int arg2;
    void *obj;
    void (*free_l)(void *obj);
    struct AVMessage *next;
} AVMessage;

typedef struct MessageQueue {
    AVMessage *first_msg, *last_msg;
    int nb_messages;
    int abort_request;
    SDL_mutex *mutex;
    SDL_cond *cond;

    AVMessage *recycle_msg;
    int recycle_count;
    int alloc_count;
} MessageQueue;

The definitions and implementations of AVMessage and MessageQueue are in ff_ffmsg_queue.h, and their related operation functions are mainly msg_xxx and msg_queue_xxx, as follows:

// AVMessage
void msg_xxx(AVMessage *msg)
// MessageQueue
void msg_queue_xxx(MessageQueue *q)

The key functions of MessageQueue are as follows:

// Initialize MessageQueue
void msg_queue_init(MessageQueue *q)
// Reset MessageQueue
void msg_queue_flush(MessageQueue *q)
// Set q->abort_request to 0 to ensure the message loop msg_loop can proceed
void msg_queue_start(MessageQueue *q)
// The msg_queue_put_xxx series of functions will call msg_queue_put_private
int msg_queue_put_private(MessageQueue *q, AVMessage *msg)
// Get the first message from the MessageQueue
int msg_queue_get(MessageQueue *q, AVMessage *msg, int block)

Message Queue Initialization#

The message queue is initialized during the creation process of the IjkPlayer, and its key function calls are as follows:

IjkMediaPlayer_native_setup->ijkmp_android_create->ijkmp_create

Here we start looking at the initialization of the message queue directly from the ijkmp_create function.

The message queue corresponds to the msg_queue member defined in FFPlayer, and during the creation process of the IjkMediaPlayer structure, the function ffp_create is called to initialize ffplayer, as follows:

// ijkplayer.c
IjkMediaPlayer *ijkmp_create(int (*msg_loop)(void*)){
    IjkMediaPlayer *mp = (IjkMediaPlayer *) mallocz(sizeof(IjkMediaPlayer));
    if (!mp)
        goto fail;
    // Create FFPlayer and initialize mp->ffplayer
    mp->ffplayer = ffp_create();
    // mp->msg_loop
    mp->msg_loop = msg_loop;
    // ...
}

Continuing to look at the implementation of the ffp_create function:

// ff_ffplay.c
FFPlayer *ffp_create(){
    // ...
    // Message queue initialization
    msg_queue_init(&ffp->msg_queue);
    ffp->af_mutex = SDL_CreateMutex();
    ffp->vf_mutex = SDL_CreateMutex();
    // Internally calls msg_queue_flush
    ffp_reset_internal(ffp);
    // ...
    return ffp;
}

In the ffp_create function, msg_queue_init is called to initialize the message queue msg_queue, which will set msg_loop's abort_request to 1. Later, when starting the message loop thread, it will set abort_request to 0.

The ffp_reset_internal internally calls msg_queue_flush to reset msg_loop, thus completing the initialization of the message queue msg_loop.

Continuing down, mp->msg_loop = msg_loop completes the assignment of the message loop function, and the message loop function passed in from ijkmp_create is message_loop, which will be introduced in the next section. At this point, the assignment of message_loop is complete.

The function pointer msg_loop is indeed a function, but when is msg_loop called?

Starting the Message Loop#

The start of the message loop is the msg_queue_start function, and its calling process is triggered when the application layer calls prepareAsync to start preparing playback. The function call process is as follows:

Mermaid Loading...

Let's look at the implementation of the ijkmp_prepare_async_l function:

// ijkplayer.c
static int ijkmp_prepare_async_l(IjkMediaPlayer *mp){
    // ...
    ijkmp_change_state_l(mp, MP_STATE_ASYNC_PREPARING);

    // Start the message loop
    msg_queue_start(&mp->ffplayer->msg_queue);

    // released in msg_loop
    ijkmp_inc_ref(mp);
    mp->msg_thread = SDL_CreateThreadEx(&mp->_msg_thread, ijkmp_msg_loop, mp, "ff_msg_loop");
    // ...
    return 0;
}

Clearly, msg_queue_start is called to start the message loop. Let's look at the implementation of msg_queue_start:

// ff_ffmsg_queue.h
inline static void msg_queue_start(MessageQueue *q){
    SDL_LockMutex(q->mutex);
    // Key point
    q->abort_request = 0;
    AVMessage msg;
    msg_init_msg(&msg);
    msg.what = FFP_MSG_FLUSH;
    msg_queue_put_private(q, &msg);
    SDL_UnlockMutex(q->mutex);
}

Here, setting abort_request to 0 indicates that messages AVMessage can be queued in and out. If this function is not called, the message loop cannot be completed. Therefore, after initializing the message loop msg_queue, msg_queue_start must be called to start the message loop.

Next, let's look at the implementation of the message retrieval function msg_queue_get. The retrieval of messages is done through this function, which continuously retrieves notifications to the application layer. The parameter block is 1 for blocking and 0 for non-blocking. According to the call, 1 is passed in here, meaning that when there are no messages in the message queue msg_queue, it will wait for messages to be added before continuing processing, as follows:

 // ff_ffmsg_queue.h
inline static int msg_queue_get(MessageQueue *q, AVMessage *msg, int block){
    AVMessage *msg1;
    int ret;
    SDL_LockMutex(q->mutex);
    for (;;) {
        // abort
        if (q->abort_request) {
            ret = -1;
            break;
        }
        // Get the first message
        msg1 = q->first_msg;
        if (msg1) {// Process messages in the queue
            q->first_msg = msg1->next;
            if (!q->first_msg)
                q->last_msg = NULL;
            q->nb_messages--;
            *msg = *msg1;
            msg1->obj = NULL;
#ifdef FFP_MERGE
            av_free(msg1);
#else
            msg1->next = q->recycle_msg;
            q->recycle_msg = msg1;
#endif
            ret = 1;
            break;
        } else if (!block) {// Exit directly
            ret = 0;
            break;
        } else {// Block and wait
            SDL_CondWait(q->cond, q->mutex);
        }
    }
    SDL_UnlockMutex(q->mutex);
    return ret;
}

The above code will only start looping to retrieve messages when q->abort_request is 0, which is why msg_queue_start is used to start the message loop.

Here, starting the message loop only ensures that messages can be queued in and out normally, but the message loop function msg_loop has not actually been run yet.

Message Loop Thread#

Remember the question left above, when is msg_loop called? The answer is that msg_loop is called in the message loop thread. Continuing from the previous text, in the ijkmp_prepare_async_l function, msg_queue_start is called first, and then the message loop thread is created, as follows:

// ijkplayer.c
static int ijkmp_prepare_async_l(IjkMediaPlayer *mp){
    // ...
    ijkmp_change_state_l(mp, MP_STATE_ASYNC_PREPARING);
    // Start the message loop
    msg_queue_start(&mp->ffplayer->msg_queue);
    // released in msg_loop
    ijkmp_inc_ref(mp);
    // Create the message loop thread
    mp->msg_thread = SDL_CreateThreadEx(&mp->_msg_thread, ijkmp_msg_loop, mp, "ff_msg_loop");
    // ...
    return 0;
}

In the above code, SDL_CreateThreadEx creates a thread named ff_msg_loop, with the thread body running the function ijkmp_msg_loop. At the same time, the member msg_thread of the IjkMediaPlayer structure is assigned. When the thread is created, it runs the thread body function ijkmp_msg_loop, as follows:

// ijkplayer.c
static int ijkmp_msg_loop(void *arg){
    IjkMediaPlayer *mp = arg;
    // Call the message loop function
    int ret = mp->msg_loop(arg);
    return ret;
}

Here, the call to the message loop function msg_loop is completed. At this point, the message loop of IjkPlayer is truly started. Next, let's continue to see how messages are sent to the application layer.

Message Loop Function#

The message loop function is passed in during the creation of the player, in IjkMediaPlayer_native_setup, as follows:

// ijkplayer_jni.c
static void IjkMediaPlayer_native_setup(JNIEnv *env, jobject thiz, jobject weak_this){
    MPTRACE("%s\n", __func__);
    // Create the corresponding IjkMediaPlayer in the C layer
    IjkMediaPlayer *mp = ijkmp_android_create(message_loop);
    // ...
}

message_loop will ultimately be assigned to the member msg_loop of the IjkMediaPlayer structure. From the previous text, we know that the message loop thread msg_thread calls the message loop function msg_loop, which is the message_loop here, and its implementation is as follows:

// ijkplayer_jni.c
static int message_loop(void *arg){
    // ...
    IjkMediaPlayer *mp = (IjkMediaPlayer*) arg;
    // Key function message_loop_n
    message_loop_n(env, mp);
    // ...
}

Continuing to look at the implementation of the key function message_loop_n:

// ijkplayer_jni.c
static void message_loop_n(JNIEnv *env, IjkMediaPlayer *mp){
    jobject weak_thiz = (jobject) ijkmp_get_weak_thiz(mp);
    JNI_CHECK_GOTO(weak_thiz, env, NULL, "mpjni: message_loop_n: null weak_thiz", LABEL_RETURN);
    while (1) {
        AVMessage msg;
        // Get a message AVMessage from the MessageQueue
        int retval = ijkmp_get_msg(mp, &msg, 1);
        if (retval < 0)
            break;

        // block-get should never return 0
        assert(retval > 0);
        // Handle various playback events
        switch (msg.what) {
        case FFP_MSG_PREPARED:
            MPTRACE("FFP_MSG_PREPARED:\n");
            // Key function
            post_event(env, weak_thiz, MEDIA_PREPARED, 0, 0);
            break;
        // ...
        default:
            ALOGE("unknown FFP_MSG_xxx(%d)\n", msg.what);
            break;
        }
        // Memory resource recovery
        msg_free_res(&msg);
    }
LABEL_RETURN:
    ;
}

It can be seen that in the message loop function, messages are continuously retrieved through ijkmp_get_msg in a loop, and then the messages are sent to the application layer via post_event:

// ijkplayer_jni.c
inline static void post_event(JNIEnv *env, jobject weak_this, int what, int arg1, int arg2){
    // postEventFromNative
    J4AC_IjkMediaPlayer__postEventFromNative(env, weak_this, what, arg1, arg2, NULL);
}

The post_event function calls the Java layer's postEventFromNative method to complete the message return, as follows:

@CalledByNative
private static void postEventFromNative(Object weakThiz, int what,
        int arg1, int arg2, Object obj) {
    if (weakThiz == null)
        return;

    @SuppressWarnings("rawtypes")
    IjkMediaPlayer mp = (IjkMediaPlayer) ((WeakReference) weakThiz).get();
    if (mp == null) {
        return;
    }

    if (what == MEDIA_INFO && arg1 == MEDIA_INFO_STARTED_AS_NEXT) {
        // this acquires the wakelock if needed, and sets the client side
        // state
        mp.start();
    }
    if (mp.mEventHandler != null) {
        Message m = mp.mEventHandler.obtainMessage(what, arg1, arg2, obj);
        // Send message
        mp.mEventHandler.sendMessage(m);
    }
}

postEventFromNative receives the messages sent from the underlying IjkPlayer and converts them into Message to be processed by EventHandler, as follows:

private static class EventHandler extends Handler {
    private final WeakReference<IjkMediaPlayer> mWeakPlayer;
    public EventHandler(IjkMediaPlayer mp, Looper looper) {
        super(looper);
        mWeakPlayer = new WeakReference<IjkMediaPlayer>(mp);
    }
    @Override
    public void handleMessage(Message msg) {
        IjkMediaPlayer player = mWeakPlayer.get();
        if (player == null || player.mNativeMediaPlayer == 0) {
            DebugLog.w(TAG,
                    "IjkMediaPlayer went away with unhandled events");
            return;
        }
        switch (msg.what) {
        case MEDIA_PREPARED:
            player.notifyOnPrepared();
            return;
        // ...
        default:
            DebugLog.e(TAG, "Unknown message type " + msg.what);
        }
    }
}

Different message types are processed accordingly. As shown above, the MEDIA_PREPARED event ultimately calls back to the corresponding callback interface, such as mOnPreparedListener, as follows:

protected final void notifyOnPrepared() {
    if (mOnPreparedListener != null)
        // Playback preparation complete event
        mOnPreparedListener.onPrepared(this);
}

At this point, the execution of the message loop function is complete.

Summary#

This article starts from the Native layer and goes all the way to the Java layer receiving messages. The most important part of the IjkPlayer's message loop is the message queue msg_queue, where all relevant event messages generated from the start to the end of playback are added to this queue. The message loop thread is responsible for retrieving messages and notifying them. If there are no messages to retrieve, it will block and wait for messages to be added before continuing the message loop process.