Camera2, MediaCodec recording mp4

PS: Do what you want to do, do at least a little bit every day, don't worry about how much, gradually make changes.

After understanding the relevant knowledge of audio and video, you can read the following two articles first:

The main content of this article is to use Android's native hardware encoding and decoding framework MediaCodec and multiplexer MediaMuxer to implement the recording of mp4 video files. The video data source is provided by Camera2. The focus here is on the encoding and multiplexing process, not just the recording of mp4. If it is only video recording, you can choose the more convenient MediaRecorder. As usual, we will learn MediaCodec in the form of a case study. Its more usage will be introduced in subsequent articles. The main content of this article is as follows:

Using Camera2
MediaCodec input methods
Encoding Camera2 data with MediaCodec
Recording process
Recording effect

Using Camera2#

Camera2 is a new camera API introduced from Android 5.0. The latest one is CameraX, which is based on Camera2. Compared with Camera2, CameraX provides a more user-friendly API. The relevant APIs mentioned in the following text can be directly referred to the following diagram, which is also a diagram of using Camera2, as follows:

MediaCodec input methods#

In order to use MediaCodec for encoding operations, the camera data needs to be input into the encoder MediaCodec. There are two ways to write data into MediaCodec, as follows:

Surface: Use Surface as the input of the encoder MediaCodec, that is, use the Surface created by MediaCodec as its input. This Surface is created by the createInputSurface method of MediaCodec. When the camera renders valid data to this Surface, MediaCodec can directly output the encoded data.
InputBuffer: Use the input buffer as the input of the encoder MediaCodec. The data to be filled here is the original frame data. For Camera2, the frame data can be directly obtained through ImageReader. The obtained Image contains information such as width, height, format, timestamp, and YUV data components, which provides higher controllability.

Encoding Camera2 data with MediaCodec#

To put it simply, the data processing method of MediaCodec is as follows: Before Android 5.0, only synchronous mode of ByteBuffer[] was supported. After that, synchronous and asynchronous modes of ByteBuffer were recommended. Here, the synchronous mode of ByteBuffer is used. The main process involved is video data encoding and multiplexing. Because it was mentioned earlier that the input of MediaCodec is completed through Surface, here we only need to obtain the already encoded data and use the multiplexer MediaMuxer to generate the Mp4 file. The key code is as follows:

// Get the index of the successfully encoded output buffer
var outputBufferId: Int = mMediaCodec.dequeueOutputBuffer(bufferInfo, 0)
if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
    // Add video track
    mTrackIndex = mMediaMuxer.addTrack(mMediaCodec.outputFormat)
    mMediaMuxer.start()
    mStartMuxer = true
} else {
    while (outputBufferId >= 0) {
        if (!mStartMuxer) {
            Log.i(TAG, "MediaMuxer not start")
            continue
        }
        // Get valid data
        val outputBuffer = mMediaCodec.getOutputBuffer(outputBufferId) ?: continue
        outputBuffer.position(bufferInfo.offset)
        outputBuffer.limit(bufferInfo.offset + bufferInfo.size)
        if (pts == 0L) {
            pts = bufferInfo.presentationTimeUs
        }
        bufferInfo.presentationTimeUs = bufferInfo.presentationTimeUs - pts
        // Write data to the multiplexer to generate the file
        mMediaMuxer.writeSampleData(mTrackIndex, outputBuffer, bufferInfo)
        Log.d(
            TAG,
            "pts = ${bufferInfo.presentationTimeUs / 1000000.0f} s ,${pts / 1000} ms"
        )
        mMediaCodec.releaseOutputBuffer(outputBufferId, false)
        outputBufferId = mMediaCodec.dequeueOutputBuffer(bufferInfo, 0)
    }
}

Recording process#

Here, Surface is used as the input of the encoder MediaCodec. The MediaCodec can only create a Surface when it is in the configuration state, which means that createInputSurface can only be called between configure and start. Refer to the following:

// Configuration state
mMediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
// Create a Surface as the input of MediaCodec. createInputSurface can only be called between configure and start to create a Surface
mSurface = mMediaCodec.createInputSurface()
// start ...

Add it to the output Surface list of SessionConfiguration, as follows:

// Create CaptureSession
@RequiresApi(Build.VERSION_CODES.P)
private suspend fun createCaptureSession(): CameraCaptureSession = suspendCoroutine { cont ->
    val outputs = mutableListOf<OutputConfiguration>()
    // Preview Surface
    outputs.add(OutputConfiguration(mSurface))
    // Add Surface for MediaCodec input
    outputs.add(OutputConfiguration(EncodeManager.getSurface()))
    val sessionConfiguration = SessionConfiguration(
        SessionConfiguration.SESSION_REGULAR,
        outputs, mExecutor, ...)
    mCameraDevice.createCaptureSession(sessionConfiguration)
}

Then initiate CaptureRequest to start preview and receive Surface output, and start encoding at the same time, as follows:

// Add preview Surface and Surface for generating Image
mCaptureRequestBuild = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW)
val sur = EncodeManager.getSurface()
mCaptureRequestBuild.addTarget(sur)
mCaptureRequestBuild.addTarget(mSurface)

// Set various parameters
mCaptureRequestBuild.set(CaptureRequest.CONTROL_VIDEO_STABILIZATION_MODE, 1) // Whether the video stabilization function is activated
// Send CaptureRequest
mCameraCaptureSession.setRepeatingRequest(
    mCaptureRequestBuild.build(),
    null,
    mCameraHandler
)
// Start encoding
EncodeManager.startEncode()

jzman