Android: How to Integrate a Decoder to Multimedia Framework

Android: How to integrate a decoder to multimedia framework

In Android SF framework, the codecs are registered through media_codecs.xml. In standard android distribution, an example media_codecs.xml can be found here. All audio-visual components are registered as OMX components.

1. Codec Registration

To register your video decoder, you would have to add a new entry under <Decoders> list. To ensure that your codec is always picked up, please ensure that your codec is listed as the first entry for the specific MIME type. An example entry for a H.264 decoder could be as below.

<Decoders>
    <MediaCodec name="OMX.ABC.XYZ.H264.DECODER" type="video/avc" >
        <Quirk name="requires-allocate-on-input-ports" />
        <Quirk name="requires-allocate-on-output-ports" />
    </MediaCodec>
    <MediaCodec name="OMX.google.h264.decoder" type="video/avc" />

Where,

a.OMX.ABC.XYZ.H264.Decoder is the name of your component

b. video/avc is the MIME type of your component. In this example, it denotes a AVC / H.264 video decoder.

c.The next 2 statements denote the quirks or special requirements of your components. In the given example, requires-allocate-on-input-ports indicates to the Stagefright framework that the component prefers to allocate the buffers on all it's input ports. Similarly, the other quirk is informing that the component will also prefer to allocate on it's output ports. For a list of supported quirks in the system, you could refer to the function OMXCodec::getComponentQuirks in OMXCodec.cpp file. These quirks translate into flags which are then read by the framework to create and initialize the components.

In the example illustration, it is shown that your OMX component is registered prior to the default Google implemented video decoder.

NOTE: If you trying this on an end device, you will have to ensure that this entry is reflected in the final media_codecs.xml file.

2. OMX Core Registration

To create your component and ensure that the correct factory method is invoked, you may have to register your OMX Core with the Stagefright framework.

To register a new core, you will have to create a new library named libstagefrighthw.so which will be located at /system/lib in your end system. This library will have to expose a createOMXPlugin symbol which will be looked by dlsym.

The registration of the OMX core is thus: OMXMaster invokes addVendorPlugin which internally invokes addPlugin("libstagefrighthw.so"). In addPlugin, the createOMXPlugin will be looked up using which the other function pointers for makeComponentInstance, destroyComponentInstance etc are initialized.

Once the OMX core is initialized, you are ready to run your own your component within the android framework. The reference for OMXMaster can be found here.

With these changes, your video decoder is integrated into the android stagefright framework.

In android multimedia framework how we decide which decoder will play this video if we have multiple decoder of same MIME type

There has been a related question on this topic where majority of the steps were covered in this answer.

In the parseXML function, the list is populated by going through the media_codecs.xml file and the codecs are listed in the order in which they are listed i.e. first codec in the file will be the first one in the list to be tried, when we have multiple codecs for same MIME type.

In case one wishes to ensure that their custom codec is employed, it should be enlisted as the first codec in the media_codecs.xml for easier integration.

How to create a stagefright plugin

From your question, it appears that you are looking at a recommendation which is specific for Ice-Cream Sandwich or earlier versions of Android. The first thing you should be clear about is the version of the android i.e. Ice-Cream Sandwich or before or JellyBean and after. The integration of codecs is different across different releases of Android.

I have already commented on your other question which is specific for JellyBean and later (Reference: Android: How to integrate a decoder to multimedia framework)

If you would like to integrate your codec in Ice-Cream Sandwich or before, the steps are already available in your question. In addition to adding the decoder into kDecoderInfo list, you may like to setup certain quirks as shown here.

For the question on OMXCodec.cpp, you can find this file at
frameworks/base/media/libstagefright/ in case of Ice-Cream Sandwich and frameworks/av/media/libstagefright/ in case of JellyBean.

If you have followed all the steps to integrate the video decoder into the Stagefright framework, then the easiest test would be to perform the following:

1. Copy a media file into SD-Card

2. In OMXCodec.cpp, enable logs by removing the comment in this statement //#define LOG_NDEBUG 0 and run a mm in the directory. Copy the rebuilt libstagefright.so to /system/lib on your device.

3. Enable logcat and start capturing logs.

4. Goto gallery, select your file and allow the standard player to play your file.

5. Check your log file if the player has selected your OMX component by searching for your component name. If found, your integration of codec into Stagefright is successful. Else, you will have to debug and find out what is the problem.

Postscript:

1. Based on your queries, I presume you aren't familiar with Android sources. Please refer to androidxref site to become familiar with AOSP distributions.

2. Unless you are planning to support a new media file-format, you will not require to support Extractor class. MediaExtractor abstracts a file-format parser and helps to de-multiplex the different tracks in a media file.

I hope with this information, you should be able to get your codec integrated and functional in Android.

Custom Wrapper Codec Integration into Android

In this post, I am using H.264 as an example, but the solution(s) can be extended to support other codecs like MPEG-4, VC-1, VP8 etc. There are 2 possible solutions to solve your problem, which I am enlisting below, each with their own pros and cons to help you take an informed decision.

Solution 1: Extend the codec to support new mode

In JellyBean, one could register the same OMX component with same MIME types as 2 different component names viz., OMX.ABC.XYZ and OMX.ABC.XYZ.secure. The former is used for normal playback and is the more commonly used component. The latter is used when the parser i.e. MediaExtractor indicates the presence of secure content. In OMXCodec::Create, after findMatchingCodecs returns a list of codecs, we can observe the choice to select .secure component as here.

Steps to follow:

1. In your platform, register another component with some new extension like OMX.H264.DECODER.decrypt or something similar. Change is required only in media_codecs.xml. The choice of whether to register a new factory method or have a common factory method is your choice.

2. From your parser, when you encounter the specific use-case, set a new flag like kKeyDecryptionRequired. For this you will have to define a new flag in Metadata.h and a corresponding quirk in OMXCodec.h.

3. Modify the OMXCodec::create method to append a .decrypt suffix similar to the .secure suffix as shown above.

4. With all changes in OMXCodec, Metadata, MediaExtractor modules, you will have to rebuild only libstagefright.so and replace the same on your platform.

Voila!! your integration should be complete. Now comes the main challenge inside the component. As part of the component implementation, you should be able to differentiate between an ordinary component creation and .decrypt component creation.

From a runtime perspective, assuming that your component is aware of the fact that it is a .decrypt component or not, you could handle the decryption as part of the OMX_EmptyThisBuffer call, where you could decrypt the data and then pass it to underlying codec.

Pros: Easy to integrate, Minimal changes in Android framework, Scalable to other codecs, No new MIME type registration required.

Cons: You need to track the future revisions of android, specifically on the new quirks, flags and choice of .decrypt extension. If Google decides to employ something similar, you will have to adapt / modify your solution accordingly.

Solution 2: Registration of new MIME Type

From your question, it is not clear if you were able to define the MIME type or not and hence, I am capturing the steps for clarity.

Steps to follow:

1. Register a new MIME type at MediaDefs as shown here. For example, you could employ a new MIME type as const char *MEDIA_MIMETYPE_VIDEO_AVC_ENCRYPT = "video/avc-encrypt";

2. Register your new component with this updated MIME type in media_codecs.xml. Please note that you will have to ensure that the component quirks are also handled accordingly.

3. In OMXCodec::setVideoOutputFormat method implementation, you will have to introduce the support for handling your new MIME type as shown for H.264 here. Please note that you will have to handle similar changes in OMXCodec to support the new MIME type.

4. In MediaExtractor, you will have to signal the MIME type for the video track using the newly defined type. With these changes, your component will be selected and created.

However, the challenge still remains: Where to perform the decryption? For this, you could as well employ the same solution as described in the previous section i.e. handle the same as part of OMX_EmptyThisBuffer call.

Pros: None that I can think of..

Cons: First, solution is not scalable. You will have to keep adding newer MIME types and keep modifying the Stagefright framework. Next, the changes in OMXCodec will require corresponding changes in MediaExtractor. Hence, even though your initial focus is on MP4 extractor, if you wish to extend the solution to other container formats like AVI, MKV, you will have to include the support for new MIME types in these extractors.

Lastly, some notes.

1. As a preferred solution, I would recommend Solution 1 as it is easy and simple.

2. I haven't touched upon ACodec based implementation of the codec. However, I do feel that Solution 1 would be a far more easier solution to implement even if such a support is required in future.

3. If you aren't modifying the OMX core, you shouldn't require to modify the libstagefrighthw.so. Just FYI, this is typically implemented by the vendors as part of their vendor specific modules as in vendor/<xyz>/hardware/.... You need to check with your platform provider on the sources for libstagefrighthw.so.

Can i encode or decode video in Android using OpenCORE or StageFright?

You should use MediaCodec api and not any lower level api. The MediaCodec api will then use the appropriate framework internally. This is because android vendors will integrate their own codecs and hardware / dsp accelerated codecs into the framework and set those as default for MediaCodec api. They will ensure their codecs work with MediaCodec api. If you use MediaCodec api it will work on all devices irrespective of the framework being used below. That is what that api is for.

Unless you specifically want to do some testing with OpenCore/Stagefright don't go there.

---

Related Topics