How to Manipulate the Camera Preview

How can I manipulate the camera preview?

I did some research on this and put together a working(ish) example. Here's what I found. It's pretty easy to get the raw data coming off of the camera. It's returned as a YUV byte array. You'd need to draw it manually onto a surface in order to be able to modify it. To do that you'd need to have a SurfaceView that you can manually run draw calls with. There are a couple of flags you can set that accomplish that.

In order to do the draw call manually you'd need to convert the byte array into a bitmap of some sort. Bitmaps and the BitmapDecoder don't seem to handle the YUV byte array very well at this point. There's been a bug filed for this but don't don't what the status is on that. So people have been trying to decode the byte array into an RGB format themselves.

Seems like doing the decoding manually has been kinda slow and people have had various degrees of success with it. Something like this should probably really be done with native code at the NDK level.

Still, it is possible to get it working. Also, my little demo is just me spending a couple of hours hacking stuff together (I guess doing this caught my imagination a little too much ;)). So chances are with some tweaking you could much improve what I've managed to get working.

This little code snip contains a couple of other gems I found as well. If all you want is to be able to draw over the surface you can override the surface's onDraw function - you could potentially analyze the returned camera image and draw an overlay - it'd be much faster than trying to process every frame. Also, I changed the SurfaceHolder.SURFACE_TYPE_NORMAL from what would be needed if you wanted a camera preview to show up. So a couple of changes to the code - the commented out code:

//try { mCamera.setPreviewDisplay(holder); } catch (IOException e)
//  { Log.e("Camera", "mCamera.setPreviewDisplay(holder);"); }

And the:

SurfaceHolder.SURFACE_TYPE_NORMAL //SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS - for preview to work

Should allow you to overlay frames based on the camera preview on top of the real preview.

Anyway, here's a working piece of code - Should give you something to start with.

Just put a line of code in one of your views like this:

<pathtocustomview.MySurfaceView android:id="@+id/surface_camera"
    android:layout_width="fill_parent" android:layout_height="10dip"
    android:layout_weight="1">
</pathtocustomview.MySurfaceView>

And include this class in your source somewhere:

package pathtocustomview;

import java.io.IOException;
import java.nio.Buffer;

import android.content.Context;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.Canvas;
import android.graphics.Paint;
import android.graphics.Rect;
import android.hardware.Camera;
import android.util.AttributeSet;
import android.util.Log;
import android.view.SurfaceHolder;
import android.view.SurfaceHolder.Callback;
import android.view.SurfaceView;

public class MySurfaceView extends SurfaceView implements Callback,
    Camera.PreviewCallback {

    private SurfaceHolder mHolder;

    private Camera mCamera;
    private boolean isPreviewRunning = false;
    private byte [] rgbbuffer = new byte[256 * 256];
    private int [] rgbints = new int[256 * 256];

    protected final Paint rectanglePaint = new Paint();

    public MySurfaceView(Context context, AttributeSet attrs) {
    super(context, attrs);
        rectanglePaint.setARGB(100, 200, 0, 0);
        rectanglePaint.setStyle(Paint.Style.FILL);
        rectanglePaint.setStrokeWidth(2);

        mHolder = getHolder();
        mHolder.addCallback(this);
        mHolder.setType(SurfaceHolder.SURFACE_TYPE_NORMAL);
    }

    @Override
    protected void onDraw(Canvas canvas) {
        canvas.drawRect(new Rect((int) Math.random() * 100,
            (int) Math.random() * 100, 200, 200), rectanglePaint);
        Log.w(this.getClass().getName(), "On Draw Called");
    }

    public void surfaceChanged(SurfaceHolder holder, int format, int width,
            int height) {
    }

    public void surfaceCreated(SurfaceHolder holder) {
        synchronized (this) {
            this.setWillNotDraw(false); // This allows us to make our own draw
                                    // calls to this canvas

            mCamera = Camera.open();

            Camera.Parameters p = mCamera.getParameters();
            p.setPreviewSize(240, 160);
            mCamera.setParameters(p);

            //try { mCamera.setPreviewDisplay(holder); } catch (IOException e)
            //  { Log.e("Camera", "mCamera.setPreviewDisplay(holder);"); }

            mCamera.startPreview();
            mCamera.setPreviewCallback(this);

        }
    }

    public void surfaceDestroyed(SurfaceHolder holder) {
        synchronized (this) {
            try {
                if (mCamera != null) {
                    mCamera.stopPreview();
                    isPreviewRunning = false;
                    mCamera.release();
                }
            } catch (Exception e) {
                Log.e("Camera", e.getMessage());
            }
        }
    }

    public void onPreviewFrame(byte[] data, Camera camera) {
        Log.d("Camera", "Got a camera frame");

        Canvas c = null;

        if(mHolder == null){
            return;
        }

        try {
            synchronized (mHolder) {
                c = mHolder.lockCanvas(null);

                // Do your drawing here
                // So this data value you're getting back is formatted in YUV format and you can't do much
                // with it until you convert it to rgb
                int bwCounter=0;
                int yuvsCounter=0;
                for (int y=0;y<160;y++) {
                    System.arraycopy(data, yuvsCounter, rgbbuffer, bwCounter, 240);
                    yuvsCounter=yuvsCounter+240;
                    bwCounter=bwCounter+256;
                }

                for(int i = 0; i < rgbints.length; i++){
                    rgbints[i] = (int)rgbbuffer[i];
                }

                //decodeYUV(rgbbuffer, data, 100, 100);
                c.drawBitmap(rgbints, 0, 256, 0, 0, 256, 256, false, new Paint());

                Log.d("SOMETHING", "Got Bitmap");

            }
        } finally {
            // do this in a finally so that if an exception is thrown
            // during the above, we don't leave the Surface in an
            // inconsistent state
            if (c != null) {
                mHolder.unlockCanvasAndPost(c);
            }
        }
    }
}

How to manipulate CameraPreview bytearray through the JNI? (OpenCV)

I want to take the bytearray "data" and pass it to the JNI and apply some OpenCV filters so that the preview changes, without returning it.

Unfortunately that's not possible. The byte array that is passed to onPreviewFrame() is just a copy of the preview frame, and any changes that you make to it will not be shown in the preview. You can test this for yourself by modifying the byte array in Java inside the onPreviewFrame() function as a test, you won't see any effect.

If you want to change the preview frame data using OpenCV and see the results in a preview window then you will need to upload the processed frame to an OpenGL texture and then render it to a GLSurfaceView, using a fragment shader to convert the NV21 data to RGB, or some other approach. Simply changing the byte array won't work.

See these questions for more information:

PreviewCallback onPreviewFrame does not change data

onPreviewFrame doesn't change the data

How to manipulate on the fly YUV Camera frame efficiently in Android?

1. Yes. To understand why, let's take a look at the bytecode Android Studio produces for your "left/right of center" nested loop:

(Annotated excerpt from a release build of blackNonROI, AS 3.2.1):

:goto_27
sub-int v2, p2, p4         ;for(int y=verMargin; y<height-verMargin; y++)
if-ge v1, v2, :cond_45
const/4 v2, 0x0
:goto_2c
if-ge v2, p3, :cond_36     ;for (int x = 0; x < hozMargin; x++)
mul-int v3, v1, p1
add-int/2addr v3, v2
.line 759
aput-byte v0, p0, v3
add-int/lit8 v2, v2, 0x1
goto :goto_2c
:cond_36
sub-int v2, p1, p3 
:goto_38
if-ge v2, p1, :cond_42     ;for (int x = width-hozMargin; x < width; x++)
mul-int v3, v1, p1
add-int/2addr v3, v2
.line 761
aput-byte v0, p0, v3
add-int/lit8 v2, v2, 0x1
goto :goto_38
:cond_42
add-int/lit8 v1, v1, 0x1
goto :goto_27
.line 764
:cond_45                   ;all done with the for loops!

Without bothering to decipher this whole thing line-by-line, it is clear that each of your small, inner loops is performing:

• 1 comparison
• 1 integer multiplication
• 1 addition
• 1 store
• 1 goto

That's a lot, when you consider that all that you really need this inner loop to do is set a certain number of successive array elements to 0.

Moreover, some of these bytecodes require multiple machine instructions to implement, so I wouldn't be surprised if you're looking at over 20 cycles, just to do a single iteration of one of the inner loops. (I haven't tested what this code looks like once it's compiled by the Dalvik VM, but I sincerely doubt it is smart enough to optimize the multiplications out of these loops.)

POSSIBLE FIXES

You could improve performance by eliminating some redundant calculations. For example, each inner loop is recalculating y * width each time. Instead, you could pre-calculate that offset, store it in a local variable (in the outer loop), and use that when calculating the indices.

When performance is absolutely critical, I will sometimes do this sort of buffer manipulation in native code. If you can be reasonably certain that mPendingFrameData is a DirectByteBuffer, this is an even more attractive option. The disadvantages are 1.) higher complexity, and 2.) less of a "safety net" if something goes wrong/crashes.

MOST APPROPRIATE FIX

In your case, the most appropriate solution is probably just to use Arrays.fill(), which is more likely to be implemented in an optimized way.

Note that the top and bottom blocks are big, contiguous chunks of memory, and can be handled by one Arrays.fill() each:

Arrays.fill(yuvData, 0, verMargin * width, 0);   //top
Arrays.fill(yuvData, width * height - verMargin * width, width * height, 0);    //bottom

And then the sides could be handled something like this:

for(int y=verMargin; y<height-verMargin; y++){
    int offset = y * width;
    Arrays.fill(yuvData, offset, offset + hozMargin, 0);  //left
    Arrays.fill(yuvData, offset + width, offset + width - hozMargin, 0);   //right
}

There are more opportunities for optimization, here, but we're already at the point of diminishing returns. For example, since the end of each row of is adjacent to the start of the next one (in memory), you could actually combine two smaller fill() calls into a larger one that covers both the right side of row N and the left side of row N + 1. And so forth.

2. Not sure. If your preview is displaying without any corruption/tearing, then it's probably a safe place to call the function from (from a thread safety standpoint), and is therefor probably as good a place as any.

3 and 4. There could be libraries for doing this task; I don't know of any offhand, for Java-based NV21 frames. You'd have to do some format conversions, and I don't think it's be worth it. Using a GPU to do this work is excessive over-optimization, in my opinion, but it may be appropriate for some specialized applications. I'd consider going to JNI (native code) before I'd ever consider using the GPU.

I think your choice to do the manipulation directly to the NV21, instead of converting to a bitmap, is a good one (considering your needs and the fact that the task is simple enough to avoid needing a graphics library).

---

Related Topics