How to use YUV (YUV_420_888) Image in Android

ImageFormat#YUV_420_888 is one of the most common image format supported by Android Cameras. It’s a multi-plane YUV (YCbCr) format represented by three separate planes in android.media.Image and the order of the planes is guaranteed to be:

• [0]: Y plane (Luma)
• [1]: U plan (Cb)
• [2]: V plane (Cr)

Figure: YUV420 representation, 8 bits for Y and 4 bits for UV (interleaved). Source: Wikipedia

This format can be used for processing the input frames before saving to disk or some other action. A very common question around YUV is how to consume it in Android. In this article, I’d describe different ways it can be used. The most common question is how to convert YUV to Bitmap or jpeg format in Android.

This article is imported from blog.minhazav.dev/how-to-convert-yuv-420-sp-android.media.Image-to-Bitmap-or-jpeg/

Theory

Y′UV was invented when engineers wanted color television in a black-and-white infrastructure. They needed a signal transmission method that was compatible with black-and-white (B&W) TV while being able to add color. The luma component already existed as the black and white signal; they added the UV signal to this as a solution.

Super cool no? Source: Wikipedia

YUV images can be represented in different formats like YUV444, YUV422, YUV4111 or YUV420p. It’s easiest to convert between RGB888 and YUV444.

pseudo-code

In general you can convert a YUV420 to RGB using following logic

void yuv2rgb(uint8_t y, uint8_t u, uint8_t v) {
    int r = y + (1.370705 * (v - 128));
    int g = y - (0.698001 * (v - 128)) - (0.337633 * (u - 128));;
    int b = y + (1.732446 * (u - 128));
    r = clamp(r, 0, 255);
    g = clamp(g, 0, 255);
    b = clamp(b, 0, 255);
    return r, g, b;
}

Convert YUV_420_888 to ——– ?

In this section, I’ll cover various how to questions. If you have gone through the theory section, the solutions should look more intuitive.

How to convert YUV_420_888 Image to Bitmap?

There are a couple of popular questions related to this in StackOverflow, like:

Most common suggestions there is to either use RenderScript based approach or a much hackier approach where we first convert the Image to YuvImage as mentioned here. And then encode to jpeg as mentioned here. Then use android.graphics.BitmapFactory to decode the jpeg byte array.

// .. get YUV420_888 Image
byte[] jpegByteArray = getJpegFromImage(image);
BitmapFactory bitmapFactory = new BitmapFactory();
BitmapFactory.Options options = new BitmapFactory.Options();
Bitmap bitmap = bitmapFactory.decodeByteArray(
     jpegByteArray, /* offset= */ 0, jpegByteArray.length);

It was when I read the latter solution I decided to write this article. If you are reading this article please do not convert the Yuv Image to jpeg just to convert it to Bitmap.

Pure Java approach

If you look at how the YUV image looks like once again:

You will see there is one U & V (chroma) value for four luma values. I have tried to use this information along with the yuv to rgb translation above to bring up some java code.

This can easily become a costly approach when handled in the java layer as soon as the image resolutions crosses a certain limit. Let’s take a look at benchmarks of this by running on a certain low end device (with viewfinder running):

Table 1: Running time for algorithm above on a certain Android device (low end)

Also, there are high chances that you might start to see logs that look like:

Background concurrent copying GC freed 1648(142KB) AllocSpace objects, 2(38MB) LOS objects, 50% free, 20MB/41MB, paused 1.135ms total 128.325ms

High frequency of such logs or high value of paused <some value> ms are an indicator of increased GC pressure.

So JVM probably is not suited to handle really big images both w.r.t latency and reliability. JVM also sets a limit on max heap memory allocations by an application in Android which can be bypassed to a higher limit but even higher limit comes with restriction. This is just FYI, converting a single image shouldn’t lead to such allocations. If you must know a YUV_420_888 image takes 1.5 bytes per pixel so an 8MP (3264 x 2488) = 8120832 pixels image should be 11.61 Mb in memory while a single ARGB_8888 bitmap would take 4 bytes per pixel leading to 30.97 Mb per image of the same size.

Renderscript approach

RenderScript is a framework for running computationally intensive tasks at high performance on Android. RenderScript is primarily oriented for use with data-parallel computation, although serial workloads can benefit as well. The RenderScript runtime parallelizes work across processors available on a device, such as multi-core CPUs and GPUs. This allows you to focus on expressing algorithms rather than scheduling work. RenderScript is especially useful for applications performing image processing, computational photography, or computer vision.

Source: developer.android.com

Android team has published an intrinsic for converting an Android YUV buffer to RGB. The input allocation is supplied as 8bit NV12 YUV byte array and the output is 4 channel 8 bit ARGB buffer which can be converted to a Bitmap. The name of intrinsic is ScriptIntrinsicYuvToRGB.

Here’s a java code sample on how to use it:

For benchmarking I’ll be looking at the setup part (i.e. creating RenderScrip allocations) separately and the actual conversions separately. In this article, I am only investigating the latency aspect of the algorithm but while using this in the applications it’d be worthwhile to track the peak heap memory usage using the Android Studio profiler.

Table 2: Running time for algorithm above on a certain Android device (low end)

RenderScript is a winner as compared to the pure java approach and considering what it’s build for it makes complete sense for this approach to be fast. Considering realtime applications this can be a fair approach even for low-end devices.

Native approach

Another approach is to leverage JNI (Java Native Interface) in Android. I don’t have exact code examples here but in my experience JNI is very well suited for low latency image processing in Android. You can also use shaders like OpenGL ES for realtime applications. For our YUV_420_888 to Bitamp conversion the API would look like:

// java delegate.
Bitmap yuv420ToBitmap(Image image) {
    int width = image.getWidth();
    int height = image.getHeight();
    // Pass this array to native code to write to.
    int argbResult = new int[width * height];
    Plane yPlane = image.getPlanes()[0];
    Plane uPlane = image.getPlanes()[1];
    Plane vPlane = image.getPlanes()[2];
    ByteBuffer yBuffer = yPlane.getBuffer();
    ByteBuffer uBuffer = uPlane.getBuffer();
    ByteBuffer vBuffer = vPlane.getBuffer();

    // call the JNI method
    yuv420ToBitmapNative(
        width,
        height,
        yBuffer,
        yPlane.getPixelStride(),
        yPlane.getRowStride(),
        uBuffer,
        uPlane.getPixelStride(),
        uPlane.getRowStride(),
        vBuffer,
        vPlane.getPixelStride(),
        vPlane.getRowStride(),
        argbResult);
    return Bitmap.createBitmap(argbResult, width, height, Config.ARGB_888);
}

// native interface
static native void yuv420ToBitmapNative(
    int width,
    int height,
    ByteBuffer yBuffer,
    int yPixelStride,
    int yRowStride,
    ByteBuffer uBuffer,
    int uPixelStride,
    int uRowStride,
    ByteBuffer vBuffer,
    int vPixelStride,
    int vRowStride,
    int[] argbResult);

And then consume this data in the native code and use the same logic as mentioned above to convert YUV_420_888 to argb. I do not have the exact numbers with me w.r.t peformance latency.

How to convert YUV_420_888 Image to YuvImage?

Here’s code example to convert android.media.Image to android.graphics.YuvImage:

Code

YuvImage toYuvImage(Image image) {
    if (image.getFormat != ImageFormat.YUV_420_888) {
        throw new IllegalArgumentException("Invalid image format");
    }

    ByteBuffer yBuffer = image.getPlanes()[0].getBuffer();
    ByteBuffer uBuffer = image.getPlanes()[1].getBuffer();
    ByteBuffer vBuffer = image.getPlanes()[2].getBuffer();

    int ySize = yBuffer.remaining();
    int uSize = uBuffer.remaining();
    int vSize = vBuffer.remaining();

    byte[] nv21 = new byte[ySize + uSize + vSize];

    // U and V are swapped
    yBuffer.get(nv21, 0, ySize);
    vBuffer.get(nv21, ySize, vSize);
    uBuffer.get(nv21, ySize + vSize, uSize);

    int width = image.getWidth();
    int height = image.getHeight();
    return new YuvImage(nv21, NV21, width, height, /* strides= */ null);
}

Running time at different resolutions

Table 4: Running time for algorithm above on a certain Android device (low end).

Overall these algorithms didn’t seem to have an inherent high cost on its own - its practically memory copy.

How to convert YUV_420_888 Image to JPEG format?

You can convert the android.media.Image to JPEG format (like single plane byte[]):

Code

byte[] toJpegImage(Image image, int imageQuality) {
    if (image.getFormat != ImageFormat.YUV_420_888) {
        throw new IllegalArgumentException("Invalid image format");
    }

    YuvImage yuvImage = toYuvImage(image);
    int width = image.getWidth();
    int height = image.getHeight();

    // Convert to jpeg
    byte[] jpegImage = null;
    try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
        yuvImage.compressToJpeg(new Rect(0, 0, width, height), imageQuality, out);
        jpegImage = out.getBytes();
    }

    return jpegImage;
}

Running time at imageQuality = 100 at different resolution

Table 5: Running time for algorithm above on a certain Android device (low end).

This article is imported from blog.minhazav.dev/how-to-convert-yuv-420-sp-android.media.Image-to-Bitmap-or-jpeg/