OpenCV on Android : Part 1

Starting with why?

We had a rather peculiar problem – we wanted to empower ground-level volunteers to scan MCQ sheet (a simple MCQ form as a bubble-sheet – for surveys or exams) without any expensive OMR machine and we wanted to explore if it would be feasible to use their existing smartphone as an enabler. Moreover, given their remote rural or semi-urban locations, we didn’t want to assume uninterrupted Internet/data connectivity either, so using online APIs wasn’t feasible.

Simply put, we wanted to use computer vision offline on an average Android phone with its camera. I used Kotlin for programming instead of Java because I wanted to learn Kotlin. Here is an article in two parts explaining my learning – technical as well as non-technical.

1. First things first: If you can, use Python for any computer vision project. It is undoubtedly a much better choice with OpenCV library. Availability of the packages such as NumPy or pandas makes it the preferred technology stack for most AI/ML projects.
2. Android will present you with tons of client side compatibility issues – different devices will throw different problems, and you cannot ever test it on all physical devices. It remains a formidable challenge, and at times you have no other option but to give up supporting few devices.

With both these disclaimers/caveats out of the way, let me get to the good news. YES, computer vision worked successfully on Android devices. We built a simple, ‘Don’t make me think’ app to scan the MCQ forms using mid-range Android smartphones.

I should also add that this is NOT a generic OMR app to read any bubble-sheet, I have built this app for a specific problem that I have discussed above.

Here is a short demo video of computer vision in action from this app –

How?

I am just listing major steps here. I cannot share the complete code for this app, but here are few Kotlin code snippets with my comments that should offer enough insights to use OpenCV library for Andorid devices.

Camera:

Since most modern phones have array of different cameras, and have tons of features/filters to manipulate the photos, I didn’t want the end users to struggle themselves with exposure, focus and so on to get the right image for processing. Likewise, I didn’t want the user’s default camera settings, filters to affect the image. So I decided to handle the device camera with auto-focus and required settings by the app itself without requiring the end-user to do all focus, exposure settings for a clear image (‘Don’t make me think’ approach again).  Within these constraints, I decided to go ahead with Android’s Camera2 API as it offers quite a good granular control over the camera hardware. You can get sample code for this from Android’s official GitHub repo to understand how it is implemented. Most of the other articles I have found online for Camera2 API discuss it with Java code, which I didn’t use. Kotlin offers a much cleaner and efficient way of building Android apps including useful Kotlin co-routines.

For the rest of this article, I am assuming that you’re well-versed with Android lifecycle concepts, its challenges and importance of keeping the main UI thread responsive at all times. Keep the user informed through small messages about the process so they are not lost when a time-consuming operation is inevitable.

Setting up OpenCV:

This is crucial. Most Android devices that ran into hardware or driver issues couldn’t initialize OpenCV properly. This needs to be done in the Android application itself so that OpenCV works fine in all the activities/fragments that need to use OpenCV APIs. This is a code snippet for initializing OpenCV in Android application –

override fun onCreate() {
    super.onCreate()
    initOpenCV()
   //other initializtion...
}

private fun initOpenCV() {
    val engineInitialized = OpenCVLoader.initDebug()
    if (engineInitialized){
        Log.i(TAG, "The OpenCV was successfully initialized in debug mode using .so libs.")
    } else {
        initAsync(OpenCVLoader.OPENCV_VERSION_3_4_0, this, object : LoaderCallbackInterface {
            override fun onManagerConnected(status: Int) {
                when(status) {
                    LoaderCallbackInterface.SUCCESS -> Log.d(TAG,"OpenCV successfully started.")
                    LoaderCallbackInterface.INIT_FAILED -> Log.d(TAG,"Failed to start OpenCV.")
                    LoaderCallbackInterface.MARKET_ERROR -> Log.d(TAG,"Google Play Store could not be invoked. Please check if you have the Google Play Store app installed and try again.")
                    LoaderCallbackInterface.INSTALL_CANCELED -> Log.d(TAG,"OpenCV installation has been cancelled by the user.")
                    LoaderCallbackInterface.INCOMPATIBLE_MANAGER_VERSION -> Log.d(TAG,"This version of OpenCV Manager is incompatible. Possibly, a service update is required.")
                }
            }

            override fun onPackageInstall(operation: Int, callback: InstallCallbackInterface?) {
                Log.d(TAG,"OpenCV Manager successfully installed from Google Play.")
            }
        })
    }
}

A small note – you might run into this error in your logs despite OpenCV getting initialized properly.

E/OpenCV/StaticHelper: OpenCV error: Cannot load info library for OpenCV

Please IGNORE this error message – ‘Info library’ is used for special Android configurations, like builds with CUDA support. It is explained by an OpenCV contributor himself.

I’ll explain the image processing and computer vision part of the Kotlin code in the next blog post.

Advertisement