Android: Automated Compatibility Testing

Content

• Why to do android compatibility testing?
• Compatibility test methods
• Create compatibility testing automation script

Why to do Android compatibility testing?

As we knows that there are thousands of different Android devices on the market. With different OS versions, screen sizes, and manufactures, the combinations of them is tremendous. In China, the situation is worse, major mobile phone manufacturers deep customize Android OS, which create more "OS versions". Therefore it is especially important to do compatibility testing before releasing an Android application.

The compatibility test mainly refers to the testing of applications main features, check whether these features works normally on all the Android devices that software vendor claim to support. With so many diversified devices, it is likely some wield exceptions will occur here and there if the app is not tested on these devices.

Compatibility test methods

There are three main methods for compatibility testing: manual testing, automated testing, and cloud platform testing:

• Manual testing is a test that is performed manually on a multi-devices.
• The automated test mainly uses the devices connected to your computer to test the installation, uninstallation, stability, and main features of the tested application.
• The cloud platform testing utilizes public device cloud platform to select device models, and then do the compatibility testing. After one uploads the application to be tested, the test on these devices can be either manual or automated.

Here we mainly talks about automatic compatibility testing with your own devices. Once you know how to do it, it won’t be different to do it similarly on public device cloud, which typically provide more devices.

Create compatibility testing automation script

One most common testing scenario is to verify install / uninstall of the application on the Android device, and whether it can be launching successfully on the target device.

We can use adb command to do the installation and uninstallation, which is commnoly used by developers. For example, we have test.apk installation package with the application name "com.sample.app", it has an activity MainActivity.

# install 
adb install test.apk

# start test
adb shell am start -n com.sample.app/.MainActivity

# uninstall 
adb uninstall com.sample.app

# install with overwrite
adb install -r test.apk

The above commands can install, start app, and uninstall the app. However, with these commands, only one device can be tested at a time.

In order to run the same test on multiple devices, especially in a Continuous Integration (CI) scenario, where a new build of app need to be tested on daily bases, the best way is to use an automation script to test it, running one script, test all available devices. An automation script is developed with the CukeTest tool for this purpose.

To set up the environmental, please first configure the android sdk (check the information on the Internet on how to install sdk)

Then you can follow the below steps:

1. Open CukeTest, File -> New Project; select Basic project for the project type, fill in the project information, and complete the creation.

2. Execute npm install adbkit –save in the root directory of the project to install "adbkit" node.js package. This is a node.js package that can run adb shell tool. More information of this package can be found on https://github.com/openstf/adbkit.

3. Change the default feature file, fill in the following to features/feature1.feature.

Feature: Devices Compatibility Test
Android phone install and uninstall

  Scenario: Install and Run app
    Given Get device information
    And install apk package
    And start MainActivity
    Then Uninstall the app

Set default timeout for steps in features/support/env.js

const { setDefaultTimeout } = require('cucumber')

// set default timeout to 60 seconds
setDefaultTimeout(60 * 1000);

Edit script content for features/step_definitions/defination1.js

const { Given, When, Then } = require('cucumber');
const path = require('path');
var adb = require('adbkit');
var client = adb.createClient();
var devices = [];

Given(/^Get device information$/, async function () {
    devices = await client.listDevices();
    console.log("devices:", devices)
});

Given(/^install apk package$/, async function () {
    let apkpath = path.join(__dirname, '../../apks/app-release.apk');

    for (let device of devices) {
        try {
            await client.install(device.id, apkpath)
            console.log("install ", apkpath, "to device", device.id);
        } catch (e) {
            console.log(device.id, "cant install the apk;")
        }

    }
});

Given(/^start MainActivity$/, async function () {
    options = {
        wait: true,
        component: "com.reactapp/.MainActivity"
    }
    for (let device of devices) {
        try {
            //start activity
            await client.startActivity(device.id, options)
        } catch (e) {
            console.log(device.id, "cant start the activity")
        }
    }
});

Then(/^Uninstall the app$/, async function () {
    let pk = 'com.reactapp'
    for (let device of devices) {
        await client.uninstall(device.id, pk)
    }
});

The scenario & code looks like the following:

And the directory structure looks like below:

After clicking “run” button, it can start package compatibility testing on all your connected android phones, and you can get console output as the report.

Summary

Using JavaScript automation script instead of raw adb commands make it easier to batch run on multiple devices. And converting it further to BDD script, you get even better the readability for the script, and you can also add more scenarios after the install scenario in this automation script to test more behaviors of application.