Cloud-Based Automated Testing: A Tutorial With Xamarin.UITest and App Center (Part 2)

In the first blog in this automated testing tutorial series, we learned about how you can use cloud-based automated testing to expand the different types of devices and operating systems you test on. We also learned how to create an automated test system for your Android and iOS using Microsoft’s App Center, a cloud-based platform, with Xamarin.UITest, a C# testing framework. 

In this post, you’ll learn about cloud testing with App Center, including how to create a test run to set the test devices, and how to set up your environment and execute automated tests on App Center.

How to create a project on App Center

To create a project on App Center so that you can run your tests for Android and iOS devices:

1. If you don’t yet have an App Center account, create one now (read here for details).

2. In App Center, click on the “Add new” button and select “Add new app.” After this, you’ll see a new section to set the app properties. 

3. Set the app name, the operating system, and the platform used to build the app. 

In the example screen below, I selected iOS as the OS, and Xamarin as the platform (note that other platforms are also supported). 

4. Click on “Add new app” in the bottom on the right corner of the window, and the new app project will be created successfully.

5. Create the Android project. Click on “App Center” in the left top corner. You should see a page that lists the apps previously created.

6. If you want to create an Android app, repeat steps 3 to 5 using the Android properties instead of iOS.

7. Now you will have both apps created as seen in the screen below.

How to create a test run for Android and iOS devices

To execute the test, you must create the test run. A test run allows you to set the list test devices you want to use in the execution for each platform. App Center provides a large list of devices from which you can choose (see here for a detailed review). 

In the steps below, you will set the devices and the operating system versions you prefer.

1. In App Center, open the Android project you created earlier. 

2. In the menu on the left side, click on “Test.”

3. Click on “New test run.”

4. Select each of the devices you want to use for your tests.

5. Click on the “Select <number> devices” button in the right bottom corner.

6. Set the test framework to “Xamarin.UITest” and click on the “Next” button.

7. IMPORTANT: Copy the command under “Running tests” and save it in a .txt file on your computer (you will use this later).

8. To create the test run for an iOS app, repeat steps 1 through 7 using the iOS properties instead of Android. Remember to save the new command in a .text file.

Setting up the local env to connect to App Center

In this section, we’ll connect your computer and App Center. 

Note! Be sure you’ve completed the steps in the preceding sections first. You need to have an account already created in order to connect with your env successfully.

1. Make sure you have Node.js installed, version 6.3 or later.

2. Install the appcenter-cli NPM package: 

npm install -g appcenter-cli

3. Run the following command to login: 

appcenter login 

4. You should see the App Center web page with your access code. Copy it. If you don’t see the access code, you need to log in to the browser, open, and then run the command one more time:

5. Once you have the access code, you need to go back to your terminal and paste the code into the prompt, and then press Enter. At this point, you should be logged in as shown in the image below.

Run the Xamarin.UI tests on App Center

Now let’s execute your tests on App Center using the test runs you created in the previous section.

1. Open the .txt file where you saved the commands created in App Center (you are going to modify both commands using the real values). 

2. Change “pathToFile.apk” to the location on your computer where you have the .apk file saved (you can use absolute paths or relative paths).. 

3. Change “pathToFile.ipa” to the location on your computer where you have the .ipa file saved (you can use absolute paths or relative paths).

4. Change “pathToUITestBuildDir” to the path where the Release directory of your Xamarin.UI Test project resides. 

For example: C:\CalculatorUITest\bin\Release. This applies for both commands.

5. Copy one of the updated commands, paste it into your terminal, and then press Enter. 

If you get the error “Cannot find test-cloud.exe” you need to specify where the test-cloud.exe resides. Then, in your commands, add the following flag: 

–uitest-tools-dir and the path to find the test-cloud.exe (you can find it at .nuget\packages\xamarin.uitest\3.0.7\tools)

Look below at an example of the final command. The iOS command would be nearly the same, except for the devices pool and the path to the .ipa file. 

appcenter test run uitest –app “user/CalculatorApp” –devices 5525cc4tb243 –app-path C:\path\calculatorapp2.apk –test-series “master” –locale “en_US” –build-dir C:\path\CalculatorUITest\bin\Release –uitest-tools-dir C:\path\.nuget\packages\xamarin.uitest\3.0.7\tools

Android devices execution

Once you have executed the command built previously, this is how your terminal should look for Android:

Test results on App Center 

Once the tests have finished, you should get a link to review the results. Alternatively, you can go directly to “App Center => appName => Test => Test runs” to see the results.

In the screenshot below, you see charts that show the number of tests and devices passed, as well as the list of test cases executed. You also see some more general information about your execution in the top right.

If you click on a test case, you will be able to see the screenshots for each device. I really like this feature.

iOS devices execution

Once you have executed the command built previously, this is how your terminal should look for iOS:

As with the Android device test, you should get a link to review the results. Alternatively, you can go directly to “App Center => appName => Test => Test runs” to see the results.

In the screenshot below, you see charts that show the number of tests and devices passed, as well as the list of test cases executed. You also see some more general information about your execution in the top right.

If you click on a test case, you will be able to see the screenshots for each iOS device. I really like this feature.

Note that if you grouped the test class by using CategoryAttribute (for example: [Category(“LoginTests”)] ) you can only run the test cases under this category. To run the test from other categories, you just need to add the following parameter in the command you built previously: –include-category “LoginTests” 

Improving the test device coverage through cloud testing with App Center 

Now you’ve learned how to work with App Center to execute your tests—and how easy it is to increase your device and operating system coverage. You’ve also learned some basic elements you need to add to your apps. With this knowledge,  you can tackle test executions with a significant number of real devices, as well as a variety of operating system versions. This is a very powerful way to increase the test device coverage you have for your tests; and this means that you’re much more likely to catch more defects earlier. You get better quality testing, faster!