Overview

Once the automated tests have been written, developers will want to upload the tests and receive feedback about the test runs. There are two ways to accomplish this:

• Using the IDE – Both Xamarin Studio and Xamarin for Visual Studio can submit tests directly to Test Cloud.
• At the command line – It is also possible to upload tests and applications to Xamarin Test Cloud at the command line. This scenario is more suited for continuous integration or when the build process is started via a command line script.

This guide will focus on how to upload UITests and mobile applications to Xamarin Test Cloud using Xamarin Studio or Visual Studio. There are two possible scenarios to be considered:

1. A complete Xamarin solution with a mobile app project and a Xamarin.UITest project – This will discuss how to submit a tests when the mobile app is written using Xamarin and the Xamarin.UITest project is a part of the solution.
2. A Xamarin.UITest project and an external IPA or APK – In this case Xamarin.UITests are written for a mobile app that was created outside of the Xamarin platform.

Limitations

At the present, Test Cloud is unable to run tests that involve the following scenarios or situations:

• Tests that run longer than 180 minutes
• Testing with Bluetooth connected
• Testing apps on network instead of Wifi
• Throttling WiFi
• Testing under different battery conditions
• Apps using the Camera
• Starting an app automatically rotated
• VPN into corporate network instead of opening up ports to the firewall
• Testing integration with other apps installed on the device

Requirements

IDE support for Xamarin Test Cloud requires Xamarin Studio 5.9 or Xamarin for Visual Studio 3.11.

It is assumed that a Test Cloud team has been created, that your e-mail address has been added to the team, and you have the Team API Key. Please see the Organizations & Teams document for more Details.

Android applications must request the INTERNET permission in AndroidManifest.XML.

Firewall Configuration

In order for tests to be submitted to Xamarin Test Cloud, the computer submitting the tests must be able to communicate with the Test Cloud servers. Firewalls must be configured to allow network traffic to and from the servers located at testcloud.xamarin.com on ports 80 and 443. This endpoint is managed by DNS and the IP address is subject to change.

In some situations, a test (or a device running the test) must communicate with web servers protected by a firewall. In this scenario the firewall must be configured to allow traffic from the following IP addresses:

• 195.249.159.238
• 195.249.159.239

Submitting UITests From a Xamarin Solution

Both Visual Studio and Xamarin Studio can submit UITests to Xamarin Test Cloud when the solution contains both a mobile Xamarin project and a UITest project. When UITests are submitted to Xamarin Test Cloud, the IDE will rebuild the application and the tests and upload the binaries, placing them in a queue for execution.

ℹ️

It is not possible to select an NUnit test Category when submitted tests to Xamarin Test Cloud through the IDE. Running tests that belong to an NUnit Category is only supported when submitting tests by using the command line.

Prior to submitting an Android project, you must ensure that the Android project is set as the startup project:

Prior to submitting an Android project, you must create the APK first and ensure that the Android project is set as the startup project:

Before submitting an iOS project, you must ensure that your build configuration is set to Debug > iPhone and that the iOS project is set as the startup project:

Next, right click on the project in the Test Pad, and select Run in Test Cloud:

This will cause the Run in Test Cloud dialog to appear. Select which application you would like to run the tests for:

The upload will begin once you click the Upload and Run button. You can see the progress in the Tests Cloud pane inside the IDE:

Next, right click on the the UITest project in the Solution Explorer, and select Run in Test Cloud from the context menu:

When you do so, Visual Studio will present you with the following dialog informating you that the workflow is being handed off to the web browser:

When you click the OK button, the application and tests will be uploaded to Test Cloud.

When the upload is finished, the IDE will hand off the workflow to the the web browser to complete the test run. If necessary, you will be prompted to log into the Test Cloud website.

First, it will be necessary to select an existing team for the test run:

After the team has been choosen, selected the devices to run the test on:

The final dialog will allow you to provide some final information about the test run itself:

Once you click Done the test run will be queued in Test Cloud. When the test run is finished, Test Cloud will send out an e-mail with a summary of the test results:

Click on View Results in order to see details about the test run on the Test Cloud website.

⚠️

Xamarin Test Cloud will terminate test runs that take longer than 180 minutes.

Submitting UITests for an External Mobile App

It is possible to write Xamarin.UITests for non-Xamarin mobile applications. In this situation, Xamarin Studio can submit tests for a solution that only has single Xamarin.UITest project. Open up the Unit Test pad, and right click on the UITest project. Select Run in Test Cloud from the context menu that appears:

The <Run In Test Cloud dialog will appear:

Select the APK or the IPA to submit along with the tests, and then click the Upload and Run button. Xamarin Studio will then proceed to compile the UITest project and upload the test assembly along with the APK or IPA to Xamarin Test Cloud.

Although it is possible to write Xamarin.UITests for an existing IPA or APK in Visual Studio, it is not possible to submit through Visual Studio.

Summary

This guide discussed how to submit tests to Xamarin Test Cloud using Xamarin Studio or Visual Studio.