Submitting UITests at the Command Line
- PDF for offline use:
- Related Articles:
Let us know how you feel about this.
0/250
last updated: 2016-12
Testing on physical devices is an important part of creating a mobile application. Calabash allows developers to create automated tests for mobile applications and Xamarin Test Cloud provides a service that will run these tests on hundreds of physical devices. This guide will discuss how to submit UITests to Xamarin Test Cloud.
Overview
This guide discusses how to submit your UITests tests to Xamarin Test Cloud using the command line. This scenario is appropriate for continuous integration servers or builds that performed at the command line using a script. It will also discuss strategies for signing your Android APK’s with custom keystores.
Limitations
At the present, Test Cloud is unable to run tests that involve the following scenarios or situations:
- Tests that run longer than 180 minutes
- Testing with Bluetooth connected
- Testing apps on network instead of Wifi
- Throttling WiFi
- Testing under different battery conditions
- Apps using the Camera
- Starting an app automatically rotated
- VPN into corporate network instead of opening up ports to the firewall
- Testing integration with other apps installed on the device
Requirements
It is assumed that UITests have been written and are running in your local environment, and that you have the test-cloud.exe application installed via NuGet.
It is assumed that a Test Cloud project has been created, your account has been added to the team for that project, and you have the Team API Key. Please see the Organizations & Teams document for more Details.
Familiarity with the command line is required.
Android applications must request the INTERNET permission in AndroidManifest.XML.
Firewall Configuration
In order for tests to be submitted to Xamarin Test Cloud, the computer submitting the tests must be able to communicate with the Test Cloud servers. Firewalls must be configured to allow network traffic to and from the servers located at testcloud.xamarin.com on ports 80 and 443. This endpoint is managed by DNS and the IP address is subject to change.
In some situations, a test (or a device running the test) must communicate with web servers protected by a firewall. In this scenario the firewall must be configured to allow traffic from the following IP addresses:
- 195.249.159.238
- 195.249.159.239
Submitting UITests
Tests are submitted to the using test-cloud.exe command line application, which is distributed as a part of the Xamarin.UITest NuGet package. This is a multi-purpose utility that will submit both Android and iOS tests, and it works on both OS X and Windows.
The following table describes some of the more important command line parameters for test-cloud.exe, which may be found in the packages directory as shown in the screenshot below. It is not a complete list of all parameters.
| Parameter | Description |
| --assembly-dir <PATH> | The path to the folder holding the test assemblies. |
| --devices <DEVICE ID> | The device ID that was provided in the Test Cloud Upload dialog. |
| --user <EMAIL> | The e-mail address of the team member submitting the tests. |
| --nunit-xml <FILENAME> | The filename to which test results are exported, formatted as NUnit results XML. Optional. |
| --sign-info <FILENAME> | Android only. Supply a signing information file that will be used to sign the Test Server APK. See the section below for more details. Optional. |
| --dsym <FILENAME> | iOS only. Will upload the dSYM files along with the application and tests. This allows for more detail in the log files. Optional. |
| --fixture <NUNIT-FIXTURE> | NUnit fixture / namespace to run. (Can be used multiple times) |
| --include <CATEGORY-NAME> | Identifies the NUnit test categories that should only be included in the test run. |
| --exclude <CATEGORY-NAME> | Identifies the NUnit test categories that should be excluded from the test run. |
| --test-chunk | Run tests in parallel by method |
| --fixture-chunk | Run tests in parallel by fixture |
| --data <FILE OR DIRECTORY> | Uploads file or directory along with assemblies. (Can be used multiple times). |
It is possible to see all the commmand line parameters available run of the two command lines from the solution directory:
- on OS X –
mono packages/Xamarin.UITest.X.X.X/tools/test-cloud.exe help submit - Windows –
packages\Xamarin.UITest.X.X.X\tools\test-cloud help submit
It is important that all commands be run from the solution directory. As an example, the following screenshot of the CreditCardValidation folder on Github identifies the solution project for the CreditCardValidation-UITest project:
Limitations
At the present, Test Cloud is unable to run tests that involve the following scenarios or situations:
- Tests that run longer than 180 minutes
- Testing with Bluetooth connected
- Testing apps on network instead of Wifi
- Throttling WiFi
- Testing under different battery conditions
- Apps using the Camera
- Starting an app automatically rotated
- VPN into corporate network instead of opening up ports to the firewall
- Testing integration with other apps installed on the device
Submitting iOS Tests
The following snippet is the simplest command line that could be used to submit an iOS application and tests to Test Cloud:
test-cloud.exe submit <PATH-TO-IPA> <TEAM API KEY> --devices=<DEVICES> --assembly-dir=<PATH-TO-TEST-ASSEMBLY-DIR> --user=<EMAIL>Submitting Android Tests
The following snippet is the simplest command line that could be used to submit an Android application and tests to Test Cloud:
test-cloud.exe submit <PATH-TO-APK> <TEAM API KEY> --devices=<DEVICES> --assembly-dir=<PATH-TO-TEST-ASSEMBLY-DIR> --user=<EMAIL>ℹ️Only release builds of an Android application should be submitted to Test Cloud.
test-cloud.exe will sign the application APK and the test server APK with the same keystore before uploading to Test Cloud. It is possible to control this signing step as described in the next section.
Signing Android APK's
test-cloud.exe will use one of three algorithms to sign the APKs prior to submission:
- Use Build-in Keystore. This is the default behaviour. test-cloud.exe will sign the application and the test server APK using it’s own keystore.
Provide your own Keystore. The second option is to provide your own keystore (along with credentials) that test-cloud.exe will use to sign the application and test server APK. The following snippet shows the additional command line parameters:
test-cloud.exe submit <PATH-TO-APK> <TEAM API KEY> keystore <PATH-TO-KEYSTORE> <KEYSTORE-PASSWORD> <KEYALIAS> <KEYALIAS-PASSWORD>
This option allows for complete control over what key will be used to sign the application, but it presents a potential security risk because the application keystore must be included in source code and the password for the keystore will be in clear text in build scripts.
- Use a signing information file. This option involves using test-cloud.exe to extract a few files from a signed test server APK, which are then stored in a small zip file, known as a signing information file. The main reason for using a signing information file is so that it can be added to source code control. This ensures that the application and test server are consistently signed with the same key information.
The following snippet shows how to create the signing information file:
test-cloud.exe gen-sign-info <PATH-TO-SIGNED_APK> <PATH_TO_KEYSTORE> <KEYSTORE-PASSWORD> <KEYALIAS> <KEYALIAS-PASSWORD> --dir <DIRECTORY-TO-SAVE-SIGN-INFO>When run, test-cloud.exe will create a file called testserver.si in the directory specified by the --dir parameter.
Summary
This guide discussed how to use test-cloud.exe to upload Xamarin.UITests and applications to Xamarin Test Cloud for testing. It also discusses several options for how to sign Android APKs prior to submitting test runs to Test Cloud.
Let us know how you feel about this.
0/250
Xamarin Workbook
If it's not already installed, install the Xamarin Workbooks app first. The workbook file should download automatically, but if it doesn't, just click to start the workbook download manually.