Your Alexa Dashboards
Sign In
?
Sign In
Your Alexa Dashboards Support

Functional Testing for a Custom Skill

The functional tests verify that:

  • The skill’s basic functionality matches the information displayed on the skill’s detail card in the Amazon Alexa app. You enter this data on the Publishing Information page for the skill on the developer portal.

    See Define Skill Metadata for more details about defining this card.

  • The skill’s core functionality works and provides useful home cards to the Amazon Alexa app.

Use the Amazon Alexa app for these tests. The Alexa app is available for Fire OS, Android, iOS, and web browsers. The Alexa app displays all of your skills currently under development, as well as all published skills.

To return to the high-level testing checklist, see Certification Requirements for Custom Skills.

3.1. Review and Test Example Phrases

Example phrases are shown on the skill’s detail card in the Alexa app to help users understand how to interact with the skill. You select these phrases from your sample utterances to ensure accuracy.

These are the phrases users are most likely to try the first time they interact with the skill. Therefore, make sure that they work well and provide a good user experience.

Detail Card for a Skill in the Alexa App
Detail Card for a Skill in the Alexa App
Test Expected Results

1.

Open the Alexa app, click Skills and then scroll or search to find your skill.

Review the skill’s detail page. Inspect all example phrases listed in the skill’s detail card.
  • At least one example phrase has been provided.
  • The first phrase clearly indicates how to begin using the skill, and includes both the wake word and invocation name.
  • Example phrases containing the wake word and invocation name must use a supported format for invoking a skill. For a list of supported formats, see Understanding How Users Invoke Custom Skills.
  • The example phrases do not contain any additional information that is not intended to be spoken by users. The example phrases must function exactly as they are written.
  • The invocation name uses title case.
  • The example phrases do not contain any emoticons, symbols, or grammatical errors.
  • Each example phrase ends with punctuation. Exclamation points are not used, as they may misguide users into thinking they must express enthusiasm.
  • The example phrases show how to launch the skill. They do not illustrate enabling the skill by voice. For example, the phrase must not be something like “Alexa, enable <skill name>.”
  • The example phrases are all written in the same language used by the Alexa account. For instance, when using an account configured with German, the phrases are written in German.

2.

Open the developer portal, edit the skill, and go to the Interaction Model page. Review the sample utterances for your skill and verify that each example phrase is included as a sample utterance.

For more about defining example phrases, see the “Defining your Example Phrases (“Try Saying”)” section of Define Skill Metadata.

If you are using the utterance for a built-in intent for an example phrase, the utterance does not need to be in your sample utterances. However, fully test the utterance and verify that your skill returns a valid response.

Each example phrase is explicitly included in the sample utterances.

For instance, if you have this example phrase:

  • Alexa, ask Tide Pooler when is high tide in Seattle.

Then you must also have this utterance:

  • OneshotTideIntent when is high tide in {City}

Any slots in the example phrase (such as Seattle in this example) must be filled with an example of a valid slot value. In this example, City is a custom slot type, and “Seattle” is a value explicitly defined for the City type.

3.

Launch the skill using each of the provided example phrases.

Each example phrase renders a contextualized, non-error response.

3.2. Skill Description and Detail Page

The skill description teaches users about what your skill can do and how to use it.

Test Expected Results

1.

Open the Alexa app, click Skills and then scroll to your skill. Review the short description shown in the list of skills
  • The short description describes the skill’s core functionality.
  • The short description is written in the same language used by the Alexa account. For instance, when using an account configured with German, the skill short description is displayed in German.

2.

Open the detail card for the skill. Review the skill’s full description shown on the skill’s detail card.
  • The skill description captures any prerequisites needed to use the skill (such as additional hardware, software, or accounts).
  • The description describes the skill’s core functionality.
  • The description does not misrepresent the skill’s functionality.
  • The description is written in the same language used by the Alexa account. For instance, when using an account configured with German, the skill description is displayed in German.
  • If the skill integrates with Raspberry Pi or a similar device, the description must:
    1. Mention that a Raspberry Pi (or similar device) is required to access core functionality of the skill.
    2. Include links to documentation explaining how to configure the Raspberry Pi with the skill.
  • The description must not claim that the skill has any anti-virus features.

3.

If either the short description or full description includes any URLs (such as directing users to more information about the skill), visit the URL in a browser and verify the page opens without error.
  • The page referenced by the description opens without error.
  • The page is displayed in the same language used by the Alexa account. For instance, when using an account configured with German, the page should display in German.

4.

If your skill’s detail page includes links to a privacy policy or terms of use, open the links in a browser.
  • The privacy policy / terms of use links shown on the skill detail page open without error.
  • The pages are displayed in the same language used by the Alexa account. For instance, when using an account configured with German, the privacy policy and terms of use must be in German.

5.

Review the skill title and icon and ensure they are relevant to the core functionality of the skill.

Both the title and icon are relevant to the skill description and are not misleading.

3.3. Home Cards

Home cards are graphical cards displayed in the Alexa app to describe or enhance the voice interaction.

Test Expected Results

1.

Walk through the skill’s functionality, invoking each intent. For each response that includes a home card, review the card in the Alexa app.

For each card:

  • The card title and content are appropriate in the context of the skill’s functionality.
  • The card title and content do not contain any code references or grammatical errors.
  • The card title and content are written in a language supported by Alexa. The current supported languages are English and German.
  • If the card includes an image, the image is correctly displayed.

2.

If any home cards include URLs, visit each URL in a browser and verify the page opens without error.

For each URL displayed in a home card:

  • The page opens in a browser without error.
  • The page is displayed in a language supported by Alexa. The current supported languages are English and German.

Note that skills normally only return home cards when responding with the information the user requested. Supportive prompts to ask the user for more information do not normally include home cards.

Home cards can be a useful tool when developing and debugging a skill, but be sure to remove any extraneous debug-type cards before submitting the skill.

3.4. Account Linking

If your skill needs to connect the identity of the end user with a user in another system (account linking), this test verifies that the skill implements the solution described in Linking an Alexa User with a User in Your System correctly. When submitting your skill, be sure to provide a valid set of account credentials with your testing instructions so our certification team can verify the account linking and functionality of your skill.

If your skill does not use account linking, you can skip this test.

Test Expected Results

1.

In the developer portal, review the Configuration page for the skill and verify that account linking has been configured correctly.
  • The Account Linking or Creation option is set to Yes.
  • The Authorization URL is set to the login page for your service.

2.

In the Alexa app, review the detail card for your skill. Verify that the skill description explains that you must log in to the other system to use the skill.

The description clearly explains the steps and information necessary to successfully link the accounts.

3.

Log in to the Alexa app as an Alexa user who has not yet enabled the skill.

Enable the skill, but do not complete the account linking process, leaving the skill enabled but not linked.

  1. Invoke any intents that do not require authentication and verify that they work.
  2. Invoke the intents that do require authentication and verify the response.
  • Each intent that does not require authentication works correctly even when accounts are not linked.
  • Each intent that does require authentication returns output speech telling the user what to do, and a link account card. Verify the card in the Alexa app.

4.

Disable the skill or log in to the Alexa app as an Alexa user who has not yet enabled the skill.

Enable the skill. Once the login page for your skill is displayed, click all links and verify that they navigate to new pages.
  • All clickable links on the login page navigate to new pages.
  • None of the clickable links open JavaScript alerts or popup windows.

5.

Disable the skill or log in to the Alexa app as an Alexa user who has not yet enabled the skill.

Enable the skill. Once the login page for your skill is displayed, log in and verify the account linking flow.
  • The login page for your skill is displayed correctly.
  • You can successfully log in using an account with the service.
  • After connecting the accounts, you can successfully invoke each intent that requires authentication.

3.5. Audio Playback

Complete these tests if your skill uses the AudioPlayer interface to stream audio.

Test Expected Results

1.

Set the volume on the device to 3 and invoke an intent that streams the audio. Listen for several minutes.
  • The audio playback is audible at volume level 3.
  • The volume of the audio playback does not vary significantly from normal Alexa text-to-speech. The volume remains consistent while playing for several minutes.
  • If the audio content includes words or speech, the content is spoken in a language supported by Alexa. The current supported languages are English and German.

2.

While listening to the audio playback, invoke the following built-in intents:

  • AMAZON.PauseIntent (say “pause”)
  • AMAZON.ResumeIntent (say “resume”)
  • The “pause” utterance stops the audio playback.
  • The “resume” utterance resumes the audio playback.

3.

While listening to the audio playback, use a remote control or other hardware device to pause and resume the audio.
  • Pressing a pause button while audio is playing stops the playback.
  • Pressing a play button while the audio is stopped resumes the playback.

4.

While listening to the audio playback, invoke the following built-in intents:

  • AMAZON.CancelIntent (say “cancel”)
  • AMAZON.LoopOffIntent (say “loop off”)
  • AMAZON.LoopOnIntent (say “loop on”)
  • AMAZON.NextIntent (say “next”)
  • AMAZON.PreviousIntent (say “previous”)
  • AMAZON.RepeatIntent (say “repeat that”)
  • AMAZON.ShuffleOffIntent(say “shuffle off”)
  • AMAZON.ShuffleOnIntent (say “shuffle”)
  • AMAZON.StartOverIntent (say “start over”)
  • AMAZON.StopIntent (say “stop”)

Each command either does something that makes sense for your skill (such as “next” advancing to the next track in a playlist) or is handled gracefully. No unexpected errors occur. See Configuring Your Skill for the AudioPlayer Directives.

3.6 Screen Display and Navigation

Complete these tests if your skill uses the Display interface to display content on Echo Show. Pay particular attention to the interaction of voice, touch, audio, and video in your skill.

Test Expected Results

1.

Open the skill using Echo Show and interact with the skill for several minutes, paying particular attention to the user experience.
  • All GUI content, whether in the form of static templates or video, must load within 10 seconds of the triggering voice or touch input.
  • GUI content (images and video) should not look stretched, grainy or pixelated.
  • The GUI content must not contradict the voice content (whether text-to-speech or audio) that is delivered alongside the GUI content. The GUI content does not need to match exactly, but must complement the voice content.
  • All GUI content should be relevant to the triggering voice input or touch input.
  • GUI content should not contain more than three actions per screen, if the screen is not displaying a list.
  • All actions present on the GUI should also be accessible via voice requests, meaning these actions are implemented as intents and have corresponding sample utterances in the language model.

2.

Place the Echo Show device at a 7-foot distance from the viewer. Interact with the skill for several minutes.
  • All GUI content should be readable from a 7-foot distance.

3.7. Video Playback

In addition to the screen display requirements, some of which also apply to video, also complete these tests if your skill uses the VideoApp interface to stream video.

Test Expected Results

1.

Set the volume on the device to 3 and invoke an intent that streams the video. Listen for several minutes.
  • The video playback is audible at volume level 3.
  • The volume of the audio playback does not vary significantly from normal Alexa text-to-speech. The volume remains consistent while playing for several minutes.
  • If the audio content includes words or speech, the content is spoken in a language supported by Alexa. The current supported languages are English and German.
  • Buffering during video playback should not exceed 10% of the total video playback time. For example, for every minute of video playback, buffering must not exceed 6 seconds.
  • During video playback, the audio should be in sync with the video.
  • The video is clearly visible and not grainy, pixelated, or distorted.

2.

While the video plays, use the progress bar on the touchscreen to navigate forward and backward.

Verify that the video resumes without error.

3.8. Duplicates of Samples or Templates

Make sure your skill is not an exact duplicate of any of the Amazon-provided samples or templates. You can use the samples and templates as a starting point for your skill, but the finished skill must be different.

For example, if you use the Trivia sample to create a trivia skill, your skill must have:

  • A different name
  • A different set of questions

The samples and templates are available on GitHub:

3.9 Device Location API

Complete these tests if your skill uses the Device Address API to obtain the location of the user’s device.

Test Expected Results

1.

Enable the skill and grant the skill the requested permissions for the device location. Test the functionality that is dependent on the device location and note the responses.
  • All skill responses are appropriate with regards to the device location. For example, a weather skill that uses the device location is expected to give weather information for the city specified in the device location (unless the user provides a different location in an utterance).
  • If the skill communicates the device location back to the user, it matches the specified device location.

2.

Enable the skill, but do not grant the device location permissions.

  1. Invoke any intents that do not require the location and verify that they work.
  2. Invoke the intents that do require the location and verify the response.
  • Each intent that does not require the device location works correctly even the user has not granted the permissions.
  • Each intent that does require the device location returns output speech telling the user to grant the permissions, and a permissions card. Verify the card in the Alexa app.

3.

If applicable, change the device location to a location or address that is not supported by the skill, then enable the skill and grant permissions. Invoke the intents that require the device location.

For example, if your skill provides food delivery in a particular city, set the device location to an address far outside your delivery area, then invoke the intents to place a delivery order.

The skill informs the user that skill does not support their location. No errors are returned.

Next Steps