Combined Template

Complete template for the Skills Acceptance Process.

The overall Skills Acceptance Process is outlined in the Mycroft Documentation.

This template is provided to make it easier for testers to complete the process and provide relevant feedback to Skill Authors. An example of the completed process can be found on this Pull Request. If you have any other questions or comments, please raise them in the Skill Testing Team Chat.

The process has been split into three parts: 1. Code review - ensuring the code is stable and secure; 2. Information review - ensuring all info is accurate and understandable; and 3. Functional review - ensuring the experience of using the Skill is intuitive.

Testers can choose to conduct one or more of these reviews at a time, and once completed the review should be posted as a comment on the relevant Pull Request (PR). Once all reviews are completed with no outstanding Actions Required, the Skill will be merged into the repo and automatically added to the Mycroft Marketplace.

Instructions for use

Each post back to a PR must contain:

  • Meta block - detail on the tester, and test environment

  • One or more review blocks - 1. Code, 2. Information, and/or 3. Functional

  • Summary block - list of any Actions Required and additional comments

Each test starts with a checkbox. To mark a test successful, add an 'x' to the markdown checkbox like so:

  • Test that has not yet been conducted, or is not considered passed

  • Test that has been completed and passed

    Please add comments about the outcome of the test, including any output generated into these block quotes following each test.

Action Required: For any actions required by the Skill Author, please use this syntax and include it in the appropriate test, as well as in the summary section at the bottom.

Meta

  • Platform:

  • Mycroft-core version:

  • Who: @your-github-handle / @your-mycroft-chat-handle

  • Datestamp: run date +%Y-%m-%d_%H:%M:%S_%Z

  • Language and dialect of tester:

0. Automated tests

Are all automated tests passing?

  • Skill tester - Jenkins

  • Continuous Integration - Travis-CI

1. Code Review - secure and stable

  • Code Quality

    Can you understand what the code is doing? Is there inline documentation? Do you have any concerns about this code running on your machine? Are there any performance issues such as nested or infinite loops? Do you have significant concerns about the overall code quality?

    NOTE: We do not enforce PEP8 Checks on Skills

>

  • Error Handling

    Are there any specific checks we make for error handling or graceful degradation?

>

  • Libraries

    Does the Skill include the correct libraries? Does it use too many libraries or dependencies?

>

  • Required Dependencies

    Check requirements.txt and requirements.sh - are the required dependencies listed? If requirements.sh is used, is some form of conditional processing done to match against multiple distros? Often Skill Authors will add requirements.txt using only an “library=1.x.x” instead of “library >=1.x.x”. Check to make sure that there is an equal or greater than in the requirements to help future-proof the Skill, unless a specific version is needed.

>

  • Settings Is the settingsmeta file well laid out? If settings are not used, has the default file been deleted? If it is the default file, the first setting section will be called "Options << Name of section". >

  • Integration Tests Does the skill include sufficient integration tests, included in the test folder?

>

  • Other Files

    Are there any other files included that are unnecessary or you are unsure of their function?

>

2. Information Review - accurate and understandable

This review checks the README for completeness - does it follow the README template and include all the relevant sections such as Intents, known issues, dependencies and so on?

  • Icon

    Does the Skill have an appropriate icon that is either included in the repo or linked from an appropriate place (eg raw.githack.com not privateicons.com or rawgit.com)?

>

  • Description

    Are the title, short description, and long description (under 'About') clear and concise? Do they provide enough information to understand what the Skill does? Does the title have the word "skill" included? This is strongly discouraged.

>

  • Examples

    Do the examples give you a clear understanding of how you can use the Skill? Is there a single example per dot-point?

>

  • Supported Devices

    If relevant, are the supported devices listed? An example might be a Skill that requires the screen of the Mark II. If the section is not present, all devices are considered supported.

>

  • Categories

    Is there at least one category listed? At least one category must be selected for the Skill to be displayed correctly in the Mycroft Marketplace.

    Is the bolded category the most appropriate for this Skill? The bold category is considered the primary category and will be used for display in the Marketplace, all others are secondary and used for search results.

>

  • Tags

    Are listed tags appropriate for this Skill?

>

  • License

    Is an appropriate LICENSE file used in the repo - such as Apache-2.0 or GPL-3.0?

>

3. Functional Review - intuitive and expected

  • Installation

Check that the Skill installs using voice commands. Mycroft will get the user to confirm which Skill should be installed if there is ambiguity in Skill names - such as duplicate names. If possible, name the Skill so that there is minimal duplication and/or conflict. You should also verify that the Skill name can be verbally pronounced by speaking the Skill name into the Mycroft command line several times, and reading the resulting transcriptions. Suggest alternative Skill names if it is difficult to verbally pronounce the Skill name. Please provide confirmation that the Skill was successfully installed and by what means (voice or mycroft-msm install), as well as what utterance was detected when invoking the install voice command.

Install method: Output:

Checking that STT transcribes correctly:

  • Settings

    If Skill includes a settingsmeta file - are the settings well laid out? Does the placeholder text make sense? This can also be checked on home.mycroft.ai/#/skill

>

  • Dialog

Check the dialog directory to ensure that from a voice user interface perspective the dialogs read well. Alway play every dialog phrase on the command line by doing say so that you can check how the dialog is spoken. It is a good idea to run the dialog phrases through mimic.

Sometimes the dialog files will need a small tweak such as a space between words, or extra vowel sounds, to sound realistic. Sometimes words don't render as expected and alternative wording should be used. Some of the tricks you might need to use include separating words by a space - such as sub sonic instead of subsonic, or broad cast instead of broadcast.

>

Skill Functions

For each function of the Skill add a new checkbox with the utterance used to invoke the functionality. Confirm the output and behaviour of each. If any setup is required to perform these tests please indicate this directly before the test is described.

  • "Are there unread messages on Mycroft Chat"

>

  • "Name Mycroft Chat channels with unread messages"

>

  • "Read all unread Mycroft Chat messages"

>

  • "Read messages for the channel {name}"

>

  • "Begin monitoring of Mycroft Chat"

>

  • "Stop monitoring of Mycroft Chat"

>

Summary

A short list of any Actions Required. It is also great to provide a short statement of your impressions. *

Once you have completed any updates, you can update this pull request by running mycroft-msk submit /path/to/skill/directory