mycroft-skillsrepo and wish to run tests to ensure they are still passing, you can add the comment
Run testto the PR and this will automatically initiate a Jenkins CI runthrough of the tests in the Skill.
mycroft-corepackage. It consists of the following files in
runner.pycan be copied to the Skill Author's working directory, where the Skill's
__init__.pyfile exists or invoked directly with the skill path as argument. Running
runner.pywill test only the Skills it finds in the directory it is in, or, if it can’t find a Skill, it will search any subdirectory.
discover_test.pyis the Python file that runs integration tests on all Skills in
/opt/mycroft/skills. It is intended for debugging that all your tests are found by the test runner.
message_tester.pyis a utility that can test a single message against the internal rule format used by the
skill_tester. It is intended for debugging rules.
test_all_skills.pytests all Skills at the Skill level, where
discover_tests.pytests at the Intent level. Because of that,
test_all_skills.pynot only runs all the Intent tests, it also determines if all Intents in a Skill are tested, i.e. if any test cases are missing. It prints a list of missing test cases for each Skill, and fails if it finds any.
test/intentdirectory. Each test case corresponds to one Utterance. All parts of the test case are optional, except the Utterance. The test case files are executed by the Integration Test Runner, one after another, and they are executed in alphabetical order. Alphabetic ordering can be used for compensating (trans)actions, for instance first add to a list, then remove from a list, to leave the list unchanged.
test/intentdirectory are written in
JSONformat, and must be named to match the pattern
__init__.pycode, is used
add (?P.+) to (?P.+) list$
Add, defined in the
add milk to the grocery list
ConfirmContextmay have been set, but to be sure they are removed, we remove them before the test starts.
set_contextcan be used to test Intents that require a context. In the example above the
_TestRunnercontext is set to "data". When the Skill has side effects that are unwanted when testing, this trick allows us to test for the
_TestRunnercontext in the Intent, and behave accordingly, for instance not executing code with side effects. Of course, other tests are then required to test the inner working of the Intent.
intent_typeis used to verify that the Utterance is passed to the right Intent, hence it must the
IntentBuilder()parameter. In this case
intentis a list of key/values that must match the
IntentBuilder().require()or similar parameters. This is used for testing that the right part of the Utterance is passed as parameters to the Skill.
expected responseis a regular expression that must match the answer that is Spoken by the Intent.
changed_contextis a list of contexts, that the Intent has set or removed. It is not possible to distinguish between set or remove context.
expected_datacan be used to check for specific data content, for example the content of a message parsed with Padatious. The example test case below will pass if a message contains an "ampm" value equal to "pm" and a "time" value equal to 6. Note that the "ampm" value is a string literal, and is quoted, while the "time" value is an integer value and is not quoted.
assertactually does the same thing, it is mentioned as an example only. The
assertshows the internal rule format (see the next paragraph).
assertkeyword. The standard keywords and values are translated into the internal rule format by the Integration Test Runner as well.
assert, one of which is of course not needed.
discover_tests.pyis intended to be run like a Python unit test, please refer to https://docs.python.org/2/library/unittest.html. Most IDEs have an easy way to run unit tests, and create nice structured test reports.
runner.pyis intended to run with a skill directory as parameter,
__init__.pyfile. When it does, it will not traverse further down from that directory, but it will search sibling directories. In effect, it will only test one Skill, if it is run in that Skill's directory.
runner.pyis an excellent tool when developing a Skill and wanting to run only the tests for that Skill.
['endsWith', 'intent_type', 'PairingIntent', 'succeeded']which means that the Intent type was found, since “succeeded” was appended to that rule part. However, we did not find the
DevicePairingPhrasein the same message. We expect the
DevicePairingPhrasein the same message because of the “and” operator. And if we look at the messages in the log above, we can’t find a message where the
message_tester.pytool is able to evaluate one message event at a time, for instance:
mycroft-coreor other Mycroft software, then the Skills Management Team can choose to override the need for automated tests to have passed when they assess the Skill.