mycroft-skills repo and wish to run tests to ensure they are still passing, you can add the comment Run test to the PR and this will automatically initiate a Jenkins CI runthrough of the tests in the Skill.mycroft-core package. It consists of the following files in mycroft-core/test/integrationtests/skills:discover_tests.pyskill_tester.pyrunner.pymessage_tester.pytest_all_skills.pyrunner.py can be copied to the Skill Author's working directory, where the Skill's __init__.py file exists or invoked directly with the skill path as argument. Running runner.py will test only the Skills it finds in the directory it is in, or, if it can’t find a Skill, it will search any subdirectory.discover_test.py is the Python file that runs integration tests on all Skills in /opt/mycroft/skills. It is intended for debugging that all your tests are found by the test runner.message_tester.py is a utility that can test a single message against the internal rule format used by the skill_tester. It is intended for debugging rules.test_all_skills.py tests all Skills at the Skill level, where discover_tests.py tests at the Intent level. Because of that, test_all_skills.py not only runs all the Intent tests, it also determines if all Intents in a Skill are tested, i.e. if any test cases are missing. It prints a list of missing test cases for each Skill, and fails if it finds any.test/intent directory. Each test case corresponds to one Utterance. All parts of the test case are optional, except the Utterance. The test case files are executed by the Integration Test Runner, one after another, and they are executed in alphabetical order. Alphabetic ordering can be used for compensating (trans)actions, for instance first add to a list, then remove from a list, to leave the list unchanged.test/intent directory are written in JSON format, and must be named to match the pattern *.intent.jsonJSON keywords:__init__.py code, is usedregex/en-us directory is: add (?P.+) to (?P.+) list$AddTaskToListKeyword is Add, defined in the vocab/en-us directory.add milk to the grocery list
UndoContext and the ConfirmContext may have been set, but to be sure they are removed, we remove them before the test starts.set_context can be used to test Intents that require a context. In the example above the _TestRunner context is set to "data". When the Skill has side effects that are unwanted when testing, this trick allows us to test for the _TestRunner context in the Intent, and behave accordingly, for instance not executing code with side effects. Of course, other tests are then required to test the inner working of the Intent.intent_type is used to verify that the Utterance is passed to the right Intent, hence it must the IntentBuilder() parameter. In this case AddTaskToListIntent.intent is a list of key/values that must match the IntentBuilder().require() or similar parameters. This is used for testing that the right part of the Utterance is passed as parameters to the Skill.expected response is a regular expression that must match the answer that is Spoken by the Intent.changed_context is a list of contexts, that the Intent has set or removed. It is not possible to distinguish between set or remove context.expected_data can be used to check for specific data content, for example the content of a message parsed with Padatious. The example test case below will pass if a message contains an "ampm" value equal to "pm" and a "time" value equal to 6. Note that the "ampm" value is a string literal, and is quoted, while the "time" value is an integer value and is not quoted.expected_dialog takes the dialog file (without the .dialog) in the same manner as when using the dialog in the Skill. See skill-personal for an example.changed_context and assert actually does the same thing, it is mentioned as an example only. The assert shows the internal rule format (see the next paragraph).assert keyword. The standard keywords and values are translated into the internal rule format by the Integration Test Runner as well.changed_context and assert, one of which is of course not needed.discover_tests.py is intended to be run like a Python unit test, please refer to https://docs.python.org/2/library/unittest.html. Most IDEs have an easy way to run unit tests, and create nice structured test reports.runner.py is intended to run with a skill directory as parameter,__init__.py file. When it does, it will not traverse further down from that directory, but it will search sibling directories. In effect, it will only test one Skill, if it is run in that Skill's directory. runner.py is an excellent tool when developing a Skill and wanting to run only the tests for that Skill.['endsWith', 'intent_type', 'PairingIntent', 'succeeded'] which means that the Intent type was found, since “succeeded” was appended to that rule part. However, we did not find the DevicePairingPhrase in the same message. We expect the DevicePairingPhrase in the same message because of the “and” operator. And if we look at the messages in the log above, we can’t find a message where the DevicePairingPhrase appear.message_tester.py tool is able to evaluate one message event at a time, for instance:mycroft-core or other Mycroft software, then the Skills Management Team can choose to override the need for automated tests to have passed when they assess the Skill.