Reviewing the Report
The details of each test scenario is printed to the terminal (stdout) as it is run, and at the end of our test a summary is printed.
features/hello-world.feature:3 Hello world
1 feature passed, 1 failed, 0 skipped
8 scenarios passed, 1 failed, 0 skipped
26 steps passed, 1 failed, 0 skipped, 0 undefined
In this example we can see that a scenario called "Hello world" from the
hello-world.featurefile failed. It also points us to the line number where this scenario is defined in that file - line 3.
The summary also outlines how many features, scenarios and steps have passed, failed or been skipped. Followed by the duration of how long all of the tests took to run.
Let's now go take a look at that failing test.
If a scenario fails, the failing
Thenstatement will be highlighted in red, and a detailed description of the failure is provided. By default, the test runner will automatically retry the failing scenario a second time before moving on.
2020-03-20 19:11:14,392 | Voight Kampff | INFO | Starting tests for mycroft-hello-world
Feature: mycroft-hello-world # features/hello-world.feature:1
Scenario: Hello world # features/hello-world.feature:3
Given an english speaking user # features/steps/utterance_responses.py:108 0.001s
When the user says "Hello world" # features/steps/utterance_responses.py:113 0.001s
Then "mycroft-hello-world" should reply with "I'm doing very well" # features/steps/utterance_responses.py:137 10.019s
Assertion Failed: Mycroft responded with: mycroft-hello-world by mycroftai
"Hi to you too"
Matching with the dialog file: how.are.you.dialog
INFO:msm.mycroft_skills_manager:Best match (1.0): mycroft-hello-world by mycroftai
The example output above shows a test that has failed. The line containing the
Thenstatement would be highlighted in red, and it has provided a description of the error. In this case Mycroft responded with "Hi to you too" which is in the
hello.world.dialogfile in the
mycroft-hello-worldSkill. It then indicates that the response string provided in the test "I'm doing very well" was actually found in the
Looks like we have an error in our test. Mycroft should respond to "Hello World" with "Hi to you too" or another phrase from
The test results are also written to JSON files that can be used with Allure, an open source test reporting framework.
For further assistance with Skill testing, please post your question on the Community Forums or in the Skills channel on Mycroft Chat.