The details of each test scenario is printed to the terminal (stdout) as it is run, and at the end of our test a summary is printed.
Failing scenarios:features/hello-world.feature:3 Hello world1 feature passed, 1 failed, 0 skipped8 scenarios passed, 1 failed, 0 skipped26 steps passed, 1 failed, 0 skipped, 0 undefinedTook 0m14.191s
In this example we can see that a scenario called "Hello world" from the
hello-world.feature file failed. It also points us to the line number where this scenario is defined in that file - line 3.
The summary also outlines how many features, scenarios and steps have passed, failed or been skipped. Followed by the duration of how long all of the tests took to run.
Let's now go take a look at that failing test.
If a scenario fails, the failing
Then statement will be highlighted in red, and a detailed description of the failure is provided. By default, the test runner will automatically retry the failing scenario a second time before moving on.
2020-03-20 19:11:14,392 | Voight Kampff | INFO | Starting tests for mycroft-hello-worldFeature: mycroft-hello-world # features/hello-world.feature:1Scenario: Hello world # features/hello-world.feature:3Given an english speaking user # features/steps/utterance_responses.py:108 0.001sWhen the user says "Hello world" # features/steps/utterance_responses.py:113 0.001sThen "mycroft-hello-world" should reply with "I'm doing very well" # features/steps/utterance_responses.py:137 10.019sAssertion Failed: Mycroft responded with: mycroft-hello-world by mycroftaiMycroft: hello.world.dialog(HelloWorldSkill)"Hi to you too"Captured stdout:Matching with the dialog file: how.are.you.dialogCaptured logging:INFO:msm.mycroft_skills_manager:Best match (1.0): mycroft-hello-world by mycroftai
The example output above shows a test that has failed. The line containing the
Then statement would be highlighted in red, and it has provided a description of the error. In this case Mycroft responded with "Hi to you too" which is in the
hello.world.dialog file in the
mycroft-hello-world Skill. It then indicates that the response string provided in the test "I'm doing very well" was actually found in the
Looks like we have an error in our test. Mycroft should respond to "Hello World" with "Hi to you too" or another phrase from
The test results are also written to JSON files that can be used with Allure, an open source test reporting framework.
See our tips for how to ask the best questions. This helps you get a more complete response faster.