GUI Framework
A visual and display framework for Mycroft running on top of KDE Plasma Technology and built using Kirigami, a lightweight user interface framework for convergent applications empowered by Qt.
In the age of information visualization is eminently essential to grab attention and create a promising communication strategy. Visual content that supports your spoken content can make it easier to present information well and more engaging for your audience and users.

Introduction

Mycroft-GUI is an open source visual and display framework for Mycroft running on top of KDE Plasma Technology and built using Kirigami a lightweight user interface framework for convergent applications which are empowered by Qt.

Getting Started

Mycroft is an open source voice assistant that can be extended and expanded to the limits of your imagination. Mycroft can run anywhere from your desktop to your automobiles or on smart devices that empower your home.
Want Mycroft to do something new? Teach Mycroft a skill, share it, and improve the experience for tens of thousands of people all over the world. This guide aims to provide you with resources to create familiar and consistent visual experiences with your expanding and innovative skills.

VISUAL SKILL DEVELOPMENT API FOUNDATION

Mycroft enabled devices with displays such as the Mark II, KDE Plasmoid provide skill developers the opportunity to create skills that can be empowered by both voice and screen interaction. The display interaction technology is based on the QML user interface markup language that gives you complete freedom to create in-depth innovative interactions without boundaries or provide you with simple templates within the Mycroft GUI framework that allow minimalistic display of text and images based on your skill development specifics and preferences.
This section of the guide is divided into two skill examples that will show you how to create:
  • In-depth QML based audio and visual interaction skills
  • Simple template based text and image skills

In-depth QML based audio and visual interaction skills

QML user interface markup language is a declarative language built on top of Qt's existing strengths designed to describe the user interface of a program: both what it looks like, and how it behaves. QML provides modules that consist of sophisticated set of graphical and behavioral building elements. In the example below we will showcase how to create a QML interface for your skill including how it interacts with your voice skill.

Before Getting Started Resources

A collection of resources to familiarize you with QML and Kirigami Framework.

Building your skill to support display

Skills for Mycroft AI are written in Python, using the skills development guide available here
Let's walk you through some basics of writing your QML user interface, this section is divided into 5 parts:

Importing Modules

A QML module provides versioned types and JavaScript resources in a type namespace which may be used by clients who import the module. Modules make use of the QML versioning system which allows modules to be independently updated. More in-depth information about QML modules can be found here Qt QML Modules Documentation
In the code snippet example below we will look at importing some of the common modules that provide the components required to get started with our Visual User Interface.
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
import org.kde.lottie 1.0
Copied!
QTQuick Module:
Qt Quick module is the standard library for writing QML applications, the module provides a visual canvas and includes types for creating and animating visual components, receiving user input, creating data models and views and delayed object instantiation. In-depth information about QtQuick can be found at Qt Quick Documentation
QTQuick.Controls Module:
The QtQuick Controls module provides a set of controls that can be used to build complete interfaces in Qt Quick. Some of the controls provided are button controls, container controls, delegate controls, indicator controls, input controls, navigation controls and more, for a complete list of controls and components provided by QtQuick Controls you can refer to QtQuick Controls 2 Guidelines
QtQuick.Layouts Module:
QtQuick Layouts are a set of QML types used to arrange items in a user interface. Some of the layouts provided by QtQuick Layouts are Column Layout, Grid Layout, Row Layout and more, for a complete list of layouts you can refer to QtQuick Layouts Documentation
Kirigami Module:
Kirigami is a set of QtQuick components for mobile and convergent applications. Kirigami is a set of high level components to make the creation of applications that look and feel great on mobile as well as desktop devices and follow the Kirigami Human Interface Guidelines
Mycroft Module:
Mycroft GUI frameworks provides a set of high level components and events system for aiding in the development of Mycroft visual skills. One of the controls provided by Mycroft GUI frameworks are Mycroft-GUI Framework Base Delegates Mycroft-GUI Framework Base Delegates Documentation
QML Lottie Module:
This provides a QML Item to render Adobe® After Effects™ animations exported as JSON with Bodymovin using the Lottie Web library. For list of all properties supported refer Lottie QML

Using Mycroft-GUI Framework Base Delegates

When you design your skill with QML, Mycroft-GUI frameworks provides you with some base delegates you should use when designing your GUI skill. The base delegates provide you with a basic presentation layer for your skill with some property assignments that can help you setup background images, background dim, timeout and grace time properties to give you the control you need for rendering an experience. In your GUI Skill you can use:
  • Mycroft.Delegate: A basic and simple page based on Kirigami.Page
    Simple display Image and Text Example using Mycroft.Delegate
    1
    import Mycroft 1.0 as Mycroft
    2
    3
    Mycroft.Delegate {
    4
    skillBackgroundSource: sessionData.exampleImage
    5
    ColumnLayout {
    6
    anchors.fill: parent
    7
    Image {
    8
    id: imageId
    9
    Layout.fillWidth: true
    10
    Layout.preferredHeight: Kirigami.Units.gridUnit * 2
    11
    source: "https://source.unsplash.com/1920x1080/?+autumn"
    12
    }
    13
    Label {
    14
    id: labelId
    15
    Layout.fillWidth: true
    16
    Layout.preferredHeight: Kirigami.Units.gridUnit * 4
    17
    text: "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."
    18
    }
    19
    }
    20
    }
    Copied!
  • Mycroft.ScrollableDelegate: A delegate that displays skill visuals in a scroll enabled Kirigami Page.
    Example of using Mycroft.ScrollableDelegate
    1
    import QtQuick 2.4
    2
    import QtQuick.Controls 2.2
    3
    import QtQuick.Layouts 1.4
    4
    import org.kde.kirigami 2.4 as Kirigami
    5
    import Mycroft 1.0 as Mycroft
    6
    7
    Mycroft.ScrollableDelegate{
    8
    id: root
    9
    skillBackgroundSource: sessionData.background
    10
    property var sampleModel: sessionData.sampleBlob
    11
    12
    Kirigami.CardsListView {
    13
    id: exampleListView
    14
    Layout.fillWidth: true
    15
    Layout.fillHeight: true
    16
    model: sampleModel.lorem
    17
    delegate: Kirigami.AbstractCard {
    18
    id: rootCard
    19
    implicitHeight: delegateItem.implicitHeight + Kirigami.Units.largeSpacing
    20
    contentItem: Item {
    21
    implicitWidth: parent.implicitWidth
    22
    implicitHeight: parent.implicitHeight
    23
    ColumnLayout {
    24
    id: delegateItem
    25
    anchors.left: parent.left
    26
    anchors.right: parent.right
    27
    anchors.top: parent.top
    28
    spacing: Kirigami.Units.largeSpacing
    29
    Kirigami.Heading {
    30
    id: restaurantNameLabel
    31
    Layout.fillWidth: true
    32
    text: modelData.text
    33
    level: 2
    34
    wrapMode: Text.WordWrap
    35
    }
    36
    Kirigami.Separator {
    37
    Layout.fillWidth: true
    38
    }
    39
    Image {
    40
    id: placeImage
    41
    source: modelData.image
    42
    Layout.fillWidth: true
    43
    Layout.preferredHeight: Kirigami.Units.gridUnit * 3
    44
    fillMode: Image.PreserveAspectCrop
    45
    }
    46
    Item {
    47
    Layout.fillWidth: true
    48
    Layout.preferredHeight: Kirigami.Units.gridUnit * 1
    49
    }
    50
    }
    51
    }
    52
    }
    53
    }
    54
    }
    Copied!

Using Mycroft Framework Components

Simple template based text and image skill displays

Designing a simple skill and only want to display text or images ? Mycroft GUI framework and Mycroft enclosure API provides ready to use QML based template wrappers that can minimalisticly display simple skills data such as text and images. In the example below we will showcase how to create a simple voice skill that displays simple text on your Mycroft enabled device with a display.
Text Example:
1
...
2
def handle_hello_world(self, message):
3
...
4
self.gui.show_text("Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmo tempor incididunt ut labore et dolore magna aliqua.")
5
...
Copied!
Image Example
1
...
2
def handle_hello_world(self, message):
3
...
4
self.gui.show_image("https://source.unsplash.com/1920x1080/?+autumn")
5
...
Copied!
HTML URL Example
1
...
2
def handle_hello_world(self, message):
3
...
4
self.gui.show_url("https://mycroft.ai")
5
...
Copied!
HTML Raw Example
1
...
2
def handle_hello_world(self, message):
3
...
4
rawhtmlexample = """<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
5
<html xmlns="http://www.w3.org/1999/xhtml">
6
<head>
7
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
8
<title>Untitled Document</title>
9
<body>
10
<h1> HTML Example </h1>
11
<p> This is an example of an HTML webpage. </p>
12
<p> <b>Tags</b> can be wrapped <i>inside other tags!</i> </p>
13
14
<p>
15
HTML doesn't care about extra spaces, tabs or newlines,
16
so we can use indentation and spacing to keep everything
17
lined up nicely.
18
</p>
19
20
<ul>
21
<li> This is how you create a bulleted list! </li>
22
<li> Item 2 </li>
23
<li> Item 3 </li>
24
</ul>
25
</body>
26
</html>
27
"""
28
self.gui.show_html(rawhtmlexample)
29
...
Copied!

Advanced skill displays using QML

Display Lottie Animations:
You can use the LottieAnimation item just like any other QtQuick element, such as an Image and place it in your scene any way you please.
QML Example
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
import org.kde.lottie 1.0
7
8
Mycroft.Delegate {
9
LottieAnimation {
10
id: fancyAnimation
11
anchors.fill: parent
12
source: Qt.resolvedUrl("animations/fancy_animation.json")
13
loops: Animation.Infinite
14
fillMode: Image.PreserveAspectFit
15
running: true
16
}
17
}
Copied!
Display Sliding Images
Contains an image that will slowly scroll in order to be shown completely
QML Example
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
7
Mycroft.Delegate {
8
background: Mycroft.SlidingImage {
9
source: "foo.jpg"
10
running: bool //If true the sliding animation is active
11
speed: 1 //Animation speed in Kirigami.Units.gridUnit / second
12
}
13
}
Copied!
Display Paginated Text
Takes a long text and breaks it down into pages that can be horizontally swiped
QML Example
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
7
Mycroft.Delegate {
8
Mycroft.PaginatedText {
9
text: string //The text that should be displayed
10
currentIndex: 0 //The currently visible page number (starting from 0)
11
}
12
}
Copied!
Display A Vertical ListView With Information Cards
Kirigami CardsListView is a ListView which can have AbstractCard as its delegate: it will automatically assign the proper spacing and margins around the cards adhering to the design guidelines.
Python Skill Example
1
...
2
def handle_food_places(self, message):
3
...
4
self.gui["foodPlacesBlob"] = results.json
5
self.gui.show_page("foodplaces.qml")
6
...
Copied!
QML Example
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
7
Mycroft.Delegate{
8
id: root
9
property var foodPlacesModel: sessionData.foodPlacesBlob
10
11
Kirigami.CardsListView {
12
id: restaurantsListView
13
Layout.fillWidth: true
14
Layout.fillHeight: true
15
model: foodPlacesModel
16
delegate: Kirigami.AbstractCard {
17
id: rootCard
18
implicitHeight: delegateItem.implicitHeight + Kirigami.Units.largeSpacing
19
contentItem: Item {
20
implicitWidth: parent.implicitWidth
21
implicitHeight: parent.implicitHeight
22
ColumnLayout {
23
id: delegateItem
24
anchors.left: parent.left
25
anchors.right: parent.right
26
anchors.top: parent.top
27
spacing: Kirigami.Units.smallSpacing
28
Kirigami.Heading {
29
id: restaurantNameLabel
30
Layout.fillWidth: true
31
text: modelData.name
32
level: 3
33
wrapMode: Text.WordWrap
34
}
35
Kirigami.Separator {
36
Layout.fillWidth: true
37
}
38
RowLayout {
39
Layout.fillWidth: true
40
Layout.preferredHeight: form.implicitHeight
41
Image {
42
id: placeImage
43
source: modelData.image
44
Layout.fillHeight: true
45
Layout.preferredWidth: placeImage.implicitHeight + Kirigami.Units.gridUnit * 2
46
fillMode: Image.PreserveAspectFit
47
}
48
Kirigami.Separator {
49
Layout.fillHeight: true
50
}
51
Kirigami.FormLayout {
52
id: form
53
Layout.fillWidth: true
54
Layout.minimumWidth: aCard.implicitWidth
55
Layout.alignment: Qt.AlignLeft | Qt.AlignBottom
56
Label {
57
Kirigami.FormData.label: "Description:"
58
Layout.fillWidth: true
59
wrapMode: Text.WordWrap
60
elide: Text.ElideRight
61
text: modelData.restaurantDescription
62
}
63
Label {
64
Kirigami.FormData.label: "Phone:"
65
Layout.fillWidth: true
66
wrapMode: Text.WordWrap
67
elide: Text.ElideRight
68
text: modelData.phone
69
}
70
}
71
}
72
}
73
}
74
}
75
}
76
}
Copied!
Using Proportional Delegate For Simple Display Skills & Auto Layout
ProportionalDelegate is a delegate which has proportional padding and a columnlayout as mainItem. The delegate supports a proportionalGridUnit which is based upon its size and the contents are supposed to be scaled proportionally to the delegate size either directly or using the proportionalGridUnit.
AutoFitLabel is a label that will always scale its text size according to the item size rather than the other way around
QML Example
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
7
Mycroft.ProportionalDelegate {
8
id: root
9
10
Mycroft.AutoFitLabel {
11
id: monthLabel
12
font.weight: Font.Bold
13
Layout.fillWidth: true
14
Layout.preferredHeight: proportionalGridUnit * 40
15
text: sessionData.month
16
}
17
18
Mycroft.AutoFitLabel {
19
id: dayLabel
20
font.weight: Font.Bold
21
Layout.fillWidth: true
22
Layout.preferredHeight: proportionalGridUnit * 40
23
text: sessionData.day
24
}
25
}
Copied!
Using Slideshow Component To Show Cards Slideshow
Slideshow component lets you insert a slideshow with your custom delegate in any skill display which can be tuned to autoplay and loop and also scrolled or flicked manually by the user.
QML Example
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
7
Mycroft.Delegate {
8
id: root
9
10
Mycroft.SlideShow {
11
id: simpleSlideShow
12
model: sessionData.exampleModel // model with slideshow data
13
anchors.fill: parent
14
interval: 5000 // time to switch between slides
15
running: true // can be set to false if one wants to swipe manually
16
loop: true // can be set to play through continously or just once
17
delegate: Kirigami.AbstractCard {
18
width: rootItem.width
19
height: rootItem.height
20
contentItem: ColumnLayout {
21
anchors.fill: parent
22
Kirigami.Heading {
23
Layout.fillWidth: true
24
wrapMode: Text.WordWrap
25
level: 3
26
text: modelData.Title
27
}
28
Kirigami.Separator {
29
Layout.fillWidth: true
30
Layout.preferredHeight: 1
31
}
32
Image {
33
Layout.fillWidth: true
34
Layout.preferredHeight: rootItem.height / 4
35
source: modelData.Image
36
fillMode: Image.PreserveAspectCrop
37
}
38
}
39
}
40
}
41
}
Copied!

Using AudioPlayer Component To Play Audio Files / Audio Streaming

AudioPlayer component is a custom wrapper around Qt Multimedia MediaPlayer, that gives the Skill Authors a basic responsive design audio player they can plug into their skills.
QML Example
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
7
Mycroft.Delegate {
8
id: root
9
skillBackgroundSource: sessionData.audioThumbnail
10
11
Mycroft.AudioPlayer {
12
id: examplePlayer
13
anchors.fill: parent
14
source: sessionData.audioSource //Set URL of audio file
15
thumbnail: sessionData.audioThumbnail //Set Thumbnail of audio
16
title: sessionData.audioTitle //Set Title of audio
17
nextAction: "author.example-player.next" //Event to drive next button action in skill
18
previousAction: "author.example-player.previous" //Event to drive previous button action in skill
19
status: sessionData.status //Current status of playing audio
20
}
21
}
Copied!

Event Handling

Mycroft GUI API provides an Event Handling Protocol between the skill and QML display which allow Skill Authors to forward events in either direction to an event consumer. Skill Authors have the ability to create any amount of custom events. Event names that start with "system." are available to all skills, like previous/next/pick.
Simple Event Trigger Example From QML Display To Skill
Python Skill Example
1
def initialize(self):
2
# Initialize...
3
self.gui.register_handler('skill.foo.event', self.handle_foo_event)
4
...
5
def handle_foo_event(self, message):
6
self.speak(message.data["string"])
7
...
8
...
Copied!
QML Example
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
7
Mycroft.Delegate {
8
id: root
9
10
Button {
11
anchors.fill: parent
12
text: "Click Me"
13
onClicked: {
14
triggerGuiEvent("skill.foo.event", {"string": "Lorem ipsum dolor sit amet"})
15
}
16
}
17
}
Copied!
Simple Event Trigger Example From Skill To QML Display
Python Skill Example
1
...
2
def handle_foo_intent(self, message):
3
self.gui['foobar'] = message.data.get("utterance")
4
self.gui['color'] = "blue"
5
self.gui.show_page("foo.qml")
6
...
7
...
Copied!
QML Example
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
7
Mycroft.Delegate {
8
id: root
9
property var fooString: sessionData.foobar
10
11
onFooStringChanged: {
12
fooRect.color = sessionData.color
13
}
14
15
Rectangle {
16
id: fooRect
17
anchors.fill: parent
18
color: "#fff"
19
}
20
}
Copied!

Using VideoPlayer Component To Play Video Files / Video Streaming

VideoPlayer component is a custom wrapper around Qt Multimedia MediaPlayer, that gives the Skill Authors a basic responsive design video player they can plug into their skills.
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
7
Mycroft.Delegate {
8
id: root
9
skillBackgroundSource: sessionData.videoThumbnail
10
11
Mycroft.VidioPlayer {
12
id: examplePlayer
13
anchors.fill: parent
14
source: sessionData.videoSource //Set URL of video file
15
nextAction: "author.example-player.next" //Event to drive next button action in skill
16
previousAction: "author.example-player.previous" //Event to drive previous button action in skill
17
status: sessionData.status //Current status of playing video
18
}
19
}
Copied!

Resting Faces

The resting face API provides skill authors the ability to extend their skills to supply their own customized IDLE screens that will be displayed when there is no activity on the screen.
Simple Idle Screen Example
Python Skill Example
1
from mycroft.skills.core import resting_screen_handler
2
...
3
@resting_screen_handler('NameOfIdleScreen')
4
def handle_idle(self, message):
5
self.gui.clear()
6
self.log.info('Activating foo/bar resting page')
7
self.gui["exampleText"] = "This Is A Idle Screen"
8
self.gui.show_page('idle.qml')
Copied!
QML Example
1
import QtQuick 2.4
2
import QtQuick.Controls 2.2
3
import QtQuick.Layouts 1.4
4
import org.kde.kirigami 2.4 as Kirigami
5
import Mycroft 1.0 as Mycroft
6
7
Mycroft.Delegate {
8
id: root
9
property var fooString: sessionData.exampleText
10
11
Kirigami.Heading {
12
id: headerExample
13
anchors.centerIn: parent
14
text: fooString
15
}
16
}
Copied!
Last modified 8mo ago