Category: Projects

The glue between speech recognition and auditory reponses

Everyone knows of Amazon Echo and Google Home, there are even a few open-source virtual assistants like Mycroft and Snips.ai. In my opinion, these all suffer from the same deficiency: they aren’t very smart.

I want to be able to talk to my house, and by talk, I mean actually talk. Sure there are a lot of skills or plug-ins made for these platforms, but I haven’t really been impressed by any of them enough to want to use them as my primary voice interaction with my house. You can hook them into Home-Assistant and Mycroft falls back to Wolfram Alpha for any unknown user intents; but can you really talk to them? If you ask Alexa “How are you doing?” do you get some predefined response or does it look at your home and network and respond with the status? No, it doesn’t.

Most people know I hate the cloud; putting your work on “someone else’s machine” is asking for privacy violations, platform shut down and other issues. All of my projects are local first. So right away Amazon Echo, Google Home, and even Siri are off the table. Mycroft and Snips are privacy by design, but if you look at the skills available for each, it’s appalling. For example, Snips has around 8 different integrations with Home-Assistant and almost every one of them is limited to lights, switches and maybe one or two other domains – this applies similarly to Mycroft.

I recently installed a machine-learning centric server in our rack with two CUDA enabled GPUs specifically for facilitating training and inference of machine learning models. Thus, it is only fitting that the platform for my assistant is a learning one. Enter Rasa, a machine learning chatbot. It is definitely a time sink, but it does exactly what I want. No regex patterns for determining user intent (looking at you Mycroft!), the ability to execute remote code for certain intents and allows you to enter multiple response templates for any intent so it doesn’t feel as robotic.

Natural Language Understanding

With Rasa, you define actions and intents, combining them with stories. Intents are exactly what you would expect: what the user wants the assistant to do. For example, you might have an intent named greet which returns the text “Hello, how are you today?”. Your stories can fork the logic based on the user’s response. “I’m doing terrible today” could yield the bot sending cute animal pictures from an API that returns random cat pictures to try to cheer you up. You get to design the flow of dialog how you see fit.

How does Rasa determine the user’s intent? Through training. You provide it with as many sample inputs as you can and associate them with the appropriate intent. As you use your bot, your inputs are logged and can be later annotated – annotating when it comes to machine learning is telling the bot whether it inferred the correct intent from the input or not. This right here is the time sink, it takes a lot of time to come up with sentences that a user might input for every intent you define.

Stories

We use SABNZBD to download much of our media and some times I’d like to know if my downloads are done. Before Rasa, I would have to navigate to the SABNZBD web front end to check the queue. With Rasa, I can ask it “are my downloads done?” and it will query the SABNZBD API to see if the queue is empty or not and report back! If you’re bored, you can set up intents and responses to play a game – like guess a number. The possibilities are endless!

For most intents, there’s one action – but some intents can trigger an entire tree of actions and follow up intents. For example, if the bot asks the user how they are doing, depending on the response the bot will respond differently.

## greet
* greet
  - utter_greet
> check_mood

## user in good mood
> check_mood
* mood_great
  - utter_happy

## user not in good mood
> check_mood
* mood_unhappy
  - utter_cheer_up
  - utter_did_that_help
> check_better

In the example above, when the user says “Hello” or “Hi” the bot greets them and asks how they are. If the user responds with “Good”, “Awesome”, etc. then the bot responds with a positive message like “That’s awesome, ist here anything I can do for you?”. However, if the user says “Terrible” or “Awful”, the bot will try to cheer the user up – in my case, cute animal pictures or funny jokes. If the user is still not cheered up, then it will randomly respond with something else to try to cheer the user up until they are happy.

Communicating With the House

In addition to the built-in actions, you can build custom actions. By default, these custom actions are in the configuration directory inside actions.py. If you plan on making custom actions, definitely spin up a custom action server because when you make a change to a custom action the Rasa service needs to be restarted with every change – with a custom action server, changes only require restarting the custom action server.

The easiest way to spin up a custom action server is via their docker image. You’ll tell Rasa to talk to the custom action server by editing the appropriate line in the config.yaml in the project directory. Once spun up, you can implement actions to your heart’s content. Be warned, Rasa only loads Action subclasses defined in the actions.py file – to work around this, I place the logic for the action in their own python file in separate packages and define the class itself inside actions.py. For example:

# actions.py
# NOTE: package must start with actions or it can't locate the eddie_actions package

from actions.eddie_actions.location import who_is_home, locate_person


class ActionLocatePerson(Action):
    def name(self) -> Text:
        return "action_locate_person"

    def run(self,
            dispatcher: CollectingDispatcher,
            tracker: Tracker,
            domain: Dict[Text, Any]) -> List[Dict[Text, Any]]:
        return locate_person(dispatcher,
                             tracker,
                             domain)

# eddie_actions/location.py
def locate_person(dispatcher: CollectingDispatcher,
                  tracker: Tracker,
                  domain: Dict[Text, Any]) -> List[Dict[Text, Any]]:
    person = next(tracker.get_latest_entity_values(PERSON_SLOT), None)

    response = requests.get(
        f"https://automation.prettybaked.com/api/states/person.{str(person).lower()}",
        headers={
            "Authorization": f"Bearer {HOME_ASSISTANT_TOKEN}"
        }
    )

    location = None

    try:
        response.raise_for_status()
        location = response.json().get('state', None)
    except requests.HTTPError as err:
        _LOGGER.error(str(err))

    if not location:
        dispatcher.utter_message(template="utter_locate_failed")
    elif location == "not_home":
        dispatcher.utter_message(template="utter_locate_success_away")
    else:
        dispatcher.utter_message(template="utter_locate_success", location=location)

    return []

I created a long-lived token for Rasa inside Home-Assistant and pass it to the container via an environment variable. I created similar actions for connecting to Tautalli (Plex metrics) for recently added media, SABNZBD (UseNet download client) for asking about download status and plan to connect it to pFSense and Unifi for network status – “Hey Eddie, how are you today?” “Not so good, network traffic is awfully high right now.”

Chitchat and Other Fun

With the goal of being able to actually talk to your assistant, general chitchat is a must. Generally, when you meet someone there are some pretty common patterns in the conversations: introductions, hobbies, jokes, etc. With Rasa’s slots, introductions are fairly easy to implement. Create an introduction intent and add some examples: “Hi there, I’m Teagan” (where Teagan is annotated as the bot respondent’s name, reply with the bot’s name and continue from there. Eddie, my assistant, definitely has some hobbies:

Every virtual assistant out there has some fun easter eggs. Any child of the 80’s/90’s who played games knows some of the iconic cheat codes. Eddie is not a fan of cheating:

Eddie is modeled after the Heart of Gold’s onboard computer. So, of course, it has to have specific knowledge:

Thoughts and Next Steps

Truthfully, it can be very tedious to train your assistant yourself. I highly recommend deploying an instance and sharing it with friends and family. You’ll see the conversations they have had, be able to annotate the user’s intents (or add new ones), fix the actions and responses, and training a better model.

Of course, Rasa is text-based by default. Once I am happy with the defined intents, stories, responses and flow of dialog it will need to be integrated with Speech-to-Text (currently looking at Deepspeech) and Text-to-Speech (espeak, MaryTTS or even Mozilla TTS). Keep an eye out for a post about integrating these services with Rasa for a true voice assistant that continually learns!