Alexa Determining Skills for Customer Needs

Its ultimately a concierge type problem.   How do we determine the best augmenting skill?    Specified need, context, technical ability,  past interactions .... Lots can be at play.  Ultimately a component of all intelligent conversations with a goal.

Rank: How Alexa Determines What Skill Can Best Meet a Customer’s Need    By Young-Bum Kim  Amazon Developer

Amazon Alexa currently has more than 40,000 third-party skills, which customers use to get information, perform tasks, play games, and more. To make it easier for customers to find and engage with skills, we are moving toward skill invocation that doesn’t require mentioning a skill by name (as highlighted in a recent post).

To enable name-free skill interaction, Alexa currently uses a two-step, scalable, and efficient neural shortlisting-reranking approach. (I described our approach to shortlisting in a post yesterday). The shortlisting step uses a scalable neural model to efficiently find the optimal (k-best) candidate skills for handling a particular utterance; the re-ranking step uses rich contextual signals to find the most relevant of those skills. We use the term “re-ranking” since we improve upon the initial confidence score provided by the shortlisting step.

This week, at the Human Language Technologies conference of the North American chapter of the Association for Computational Linguistics (NAACL 2018), my colleagues and I presented a paper, “A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding,” that describes our approach.

hyprank_overview.png
A high-level flow of the two-step shortlisting-reranking approach

The Challenge
The problem here is essentially a domain classification problem over the k-best candidate skills returned by the shortlisting system, which we call Shortlister. The goal of Shortlister is to achieve high recall — to identify as many pertinent skills as possible — with maximum efficiency. On the other hand, the goal of the reranking network, HypRank, is to use rich contextual signals to achieve high precision — to select the most pertinent skills. Designing HypRank comes with its own challenges:

•    Hypothesis representation: It needs to use available contextual signals to produce an effective hypothesis representation for each skill in the k-best list;
•    Cross-hypothesis feature representation: It needs to efficiently and automatically compare features, such as a skill’s intent confidence, to those of other candidate skills in the k-best list; 
•    Generalization: It needs to be language-agnostic; and
•    Robustness: It needs to be able to accommodate changes, such as independent modifications to Shortlister or to the natural-language-understanding models that provide skill-specific semantic interpretation of utterances. .... "