Benchmarking Voice Understanding

 Good points made.   I have been using Google voice assistant versus Amazon Alexa for a few years now.  Only now and then using Siri.   I see more 'balking' by Alexa (that is she does not answer coherently at all)  than Google assistant, but then more 'understanding'.  Alexa is in general more 'human' in conversation.  After that I don't see adequate contextual understanding from either in general.  It all depends on how important and risky the dependent decisions are.  Google does a good job of multilingual understanding when properly set up.   Here voicebot.ai has taken a broader look that is worth looking at.   Neither in my opinion can understand and answer that I would call 'complex questions'.

Understanding Is Crucial for Voice and AI: Testing and Training are Key To Monitoring and Improving It      By John Kelvie in Voicebot.ai

BENCHMARKING VOICE ASSISTANTS

How well does your voice assistant understand and answer complex questions? It is often said, making complex things simple is the hardest task in programming, as well as the highest aim for any software creator. The same holds true for building for voice. And the key to ensuring an effortlessly simple experience for voice is the accuracy of understanding, achieved through testing and training.

To dig deeper into the process of testing and training for accuracy, Bespoken undertook a benchmark to test Amazon Echo Show 5, Apple iPad Mini, Google Nest Home Hub. This article explores what we learned through this research and the implications for the larger voice industry based on other products and services.

For the benchmark, we took a set of nearly 1,000 questions from the ComQA dataset and ran them against the three most popular voice assistants: Amazon Alexa, Apple Siri, and Google Assistant. The results were impressive – these questions were not easy, and the assistants handled them often with aplomb:  ... "

GPT-3 and Crypto Assets

Just recently been talking digital Assets and how advances in crypto might influence them.  Here a piece from Coindesk that talks some of the issues, considering it.   I am thinking this, let me know if you have comments. Full opinion piece at the link.

Crypto Needn’t Fear GPT-3. It Should Embrace It
Jul 22, 2020 at 17:47 UTC   By Jesus Rodriquez  in Coindesk

Jesus Rodriguez is the CEO of IntoTheBlock, a market intelligence platform for crypto assets. He has held leadership roles at major technology companies and hedge funds. He is an active investor, speaker, author and guest lecturer at Columbia University. 

During the last few days, there has been an explosion of commentary in the crypto community about OpenAI’s new GPT-3 language generator model. Some of the comments express useful curiosity about GPT-3, while others are a bit to the extreme, asserting that the crypto community should be terrified about it. 

The interest is somewhat surprising because the GPT models are not exactly new and they have been making headlines in the machine learning community for over a year now. The research behind the first GPT model was published in June 2018, followed by GPT-2 in February 2019 and most recently GPT-3 two months ago. 

See also: What Is GPT-3 and Should We Be Terrified?

I think it is unlikely that GPT-3 by itself can have a major impact in the crypto ecosystem. However, the techniques behind GPT-3 represent the biggest advancement in deep learning in the last few years and, consequently, can become incredibly relevant to the analysis of crypto-assets. In this article, I would like to take a few minutes to dive into some of the concepts behind GPT-3 and contextualize it to the crypto world.  .... 

Value and Methods of Text Summarization

Ultimately a powerful concept.    We examined its use with group meetings and focus groups, to gather important topic information, a kind of more focused crowdsourcing of information.   Often aimed at specific goals.  Also was sometime used with documents supporting the topics.   Indeed a 'holy grail' if it can be done systematically and well.  Below from KDNuggets:

This article will present the main approaches to text summarization currently employed, as well as discuss some of their characteristics.

The bona fide semantic understanding of human language text, exhibited by its effective summarization, may well be the holy grail of natural language processing (NLP). That statement isn't as hyperbolic as it sounds: as true human language understanding definitely is the holy grail of NLP, and genuine effective summarization of said human language would necessarily entail true understanding, transitivity would back me up on this.

Unfortunately — or perhaps not, depending on your outlook — honest to goodness "understanding" of human language is not something we can currently count on for text summarization. However, the show must go on, and there currently exist an array of actual techniques for summarizing text, some of which stretch back decades. These techniques take different approaches to reaching the same goal, and can be classified into a fairly narrow set of categories for pursuing their shared goal.

This article will present the main approaches to text summarization currently employed, as well as discuss some of their characteristics: ... '

Alexa Self Learning to Correct Mistakes

In all conversation there is adjustments of our interactions.Good piece here that shows how this is being proposed for a common assistant.

Amazon Uses Self-Learning to Teach Alexa to Correct its Own Mistakes

The digital assistant incorporates a reformulation engine that can learn to correct responses in real time based on customer interactions .    By Jesus Rodriguez in Towards Data Science

 Digital assistant such as Alexa, Siri, Cortana or the Google Assistant are some of the best examples of mainstream adoption of artificial intelligence(AI) technologies. These assistants are getting more prevalent and tackling new domain-specific tasks which makes the maintenance of their underlying AI particularly challenging. The traditional approach to build digital assistant has been based on natural language understanding(NLU) and automatic speech recognition(ASR) methods which relied on annotated datasets. Recently, the Amazon Alexa team published a paper proposing a self-learning method to allow Alexa correct mistakes while interacting with users.

The rapid evolution of language and speech AI methods have made the promise of digital assistants a reality. These AI methods have become a common component of any deep learning framework allowing any developer to build fairly sophisticated conversational agents. However, the challenges are very different when operating at the scale of a digital assistant like Alexa. Typically, the accuracy of the machine learning models in these conversational agents is improved by manually transcribing ... '

New Tools Annonced for Alexa NLU Dev

Impressed by the number of new capabilities being rolled out for skills delivery in Alexa.  Yet I still see quite a few foundational problems with natural language understanding on Alexa, which I use at the skill and foundation level every day.   Makes for a shaky impression during demonstrations.  Does this mean they have hit some fundamental limitation of technology for now?

Build, Test, and Tune Your Skills with Three New Tools  (Full detail at link) 
October 09, 2019
By Leo Ohannesian

We’re excited to announce the General Availability of two tools which focus on your voice model’s accuracy: Natural Language Understanding (NLU) Evaluation Tool and Utterance Conflict Detection. We are also excited to announce that you will now be able to build your own quality and usage reporting with the Get Metrics API, now in Beta. These tools help complete the suite of Alexa skill testing and analytics tools that aide in creating and validating your voice model prior to publishing your skill, detect possible issues when your skill is live, and help you refine your skill over time.

The NLU Evaluation Tool helps you batch test utterances and compare how they are interpreted by your skill’s NLU model against your expectations. The tool has three use cases:

Prevent overtraining NLU models: overtraining your NLU model with too many sample utterances and slot values can reduce accuracy. Instead of adding exhaustive sample utterances to your interaction model, you can now run NLU Evaluations with utterances you expect users to say. If any utterance resolves to the wrong intent and/or slot, you can improve accuracy of your skill’s NLU model by only adding those utterances as new training data (by creating new sample utterances and/or slots).

Regression tests - you can create regression tests and run them after adding new features to your skills to ensure your customer experience stays intact.

Accuracy measurements - you can measure the accuracy of your skill’s NLU model by running an NLU Evaluation with anonymized frequent live utterances surfaced in Intent History (production data), and then measure the impact on accuracy for any changes you make to their NLU model.

Utterance Conflict Detection helps you detect utterances which are accidentally mapped to multiple intents, which reduces accuracy of your Alexa skill’s Natural Language Understanding (NLU) model. This tool is automatically run on each model build and can be used prior to publishing the first version of your skill or as you add intents and slots over time - preventing you from building models with unintended conflicts.  ..... "