Alexa Announces an A/B Testing Service

Could have used this.

Announcing the general availability of the Alexa A/B testing service

Staff Writer Sep 30, 2022

The Alexa Skills A/B testing service, which allows skill builders to make informed design choices is generally available today

In the third quarter of 2021, the development team at Vocala had to make a critical decision: they had to determine the ideal length of a prompt that would motivate users to subscribe to the paid version of their skill.

Vocala is a leading voice design studio based in the United Kingdom. Over the last few years, Vocala has developed Interflora, the United Kingdom’s first retail sector voice-based skill, in addition to developing a recruiting skill for the British Royal Navy. Vocala is also the publisher of the Yak Yak Games Skill that has a 4.5 rating on the Amazon UK store. Yak Yak Games is a curated collection of leading games such as Deal or No Deal, Song Blast, and Jeff Stelling's Sports Quiz. The skill has a seven-day free trial, at the end of which customers can sign up for a paid version with a monthly paid subscription.

“We wanted to test the prompt that would result in the greatest number of paid conversions,” says James Holland, lead voice developer at Vocala. “More specifically, we wanted to run an A/B test to determine whether a longer or shorter prompt would help us achieve our goal.”

A/B testing is a commonly used experimentation process to make design-related decisions. For web or mobile design, these variables could include the placement of a form on the page, or the color of a Register button. For skill builders, A/B testing can be utilized to arrive at the optimal utterances in order to achieve objectives such as reduced customer friction, increased daily repeat visits, and greater revenue.

Historically, designing and deploying A/B tests for Alexa skills has been a cumbersome process. Setting up infrastructure for an A/B test could take as long as two weeks.

“Working on multiple projects at one time, and responding to clients who want quick turnarounds, we don’t always have the bandwidth for lengthy A/B experiments,” said Holland.

As part of their efforts to continually improve Alexa’s offerings for developers, Alexa solution architects meet regularly with skill builders like Vocala. In response to recent developer feedback related to the importance of experimentation, today, Amazon is launching the Alexa Skill A/B testing service. The new service allows skill builders to design A/B experiments with the goal of maximizing in-skill purchases, repeat visits, and number of dialogs for a session. ... '