Evaluating the Performance of a Speech Recognition based System

by   Vinod Kumar Pandey, et al.

Speech based solutions have taken center stage with growth in the services industry where there is a need to cater to a very large number of people from all strata of the society. While natural language speech interfaces are the talk in the research community, yet in practice, menu based speech solutions thrive. Typically in a menu based speech solution the user is required to respond by speaking from a closed set of words when prompted by the system. A sequence of human speech response to the IVR prompts results in the completion of a transaction. A transaction is deemed successful if the speech solution can correctly recognize all the spoken utterances of the user whenever prompted by the system. The usual mechanism to evaluate the performance of a speech solution is to do an extensive test of the system by putting it to actual people use and then evaluating the performance by analyzing the logs for successful transactions. This kind of evaluation could lead to dissatisfied test users especially if the performance of the system were to result in a poor transaction completion rate. To negate this the Wizard of Oz approach is adopted during evaluation of a speech system. Overall this kind of evaluations is an expensive proposition both in terms of time and cost. In this paper, we propose a method to evaluate the performance of a speech solution without actually putting it to people use. We first describe the methodology and then show experimentally that this can be used to identify the performance bottlenecks of the speech solution even before the system is actually used thus saving evaluation time and expenses.


page 1

page 2

page 3

page 4


Deaf, Hard of Hearing, and Hearing Perspectives on using Automatic Speech Recognition in Conversation

Many personal devices have transitioned from visual-controlled interface...

From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition

Consumer speech recognition systems do not work as well for many people ...

A Multi-Biometrics for Twins Identification Based Speech and Ear

The development of technology biometrics becomes crucial more. To define...

Feasibility of Using Automatic Speech Recognition with Voices of Deaf and Hard-of-Hearing Individuals

Many personal devices have transitioned from visual-controlled interface...

Introducing the Talk Markup Language (TalkML):Adding a little social intelligence to industrial speech interfaces

Virtual Personal Assistants like Siri have great potential but such deve...

A Metric to Classify Style of Spoken Speech

The ability to classify spoken speech based on the style of speaking is ...

Eliciting Spoken Interruptions to Inform Proactive Speech Agent Design

Current speech agent interactions are typically user-initiated, limiting...

Please sign up or login with your details

Forgot password? Click here to reset