Summary of Our Prototype System Performance

   

Our prototype system allows users to query for instructions around the Starkville/MSU area. It uses the DARPA communicator architecture with major components for automatic speech recognition (ASR), natural language understanding (NLU), Dialog Management (DM), and a database backend. We derived the language model and grammar for the ASR and NLU components from a pilot corpus of 276 sentences spontaneously entered by users over a series of three experiments. Each of the three pilot experiments consisted of two phases, 1) initial data gathering and system testing, followed by 2) retesting the system on the initial data after enhancements were made to the grammar and language model. These experiments focused on reducing the rate of out of vocabulary (OOV) utterances and parsing errors for the NLU component. Results for each of the three pilot experiments are shown below:

   

Two overall system tests were also performed after completing the pilot experiments. Results of these tests are shown below:

   

We continued to refine the NLP component through a series of Wizard of Oz (WOZ) experiments with a simulated speech recognizer. Users participated in multiple scenarios performing a variety of carefully defined tasks in which they queried for information about the Starkville/MSU area. Analysis of data collected in these experiments resulted in significant improvements to the NLP coverage. The number of query types covered increased 12% while coverage of distance and direction queries increased 16% and 35% respectively. Enhanced coverage was tested first in natural language text mode, followed by tests with ASR inserted.