APPLICATIONS OF RISK MINIMIZATION
TO SPEECH RECOGNITION
	Aravind Ganapathiraju	Jonathan Hamaker and Joseph Picone
	Conversay	Inst. for Signal and Information Processing
	15375 NE 90th St.	Mississippi State University
	Redmond, WA, USA	Mississippi State, MS 39762, USA
	aganapathiraju@conversay.com	{hamaker, picone}@isip.msstate.edu
ABSTRACT
Statistical techniques based on Hidden Markov models (HMMs) with Gaussian emission densities have dominated the signal processing and pattern recognition literature for the past 20 years. However, HMMs suffer from an inability to learn discriminative information and are prone to overfitting and over-parameterization. Recent work in machine learning has focused on models, such as the support vector machine (SVM), that automatically control generalization and parameterization as part of the overall optimization process. SVMs have been shown to provide significant improvements in performance on small pattern recognition tasks compared to a number of conventional approaches.
In this paper, we describe an application of SVMs to a large‑scale speech recognition task and demonstrate a state-of-the-art word error rate of 10.6% on a continuous alphadigit task. We also show an SVM classifier provides a significant improvement in performance on a static pattern classification task based on the Deterding vowel data. Unifying themes in this work are the concepts of risk minimization and margin maximization that guarantee estimation of classifiers with good generalization properties.
Please direct all correspondence to: Joseph Picone, 413 Simrall, Hardy Road, Box 9571, Mississippi State, MS 39762, USA, Email: picone@isip.msstate.edu, Phone: (662) 325-3149, Fax: (662) 325-2298.

1.    
INTRODUCTION
Speech recognition systems are now essentially an application of machine learning and pattern recognition technology. Simplistic techniques based on the design of smart feature extraction algorithms and physiological models have given way to powerful statistical methods which automatically learn in data-driven modes. We can point to a progression of techniques over the past 40 years: ad-hoc measurements (circa 1960s), Fourier methods (circa late 1960s and early 1970s), maximum entropy and least mean square error methods (circa late 1970s), temporal modeling techniques such as dynamic time warping (circa late 1970s and early 1980s), and Markov models (circa mid-1980s and early 1990s).
By the mid-1990s, the field of machine learning had grown tremendously, generating many new competing techniques for the generation of technology beyond Markov models. Most of these techniques are rooted in principles of maximum likelihood, and use the Expectation Maximization (EM) algorithm to estimate parameters. Modern day systems, though impressive in their architecture and performance, involve enormous amounts of computation, utilize large numbers of parameters in complex statistical models, and yet are still relatively fragile. The essential challenges in signal processing problems such as speech recognition are controlling generalization, direct discriminative training, and sparsity. In this paper, we address the first two issues, and leave the third for a companion work []. In Section , we review the dominant statistical approach to speech recognition based on Bayes’ Rule. In Section 3, we demonstrate this is a specific instance of a general approach based on risk minimization, while in Section  we introduce the support vector machine. In Sections  and  we present some experimental results on speech recognition problems in which the training data is highly ambiguous. We conclude with a discussion of some ongoing research on extensions to the basic risk minimization framework.
2.    MAXIMUM LIKELIHOOD-BASED SPEECH RECOGNITION
Modern speech recognition systems use a statistical approach [] based on Bayes’ Rule. If  is the acoustic evidence that is provided to the system and  is a sequence of words, then the recognition system chooses a word string  such that
(1)	.
The probability, , is typically provided by an acoustic model while  gives the a priori probability of the word sequence  being spoken as predicted by a language model.
The acoustic modeling components of a speech recognizer are based on hidden Markov models (HMMs) [,]. The power of an HMM representation lies in its ability to model the temporal evolution of a signal via an underlying Markov process. The ability of an HMM to statistically model the acoustic and temporal variability in speech has been integral to their success. The probability distribution associated with each state in an HMM models the variability which occurs in speech across speakers or phonetic context. This distribution is typically a Gaussian mixture model (GMM) since a GMM provides a sufficiently general parsimonious parametric model as well as a solid mathematical framework for estimation and analysis.
A key to the widespread use of HMMs to model speech can be attributed to the availability of efficient parameter estimation procedures []. 		If we assume that parameters of an HMM are fixed but unknown, we can pose the problem of parameter estimation as one that maximizes the probability that the model generates the observed data (a posteriori probability). The approach for a typical HMM parameter estimation process then becomes maximization of the likelihood of the data given the model, traditionally known as maximum likelihood (ML) estimation []. One of the most compelling reasons for the success of ML and HMMs has been the existence of iterative methods to estimate the parameters that guarantee convergence. The EM algorithm provides an iterative framework for ML estimation with good convergence properties, though it does not guarantee finding the global maximum [].
There are, however, problems with an ML formulation for problems such as speech recognition. ML does not necessarily translate to better classifiers. The goal in building a classifier is to learn to distinguish between two or more distinct classes of data. ML improves a classifier’s ability to represent a specific class. In this light, ML-based estimation is tangential to the goal of classifiers in general and speech recognizers in particular. The ML estimation process tries to optimize the modeling ability of the acoustic models without access to a measure of their classification ability. In reality, better classification is the ultimate goal of the speech recognizer. ML estimation is unable to guarantee this because it is not discriminatively trained — the model parameters are estimated based as an optimization process that only guarantees that the model describes the data accurately and not necessarily the classification power of the model.
A simple example, shown in Figure   illustrates this problem. The two classes shown are derived from completely separable uniform distributions. ML is used to fit Gaussians to these classes and a simple Bayes classifier is built. However, we see that the decision threshold occurs inside the range of class 2. This means that the probability of error is significant. However if we were to simply recognize that the range of data points in class 1 is less than 3.3 and that no data point in class 2 occurs within this range, we can achieve perfect classification. In this example any amount of effort expended in learning a better Gaussian model will not help achieve perfect classification. More dramatic examples can be constructed to show that learning decision regions discriminatively will help improve classification. The conclusion from the above example is not necessarily that using a Gaussian is an incorrect choice. However, it is clear from such examples that discrimination is a key ingredient for creating robust and more accurate acoustic models.
The primary difference between ML-based HMM parameter estimation and discriminative techniques is that the objective criterion for the latter includes the probability of the data given that the wrong model was used []. Under discriminative‑based estimation, the optimization process can train a model to effectively reject out-of-class examples while simultaneously learning to optimally represent in-class examples. There has already been significant progress in the area of discriminative techniques for the estimation of HMM parameters. Optimizing criteria such as Maximum Mutual Information (MMI) [] and Minimum Classification Error (MCE) [] have emerged in the past five years as powerful techniques. MCE is especially elegant in that it uses the fact that in order to achieve good classification, the estimation of the posterior probabilities is not as important as it appears. ML and MMI suffer from the fact that both the estimation procedures expend effort in modeling posteriors while not guaranteeing improved classification performance.
Artificial neural networks (ANNs) represent a class of discriminative techniques that have been successfully applied to speech recognition [-]. Though ANNs attempt to overcome many of the problems previously described, their shortcomings with respect to applications such as speech recognition are well‑documented [,]. Some of the most notable deficiencies include design of optimal model topologies, slow convergence during training and a tendency to overfit the data. However, it is important to note that many of the fundamental ideas presented in this paper (e.g., soft margin classifiers) have similar implementations within an ANN framework. In most classifiers, the controlled trade-off between overfitting and good closed-loop performance is vital to the success of systems in real-world conditions. 
3.    RISK MINIMIZATION
One would expect better performance with classifiers that estimate decision surfaces directly (rather than a probability distribution across the training data). A Support Vector Machine (SVM) is one such machine learning technique []. SVMs have demonstrated good performance on several classical pattern recognition problems []. Through the principle of structural risk minimization (SRM), the SVM framework provides significant advantages over ML techniques. Unlike ML, the SRM paradigm provides for control of the trade-off between generalization and closed-loop optimality. The design of a classifier is essentially a process of learning a mapping of the input data to class labels, often achieved using an optimization constrained by domain knowledge. Two commonly used minimization criteria are empirical risk minimization (ERM) and structural risk minimization (SRM). SVMs are estimated based on SRM. 
Let us assume that the training data consists of pairs,  where the ‘s are the input observations and ‘s are the target classes. The goal of a learning machine is to learn the mapping . We assume that the training data has been drawn randomly and independently based on the joint distribution . To learn the unknown mapping, we can either estimate a function that is close to the joint distribution under an appropriate metric or we can learn an optimal predictor of the system’s output. For the purposes of data classification, we pursue the latter approach.
Let us define a term, risk, which measures the quality of the chosen function. The operating space is a subset  of an n-dimensional vector space , which is the cross-product of the input vector space and the output space. Let us also assume the existence of a set of functions  and a functional , that measures the risk as,
(2)	
where  is an appropriately defined loss function and  is the joint probability distribution. In the case of a pattern classification problem, the functions  can be viewed as functions which predict the output class given the input observations. The quality of the choice of the function  can be evaluated in terms of the functional  using the loss  as a cost function.
The problem of risk minimization can then be mathematically defined as minimizing the functional given by () for a specific training data set. This minimization is not possible in the strict sense because of the lack of knowledge about . In practice, the minimization process involves finding the optimal parameterization for , which can be represented as . The minimization finds the best parameterization  such that  is the optimal function from the set . This parameterization does not restrict us to parametric forms since  can be a scalar, a vector or any other abstract functional element.
The risk can now be rewritten as
(3)	,
where,
(4)	.
The function  is called the loss function. The choice of the loss function is typically problem specific. For example, it could be a simple indicator function which takes a value 1 when the function  predicts the correct output observation and a value 0 otherwise. 
	Choosing an appropriate value for  to minimize the functional defined in () is called risk minimization. Minimizing the risk functional is not a trivial problem because the form of the parameterization of the joint distribution  is not known a priori. The problem can be simplified if we minimize a variation of (). We can minimize the measured mean risk, or empirical risk:
(5)	.
For many pattern recognition problems, the loss function is an indicator function: .
We assume that we have access to  training observations .  is therefore the mean error computed from a fixed number of training samples under the assumption that the training samples are uniformly distributed. Minimization of () is called empirical risk minimization (ERM) and is one of the most commonly used optimization procedures in machine learning. ERM is computationally simpler than attempting to minimize the actual risk as defined in () since it circumvents the need for the estimation of the joint probability density function . 
Vapnik [] proved that a bound exists between these two forms for risk:
(6)	
where  is a function of , a scalar value referred to as the Vapnik-Chervonenkis (VC) dimension.  is a measure of the capacity of a learning machine to learn any training set. The VC dimension is directly related to the generalization ability of a learning machine [] and is typically proportional to the number of free parameters (weights, mixture components, etc.) in the system. The reason VC dimension is said to be a measure of the generalization ability of a system is that a tight bound on the actual risk implies good generalization. Good generalization is only possible when  is small. However a large value of  implies that actual risk may be very different from the empirical risk, which in turn results in poor generalization.
For loss functions in the form of indicator functions, which is true in the case of classifiers, the second term in the r.h.s. of equation (), , is,
(7)	
where  is the parameter set that defines the learning machine and  is the measure of the difference between the expected and empirical risk []. The error term, , can be written in terms of the VC dimension and the size of the training set as, 
(8)	,
where  is a confidence value (with a range ).
Equation () provides us with a good criterion for comparing system configurations optimized using empirical risk minimization. When  is large,  and  are both small. This implies that the expected risk tends towards the empirical risk. With this, we can guarantee both a small empirical risk (training error) and good generalization — an ideal situation for a learning machine. On the other hand, when  is small, both  and  are large. Under this condition, a small empirical risk does not guarantee a small expected risk. Thus, the system is not guaranteed to generalize well. To achieve a tight bound on the expected risk, both terms on the r.h.s. of () need to be minimized simultaneously. From (), we see that the error term, , monotonically increases with the VC dimension. This implies that the confidence in the empirical risk decreases monotonically with the VC dimension, as does the generalization ability, as shown in Figure  . Hence, making the VC dimension a controlling variable for generalization is a reasonable choice.
4.    Support Vector Classifiers
Figure   shows a typical 2-class classification example where the examples are perfectly separable using a linear decision region.  and  define two hyperplanes. The distance separating these hyperplanes is called the margin. The closest in-class and out-of-class examples lie on these two hyperplanes. ERM can be used to find a good hyperplane. However, several such hyperplanes exist, each of which satisfy the requirements of ERM. Adding an additional requirement that the optimal hyperplane should have good generalization properties can help choose the best hyperplane. The SRM principle imposes structure on the optimization process by ordering the hyperplanes based on the margin. The optimal hyperplane is the one that maximizes the margin while minimizing the empirical risk. This indirectly guarantees better generalization []. Figure   illustrates the differences between using ERM and SRM.
Let  be a vector that is normal to the separating hyperplane. The  training examples are represented as pairs  where . The points that lie on the separating hyperplane satisfy
(9)	
where  is the distance of the hyperplane from the origin. Let the margin of the SVM be defined as the distance from the separating hyperplane to the closest positive and negative examples. The SVM training paradigm finds the separating hyperplane which gives the maximum margin. Once the hyperplane is obtained, all the training examples satisfy the following inequalities:
(10)	
(11)	.
The above equations can be compactly represented as a single inequality,
(12)	.
Examining the above equations with respect to Figure  , we see that all points satisfying the equality condition in () lie on . Similarly, all points satisfying the equality condition () lie on . The distance between  and , also called the margin, is therefore two units. Since the normal to the hyperplane is not constrained to be of unit-norm, we need to normalize this margin by the norm of the normal vector to the hyperplane, . Therefore the margin as defined by the hyperplane is . For a completely separable data set, no points fall between  and . To maximize the margin we need to therefore maximize . 																						The training points for which the equality in () holds are called support vectors.
The classifier is defined in terms of the training examples. However all training examples do not contribute to the definition of the classifier. Only those examples that satisfy the equality in () are needed to define the separating hyperplane. In practice, the proportion of support vectors is small, making the classifier sparse. The data set itself defines how complex the classifier needs to be. This is in stark contrast to systems such as neural networks and HMMs where the complexity of the system is typically predefined or chosen through a cross-validation process. Since there are  dot products involved in the definition of the classifier, where  is the number of support vectors, the classification task scales linearly with the number of support vectors.
Thus far we have only considered the case where the training data is completely separable using a linear margin. However, we know that real‑world classification problems typically involve data which is non-linearly separable or completely non-separable. Given such a training set, we still need to estimate the classifier that maximizes the margin and minimizes the errors on the training set. Optimization of non-separable data is typically accomplished by the use of soft decision classifiers where the classification of an example is tagged with a probability. However, we must introduce slack variables to find an optimal solution. An optimal classifier for this non-separable case exists and is described in detail in [-].
5.    APPLICATIONS TO SPEECH RECOGNITION
Hybrid approaches for speech recognition [] provide a flexible paradigm to evaluate new acoustic modeling techniques. These systems are not able to eliminate the HMM framework entirely because classification models such as SVMs cannot model the temporal structure of speech effectively. Most contemporary connectionist systems use the neural networks only to estimate posterior probabilities and use the HMM structure to model temporal evolution [-]. In integrating SVMs into such a hybrid system, several issues arise:
Posterior Estimation: One drawback of an SVM is that it provides an m-ary decision. Most signal processing applications need a posterior probability that captures our uncertainty in the classification. This issue is particularly significant in speech recognition because there is a large degree of overlap in the feature space. SVMs provide a distance or discriminant which can be used to compare classifiers. This is unlike connectionist systems whose output activations are estimates of the posterior class probabilities [,]. One of the main concerns in using SVMs for speech recognition is the lack of a clear relationship between distance from the margin and the posterior class probability. Recent work on using moderated SVM outputs as estimates of the posterior probability has had success at the expense of increased computations [].
From a variety of options available for converting the posterior to a probability [], we chose unmoderated probability estimates based on maximum likelihood fitting. We used a sigmoid distribution to map the output distances to posteriors:
(13)	.
 and  are parameters that need to be estimated (using any suitable nonlinear function estimator). Note that we have assumed that the prior class probabilities are equal. An issue that arises from this formulation of estimating posteriors is that the distance estimates are heavily biased by the training data. In order to avoid biased estimates, a cross-validation set must be used to estimate the parameters of the sigmoid []. The size of this data set can be determined based on the amount of training data that is available for the classifier.
Classifier Design: Thus far we have not addressed a fundamental issue in classifier design — should the classifiers be one-vs-one classifiers, which learn to discriminate one class from another class, or one-vs-all classifiers, which learn to discriminate one class from all other classes? One-vs-one classifiers are typically smaller and less complex and can be estimated using fewer resources than one-vs-all classifiers. When the number of classes is  we need to estimate  one-vs-one classifiers as compared to  one-vs-all classifiers. On several standard classification tasks it has been proven that one-vs-one classifiers are marginally more accurate than one-vs-all classifiers [,]. Nevertheless, for computational efficiency, we chose to use one-vs-all classifiers in all experiments reported here.
Segmental Modeling: A logical step in building a hybrid system would be to replace the Bayes classifier in a traditional HMM system with an SVM classifier at the frame-level. However, the amount of training data and the confusion inherent in frame-level acoustic feature vectors prevents this at the current time. Consider training a classifier to discriminate the phone ‘s’ from all other phones in a training set consisting of 40 hours of speech. At a frame rate of 100 frames per second,  frames of training data are available for each classifier. Though very efficient optimizers are used to train an SVM, this amount of data is still beyond present day computational resources (on the order of months even on extremely fast processors).
In our work, we have chosen to use a segment-based approach which solves the data size issue. The other motivating factor for most segment-based approaches is that the acoustic model needs to capture both temporal and spectral structure of speech which is clearly missing in frame-level classification schemes. Segmental approaches also overcome the assumption of conditional independence between frames of data in traditional HMM systems. Segmental data takes better advantage of the correlation in adjacent frames of data that is inherent in speech.
A related problem we face is the variable length or duration problem. Since segment duration is an important speech-related feature which is correlated with the word choice and speaking rate, our classifier cannot simply discard this information. A simple but effective approach motivated by the 3-state HMMs used in most state-of-the-art speech recognition systems is to assume that the segments (phones in most cases) are composed of a fixed number of sections [,]. The first and third sections model the transition into and out of the segment, while the second section models the stable portion of the segment. We use segments composed of three sections in all recognition experiments reported in this work.
N-best List Rescoring: As a first step towards building a stand-alone hybrid SVM/HMM system, we have explored a simple N‑best list rescoring paradigm. Assuming that we have already trained the SVM classifiers for each phone in the model inventory, we generate N-best lists using a conventional HMM system. A model-level alignment for each hypothesis in the N-best list is then generated using the HMM system. Segment-level feature vectors are generated from these alignments. These segments are then classified using the SVMs. Posterior probabilities are computed using the sigmoid approximation previously discussed. These probabilities are used to compute the utterance likelihood of each hypothesis in the N-best list. The N-best list is reordered based on the likelihood and the top hypothesis is used to calibrate the performance of the system.
6.    EXPERIMENTAL RESULTS
We have evaluated our SVM technology on three popular tasks. We first experimented on the Deterding static vowel classification task [] which is a common benchmark used for new classifiers. This was a good data set to evaluate the efficacy of static SVM classifiers on speech data since it is a standard benchmark for several non-linear classifiers. In this evaluation, the speech data was collected at a 10 kHz sampling rate and low pass filtered at 4.7 kHz. The signal was then transformed to 10 log-area parameters, giving a 10‑dimensional input space. A window duration of 50 msec. was used for generating the features. The training set consisted of 528 frames from eight speakers and the test set consisted of 462 frames from a different set of seven speakers. The speech data consisted of 11 vowels uttered by each speaker in a h*d context. 
Table   presents results comparing two types of polynomial kernels to a variety of other classification schemes. A detailed description of the conditions for this experiment are described in []. Performance using both kernels is better than most nonlinear classification schemes. The best performance we report is 35%. This is worse than the best performance reported on this data set (30% using a speaker adaptation scheme called Separable Mixture Models []). However, it is significantly better than the best neural network classifiers (Gaussian Node Network) that produce a misclassification rate of 44% []. We have observed similar trends on a variety of static classification tasks.
We next evaluated this approach on two continuous speech recognition tasks: OGI Alphadigits (AD) [] and SWITCHBOARD (SWB) []. Both are telephone databases collected from a wide variety of speakers and channel conditions, and are extremely challenging from an acoustic modeling standpoint. Alphadigits consists of spoken letters and numbers which are extremely confusable for telephone‑quality speech (e.g., “p” vs. “b”). SWB consists of conversational speech that is even difficult for humans to transcribe. Results for this task are shown in Table .
For these experiments, a baseline HMM system was created using a public-domain ASR system []. The baseline HMM system was also used to generate the segmental training data for the SVM models by Viterbi-aligning the training reference transcription to the acoustic data. The time marks derived from this Viterbi alignment were used to extract the segments. Before extraction, each feature dimension was normalized to the range [-1,1] to account for unequal variance amongst the feature dimensions. For each phoneme instance in the training transcription, a segment was extracted. This segment was divided into three parts as previously described.
For each phoneme in the lexical set, an SVM model was trained to discriminate between that phoneme and all other phonemes (one-vs-all models) giving a total of 29 models. From the segmental data extracted above, we chose a training set for each model. This training set consisted of equal amounts of within-class and out-of-class data. All within-class data available for that phoneme was used. To reduce the amount of training data while providing sufficient information regarding the various classes, the out-of-class data was randomly chosen such that one half of the out-of-class data came from phonemes that were phonetically similar to the phoneme of interest and one half came from all other phonemes.
In Table , we compare the performance of our HMM and SVM systems under a variety of conditions. The improvements provided by the hybrid SVM system are statistically significant and very promising. The complexity of the hybrid system is comparable to the complexity of a cross-word HMM system using state-tying. In contrast, the hybrid system utilizes only monophone models, though the use of segmental data does provide implicit context for the model. The number of free parameters in the hybrid system, primarily the Lagrange multipliers and support vectors, is linearly dependent on the amount of training data. This is another reason why heuristics need to be applied to trim the out-of-class data to a more manageable size [].
As a point of reference, we also produced results using a reference segmentation or reference transcription as part of the N-best lists. In these oracle experiments, the segmentations are produced by forced-alignment of the reference transcription using the baseline HMM system. The results of these experiments provide a nice analysis tool as they give us a presumptive lower bound on the achievable error by the hybrid system. The actual lower bound is the N-best list error rate, but it is a good assumption that we won’t do better than a system with perfect knowledge of the reference segmentation.
It is important to notice that using the correct segmentation for the SVM model is critical in achieving good performance. However, when we allow the SVM decide the best segmentation and hypothesis combination by using the N-best segmentations the performance gets worse, as shown in the last rows in Table . This apparent anomaly indicates the need to incorporate variations in segmentation into the classifier estimation process — recall that the SVM only saw “correct” segmentations of in-class and out-of-class data during training while during decoding it may see many “incorrect” segment hypotheses.
Relaxing this strong interdependence between the segmentation and the SVM performance is a point for further research. The oracle experiment shows that further improvements are possible using this paradigm if certain key issues are addressed. Primary among them is the need for better integration of segmentation information into the system using concepts such as segment-graphs. One way to mitigate this issue is to iteratively train the classifiers by going through a estimate-classify process where classification errors can be fed back into the system.
7.    CONCLUSIONS
This paper addresses the use of Support Vector Machines as a viable classifier in a continuous speech recognition system. The technology has been successfully applied to several speech recognition tasks. A hybrid SVM/HMM system has been developed which uses SVMs to post-process data generated by a conventional HMM system. The results obtained in the experiments clearly indicate the classification power of SVMs and affirm the use of SVMs for acoustic modeling. The oracle experiments reported here clearly shows the potential of this hybrid system while highlighting the need for further research into the segmentation issue.
Our ongoing research into new acoustic modeling techniques for speech recognition is taking two directions. First, it is clear that for a formalism such as SVMs, there is a need for an iterative SVM classifier estimation process embedded within the learning loop of the system to expose the classifier to the same type of ambiguity observed during recognition. Mismatched training and evaluation conditions are dangerous for recognition experiments.
Second, the basic SVM formalism suffers from three fundamental problems: scalability, sparsity, and Bayesian-ness. Recent related research [] based on relevance vector machines (RVM) directly addresses the last two issues. Promising results are shown in Table  . While the Bayesian RVM model achieves superior performance and superior sparsity, scalability of the training routines to large data sets remains an issue. Current research is aimed at developing faster training and computational paradigms for RVMs.
Finally, the algorithms, software, and recognition systems described in this work are available in the public domain as part of our speech recognition toolkit []. Detailed explanations of the theory and experiments described here can also be found in [].
8.    ACKNOWLEDGEMENTS
The inspiration for this work grew out of discussions with Vladimir Vapnik at the 1997 Summer Workshop at the Center for Speech and Language Processing at the Johns Hopkins University. We are also grateful to Naveen Parihar and Bohumir Jelinek for their contributions to the software infrastructure required for the experiments described in Table .
9.    

REFERENCES
[1]	J. Hamaker and J. Picone, “Advances in Speech Recognition Using Sparse Bayesian Methods,” submitted to the IEEE Transactions on Speech and Audio Processing, January 2003.
[2]	Jelinek, F. (1997). Statistical Methods for Speech Recognition, MIT Press, Cambridge, Massachusetts, USA.
[3]	Rabiner, L.R. and Juang, B.H. (1993). Fundamentals of Speech Recognition, Prentice Hall, Englewood Cliffs, New Jersey, USA.
[4]	Dempster, A.P., Laird, N.M. and Rubin, D.B. (1977). Maximum Likelihood Estimation from Incomplete Data. Journal of the Royal Statistical Society, vol. 39, no. 1, pp. 1-38.
[5]	McDermott, E. (1997). Discriminative Training for Speech Recognition, Ph. D. Thesis, Waseda University, Japan.
[6]	Woodland, P. and Povey, D. (2000). Very Large Scale MMIE Training for Conversational Telephone Speech Recognition. Proceedings of the 2000 Speech Transcription Workshop, University of Maryland, MD, USA.
[7]	Bourlard, H.A. and Morgan, N. (1994). Connectionist Speech Recognition — A Hybrid Approach, Kluwer Academic Publishers, Boston, USA.
[8]	Bridle, J.S. and Dodd, L. (1991). An Alphanet Approach to Optimizing Input Transformations for Continuous Speech Recognition. Proceedings of the International Conference on Acoustics, Speech and Signal Processing, vol. 1, pp. 277-280.
[9]	Robinson, A.J. (1989). Dynamic Error Propagation 	Networks, Ph.D. Thesis, Cambridge University, UK.
[10]	Ganapathiraju, A. (2002). Support Vector Machines for Speech Recognition, Ph. D. Thesis, Mississippi State University, Mississippi State, Mississippi, USA.
[11]	Vapnik, V.N. (1998). Statistical Learning Theory, John Wiley, New York, NY, USA.
[12]	Burges, C.J.C. (1999). A Tutorial on Support Vector Machines for Pattern Recognition, http://svm.research.bell-labs.com/SVMdoc.html, AT&T Bell Labs.
[13]	Vapnik, V.N. (1995). The Nature of Statistical Learning Theory, Springer-Verlag, New York, NY, USA.
[14]	Tebelskis, J. (1995). Speech Recognition using Neural Networks, Ph. D. Thesis, Carnegie Mellon University, Pittsburgh, USA.
[15]	Cook, G.D. and Robinson, A.J. (1997). The 1997 ABBOT System for the Transcription of Broadcast News. Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop, Lansdowne, Virginia, USA.
[16]	Ström, N., Hetherington, L., Hazen, T.J., Sandness, E. and Glass, J. (1999). Acoustic Modeling Improvements in a Segment-Based Speech Recognizer. Proceedings of IEEE ASRU Workshop, Keystone, CO, USA.
[17]	Ganapathiraju, A., Hamaker, J. and Picone, J. (1998). Support Vector Machines for Speech Recognition. Proceedings of the International Conference on Spoken Language Processing, pp. 2923-2926, Sydney, Australia.
[18]	Ganapathiraju, A., Hamaker, J. and Picone, J. (2000a). A Hybrid ASR System Using Support Vector Machines. International Conference of Spoken Language Processing, Beijing, China.
[19]	Kwok, J. (1999). Moderating the Outputs of Support Vector Machine Classifiers. IEEE Transactions on Neural Networks, vol. 10, no. 5.
[20]	Platt, J. (1999). Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods, In Advances in Large Margin Classifiers, MIT Press, Cambridge, MA, USA.
[21]	Allwein, E. Schapire, R.E. and Singer, Y. (2000). Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers. Journal of Machine Learning Research, vol. 1(Dec), pp. 113-141.
[22]	Weston, J. and Watkins, C. (1999). Support Vector Machines for Multi-Class Pattern Recognition. Proceedings of the Seventh European Symposium On Artificial Neural Networks.
[23]	Chang, J. and Glass, J. (1997). Segmentation and Modeling in Segment-based Recognition, Proceedings of Eurospeech, pp. 1199-1202, Rhodes, Greece.
[24]	Deterding, D., Niranjan, M. and Robinson, A.J. (2000). Vowel Recognition (Deterding data). Available at http://www.ics.uci.edu/pub/machine-learning-databases/undocumented/ connectionist-bench/vowel.
[25]	Tenenbaum, J. and Freeman, W.T. (1997). Separating Style and Content. Advances in Neural Information Processing Systems 9, MIT Press, Cambridge, MA, USA.
[26]	Cole, R. (1998). Alphadigit Corpus v1.0. http://www.cse.ogi.edu/CSLU/corpora/alphadigit, Center for Spoken Language Understanding, Oregon Graduate Institute, USA.
[27]	Godfrey, J., Holliman, E. and McDaniel, J. (1992). SWITCHBOARD: Telephone Speech Corpus for Research and Development. Proceedings of the International Conference on Acoustics, Speech and Signal Processing, vol. 1, pp. 517-520, San Francisco, California, USA.
[28]	Hamaker, J., Picone, J. and Ganapathiraju, A. (2002). A Sparse Modeling Approach to Speech Recognition Based on Relevance Vector Machines, Proceedings of the International Conference of Spoken Language Processing, vol. 2, pp. 1001-1004, Denver, Colorado, USA.
Figure  1.    An example of a two-class problem where a maximum likelihood-derived decision surface is not optimal (adapted from []). In the exploded view, the shaded region indicates the error induced by modeling the separable data by Gaussians estimated using maximum likelihood. This case is common for data, such as speech, where there is overlap in the feature space or where class boundaries are adjacent.
Figure  2.    The goal in training a classification system is to minimize the expected risk. As the VC dimension of the classifier increases, our confidence in the generalization ability of the machine decreases. The learning machine becomes too complex and suffers from overfitting of the training set. This demonstrates the principle of Occam’s razor where the simplest (lowest VC dimension) classifier is chosen that will attain a low empirical risk. The ability to automatically learn the optimal trade-off between these two factors is the most compelling feature of the SVM theory.
Figure  3.    Difference between empirical risk minimization and structural risk minimization for a simple example involving a hyperplane classifier. Each hyperplane (,  and ) achieves perfect classification and, hence, zero empirical risk. However,  is the optimal hyperplane because it maximizes the margin — the distance between the hyperplanes  and . Maximizing the margin indirectly results in better generalization.
Figure  4.    Definition of a linear hyperplane classifier. SVMs are constructed by maximizing the margin. The support vectors are shown as concentric circles.

Approach	Error Rate	
K-Nearest Neighbor	44%	
Gaussian Node Network	44%	
SVM: Polynomial Kernels	49%	
SVM: RBF Kernels	35%	
Separable Mixture Models	30%	
RVM: RBF Kernels	30%	

Table 1.    
The SVM classifier using an RBF kernel provides significantly better performance than other comparable classifiers.
Table 2.    
Experimental results comparing the baseline HMM system to a new hybrid SVM/HMM system. The hybrid system marginally outperforms the HMM system when used as a post-processor to the output of the HMM system. However, when the reference transcription is included, SVMs offer a significant improvement in performance.
Table 3.    
Performance comparison of SVMs and RVMs on an Alphadigit recognition data. A training set of 10,000 examples was used to train both models. The RVM model achieves a significantly lower word error rate while using less than one-tenth the number of parameters. This translates directly into improved real-time performance during testing.

Information Source		HMM		Hybrid		
Transcription	Segmentation	AD	SWB	AD	SWB	
N-best	Hypothesis*	11.9	41.6	11.0	40.6	
N-best	N-best	11.9	42.3	11.8	42.1	
N-best + Ref.	Reference**	—	—	3.3	5.8	
N-best + Ref.	N-best + Ref.	11.9	38.6	9.1	38.1	

*	For the baseline HMM system, recognition is performed by rescoring a word-lattice composed with the N-best list.
**	For the baseline HMM system, experiments were not performed due to the lack of a time-aligned decoding mechanism.

Approach	Word Error Rate	Avg # Parameters	Training
Time	Testing
Time	
SVM: RBF Kernels	15.5%	994	3 hours	1.5 hours	
RVM: RBF Kernels	14.8%	72	5 days	5 minutes