Kernel Methods and the Kernel Trick
Kernel methods are a powerful tool for the evaluation of many different machine learning models. In this article we will discuss their applications as well as some of the mathematical intuition behind them.
“Many linear parametric models can be re-cast into an equivalent ‘dual representation’ in which the predictions are also based on linear combinations of a kernel function evaluated at the training data points.” For models based on a fixed nonlinear feature space mapping φ(x), the kernel function is given by:
k(x, x’ ) = φ(x)^Tφ(x’) (Bishop, 2006)
With the power of these dual representations, we can represent high dimensional data in a much simpler way by avoiding the introduction of the feature vector φ(x) which often has very high dimensionality.
Kernel methods are a class of machine learning algorithms that make use of high-dimensional data but are more computationally efficient than other methods. Most algorithms in machine learning are based on some kind of kernel function, and in practice these methods can be viewed as approximations to the desired result. They are widely used for feature extraction, linear and nonlinear classification, clustering and dimensionality reduction.
These methods use ideas from statistics to better deal with problems where dimensionality is high, or where more than two classes are present. For example, kernel PCA is used to help with classification of high dimensional data whose decision boundary is described by a non-linear function. In a situation like this, the kernel function becomes important as it can map this non-linear function to a dimension or feature space in which this decision boundary function becomes linear. In doing so we increase the dimension we are working with, which we combat by using the kernel trick. Our kernel function accepts inputs in the original lower dimensional space and returns values in the higher dimensional space which we established to create linearity in the decision boundary. The process for kernel PCA is as follows:
Kernel methods are being used more frequently as techniques in machine learning. Kernelized multivariate methods have been applied to diverse problems in statistics, signal processing and pattern recognition as they provide a way to represent nonlinear relationships in data. Additionally, they can be used to reduce noise in a dataset as well as find subsets of data that are important but not represented by the other features.
All in all, kernel methods are a useful mechanism for simplifying model training by making use of a feature space rather than the input space, which in turn may result in an increase in the performance of machine learning models.
Sources/Further Links:
Bishop, Christopher M. Pattern Recognition and Machine Learning
