Supervised Classification¶
Every classifier must be initialized with a specific set of
parameters. Two distinct methods are deployed for the training
(compute()
) and the testing (predict()
)
phases. Whenever possible, the real valued prediction is stored in the
realpred variable.
Support Vector Machines (SVMs)¶
-
class
mlpy.
Svm
(kernel='linear', kp=0.1, C=1.0, tol=0.001, eps=0.001, maxloops=1000, cost=0.0, alpha_tversky=1.0, beta_tversky=1.0, opt_offset=True)¶ Support Vector Machines (SVM).
Example: >>> import numpy as np >>> import mlpy >>> xtr = np.array([[1.0, 2.0, 3.0, 1.0], # first sample ... [1.0, 2.0, 3.0, 2.0], # second sample ... [1.0, 2.0, 3.0, 1.0]]) # third sample >>> ytr = np.array([1, -1, 1]) # classes >>> mysvm = mlpy.Svm() # initialize Svm class >>> mysvm.compute(xtr, ytr) # compute SVM 1 >>> mysvm.predict(xtr) # predict SVM model on training data array([ 1, -1, 1]) >>> xts = np.array([4.0, 5.0, 6.0, 7.0]) # test point >>> mysvm.predict(xts) # predict SVM model on test point -1 >>> mysvm.realpred # real-valued prediction -5.5 >>> mysvm.weights(xtr, ytr) # compute weights on training data array([ 0., 0., 0., 1.])
Initialize the Svm class
Parameters: - kernel : string [‘linear’, ‘gaussian’, ‘polynomial’, ‘tr’, ‘tversky’]
kernel
- kp : float
kernel parameter (two sigma squared) for gaussian and polynomial kernel
- C : float
regularization parameter
- tol : float
tolerance for testing KKT conditions
- eps : float
convergence parameter
- maxloops : integer
maximum number of optimization loops
- cost : float [-1.0, ..., 1.0]
for cost-sensitive classification
- alpha_tversky : float
positive multiplicative parameter for the norm of the first vector
- beta_tversky : float
positive multiplicative parameter for the norm of the second vector
- opt_offset : bool
compute the optimal offset
-
compute
(x, y)¶ Compute SVM model
Parameters: - x : 2d ndarray float (samples x feats)
training data
- y : 1d ndarray integer (-1 or 1)
classes
Returns: - conv : integer
svm convergence (0: false, 1: true)
-
predict
(p)¶ Predict svm model on a test point(s)
Parameters: - p : 1d or 2d ndarray float (samples x feats)
test point(s)training dataInput
Returns: - cl : integer or 1d ndarray integer
class(es) predicted
Attributes: - Svm.realpred : float or 1d ndarray float
real valued prediction
-
weights
(x, y)¶ Return feature weights
Parameters: - x : 2d ndarray float (samples x feats)
training data
- y : 1d ndarray integer (-1 or 1)
classes
Returns: - fw : 1d ndarray float
feature weights
Note
For tr kernel (Terminated Ramp Kernel) see [Merler06].
K Nearest Neighbor (KNN)¶
-
class
mlpy.
Knn
(k, dist='se')¶ k-Nearest Neighbor (KNN).
Example:
>>> import numpy as np >>> import mlpy >>> xtr = np.array([[1.0, 2.0, 3.1, 1.0], # first sample ... [1.0, 2.0, 3.0, 2.0], # second sample ... [1.0, 2.0, 3.1, 1.0]]) # third sample >>> ytr = np.array([1, -1, 1]) # classes >>> myknn = mlpy.Knn(k = 1) # initialize knn class >>> myknn.compute(xtr, ytr) # compute knn 1 >>> myknn.predict(xtr) # predict knn model on training data array([ 1, -1, 1]) >>> xts = np.array([4.0, 5.0, 6.0, 7.0]) # test point >>> myknn.predict(xts) # predict knn model on test point -1 >>> myknn.realpred # real-valued prediction 0.0
Initialize the Knn class.
Parameters: - k : int (odd > = 1)
number of NN
- dist : string (‘se’ = SQUARED EUCLIDEAN, ‘e’ = EUCLIDEAN)
adopted distance
-
compute
(x, y)¶ Store x and y data.
Parameters: - x : 2d ndarray float (samples x feats)
training data
- y : 1d ndarray integer (-1 or 1 for binary classification)
: 1d ndarray integer (1, ..., nclasses for multiclass classificatio) classes
Returns: 1
Raises: - ValueError
if not (1 <= k <= #samples)
- ValueError
if there aren’e at least 2 classes
- ValueError
if, in case of 2-classes problems, the lables are not 1 and -1
- ValueError
if, in case of n-classes problems, the lables are not int from 1 to n
-
predict
(p)¶ Predict knn model on a test point(s).
Parameters: - p : 1d or 2d ndarray float (sample(s) x feats)
test sample(s)
Returns: the predicted value(s) on success: integer or 1d numpy array integer (-1 or 1) for binary classification integer or 1d numpy array integer (1, ..., nclasses) for multiclass classification 0 on succes with non unique classification -2 otherwise
Raises: - StandardError
if no Knn method computed
Fisher Discriminant Analysis (FDA)¶
Described in [Mika01].
-
class
mlpy.
Fda
(C=1)¶ Fisher Discriminant Analysis.
Example:
>>> import numpy as np >>> import mlpy >>> xtr = np.array([[1.0, 2.0, 3.1, 1.0], # first sample ... [1.0, 2.0, 3.0, 2.0], # second sample ... [1.0, 2.0, 3.1, 1.0]]) # third sample >>> ytr = np.array([1, -1, 1]) # classes >>> myfda = mlpy.Fda() # initialize fda class >>> myfda.compute(xtr, ytr) # compute fda 1 >>> myfda.predict(xtr) # predict fda model on training data array([ 1, -1, 1]) >>> xts = np.array([4.0, 5.0, 6.0, 7.0]) # test point >>> myfda.predict(xts) # predict fda model on test point -1 >>> myfda.realpred # real-valued prediction -42.51475717037367 >>> myfda.weights(xtr, ytr) # compute weights on training data array([ 9.60629896, 9.77148463, 9.82027615, 11.58765243])
Initialize Fda class.
Parameters: - C : float
regularization parameter
-
compute
(x, y)¶ Compute fda model.
Parameters: - x : 2d numpy array float (sample x feature)
training data
- y : 1d numpy array integer (two classes, 1 or -1)
classes
Returns: 1
-
predict
(p)¶ Predict fda model on test point(s).
Parameters: - p : 1d or 2d ndarray float (sample(s) x feats)
test sample(s)
Returns: - cl : integer or 1d numpy array integer
class(es) predicted
Attributes: - self.realpred : float or 1d numpy array float
real valued prediction
-
weights
(x, y)¶ Return feature weights.
Parameters: - x : 2d ndarray float (samples x feats)
training data
- y : 1d ndarray integer (-1 or 1)
classes
Returns: - fw : 1d ndarray float
feature weights
Spectral Regression Discriminant Analysis (SRDA)¶
Described in [Cai08].
-
class
mlpy.
Srda
(alpha=1.0)¶ Spectral Regression Discriminant Analysis (SRDA).
Example:
>>> import numpy as np >>> import mlpy >>> xtr = np.array([[1.0, 2.0, 3.1, 1.0], # first sample ... [1.0, 2.0, 3.0, 2.0], # second sample ... [1.0, 2.0, 3.1, 1.0]]) # third sample >>> ytr = np.array([1, -1, 1]) # classes >>> mysrda = mlpy.Srda() # initialize srda class >>> mysrda.compute(xtr, ytr) # compute srda 1 >>> mysrda.predict(xtr) # predict srda model on training data array([ 1, -1, 1]) >>> xts = np.array([4.0, 5.0, 6.0, 7.0]) # test point >>> mysrda.predict(xts) # predict srda model on test point -1 >>> mysrda.realpred # real-valued prediction -6.8283034257748758 >>> mysrda.weights(xtr, ytr) # compute weights on training data array([ 0.10766721, 0.21533442, 0.51386623, 1.69331158])
Initialize the Srda class.
Parameters: - alpha : float(>=0.0)
regularization parameter
-
compute
(x, y)¶ - Compute Srda model.
- Initialize array of alphas a.
Parameters: - x : 2d ndarray float (samples x feats)
training data
- y : 1d ndarray integer (-1 or 1)
classes
Returns: 1
Raises: - LinAlgError
if x is singular matrix in __PenRegrModel
-
predict
(p)¶ Predict Srda model on test point(s).
Parameters: - p : 1d or 2d ndarray float (sample(s) x feats)
test sample(s)
Returns: - cl : integer or 1d numpy array integer
class(es) predicted
Attributes: - self.realpred : float or 1d numpy array float
real valued prediction
-
weights
(x, y)¶ Return feature weights.
Parameters: - x : 2d ndarray float (samples x feats)
training data
- y : 1d ndarray integer (-1 or 1)
classes
Returns: - fw : 1d ndarray float
feature weights
Penalized Discriminant Analysis (PDA)¶
Described in [Ghosh03].
-
class
mlpy.
Pda
(Nreg=3)¶ Penalized Discriminant Analysis (PDA).
Example:
>>> import numpy as np >>> import mlpy >>> xtr = np.array([[1.0, 2.0, 3.1, 1.0], # first sample ... [1.0, 2.0, 3.0, 2.0], # second sample ... [1.0, 2.0, 3.1, 1.0]]) # third sample >>> ytr = np.array([1, -1, 1]) # classes >>> mypda = mlpy.Pda() # initialize pda class >>> mypda.compute(xtr, ytr) # compute pda 1 >>> mypda.predict(xtr) # predict pda model on training data array([ 1, -1, 1]) >>> xts = np.array([4.0, 5.0, 6.0, 7.0]) # test point >>> mypda.predict(xts) # predict pda model on test point -1 >>> mypda.realpred # real-valued prediction -7.6106885609535624 >>> mypda.weights(xtr, ytr) # compute weights on training data array([ 4.0468174 , 8.0936348 , 18.79228266, 58.42466988])
Initialize Pda class.
Parameters: - Nreg : int
number of regressions
-
compute
(x, y)¶ Compute Pda model.
Parameters: - x : 2d ndarray float (samples x feats)
training data
- y : 1d ndarray integer (-1 or 1)
classes
Returns: 1
Raises: - LinAlgError
if x is singular matrix in __PenRegrModel
-
predict
(p)¶ Predict Pda model on test point(s).
Parameters: - p : 1d or 2d ndarray float (sample(s) x feats)
test sample(s)
Returns: - cl : integer or 1d numpy array integer
class(es) predicted
Attributes: - self.realpred : float or 1d numpy array float
real valued prediction
-
weights
(x, y)¶ Compute feature weights.
Parameters: - x : 2d ndarray float (samples x feats)
training data
- y : 1d ndarray integer (-1 or 1)
classes
Returns: - fw : 1d ndarray float
feature weights
Diagonal Linear Discriminant Analysis (DLDA)¶
-
class
mlpy.
Dlda
(nf=0, tol=10, overview=False, bal=False)¶ Diagonal Linear Discriminant Analysis.
Example:
>>> from numpy import * >>> from mlpy import * >>> xtr = array([[1.1, 2.4, 3.1, 1.0], # first sample ... [1.2, 2.3, 3.0, 2.0], # second sample ... [1.3, 2.2, 3.5, 1.0], # third sample ... [1.4, 2.1, 3.2, 2.0]]) # fourth sample >>> ytr = array([1, -1, 1, -1]) # classes >>> mydlda = Dlda(nf = 2) # initialize dlda class >>> mydlda.compute(xtr, ytr) # compute dlda 1 >>> mydlda.predict(xtr) # predict dlda model on training data array([ 1, -1, 1, -1]) >>> xts = array([4.0, 5.0, 6.0, 7.0]) # test point >>> mydlda.predict(xts) # predict dlda model on test point -1 >>> mydlda.realpred # real-valued prediction -21.999999999999954 >>> mydlda.weights(xtr, ytr) # compute weights on training data array([ 2.13162821e-14, 0.00000000e+00, 0.00000000e+00, 4.00000000e+00])
Initialize Dlda class.
Parameters: - nf : int (1 <= nf >= #features)
the number of the best features that you want to use in the model. If nf = 0 the system stops at a number of features corresponding to a peak of accuracy
- tol : int
in case of nf = 0 it’s the number of steps of classification to be calculated after the peak to avoid a local maximum
- overview : bool
set True to print informations about the accuracy of the classifier at every step of the compute
- bal : bool
set True if it’s reasonable to consider the unbalancement of the test set similar to the one of the training set
-
compute
(x, y, mf=0)¶ Compute Dlda model.
Parameters: - x : 2d ndarray float (samples x feats)
training data
- y : 1d ndarray integer (-1 or 1)
classes
- mf : int
number of classification steps to be calculated more on a model already computed
Returns: 1
Raises: - LinAlgError
if x is singular matrix
-
predict
(p)¶ Predict Dlda model on test point(s).
Parameters: - p : 1d or 2d ndarray float (sample(s) x feats)
test sample(s)
Returns: - cl : integer or 1d numpy array integer
class(es) predicted
Attributes: - self.realpred : float or 1d numpy array float
real valued prediction
-
weights
(x, y)¶ Return feature weights.
Parameters: - x : 2d ndarray float (samples x feats)
training data
- y : 1d ndarray integer (-1 or 1)
classes
Returns: - fw : 1d ndarray float
feature weights, they are going to be > 0 for the features chosen for the classification and = 0 for all the others
[Vapnik95] | V Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, 1995. |
[Cristianini] | N Cristianini and J Shawe-Taylor. An introduction to support vector machines. Cambridge University Press. |
[Merler06] | S Merler and G Jurman. Terminated Ramp - Support Vector Machine: a nonparametric data dependent kernel. Neural Network, 19:1597-1611, 2006. |
[Nasr09] |
|
[Mika01] | S Mika and A Smola and B Scholkopf. An improved training algorithm for kernel fisher discriminants. Proceedings AISTATS 2001, 2001. |
[Cristianini02] | N Cristianini, J Shawe-Taylor and A Elisseeff. On Kernel-Target Alignment. Advances in Neural Information Processing Systems, Volume 14, 2002. |
[Cai08] | D Cai, X He, J Han. SRDA: An Efficient Algorithm for Large-Scale Discriminant Analysis. Knowledge and Data Engineering, IEEE Transactions on Volume 20, Issue 1, Jan. 2008 Page(s):1 - 12. |
[Ghosh03] | D Ghosh. Penalized discriminant methods for the classification of tumors from gene expression data. Biometrics on Volume 59, Dec. 2003 Page(s):992 - 1000(9). |