Metrics/Scoring

• Three Scikit APIs of note:

  • estimator score methods: provides a default evaluation method.
  • scoring parameter: used by Cross Validation tools.
  • metric functions: used to compute prediction error in specific situations.

• dummy estimators provide baseline metric values for random predictions.

• pairwise metrics provide metrics between samples. They are not estimators or predictors.

scoring - defines model evaluation rules

• Model selection & evaluation tools (Grid Search, cross_val_score) use scoring to decide which metric to apply to an estimator.

• functions ending in _score: higher values = "better".

• functions ending with _error or _loss: lower values = "better".

• Sometimes you'll need to wrap a scoring function with make_scorer to make it "callable" for model evaluations. The fbeta_score method is an example:

from sklearn.metrics import fbeta_score, make_scorer
from sklearn.model_selection import GridSearchCV
from sklearn.svm import LinearSVC

ftwo_scorer = make_scorer(fbeta_score, beta=2)

grid = GridSearchCV(LinearSVC(), 
                    param_grid = {'C': [1, 10]},
                    scoring    = ftwo_scorer, cv=5)

Custom Scoring functions with make_scorer

• Build a custom scoring object with these components:
  • a python function
  • whether the function returns a score or a loss (greater_is_better is True (default) or False respectively.)
  • whether the function requires continuous decision certainties (needs_threshold=True). (For classification metrics only.)
  • any additional params.

import numpy as np
from sklearn.dummy import DummyClassifier

def my_custom_loss_func(y_true, y_pred):
    diff = np.abs(y_true - y_pred).max()
    return np.log1p(diff)

# score will negate the return value of my_custom_loss_func,
# which will be np.log(2), 0.693, given the values for X
# and y defined below.

score = make_scorer(my_custom_loss_func, greater_is_better=False)
X,y   = [[1], [1]], [0, 1]
clf   = DummyClassifier(strategy='most_frequent', 
                        random_state=0).fit(X, y)

my_custom_loss_func(y, clf.predict(X))
score(clf, X, y)

-0.6931471805599453

Multiple metrics

• Available in GridSearchCV, RandomizedSearchCV, cross_validate. Multiple ways to specify them.

# specified as iterable of strings
scoring = ['accuracy', 'precision']

# specified as a dict
from sklearn.metrics import accuracy_score
from sklearn.metrics import make_scorer
scoring = {'accuracy': make_scorer(accuracy_score),
           'prec': 'precision'}

# specified as a callable
from sklearn.model_selection import cross_validate
from sklearn.metrics import confusion_matrix
from sklearn.datasets import make_classification

def confusion_matrix_scorer(clf, X, y):
     y_pred = clf.predict(X)
     cm = confusion_matrix(y, y_pred)
     return {'tn': cm[0, 0], 'fp': cm[0, 1],
             'fn': cm[1, 0], 'tp': cm[1, 1]}
    
X, y = make_classification(n_classes=2, 
                                    random_state=0)
svm = LinearSVC(random_state=0)

cv_results = cross_validate(svm, X, y, cv=5,
                            scoring=confusion_matrix_scorer)

print(cv_results['test_tp'])
print(cv_results['test_fn'])

[10  9  8  7  8]
[0 1 2 3 2]