Performance Evaluation

Given a classifier trained on a particular training set (Training Set), it must be evaluated on another set (Validation Set or Certification Set). This comparison makes it possible to derive metrics for evaluating the classifier and comparing different classifiers. It is essential that performance metrics be computed on a set of samples not used during training (the validation set), in order to detect problems such as data overfitting, that is, a failure to generalize.

Once the classifier parameters have been fixed, the contingency table (Confusion Matrix) can be constructed:

		True Value
		p	n
Classification	p'	TP	FP
	n'	FN	TN

False Positives (FP) are also called false alarms, while False Negatives (FN) are called misses.

The following performance metrics are typically derived from the table:

• Accuracy is the ratio of the number of correct predictions to the total number of predictions = (TP+TN)/(TP+TN+FN+FP);
• Error Rate is the ratio of the total number of incorrect predictions to the total number of predictions = (FP+FN) / (TP+TN+FN+FP);
• Precision is the probability that a positive returned by the classifier is correct = TP / (TP+FP);
• Recall (or hit-rate or TPR) is the percentage of correctly recognized positives = TP / (TP+FN);
• Miss-Rate, or FNR, is the complement of Recall = 1-Recall = FN/(TP+FN).

Each classifier has one or more parameters that, when modified, change the ratio between correct detections and the number of false positives. It is therefore difficult to compare two classifiers objectively: at the same threshold, one may produce more correct detections than the other but also a greater number of false positives. To compare the performance of different binary classifiers obtained from different training sessions, curves are normally used to show performance as this internal classifier threshold varies.

The performance curves that may be encountered include:

• The ROC curve (Receiver Operating Characteristic) is a Cartesian graph whose abscissa represents the number of false positives (as a percentage, FPR, per frame, or in absolute terms) and whose ordinate represents the percentage of correct detections (True Positive Rate, TPR) generated by the classifier as the threshold varies. The diagonal connecting points $(0,0)$ and $(1,1)$ represents the expected performance of a random classifier^5.1.
• Precision-Recall (PRC), which focuses primarily on positives. The area under the PRC is called Average Precision (AP). In real detection problems, the number of True Negatives is very large, so it is necessary to focus on false positives. The PRC curve has the major advantage of concealing the scale of False Positives: these are normally reported per frame, per minute, or using another unit of measurement.
• Detection error Tradeoff (DET), which represents negative errors (misses) and positive errors (false alarms) on the axes. It is a curve that focuses exclusively on errors.

Finally, it should be noted that these metrics apply to any class of problems involving the concept of a correct or incorrect result. They therefore apply not only to classifiers but also, for example, to feature-point matching and other tasks.

More recently, to enable more streamlined comparisons of classifier performance, functions have been proposed that, when applied to ROC curves, yield a single scalar representing a classification-quality score. These functions are usually averages of samples from the ROC curve over the regions of practical interest.

Footnotes

... classifier^5.1

A good classifier produces a ROC curve predominantly above this diagonal; a curve below the diagonal indicates that the classifier systematically ranks the examples incorrectly and that inverting the decision rule could yield performance better than chance