Performance Evaluation

Given a classifier trained on a particular training set (Training Set), it must be evaluated on another set (Validation Set or Certification Set). This comparison makes it possible to derive metrics for evaluating the classifier and comparing different classifiers. It is essential that performance metrics be computed on a set of samples not used during training (the validation set), in order to detect problems such as data overfitting, that is, a failure to generalize.

Once the classifier parameters have been fixed, the contingency table (Confusion Matrix) can be constructed:

		True Value
		p	n
Classification	p'	VP	FP
	n'	FN	VN

False Positives (FP) are also referred to as false alarms. False Negatives (FN) are referred to as misses.

The table is normally used to derive performance metrics such as:

• Accuracy is the ratio of the number of correct predictions to the total number of predictions = (VP+VN)/(VP+VN+FN+FP);
• Error Rate is the ratio of the total number of incorrect predictions to the total number of predictions = (FP+FN) / (VP+VN+FN+FP);
• Precision is the probability that a positive returned by the classifier is correct = VP / (VP+FP);
• Recall (or hit-rate or TPR) is the percentage of positives correctly recognized = VP / (VP+FN);
• Miss-Rate or FNR is the complement of Recall = 1-Recall = FN/(VP+FN).

Each classifier has one or more parameters that, when modified, change the ratio between correct detections and the number of false positives. It is therefore difficult to compare two classifiers objectively because, at the same threshold, one may produce more correct detections than the other but also a larger number of false positives. To compare the performance of different binary classifiers obtained from different training sessions, curves are therefore normally used to show performance as a function of the classifier's internal threshold.

The performance curves that can be used are:

• The ROC curve (Receiver Operating Characteristic) is a Cartesian graph in which the abscissa represents the number of false positives (percentage FPR, per frame, or absolute) and the ordinate represents the percentage of correct detections (True Positive Rate, TPR) generated by the classifier as the threshold varies. For a classifier to qualify as such, it must have an ROC curve better than that of a random classifier, namely the line connecting points $(0,0)$ and $(1,1)$ in the ROC graph.
• Precision-Recall (PRC) focuses the analysis primarily on positive cases. The area under the PRC is called Average Precision (AP). In real detection problems, the number of True Negatives is extremely large, so it is necessary to focus on false positives. The PRC curve has the major advantage of concealing the scale of the False Positives: these are normally specified per frame, per minute, or in another unit of measurement.
• Detection Error Tradeoff (DET) makes it possible to represent negative errors (misses) and positive errors (false alarms) on the axes. It is a curve that focuses the analysis purely on errors.

Finally, it should be noted that these metrics apply to any class of problems involving the concept of a correct or incorrect result. They are therefore applicable not only to classifiers but also, for example, to feature-point matching and other tasks.

More recently, to facilitate comparisons of classifier performance, functions have been proposed that, when applied to ROC curves, yield a single scalar representing a measure of classification quality. These functions are generally averages of samples from the ROC curve restricted to the regions of practical interest.