ROC Analysis

Typical recognition system may yield decisions at multiple operating points. ROC analysis enables the system designer to select a desirable operating point based on the recognition performance estimated on an independent test set.

Introduction

ROC analysis comprises estimation of algorithm performance over a set of operating points. Traditionally, both the admissible operating points and the related performances are estimated from the soft outputs of the classifier executed on the independent test set. Under PRSD Toolbox, the ROC analysis is invoked via the sdroc function. Example of construction of ROC curve for a simple two-class problem. 

>> a=gendatb; a=setlablist(a,strvcat({'apple','pear'}))
Banana Set, 100 by 2 dataset with 2 classes: [50  50]
>> [tr,ts]=gendat(a,0.5)
Banana Set, 50 by 2 dataset with 2 classes: [25  25]
Banana Set, 50 by 2 dataset with 2 classes: [25  25]
>> w=fisherc(tr) % train the classifier
Fisher, 2 to 2 trained classifier --> affine

>> out=ts*w % soft-outputs estimated on the test set
Banana Set, 50 by 2 dataset with 2 classes: [25  25]
>> r=sdroc(out)
ROC (31 w-based op.points, 3 measures), curop: 1
est: 1:err(apple)=0.08, 2:err(pear)=0.16, 3:mean-error [0.50,0.50]=0.12

The result of the ROC analysis is the object r of class sdroc. It contains the set of operating points and the corresponding performance estimates. The operating points may be accessed directly using sddecide: 

>> wd=sddecide(r)
Weight-based decision (2 classes, 31 ops) at op 1, 2 to 1 trained  mapping   --> sddecide
>> lab=ts(1:3,:)*w*wd % perform decisions

lab =

apple
apple
apple

Accesing the performance estimates

The sdroc object provides direct access to the performance estimates for all stored operating points. The object behaves like a Matlab matrix with rows representing the operating points and columns the performance measures. The ROC estimated above contains 31 operating points and three measures, namely the error for each of the classes and the mean error assuming equal priors. We can access the estimates for the first four operating points by: 

>> r(1:4)

ans =

    0.0800    0.1600    0.1200
    1.0000         0    0.5000
    0.8000         0    0.4000
    0.4800         0    0.2400

The soft-output weights used by these four operating points are: 

>> t=getopsdata(r); % returns all the weights i.e. 31x2 matrix
>> t(1:4,:)

ans =

    0.5000    0.5000
         0    1.0000
    0.0345    0.9655
    0.0690    0.9310

The first operating point represents the situation where both class outputs are weighted equally. The second point corresponds to the extreme situation where the class (apple) is enitrely neglected.

The ROC curve may be plotted using: 

>> sddrawroc(r)

The blue dots represent the available operating points. The black dot corresponds to the current operating point which is automatically set to the equal-weight situation. Current operating point may be changed by clicking. By moving the mouse pointer, we focus on individual operating points with the gray marker. The ROC plot title shows the index and the error estimates for the focus point.

By pressing the w key, we switch to the "weight" mode where the title shows the soft-output weights. The decision mapping with the current operating point may be saved to the Matlab workspace using the s key. The cursor keys allow for switching between different performance measures stored in the ROC object.

Constructing the thresholding-based ROC

Thresholding-based ROC is created by passing the dataset with a single soft-output corresponding to the desired thresholding target. The dataset feature label specifies the target, the other class present in the dataset the non-target for the decision. 

>> w=qdc(tr) % train the Gaussian model per class
Bayes-Normal-2, 2 to 2 trained  mapping   --> normal_map
>> out=ts*w
Banana Set, 50 by 2 dataset with 2 classes: [25  25]

>> r1=sdroc(out(:,1)) % thresholding the 'apple' model
ROC (30 thr-based op.points, 3 measures), curop: 1
est: 1:err(apple)=0.00, 2:err(pear)=1.00, 3:mean-error [0.50,0.50]=0.50
>> sddrawroc(r1)

When performing the thresholding-based decisions, we must pass only the target output to the sddecide mapping: 

>> out=ts*w % soft-outputs for both "apple" and "pear" models
Banana Set, 50 by 2 dataset with 2 classes: [25  25]
>> wd % decision mapping created via the ROC plot
Thr-based decision on apple (30 ops) at op 16, 1 to 1 trained  mapping   --> sddecide
>> out(1:4,1)*wd % we use only the first column i.e. output of the "apple" model

ans =

pear 
pear 
pear 
pear

Multi-class ROC analysis

PRSD Toolbox ROC analysis framework handles the multi-class situations similarly to the two-class case. When estimating a multi-class ROC, both the operating points and the related estimates are constructed by the optimization algorithm. Large number of randomly generated operating points is evaluated minimizing the maximum per-class error. The returned set is pruned preserving only the solutions with unique per-class errors. Although this algorithm is, of course, suboptimal, it allows fast investigation of error conditions for multi-class problems. 

>> a=gendatm(1000) % simple multi-class dataset
Multi-Class Problem, 1000 by 2 dataset with 8 classes: [125  117  125  132  138  112  125  126]
>> [tr,ts]=gendat(a,0.5)
Multi-Class Problem, 502 by 2 dataset with 8 classes: [63  59  63  66  69  56  63  63]
Multi-Class Problem, 498 by 2 dataset with 8 classes: [62  58  62  66  69  56  62  63]
>> w=qdc(tr)
Bayes-Normal-2, 2 to 8 trained  mapping   --> normal_map

>> out=ts*w % soft classifier outputs on the test set
Multi-Class Problem, 498 by 8 dataset with 8 classes: [62  58  62  66  69  56  62  63]
>> r=sdroc(out)
.....
ROC (9 w-based op.points, 8 measures), curop: 1
est: 1:err(a)=0.16, 2:err(b)=0.14, 3:err(c)=0.11, 4:err(d)=0.27, 5:err(e)=0.10, 6:err(f)=0.11,
 7:err(g)=0.21, 8:err(h)=0.05

Note that the 9 points found are very limited subset from the full 7D mesh applicable in the 8-class problem.

Next step: The basics: Algorithm evaluation