Application of the Dice Coefficient to Accuracy Assessment of Object-Based Image Classification

Abstract

A methodology is proposed to assess the accuracy of individual classes within the context of an object-based image classification scenario. The Dice Coefficient (DC) and bootstrapping techniques are employed to assess the level and statistical significance of overlap between reference and candidate image object pairs. Two approaches are used to optimize object extraction parameterization. First, rates of acceptable matches observed for the ensemble of reference objects can be used to estimate conventional measures of performance such as aggregate producer and user accuracies. Second, a novel assessment methodology is proposed that analyzes the significance of changes in the DC of individual overlap cases with changing ordinal threshold. This technique provides useful insights into the gain/loss trade-offs of acceptable matches with changing threshold level. Practical application of these methodologies is presented for the case of evaluating one-to-one reference/image object correspondence. An in-depth accuracy analysis is presented of the identification of 543 core hole drilling sites associated with oil sands development from RapidEye imagery using an image object extraction methodology based on grey-level ordinal thresholding. Although producer accuracy is limited to a maximum value of 69% due to adjacency of many core sites with other manmade structures, a simple shape regularity constraint (fraction of image object pixels that are boundary pixels) results in high user accuracy (87%). Finally, 2 additional issues are raised and discussed. First, selection of an acceptable match (i.e., DC) threshold must take into account differences between reference and image objects arising from their differing extraction approaches. This primarily impacts the boundary pixel portion of an object, which in turn is dependent on object size and shape. Second, for scenarios of targeted object classification, (i.e., most of an image is unclassified), an alternate strategy is utilized for reference-data acquisition. This involves acquiring comprehensive reference information for selected subsites to ensure proper estimates of commission.