[数据集][COCO]目标检测任务评估

发表于 更新于 分类于
本文字数： 5.7k 阅读时长 ≈ 10 分钟

Image Localization Dataset是一个目标检测数据集，其标注文件使用了PASCAL VOC定义的格式，将其转换成COCO格式

Image Localization Dataset

其标注文件和图像文件放置在同一目录下，部分文件结构如下：

1
2
3
4
5
6
├── cucumber_10.jpg
├── cucumber_10.xml
├── cucumber_11.jpg
├── cucumber_11.xml
。。。
。。。

共包含了3个类别：cucumber、eggplant和mushroom

COCO格式

输入数据json文件包含了3个节点

1
2
3
4
5
6
{

    "images":            [image],
    "annotations":  [annotation],
    "categories":      [category],
}

其中image节点的数据结构如下：

1
2
3
4
5
6
{
    "id":                                    int,
    "width":                            int,
    "height":                           int,
    "file_name":                   str,
}

annotation节点的数据结构如下：

1
2
3
4
5
6
7
8
9
{
    "id":                                  int,
    "image_id":                   int,
    "category_id":              int,
    "area":                             float,
    "bbox":                            [x,y,width,height],
    "iscrowd":                      0,
    "segmentation":          []
}

category节点的数据结构如下：

1
2
3
4
5
{
    "id":                                 int,
    "name":                         str,
    "supercategory":       "",
}]

实现

完整的实现代码参考Evaluation-Metrics/py/metrics/coco

测试代码参考Evaluation-Metrics/py/test_map_coco.py

实现步骤：

  1. 将标注文件转换成COCO格式的json文件 - gt.json
  2. 将训练结果转换成COCO格式的json文件 - dt.json
  3. 载入gt.json和dt.json，计算评价指标
1
2
3
4
5
6
7
8
# initialize CocoEval object
E = COCOeval(cocoGt, cocoDt, iouType='bbox')
# run per image evaluation
E.evaluate()
# accumulate per image results
E.accumulate()
# display summary metrics of results
E.summarize()

参数解析

COCO提供了一个参数类Param来调整计算

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
class Params:
    '''
    Params for coco evaluation api
    '''
    def setDetParams(self):
        self.imgIds = []
        self.catIds = []
        # np.arange causes trouble.  the data point on arange is slightly larger than the true value
        self.iouThrs = np.linspace(.5, 0.95, int(np.round((0.95 - .5) / .05)) + 1, endpoint=True)
        self.recThrs = np.linspace(.0, 1.00, int(np.round((1.00 - .0) / .01)) + 1, endpoint=True)
        self.maxDets = [1, 10, 100]
        self.areaRng = [[0 ** 2, 1e5 ** 2], [0 ** 2, 32 ** 2], [32 ** 2, 96 ** 2], [96 ** 2, 1e5 ** 2]]
        self.areaRngLbl = ['all', 'small', 'medium', 'large']
        self.useCats = 1

    。。。
    。。。

    def __init__(self, iouType='segm'):
        if iouType == 'segm' or iouType == 'bbox':
            self.setDetParams()
        elif iouType == 'keypoints':
            self.setKpParams()
        else:
            raise Exception('iouType not supported')
        self.iouType = iouType
        # useSegm is deprecated
        self.useSegm = None

IoU设置

self.iouThrs = np.linspace(.5, 0.95, int(np.round((0.95 - .5) / .05)) + 1, endpoint=True) 输出：array([0.5 , 0.55, 0.6 , 0.65, 0.7 , 0.75, 0.8 , 0.85, 0.9 , 0.95])

可以输入固定的IoU进行计算

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# initialize CocoEval object
E = COCOeval(cocoGt, cocoDt, iouType='bbox')
# set parameters as desired
E.params.iouThrs = [0.5]
# run per image evaluation
E.evaluate()
# accumulate per image results
E.accumulate()
# display summary metrics of results
E.summarize()
###################### 输出
 Average Precision  (AP) @[ IoU=0.50:0.50 | area=   all | maxDets=100 ] = 0.976
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.50 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=100 ] = 0.751
 Average Precision  (AP) @[ IoU=0.50:0.50 | area= large | maxDets=100 ] = 0.984
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=   all | maxDets=  1 ] = 0.973
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=   all | maxDets= 10 ] = 0.989
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=   all | maxDets=100 ] = 0.989
 Average Recall     (AR) @[ IoU=0.50:0.50 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=100 ] = 0.917
 Average Recall     (AR) @[ IoU=0.50:0.50 | area= large | maxDets=100 ] = 0.994

从运行结果可以发现AP/AR大幅提高了

maxDets设置

self.maxDets = [1, 10, 100]

使用3个不同的maxDets参数，可以调整最大检测数目

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# initialize CocoEval object
E = COCOeval(cocoGt, cocoDt, iouType='bbox')
# set parameters as desired
E.params.maxDets = [10, 100, 300]
E.params.iouThrs = [0.5]
# run per image evaluation
E.evaluate()
# accumulate per image results
E.accumulate()
# display summary metrics of results
E.summarize()
############################# 输出
 Average Precision  (AP) @[ IoU=0.50:0.50 | area=   all | maxDets=100 ] = 0.976
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=300 ] = -1.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=300 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.50 | area= small | maxDets=300 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=300 ] = 0.751
 Average Precision  (AP) @[ IoU=0.50:0.50 | area= large | maxDets=300 ] = 0.984
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=   all | maxDets= 10 ] = 0.989
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=   all | maxDets=100 ] = 0.989
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=   all | maxDets=300 ] = 0.989
 Average Recall     (AR) @[ IoU=0.50:0.50 | area= small | maxDets=300 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=300 ] = 0.917
 Average Recall     (AR) @[ IoU=0.50:0.50 | area= large | maxDets=300 ] = 0.994

如何计算某一类别的AP/AR

self.catIds = []

可以指定哪个类别的AP/AR计算，实现如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# initialize CocoEval object
E = COCOeval(cocoGt, cocoDt, iouType='bbox')
# set parameters as desired
E.params.catIds = [1]
E.params.maxDets = [10, 100, 300]
E.params.iouThrs = [0.5]
# run per image evaluation
E.evaluate()
# accumulate per image results
E.accumulate()
# display summary metrics of results
E.summarize()
################################ 输出
 Average Precision  (AP) @[ IoU=0.50:0.50 | area=   all | maxDets=100 ] = 0.990
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=300 ] = -1.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=300 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.50 | area= small | maxDets=300 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=300 ] = 1.000
 Average Precision  (AP) @[ IoU=0.50:0.50 | area= large | maxDets=300 ] = 0.994
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=   all | maxDets= 10 ] = 1.000
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=   all | maxDets=100 ] = 1.000
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=   all | maxDets=300 ] = 1.000
 Average Recall     (AR) @[ IoU=0.50:0.50 | area= small | maxDets=300 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=300 ] = 1.000
 Average Recall     (AR) @[ IoU=0.50:0.50 | area= large | maxDets=300 ] = 1.000

相关阅读