[数据集][COCO]目标检测任务评估

参考:1. Detection Evaluation

Image Localization Dataset是一个目标检测数据集,其标注文件使用了PASCAL VOC定义的格式,将其转换成COCO格式

Image Localization Dataset

其标注文件和图像文件放置在同一目录下,部分文件结构如下:

1
2
3
4
5
6
├── cucumber_10.jpg
├── cucumber_10.xml
├── cucumber_11.jpg
├── cucumber_11.xml
。。。
。。。

共包含了3个类别:cucumber、eggplant和mushroom

COCO格式

参考:[数据集]COCO简介

输入数据json文件包含了3个节点

1
2
3
4
5
6
{

"images": [image],
"annotations": [annotation],
"categories": [category],
}

其中image节点的数据结构如下:

1
2
3
4
5
6
{
"id": int,
"width": int,
"height": int,
"file_name": str,
}

annotation节点的数据结构如下:

1
2
3
4
5
6
7
8
9
{
"id": int,
"image_id": int,
"category_id": int,
"area": float,
"bbox": [x,y,width,height],
"iscrowd": 0,
"segmentation": []
}

category节点的数据结构如下:

1
2
3
4
5
{
"id": int,
"name": str,
"supercategory": "",
}]

实现

完整的实现代码参考Evaluation-Metrics/py/metrics/coco

测试代码参考Evaluation-Metrics/py/test_map_coco.py

实现步骤:

  1. 将标注文件转换成COCO格式的json文件 - gt.json
  2. 将训练结果转换成COCO格式的json文件 - dt.json
  3. 载入gt.jsondt.json,计算评价指标
1
2
3
4
5
6
7
8
# initialize CocoEval object
E = COCOeval(cocoGt, cocoDt, iouType='bbox')
# run per image evaluation
E.evaluate()
# accumulate per image results
E.accumulate()
# display summary metrics of results
E.summarize()

参数解析

COCO提供了一个参数类Param来调整计算

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
class Params:
'''
Params for coco evaluation api
'''
def setDetParams(self):
self.imgIds = []
self.catIds = []
# np.arange causes trouble. the data point on arange is slightly larger than the true value
self.iouThrs = np.linspace(.5, 0.95, int(np.round((0.95 - .5) / .05)) + 1, endpoint=True)
self.recThrs = np.linspace(.0, 1.00, int(np.round((1.00 - .0) / .01)) + 1, endpoint=True)
self.maxDets = [1, 10, 100]
self.areaRng = [[0 ** 2, 1e5 ** 2], [0 ** 2, 32 ** 2], [32 ** 2, 96 ** 2], [96 ** 2, 1e5 ** 2]]
self.areaRngLbl = ['all', 'small', 'medium', 'large']
self.useCats = 1

。。。
。。。

def __init__(self, iouType='segm'):
if iouType == 'segm' or iouType == 'bbox':
self.setDetParams()
elif iouType == 'keypoints':
self.setKpParams()
else:
raise Exception('iouType not supported')
self.iouType = iouType
# useSegm is deprecated
self.useSegm = None

IoU设置

self.iouThrs = np.linspace(.5, 0.95, int(np.round((0.95 - .5) / .05)) + 1, endpoint=True)
输出:array([0.5 , 0.55, 0.6 , 0.65, 0.7 , 0.75, 0.8 , 0.85, 0.9 , 0.95])

可以输入固定的IoU进行计算

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# initialize CocoEval object
E = COCOeval(cocoGt, cocoDt, iouType='bbox')
# set parameters as desired
E.params.iouThrs = [0.5]
# run per image evaluation
E.evaluate()
# accumulate per image results
E.accumulate()
# display summary metrics of results
E.summarize()
###################### 输出
Average Precision (AP) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.976
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.50 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=100 ] = 0.751
Average Precision (AP) @[ IoU=0.50:0.50 | area= large | maxDets=100 ] = 0.984
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets= 1 ] = 0.973
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets= 10 ] = 0.989
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.989
Average Recall (AR) @[ IoU=0.50:0.50 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=100 ] = 0.917
Average Recall (AR) @[ IoU=0.50:0.50 | area= large | maxDets=100 ] = 0.994

从运行结果可以发现AP/AR大幅提高了

maxDets设置

self.maxDets = [1, 10, 100]

使用3个不同的maxDets参数,可以调整最大检测数目

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# initialize CocoEval object
E = COCOeval(cocoGt, cocoDt, iouType='bbox')
# set parameters as desired
E.params.maxDets = [10, 100, 300]
E.params.iouThrs = [0.5]
# run per image evaluation
E.evaluate()
# accumulate per image results
E.accumulate()
# display summary metrics of results
E.summarize()
############################# 输出
Average Precision (AP) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.976
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=300 ] = -1.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=300 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.50 | area= small | maxDets=300 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=300 ] = 0.751
Average Precision (AP) @[ IoU=0.50:0.50 | area= large | maxDets=300 ] = 0.984
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets= 10 ] = 0.989
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.989
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=300 ] = 0.989
Average Recall (AR) @[ IoU=0.50:0.50 | area= small | maxDets=300 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=300 ] = 0.917
Average Recall (AR) @[ IoU=0.50:0.50 | area= large | maxDets=300 ] = 0.994

如何计算某一类别的AP/AR

self.catIds = []

可以指定哪个类别的AP/AR计算,实现如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# initialize CocoEval object
E = COCOeval(cocoGt, cocoDt, iouType='bbox')
# set parameters as desired
E.params.catIds = [1]
E.params.maxDets = [10, 100, 300]
E.params.iouThrs = [0.5]
# run per image evaluation
E.evaluate()
# accumulate per image results
E.accumulate()
# display summary metrics of results
E.summarize()
################################ 输出
Average Precision (AP) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.990
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=300 ] = -1.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=300 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.50 | area= small | maxDets=300 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=300 ] = 1.000
Average Precision (AP) @[ IoU=0.50:0.50 | area= large | maxDets=300 ] = 0.994
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets= 10 ] = 1.000
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 1.000
Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=300 ] = 1.000
Average Recall (AR) @[ IoU=0.50:0.50 | area= small | maxDets=300 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=300 ] = 1.000
Average Recall (AR) @[ IoU=0.50:0.50 | area= large | maxDets=300 ] = 1.000
坚持原创技术分享,您的支持将鼓励我继续创作!