为了解决困难目标发现与识别的问题，街景理解团队提出了一种基于样本感知、挖掘、净化的困难目标发现框架（Perceive, Excavate and Purify，简称PEP），如下图5所示：图5 基于样本自动挖掘的困难目标发现框架首先，利用特征金字塔主干网络提取不同尺度的特征，然后将不同尺度特征分别输入到三个分支：实例感知分支、实例描述分支、特征学习分支。实例感知分支对主干特征每个像素点进行分类，初步确定像素点位置是否存在实例；实例描述分支学习不同实例的原始特征描述子，同时利用样本挖掘子网络来挖掘困难目标，并表征为提取描述子。此外，引入实例关联子网络来提高相同实例的相似性、降低不同实例的相似性，实现目标净化，进一步提高分割性能。最后，原始的和挖掘的实例描述子与特征学习分支的通用特征实施卷积操作得到各个目标的分割结果。最终，与主流的分割方法相比，基于样本自动挖掘的困难目标发现方法在COCO[40]等数据集上达到领先的精度。
面向分割任务的基于时空对齐的复杂目标精准分割方法、基于样本自动挖掘的困难目标发现方法完善为2篇学术论文：《Motion-state Alignment for Video Semantic Segmentation》[44]、《Perceive, Excavate and Purify: A Novel Object Mining Framework for Instance Segmentation》[45]，被CVPR 2023接收为Workshop论文（8页长文）。
[1] Cordts, Marius and Omran, Mohamed and Ramos, Sebastian and Rehfeld, Timo and Enzweiler, Markus and Benenson, Rodrigo and Franke, Uwe and Roth, Stefan and Schiele, Bernt. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.[2] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding. In ICCV, 2021.[3] Kirillov, Alexander and He, Kaiming and Girshick, Ross and Rother, Carsten and Doll{'a}r, Piotr. Panoptic segmentation. In CVPR, 2019.[4] Long, Jonathan, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.[5] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.[6] Chen, Liang-Chieh and Papandreou, George and Kokkinos, Iasonas and Murphy, Kevin and Yuille, Alan L. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.[7] Yuan, Yuhui, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In ECCV, 2020.[8] Xie, Enze and Wang, Wenhai and Yu, Zhiding and Anandkumar, Anima and Alvarez, Jose M and Luo, Ping. SegFormer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.[9] He, Kaiming and Gkioxari, Georgia and Doll{'a}r, Piotr and Girshick, Ross. Mask r-cnn. In ICCV, 2017.[10] Bolya, Daniel and Zhou, Chong and Xiao, Fanyi and Lee, Yong Jael. Yolact: Real-time instance segmentation. In ICCV, 2019.[11] Bolya, Daniel and Zhou, Chong and Xiao, Fanyi and Lee, Yong Jae. Solo: Segmenting objects by locations. In ECCV, 2020.[12] Mohan, Rohit and Valada, Abhinav. Efficientps: Efficient panoptic segmentation. IJCV, 2021.[13] Cheng, Bowen and Collins, Maxwell D and Zhu, Yukun and Liu, Ting and Huang, Thomas S and Adam, Hartwig and Chen, Liang-Chieh. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In CVPR, 2020.[14] Xiong, Yuwen and Liao, Renjie and Zhao, Hengshuang and Hu, Rui and Bai, Min and Yumer, Ersin and Urtasun, Raquel. Upsnet: A unified panoptic segmentation network. In CVPR, 2019.[15] Zhou, Tianfei and Porikli, Fatih and Crandall, David J and Van Gool, Luc and Wang, Wenguan. A survey on deep learning technique for video segmentation. TPAMI, 2022.[16] Caelles, Sergi and Maninis, Kevis-Kokitsi and Pont-Tuset, Jordi and Leal-Taix{'e}, Laura and Cremers, Daniel and Van Gool, Luc. One-shot video object segmentation. In CVPR, 2017.[17] Zhou, Tianfei and Li, Jianwu and Wang, Shunzhou and Tao, Ran and Shen, Jianbing. Matnet: Motion-attentive transition network for zero-shot video object segmentation. TIP, 2020.[18] Wang, Qiang and Zhang, Li and Bertinetto, Luca and Hu, Weiming and Torr, Philip HS. Fast online object tracking and segmentation: A unifying approach. In CVPR, 2019.[19] Oh, Seoung Wug and Lee, Joon-Young and Xu, Ning and Kim, Seon Joo. Video object segmentation using space-time memory networks. In ICCV, 2019.[20] Yang, Zongxin, Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. In NeurIPS 2021.[21] Cheng, Ho Kei, Yu-Wing Tai, and Chi-Keung Tang. Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion. In CVPR, 2021.[22] Ye, Linwei and Rochan, Mrigank and Liu, Zhi and Zhang, Xiaoqin and Wang, Yang. Referring segmentation in images and videos with cross-modal self-attention network. TPAMI, 2021.[23] Yu, Changqian and Wang, Jingbo and Peng, Chao and Gao, Changxin and Yu, Gang and Sang, Nong. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In ECCV 2018.[24] Fan, Mingyuan and Lai, Shenqi and Huang, Junshi and Wei, Xiaoming and Chai, Zhenhua and Luo, Junfeng and Wei, Xiaolin. Rethinking bisenet for real-time semantic segmentation. In CVPR, 2021.[25] Zhao, Hengshuang and Shi, Jianping and Qi, Xiaojuan and Wang, Xiaogang and Jia, Jiaya. Pyramid scene parsing network. In CVPR, 2017.[26] Hao Wang, Weining Wang, and Jing Liu. Temporal memory attention for video semantic segmentation. In ICIP 2021.[27] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In CVPR, 2016.[28] SouYoung Jin, Aruni RoyChowdhury, Huaizu Jiang, Ashish Singh, Aditya Prasad, Deep Chakraborty, and Erik Learned- Miller. Unsupervised hard example mining from videos for improved object detection. In ECCV, 2018.[29] Yumin Suh, Bohyung Han, Wonsik Kim, and Kyoung Mu Lee. Stochastic class-based hard example mining for deep metric learning. In CVPR, 2019.[30] Lin, Di and Dai, Jifeng and Jia, Jiaya and He, Kaiming and Sun, Jian. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In CVPR, 2016.[31] Chen, Xi and Zhao, Zhiyan and Zhang, Yilei and Duan, Manni and Qi, Donglian and Zhao, Hengshuang. Focalclick: Towards practical interactive image segmentation. In CVPR, 2022.[32] Zhang, Chi and Lin, Guosheng and Liu, Fayao and Yao, Rui and Shen, Chunhua. Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. In CVPR, 2019.[33] He, Shuting, Henghui Ding, and Wei Jiang. Primitive generation and semantic-related alignment for universal zero-shot segmentation. In CVPR, 2023.[34] Zhang, Xiangyu and Zhou, Xinyu and Lin, Mengxiao and Sun, Jian. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.[35] Howard, Andrew G and Zhu, Menglong and Chen, Bo and Kalenichenko, Dmitry and Wang, Weijun and Weyand, Tobias and Andreetto, Marco and Adam, Hartwig. Mobilenets: Efficient convolutional neural networks for mobile vision applications. Arxiv, 2017.[36] Cheng, Bowen, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, 2021.[37] Jain, Jitesh and Li, Jiachen and Chiu, Mang Tik and Hassani, Ali and Orlov, Nikita and Shi, Humphrey. Oneformer: One transformer to rule universal image segmentation. In CVPR, 2023.[38] Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C and Lo, Wan-Yen and others. Segment anything. Arxiv, 2023.[39] Gabriel J Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recognition using structure from motion point clouds. In ECCV, 2008.[40] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.[41] OpenAI.https://openai.com/blog/chatgpt. 2022.[42] Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj{"o}rn. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.[43] Hu, Yihan and Yang, Jiazhi and Chen, Li and Li, Keyu and Sima, Chonghao and Zhu, Xizhou and Chai, Siqi and Du, Senyao and Lin, Tianwei and Wang, Wenhai and others. Planning-oriented autonomous driving. In CVPR, 2023.[44] Su, Jinming and Yin, Ruihong and Zhang, Shuaibin and Luo, Junfeng. Motion-state Alignment for Video Semantic Segmentation. In CVPRW, 2023.[45] Su, Jinming and Yin, Ruihong and Chen, Xingyue and Luo, Junfeng. Perceive, Excavate and Purify: A Novel Object Mining Framework for Instance Segmentation. In CVPRW, 2023.
金明 旺旺等约 25 分钟