摘 要：近年来，以CLIP为代表的视觉-语言大模型在众多下游场景中显示出了出色的零样本推理能力，然而将CLIP模型迁移至需要像素水平图-文理解的参考图像分割中非常困难，其根本原因在于CLIP关注图像-文本整体上的对齐情况，却丢弃了图像中像素点的空间位置信息。鉴于此，以CLIP为基础模型，提出了一种单阶段、细粒度、多层次的零样本参考图像分割模型PixelCLIP。具体地，采取了多尺度的图像特征融合，既聚集CLIP中不同视觉编码器提取的图像像素级特征，同时又考虑CLIP中固有的图像整体语义特征。在文本信息表征上，不但依靠CLIP-BERT来保持物体种类信息，还引入LLaVA大语言模型进一步注入上下文背景知识。最后，PixelCLIP通过细粒度跨模态关联匹配，实现像素水平的参考图像分割。充分的数值分析结果验证了该方法的有效性。

关键词：零样本；CLIP；像素级；单阶段；参考图像分割

中图分类号：TP391"" 文献标志码：A""" 文章编号：1001-3695（2025）04-038-1248-07

doi： 10.19734/j.issn.1001-3695.2024.06.0254

Zero-shot referring image segmentation based on fine-tuning image-text model CLIP

Liu Jie1， 2， Qiao Wensheng1， Zhu Peipei1， Lei Yinjie3， Wang Zixuan3

（1. Southwest China Institute of Electronic Technology， Chengdu 610036， China; 2. School of Resources amp; Environment， University of Electronic Science amp; Technology of China， Chengdu 611731， China; 3. School of Electronics amp; Information Engineering， Sichuan University， Chengdu 610065， China）

Abstract：

In recent years， large vision-language models represented by CLIP have demonstrated excellent zero-shot inference capabilities in numerous downstream scenarios. However， transferring the CLIP model to reference image segmentation， which requires pixel-level image-text understanding， is very challenging. The fundamental reason lies in the fact that CLIP focuses on the overall alignment between images and text while discarding the spatial position information of pixels in the image. In view of this， this paper proposed a single-stage， fine-grained， multi-level zero-shot reference image segmentation model called Pixel-CLIP based on the CLIP model. Specifically， this paper adopted multi-scale image feature fusion， which not only aggregated pixel-level image features extracted by different visual encoders in CLIP， but also considered the inherent overall semantic features of images in CLIP. In terms of textual information representation， this paper relied not only on CLIP-BERT to maintain object category information， but also introduced the LLaVA large language model to further inject contextual background knowledge. Ultimately， PixelCLIP achieves pixel-level reference image segmentation by realizing fine-grained cross-modal associative matching. Extensive experiments indicate the validity of PixelCLIP.

Key words：zero-shot; CLIP; pixel-level; one-stage; referring image segmentation

0 引言

深度学习的最新进展彻底改变了计算机视觉和自然语言处理，并解决了视觉和语言领域的各种任务［1］。最近多模态模型（如CLIP［2］）取得成功的一个关键因素是在大量图像和文本对上进行对比图像-文本预训练。它们在广泛的任务上表现出了显著的零样本可移植性，如目标检测［3］、语义分割［4］、图像字幕［5］、视觉问答［6］等。尽管预训练的多模态大模型具有良好的可移植性，但在处理诸如参考图像分割等像素级密集预测任务时依旧具有挑战性。参考图像分割［7］是指在给定一个描述某区域的自然语言表达式的参考下实现分割图像特定部分，是众所周知的具有挑战性的视觉和语言任务之一。……