多维度交叉注意力融合的视听分割网络
2025-08-03李凡凡张垣垣章永龙朱俊武

doi:10.19734/j.issn.1001-3695.2024.08.0369
Audio-visual segmentation network with multi-dimensional cross-attention fusion
LiFanfan,Zhang Yuanyuan,Zhang Yonglong,Zhu Junwu† (School of Information Engineering,Yangzhou University,Yangzhou Jiangsu 2251Oo,China)
Abstract:Audio-visual segmentation (AVS)aimsto locateandaccuratelysegmentthesoundingobjects inimagesbasedon both visualandauditoryinformation.Whilemostexistingresearch focusesprimarilyonexploring methods foraudio-visualinformationfusio,thereisinsuicientin-depthexplorationoffine-grinedaudio-visualanalysis,particularlyinaligingcontinuousaudiofeatures withspatialpixel-level information.Therefore,thispaperproposedanaudio-visualsegmentationatention fusion(AVSAF)method basedoncontrastive learning.Firstly,themethodusedmulti-ead crossattentionmechanismand memorytokentoconstructaaudio-visualtokenfusionmodule toreducethelossofmulti-modalinformation.Secondlyitintro ducedcontrastivelearning tominimizethediscrepancybetweenaudioandvisualfeatures,enhancing theiralignment.Aduallayerdecoderwasthenemployedtoaccuratelypredictandsegment thetarget’sposition.Finalyitcarredoutalargeumber of experiments on the S4 and MS3 sub-datasets of the AVSBenge-Object dataset.The J -valueisincreasedby3.O4and4.71 percentage pointsrespectively,and the F valueis increased by 2.4 and3.5percentage points respectively,which fully proves the effectiveness of the proposed method in audio-visual segmentation tasks.
Key words:audio-visual segmentation;multi-modal;contrastive learning;attention mechanism
0引言
人类的感知是多维的,包括视觉、听觉、触觉、味觉和嗅觉。其中,视觉和听觉是日常生活中非常重要的感知方式,两者的结合帮人类更全面地了解周围环境。在数据量爆炸性增长以及智能设备广泛应用的时代下,视听融合技术得到了广泛的发展。随着视听像素级标签的引入,Zhou等人[]提出了新的视听融合任务,称为视听分割(audio-visualsegmentation,AVS)。AVS旨在根据视频和音频实现对环境中对象和语义信息的准确分割,使得模型预测的像素图与给定的音频相匹配。例如,在图1中,AVS可以识别正在啼哭的婴儿并分割出对应的像素区域。AVS在多个应用领域具有重要意义,如视频音频编辑和安全监控等。
AVS通过结合视频和音频数据,从图像分割的角度挖掘视听之间的潜在关系,实现对环境中对象和语义信息的精确分割。其中,Zhou等人[1提出一种基于时间像素级视听交互的编码器解码器网络,实现视听分割的目标,构建了AVSBench-Object数据集,不仅提高对环境信息处理的理解,还为开发更具智能化的视听感知系统提供了新的思路和方法。Mao等人[2]提出一种基于对比学习的潜在扩散模型用于音视频分割,进一步提升了音视频分割任务的精度,但该模型在融合视觉和听觉特征时,缺乏对不同模态间细微差别的捕捉。Li等人[3]提出了一种视听组合的时空依赖关系的融合方法,在捕捉音频和视觉信号的时间动态变化和空间布局方面表现出了优越性,但在连续音频特征与空间像素级信息的对齐上存在挑战。……
