刘大运　房国志　骆天依　魏华杰　王倩　李修政　李骜

摘   要：為了解决唇语识别中唇部特征提取和时序关系识别存在的问题，提出了一种双向长短时记忆网络（BiLSTM）和注意力机制（Attention Mechanism）相结合的深度学习模型。首先将唇部20个关键点得到的唇部不同位置的高度和宽度作为唇部的特征，使用BiLSTM对唇部特征序列进行时序编码，然后利用注意力机制来发掘不同时刻唇部时序特征对于整体唇语识别的不同权重，最后利用Softmax进行分类。在公开的唇语识别数据集GRID和MIRACL-VC上与传统的唇语识别模型进行实验对比。在GRID数据集上准确率至少提高了13.4%，在MIRACL-VC单词数据集上准确率至少提高了15.3%，短语数据集上准确率至少提高了9.2%。同时还与其他编码模型进行了实验对比，实验结果表明该模型能有效地提高唇语识别的准确率。

关键词：唇语识别;双向长短时记忆网络;注意力机制;深度学习;时序编码

中图分类号：TP391                                             文献标识码：A

Research on Lip-reading Based on BiLSTM-Attention

LIU Da-yun1，FANG Guo-zhi2？覮，LUO Tian-yi3，WEI Hua-jie1，Wang Qian1，Li Xiu-zheng1，Li Ao1

（1. School of Computer Science and Technology，Harbin University of Science

and Technology，Harbin，Heilongjiang 150080，China;

2. School of Measurement and Control Technology and Communication Engineering，

Harbin University of Science and Technology，Harbin，Heilongjiang 150080，China;

3. School of Automation，Harbin University of Science and Technology，Harbin，Heilongjiang 150080，China）

Abstract：In order to solve the existing problems in lip feature extraction and temporal relation recognition during the research of lip-reading，a deep learning model based on bi-directional long short-term memory（BiLSTM） and attention mechanism（Attention） is proposed. Firstly，the height and width of the different positions of the lip obtained from the 20 key points of the lip are taken as the characteristics of the lip. Secondly，the BiLSTM model is utilized to encode temporal information. Thirdly，the attention mechanism is used to explore different weights of lip sequential features at different times toward the overall lip language recognition. Finally，we use Softmax classifier to classify. Compared with the conventional lip-learning models at the current lip language recognition database GRID and MIRACL-VC，we find the recognition accuracy rate is more than 13.4% higher than that on GRID. In the MIRACL-VC word database，the accuracy rate increased by at least 15.3%，and the accuracy rate in the phrase database increased by at least 9.2%. At the same time，compared with other coding models，the experimental results show that this model can effectively improve the accuracy of lip-reading.

Key words：lip-reading;bi-directional long short-term memory;attention mechanism;deep learning;sequential coding

唇语识别集自然语言处理与机器视觉于一体，其研究的主要内容是从视频图像中识别出讲话人所说的内容。近年来，唇语作为人类表达信息的重要载体，唇语识别技术获得了广泛的关注，已经被广泛应用到案件侦破、安检、辅助语音识别等[1.2.3]领域。

同语音识别相比，唇语识别存在更多困难。不同说话人在不同时刻和不同环境下，即使说话内容完全相同，其唇动信息也存在巨大差异。视觉特征的提取对于唇语识别来说尤为关键，明显的特征应该具有类内一致性和类间差异性，一定程度上简化后续的识别过程以及提升识别准确率。……