Jize Yin, Senlin Luo, Zhouting Wu and Limin Pan

(Information System and Security & Countermeasures Experimental Center, Beijing Institute of Technology, Beijing 100081, China)

Abstract: Unlike named entity recognition (NER) for English, the absence of word boundaries reduces the final accuracy for Chinese NER. To avoid accumulated error introduced by word segmentation, a deep model extracting character-level features is carefully built and becomes a basis for a new Chinese NER method, which is proposed in this paper. This method converts the raw text to a character vector sequence, extracts global text features with a bidirectional long short-term memory and extracts local text features with a soft attention model. A linear chain conditional random field is also used to label all the characters with the help of the global and local text features. Experiments based on the Microsoft Research Asia (MSRA) dataset are designed and implemented. Results show that the proposed method has good performance compared to other methods, which proves that the global and local text features extracted have a positive influence on Chinese NER. For more variety in the test domains, a resume dataset from Sina Finance is also used to prove the effectiveness of the proposed method.

Key words: Chinese; named entity recognition (NER); character-level; bidirectional long short-term memory; soft attention model

Named entity recognition (NER) is a subtask of information extraction (IE), one of the main tasks in natural language processing (NLP). NER was established on the sixth message understanding conference (MUC-6)[1]and aims to recognize named entities like person names, location names and organization names in text. Many downstream applications like relation extraction[2]and machine translation[3]are based on NER. Recently, Chinese NER has received attention in light of the elevation of China’s international status[4-9].

Unlike English NER, Chinese NER must deal with unstructured text without any word boundaries. Therefore, the processing procedure for Chinese NER usually begins with a word segmentation step[10-13]. Recent research efforts also verify the importance of word segmentation information[14-16]. However, Kuru et al. describe a character-level tagger and evaluate it on different NER tasks. The results across seven languages including Arabic, Czech, Dutch, English, German, Spanish and Turkish show that viewing characters as the representation is better than considering words as the representation[17], which provides a new thought that Chinese NER can be improved by adequately extracting only character-level features. There have also been several works drawing attention to character-level feature extraction for Chinese NER[5-7].

For character- or word-level feature extraction, deep models are helpful[18]and there have recently been several works on deep neural networks applied in NER recently[15-17,19]. As for the methods of model training, although semi-supervised and unsupervised learning algorithms are widely used in NER[4,16,20-21], supervised learning algorithms are still popular in NER because of their simplicity and reliability[9,15,17,19].

In this article, a deep model that combines a bidirectional long short-term memory (BLSTM) with a soft attention model (AM) is proposed to improve character-level feature extraction performance. A Chinese NER method is given based on this model. The experimental results based on the Microsoft Research Asia (MSRA) dataset prove the validity of the character-level feature extraction and show the proposed method’s good performance. Further experimental results based on a resume dataset from Sina Finance prove the effectiveness of the proposed method for various test domains.

1 Related Work

1.1 Deep neural networks for NER

Some effective NER methods are shown in Tab.1. There have been many hand-crafted feature-based models used for NER such as hidden Markov models (HMM)[22], decision trees[23], maximum entropy models (ME)[24], support vector machines (SVM)[25-28]and conditional random fields (CRF)[29]. Recently, there have been several research efforts that have generated improved models[30-31]. To avoid hand-crafted features[11,32-33]and achieve automatic text feature extraction for NER, many new deep models based on neural network models like long short-term memory (LSTM)[15], BLSTM[16-17]and convolutional neural network (CNN)[19]have been proposed and proven to be effective.

Tab.1 Several existing NER methods

Several feature extraction methods based on deep models have led to good results in NER.Chiu et al. provide a neural network architecture that automatically detects word- and character-level features. This architecture includes a BLSTM to extract word-level features and CNNs to extract character-level features[19]. However, Chiu et al.’s method is for English with clear word boundaries. For some languages, character-level features might be better than word-level features. To compare character-level features with word-level features, Kuru et al. use a deep model consisting of stacked bidirectional LSTMs to get characters and output tag probabilities. These probabilities are then converted to named entity tags by a Viterbi decoder. The experimental results based on seven languages prove that character-level features are better than word-level features[17]. Similarly, Lu et al. treat Chinese sentences as sequences of characters and present a position-sensitive skip-gram model to learn multi-prototype Chinese character embeddings. The multi-prototype Chinese character embeddings are then used as features in the Chinese NER task, which results in an improvement in recognition[6]. It is a way to extract features by using unsupervised learning, and there are other methods with unsupervised character representations. Dong et al. make their model utilize both character-level and radical-level representations[5]. Zheng et al. transfer the context-specific and multi-prototype character representations into an NER task, which proves the ability of unsupervised character representation to enhance supervised learning[7]. Furthermore, Zheng et al. continue to work on unsupervised character representations and observe an improvement in the supervised learning tasks like NER[8]. However, the methods using unsupervised character representations need an additional process of unsupervised learning, all except basic supervised learning. Moreover, the above methods treat all the characters equally, which ignores the varying importance of characters in text.

Extra knowledge has been used in the NER task. For Chinese, the absence of word boundary information is treated as a problem, so several research efforts on Chinese NER use word boundary information as extra knowledge. Peng et al. use word boundary tags as features to provide richer information for an NER system, which is realized by jointly training an NER model and a Chinese word segmentation (CWS) model based on a LSTM-CRF model[15]. Furthermore, Zhang and Yang propose a good model named lattice LSTM[9]. This model receives a character-level one-hot vector sequence as the input, seeks candidate words and processes them in the form of word embeddings. That means Zhang and Yang still have to rely on the word boundary information like Peng et al.[15]. In general, the NER corpus does not have any other information except named entity information. So, the biggest disadvantage is that this kind of method needs more external labor.

In conclusion, there are still some problems that need to be solved in NER, especially for Chinese. In this article, a Chinese NER method based on a deep model combining a BLSTM with a soft attention model is proposed and character-level features are extracted to identify named entities. Without adding word boundary information, other features named local text features are extracted to make an improvement in Chinese NER. Supervised learning is enough and there is no need for other learning algorithms such as semi-supervised learning[4,16]or unsupervised learning[7-8]. Experimental results present the proposed method’s satisfactory performance.

1.2 Attention mechanism

The attention mechanism in neural networks, a selection mechanism that can allocate limited information processing capacity, is based on the visual attention mechanism exhibited in humans. Using an attention mechanism means that some important information will be addressed while other information will be ignored. There have been several applications of attention mechanism in different fields[34-37]. For some special applications in NLP, the important parts of text should be given high weights and the useless parts should be given low weights.

Bahdanau et al. propose an attention model to search for parts of a source sentence automatically so that the sentence-length feature can be useful for neural machine translation[38]. This attention model is generally called a soft attention model, compared to its opposite model called a hard attention model[39]. Shang et al. use the soft attention model to build a local scheme that is combined with a global scheme to be one part of the general encoder-decoder framework taken by the neural responding machine (NRM), a neural network-based response generator for short-text conversation[40]. A soft attention model gives weights to every word or character in the source sentence, but a hard attention model needs only to find the critical word or character in the source sentence and ignore the rest. Luong et al. build a global attention model based on the soft attention model and a local attention model based on the tradeoff between the soft and hard attention models for neural machine translation[41]. Moreover, there are many improved models and new mechanisms based on the attention mechanism for special applications[42-56].

In this article, a deep model combining a soft attention model with a BLSTM like what Bahdanau et al. provide[38]is proposed so that the important parts of the text can be weighted more heavily than the rest. There is a small difference between the proposed model and Bahdanau et al.’s model. The proposed model uses a soft attention model considering the opposite directions of the two LSTMs that comprise a BLSTM, but Bahdanau et al.’s model doesn’t. The global and local text features extracted by the proposed model are different from what the global and local attention models extract[41]. The global and local attention models are distinguished by the attention window sizes, but the global and local text features are distinguished by whether or not the soft attention model is used. By using the proposed model, a good performance of the Chinese NER method is obtained.

2 Method

2.1 Framework

As shown in Fig.1, there are three parts in the proposed Chinese NER method: character vector conversion, feature extraction and named entity tagging. In particular, the feature extraction consists of both global text feature extraction via a BLSTM and local text feature extraction via a soft attention model.

Fig.1 Proposed method of Chinese NER

2.2 Character vector conversion

For a character-based language like Chinese, each character should be converted to a unique vector so that the raw text can be used for mathematical calculation. To achieve this goal, a one-hot model is used. First, a character dictionary is built by giving sequence numbers to different characters in it. For instance, if A is the first character in the dictionary, No. 1 will be given to A. No. 2 will be given to the second character, and so on. Next, a zero vector whose dimension is the number of different characters is built. Finally, a one-hot vector occurs by replacing zero at the position to which a character’s number corresponds with one. After that, the raw text is converted to a character vector sequence.

2.3 Feature extraction

Feature extraction can extract important features from text and store them in the vectors of real numbers which are called feature vectors. In this article, NER is viewed as a sequence labeling problem, so there is a feature vector generated as output for each given character vector.

In the proposed Chinese NER method, the feature extraction consists of global text feature extraction and local text feature extraction.

Global text features can be abstracted into a text sequential feature from a character sequence. In the grammatical sense, the text sequential feature decides the characters’ order in text because people write or read a text message in some order. In the semantic sense, there is practical meaning when the characters are arranged in some order. The global text features including sufficient grammatical and semantic information are extracted as a bidirectional text sequential feature by a BLSTM.

Local text features can be regarded as remaining text features that get rid of the unnecessary information and they are determined by the chosen parts of text. In the grammatical sense, the unnecessary information might destroy the common grammatical structures. In the semantic sense, the unnecessary information can be easily replaced or deleted because it has a marginal effect on the core sense of the whole text. The local text features, including sufficient grammatical and semantic information, are extracted by a soft attention model.

2.3.1Long short-term memory

A long short-term memory is a specially designed neural network model proposed by Sepp Hochreiter and Jürgen Schmidhuber[57]. It obtains the output of the last moment and the input of the present moment like what a recurrent neural network (RNN) does, so it has the ability to process sequential data. But compared with an RNN, a LSTM can remember more information and is usually good at processing long text by adjusting the entire structure with a cell that connects to other functional structures of the LSTM. These specially designed functional structures are called gates, which include an input gate, a forget gate and an output gate.

In the process of Chinese NER, the inputted text contains many character sequences, so it’s easy to convert text processing to character sequence processing. A LSTM can extract the unidirectional sequential feature from a character sequence after the character sequence is converted to a vector sequence of real numbers.

First, a LSTM should decide which part of the new information needs to be stored in the cell. The input gate of the LSTM can make it and is shown in

it=σ(Wvivt+Whiht-1+Wcict-1+bi)

(1)

where itis the present state of the input gate andσis the logistic sigmoid function. The vector vtis the present input whose weight is Wvi. The vector ht-1is the output of the last moment (hidden state vector) whose weight is Whi. The vector ct-1is the cell state of the last moment whose weight is Wci. The vector biis the bias.

Meanwhile some old information in the cell needs to be forgotten. The forget gate of the LSTM can perform this function and is shown in

ft=σ(Wvfvt+Whfht-1+Wcfct-1+bf)

(2)

where ftis the present state of the forget gate. The present input vthas a weight Wvf, the output of the last moment ht-1has a weight Whf, the cell state of the last moment ct-1has a weight Wcf, and the vector bfis the bias.

Next, the cell needs to be updated and its present state ctis shown in

ct=ftct-1+ittanh(Wvcvt+Whcht-1+bc)

(3)

where tanh is the hyperbolic tangent function. The present input vthas a weight Wvc, the output of the last moment ht-1has a weight Whc, and the vector bcis the bias.

The LSTM needs to decide which part of the information should constitute the output. The output gate of the LSTM can make it and is shown in

ot=σ(Wvovt+Whoht-1+Wcoct+bo)

(4)

where otis the present state of the output gate. The present input vthas a weight Wvo, the output of the last moment ht-1has a weight Who, the present cell state cthas a weight Wco, and the vector bois the bias.

Finally, the present hidden state vector htoutput by the LSTM is shown in

ht=ottanhct

(5)

The hidden state vector sequence output by the LSTM is the unidirectional sequential feature vector sequence.

2.3.2Bidirectional long short-term memory

A bidirectional long short-term memory is a neural network model proposed by Alex Graves and Jürgen Schmidhuber[58]. This model consists of two LSTMs named forward LSTM and backward LSTM. The forward LSTM processes the sequential data in the forward direction and the backward LSTM processes the sequential data in the opposite direction.The BLSTM’s structure is shown in Fig.2. For each moment, the hidden state vector from the forward LSTM and the hidden state vector from the backward LSTM are combined to form the final output.

Fig.2 BLSTM’s structure

A BLSTM can output bidirectional text sequential feature vector sequences {h1, …, ht-1, ht,ht+1, ht+2, …, hT} (Tbeing the length of the output) after getting the computed text input {v1, …, vt-1, vt,vt+1, vt+2, …, vT} (Tbeing the length of the input).

2.3.3Soft attention model

For Chinese text processing, the attention mechanism works by assigning different mathematical weights to different characters. In this article, the soft attention model proposed by Bahdanau et al.[38]is used to extract local text features, whose structure is shown in Fig.1.

The soft attention model based on the BLSTM that obtains the character vector sequence {v1, v2, …, vT}, whereTis the length of the character vector sequence, consists of two parts: one for the forward LSTM and the other for the backward LSTM. The two parts have the same structure and function. Considering the part for the forward LSTM, all the hidden state vectors from the forward LSTM {h1, h2, …,hT}, whereTis the length of the output from the forward LSTM, are given corresponding weights {αt,1,αt,2, …,αt,T}, wheretrepresents the present moment andTis the length of the output from the forward LSTM. The sum of the weights is the attention state atof the present moment.

(6)

The weightαtjcan be calculated by

(7)

And the variableetjcan be calculated by

etj=VTtanh(W1at-1+W2hj+bt)

(8)

where tanh is the hyperbolic tangent function.etjrepresents the associated energy that is used to measure the relationship between the input in the positionjand the output in the positiont. The associated energy is based on the attention state of the last moment at-1with the weight W1and the hidden state vector hjwith the weight W2. The vector btis the bias. The vector V is the weight for the combination of all the features.

Finally, an attention state sequence {a1, a2, …, aT}, whereTis the length of the output from the forward LSTM, is output. Each attention state has local text features based on the sequential feature. In the same way, there is an attention state sequence for the backward LSTM. By combining both of the attention state sequences, there is a global and local text feature vector sequence {x1, x2, …, xT}, whereTis the length of the input character vector sequence, as the output of feature extraction.

2.4 Named entity tagging

Named entity tagging is the process of giving a tag to each text feature vector.In this article, a linear chain CRF is used, which is shown in Fig.1.

CRFs, a type of discriminative undirected probabilistic graphical model, are proposed by Lafferty et al.[59]. If x={x1,x2,…,xn} is the observed variable sequence and y={y1,y2,…,yn} is the corresponding tag variable sequence, a conditional probability modelP(y|x) can be obtained. If the modelP(y|x) meets the Markov property, as is shown

P(yi|x,y1,y2,…,yn)=P(yi|x,yi-1,yi+1)

(9)

then the modelP(y|x) is a linear chain CRF.

The conditional probability functionP(y|x) is shown in

(10)

using exponential potential functions and feature functions.

The functiontj(yi-1,yi,x,i) is a transition feature function whose weight isλj, which is used to represent the relationship between the adjacent tag variables and the influence from the observed variable sequence. The functionsk(yi,x,i) is a status feature function whose weight isμk, which is used to represent the influence from the observed variable sequence. The functionZ(x) shown in

(11)

is a normalizing factor used to make sure that Eq.(10) is defined correctly.

Sequential log-likelihood loss is used to train the model. After finishing model training, the linear chain CRF outputs a tag sequence based on the text feature vector sequence by using the Viterbi algorithm. Final named entities are determined according to the tag sequence.

3 Experiments and Analysis

3.1 Purpose

To prove the validity of the feature extraction model which combines a BLSTM with a soft attention model, some experiments based on the MSRA dataset are designed and implemented. The contrast models include a LSTM-CRF model, a BLSTM-CRF model, a LSTM-AM-CRF model and a BLSTM-AM-CRF model. Other research results are also referenced to prove that the proposed Chinese NER method in this article performs well. Further experiments based on a resume dataset from Sina Finance attempt to prove the effectiveness of the proposed method for more variety in test domains.

3.2 Experimental settings

The experimental datasets shown in Tab.2 include the MSRA dataset[60]and a Chinese resume dataset annotated by Zhang and Yang[9]. While the MSRA dataset is in the news domain, the resume dataset from Sina Finance consists of resumes of senior executives from listed companies in the Chinese stock market[9].

Hyper-parameters for the proposed method are shown in Tab.3. The optimization method in this article is adaptive moment estimation with a learning rate of 0.01.

Tab.3 Hyper-parameter values

The experimental software environment is shown in Tab.4.

The experimental hardware environment is shown in Tab.5.

Tab.4 Experimental software environment

① https://en.wikipedia.org/wiki/TensorFlow. ② https://en.wikipedia.org/wiki/NumPy.

Tab.5 Experimental hardware environment

3.3 Evaluation method

Precision (P), recall (R), and F-measure (F1) are used to evaluate the experimental results.Pis the ratio of the number of named entities correctly identified (true positive,TP) to the number of named entities found in text (true positive & false positive,TP+FP), which is represented as the equationP=TP/(TP+FP).Ris the ratio of the number of named entities correctly identified (true positive,TP) to the number of named entities actually existing in text (true positive & false negative,TP+FN), which is represented as the equationR=TP/(TP+FN).F1is the harmonic mean based onPandR, which is represented as the equationF1=2PR/(P+R).

3.4 Experimental procedure

In this article, four models are trained and tested based on the MSRA dataset and the Chinese resume dataset. The four models use a LSTM, a BLSTM, a model combining a LSTM with a soft attention model, and a model combining a BLSTM with a soft attention model to extract text features respectively. All of them use a linear chain CRF to tag named entities. There have been several research publications on Chinese NER based on the MSRA dataset[4-9,11,32-33].

3.5 Results and discussion

The main results on the MSRA dataset are shown in Tab.6. Compared to previous methods described in Section 1.1, the proposed method in this article has shown competitive results. Compared to method 1, method 2 obtains a good performance with an 86.84% F1-score. It is obvious that a BLSTM performs better than a LSTM in text feature extraction, which confirms the theory of bidirectional text sequential feature in this article. As for method 3, adding a soft attention model to the LSTM-CRF model increases the F1-score of Chinese NER to 85.14%. The soft attention model also works with a BLSTM-CRF model, as shown by comparing method 2 with method 4. Therefore, extracting local text features is effective. Moreover, the final proposed method receives an 88.27% F1-score, better than other three methods.

Main results on resume NER are shown in Tab.7. The proposed method performs better than the state-of-the-art method[9]with a 95.24% F1-score. It can be seen that the proposed method works well in the domain of Sina Finance. There is only one pattern different from that shown in Tab.6. Based on the resume NER dataset, method 4 seems worse than method 2. Method 2 and method 4 both use a BLSTM and a linear chain CRF with the same hyper-parameters. However, an attention model is used in method 4, but not used in method 2. Therefore, the whole deep model used in method 4 is more complex than that used in method 2. Because the resume NER dataset is small, the deep model used in method 2 with given hyper-parameters shown in the article can perfectly fit the training corpus, which leads to a 100% F1-score during model training as we have already seen. Generally, under the circumstance of perfectly fitting the training dataset, the more complex a model is, the worse its generalization ability is. That’s why method 4 has worse performance on the test dataset compared with method 2.

Tab.6 Main results on MSRA

Tab.7 Main results on resume NER

In other words, if the development dataset of the resume NER corpus is used to adjust the common hyper-parameters in method 2 and method 4, both the deep models respectively used in method 2 and method 4 could be simpler, but the one used in method 4 is still more complex. So, if the deep model used in method 2 cannot fit the training dataset perfectly, but the one used in method 4 can, then method 4 is likely to perform better on the test dataset compared with method 2.

4 Conclusions

In this article, a deep model extracting character-level features is carefully built to avoid the accumulated error introduced by word segmentation. This problem is the basis for the new proposed Chinese NER method. This method converts the raw text to a character vector sequence with the help of a one-hot model, extracts global text features that are abstracted into a bidirectional text sequential feature with a BLSTM and extracts local text features without the unnecessary information using a soft attention model. Because NER can be viewed as a sequence labeling problem, a linear chain CRF is used to label all the characters with the help of the global and local text features. Results on the MSRA dataset prove the fact that the global and local text features have a positive influence on Chinese NER. Results on a resume NER dataset from Sina Finance prove the robustness of the proposed method in various test domains.

The global text features are abstracted into a text sequential feature from a character sequence in this article, but people might read and organize the words or characters not one by one but with a special structure like a tree structure. Therefore, other text structures need to be considered to replace the sequential structure in the future.