Huifang Li, Jianghang Huang, Jingwei Huang, Senchun Chai, Leilei Zhao, Yuanqing Xia

Abstract: Industrial Internet of Things (IoT) connecting society and industrial systems represents a tremendous and promising paradigm shift. With IoT, multimodal and heterogeneous data from industrial devices can be easily collected, and further analyzed to discover device maintenance and health related potential knowledge behind. IoT data-based fault diagnosis for industrial devices is very helpful to the sustainability and applicability of an IoT ecosystem. But how to efficiently use and fuse this multimodal heterogeneous data to realize intelligent fault diagnosis is still a challenge.In this paper, a novel Deep Multimodal Learning and Fusion (DMLF) based fault diagnosis method is proposed for addressing heterogeneous data from IoT environments where industrial devices coexist. First, a DMLF model is designed by combining a Convolution Neural Network (CNN) and Stacked Denoising Autoencoder (SDAE) together to capture more comprehensive fault knowledge and extract features from different modal data. Second, these multimodal features are seamlessly integrated at a fusion layer and the resulting fused features are further used to train a classifier for recognizing potential faults. Third, a two-stage training algorithm is proposed by combining supervised pre-training and fine-tuning to simplify the training process for deep structure models. A series of experiments are conducted over multimodal heterogeneous data from a gear device to verify our proposed fault diagnosis method. The experimental results show that our method outperforms the benchmarking ones in fault diagnosis accuracy.

Keywords: fault diagnosis; deep learning; multimodal heterogeneous data; multimodal fused features

1 Introduction

With rapid development of Internet of Things(IoT), a variety of industrial equipment, as an indispensable powerful support for the stable and sustainable development of IoT ecosystems, becomes increasingly complex [1]. Once a fault occurs in the equipment and is not eliminated in time, it may pose a great threat to the safe operations of industrial production [2]. Therefore, the timely and accurate fault diagnosis of industrial equipment has very important practical significance [3]. And fortunately, IoT technology has greatly facilitated the collection of the state and running environment-related data for the industrial devices, which enables to provide a comprehensive information perception of the physical world for realizing data-driven industrial fault diagnosis, device prognostic and health management [4]. However, the collected data is often multimodal and heterogeneous, and how to use and integrate multimodal data has been a great challenge for fault diagnosis.

During the past few decades, traditional Machine Learning (ML)-based diagnosis methods such as Principal Component Analysis (PCA),Artificial Neural Network (ANN), Support Vector Machine (SVM) and extreme learning machine have been widely used [5−8]. And, some traditional ML algorithm-based research achievements have been applied to fault diagnosis using multimodal data. For example, S. Ma et al. [9]collected multimodal bearing vibration data under different rotation speeds and noise backgrounds, then processed different modal data with SVM, k-Nearest Neighbor (KNN) and decision tree respectively, and finally obtain the fault classification result through the majority voting of classifiers. Aiming at the irregularity of density in multimodal data, most methods utilized Euclidean Distance to detect faults by comparing its similarity with others. Liu et al. [10]presented a weighted-distance based strategy to select neighbors. Li et al. [11] took the acquired vibration signal and current signal as original ones, and extracted the time domain and wavelet packet entropy features. The Kernel Principal Component Analysis (KPCA) method is then used to reduce the dimensionality and fuse the features which are fed to SVM for fault recognition. Although the above-mentioned methods achieve a certain success, their performances depend heavily on the quality of extracted and selected features. In most cases, feature selection depends a lot on signal processing techniques [12]and the engineering experiences of diagnostic domain experts [13], and also is time-consuming and labor-intensive. Besides, ANN and SVM are shallow learning models, their high computational and space complexities make them difficult to effectively learn complex nonlinear relationships.In other words, they are only suitable for processing small batch data. Their feature selection and extraction are hard to handle complex fault diagnosis problems under multimodal data [14].

Recently, deep learning (DL) has gradually been applied to intelligent fault diagnosis [15].Deep lerning has a great performance to overcome the inherent disadvantages of traditional ones [16], and can automatically learn the valuable and high-level features from raw data. In other words, DL-based fault diagnosis methods can get rid of the dependence on some prior knowledge about advanced signal processing techniques and domain experts, and learn the complex nonlinear relationships between the type of faults and the running data of industrial equipment. By taking the nonlinear and non-stationary characteristics of fault vibration signal in rolling bearing systems into consideration, Tan et al. [17] proposed a stacked auto-encoder based intelligent fault diagnosis model. Liu et al. [18] developed a fault diagnosis method for a rotor platform through an infrared image and CNN classification algorithm, which can automatically select the image features to identify a fault. Y.Dong et al. [19] presented a Deep Belief Network(DBN) based fault diagnosis model for a gas pressure regulator, and tested its performance with a 288-dimensional pressure dataset. There are many impressive research efforts that have been conducted in the DL-based intelligent diagnosis area. However, most of them focus on single modality data obtained from one observation channel [17−19].

As the wide application of IoT, wireless communication, intelligent sensing, mobile and edge computing, the way for collecting fault information about industrial equipment becomes more and more diversiform and multi-sourced [20]. For the same target, a variety of data can be obtained from different observation channels, and then be utilized to describe the running state of the target devices [21]. For complex equipment, a single sensor can never acquire all the necessary data [22]. In many situations, different sensors installed on equipment can be used to collect different modalities of data related to its status and performance. For fault diagnosis, single modal data is hard to fully represent the running state of equipment, and thus confronts the challenge to be used for supporting health management well.Therefore, researchers have now integrated different deep learning networks together in different ways [23, 24] such that the DL model will have the characteristics of multiple networks at the same time, and the extracted features are more effective due to their individual advantages in working with different data, which ultimately improve the overall performance of fault diagnosis.However, most of these integration methods simply stack different networks in sequence.When further feature extraction is performed over different networks, a certain degree of feature loss may occur unavoidably. Wang et al. [25]proposed a fault diagnosis method by using deep learning multimodal fusion, where they utilized the CNN and long short-term memory (LSTM)model to extract features with spatial and temporal characteristics separately. The feature extractor with multi-networks is built to extract features efficiently, and then resulting features are fed into the classifier for fault classification[26, 27]. But they mainly deal with the same type of data.

In order to capture the joint information from heterogeneous data of different modalities,and characterize equipment operation states more comprehensively, this paper presents a new framework for intelligent fault diagnosis, as shown in Fig.1. In this framework, different DL models are used to extract different features from heterogeneous data with different modalities.Then, the extracted features are combined into a fusion one as an input fed to the feature fusion layer for fault classification. This framework gives overall considerations to the characteristics of different modals of data, and then leverages the advantages of different DL methods in working with different data. Second, it can process multimodal heterogeneous data, such as structured data, e.g., the current and voltage, and unstructured data, e.g., the sound and image, then integrate features extracted from multimodal data seamlessly at a fusion layer for fault recognition. The feature fusion method is able to make full use of not only the complementary information resulting from different data, but also the nonlinear relationships among features, and thus characterize equipment operation states more comprehensively. The main contributions of this paper are:

Fig.1 Intelligent fault diagnosis framework

1) A novel Deep Multimodal Learning and Fusion (DMLF) model is proposed, which does not depend on any prior knowledge and is highly suitable for processing massive heterogeneous multimode signals collected through IoT in the field of fault diagnosis.

2) An effective model training algorithm with supervised pretraining and supervised finetuning characteristics, is developed for the proposed DMLF to overcome the difficulty in finding a global minimum within complex solution space of our DMLF.

The rest of this paper is organized as follows.In Section 2, SDAE and CNN are briefly introduced. In Section 3, we describe the proposed DMLF. In Section 4, experimental and comparison results are discussed. In Section 5, we conclude our paper and indicate our future work.

2 Deep Learning Methods

In recent years, DL technologies are more increasingly used for fault diagnosis, where each DL method has its own preference to work with different modal data in diagnosing the health status of rotating machinery. It is exactly to make full use of the advantages of deep learning models in processing multimodal data, in this paper, two deep learning methods are mainly adopted, i.e., convolutional neural networks (CNNs)and stacked denoising autoencoders (SDAEs).

2.1 Convolutional Neural Network

As we all know, CNNs have outstanding performances in image understanding, speech recognition [28], and nature language processing. Neurons forming a CNN structure have weights and biases, which are changeable and learnable through training [29]. A CNN structure consists of three types of layers, i.e., convolutional layers,pooling layers (subsampling layers), and fully connected layers. The convolutional layers mainly perform a convolutional operation. Each neuron in a convolutional layer takes a restricted region from the whole image as its input and mines the correlations among local features [30].The pooling layer uses a mean pooling or maximum pooling operation to reduce the dimension and computational complexity. CNN adopts the strategy of local perception and parameter sharing to reduce the risk of overfitting.

1) Convolutional Layers: The convolutional layers contain a number of learnable convolutional kernels, which convolute the input from the previous layer through a set of weights and generate new feature maps as the input to the next layer. All the neurons share their weights for the same convolutional kernel, which reduces the optimization time and complexity of CNNs. Suppose the sound image dataXis the input andYis the labels of images. Then the output of a convolutional layer can be calculated as follows [31]:

2.2 Stacked Denoising Autoencoder

SDAE consists of several Denoising Autoencoders (DAEs) stacked together, and the hidden layer of its former is used as the input layer of the latter, as shown in Fig.2, where the DAE is a special neural network that can reconstruct data in the corrupted input.

Fig.2 Stacked denoising autoencoder

SDAE first gets initial paramaters through different DAEs, where the DAE is a kind of unsupervised neural networks whose main idea is to reconstruct the original input from a corrupted one. Then we train the DAE to reconstruct a clean input from the corrupted one, so as to enhance its ability in resisting noise [32]. An DAE includes two parts: encoder and decoder [33], as shown in Fig.3.

Fig.3 Denoising autoencoder

The input of DAE is corrupted by some kinds of noises, such as Gaussian noise, i.e.:

The role of an encoder is to map an input vector to a hidden layer and then get a new feature expression. The function is expressed as follows:

wherebjandajis the biases and output ofj-th neuron in the hidden layer,wijis the weight betweeni-th neuron in input layer andj-th neuron in hidden layer.fis the activation function.

The role of a decoder is to map the output of a hidden layer back to its original input. It is defined as follows:

For SDAE, constructing a model involves two steps: Firstly, all DAEs will be trained in turn through an unsupervised training algorithm,and then their encoding parts and an output layer are stacked into an SDAE. Secondly, the resulting SDAE's parameters will be fine-tuned by using a supervised learning algorithm.

3 Proposed Fault Diagnosis Method

For capturing the relatively comprehensive fault information from heterogeneous data, i.e., structured and unstructured data, to achieve accurate diagnostic results, DMLF leverages CNN and SDAE algorithms to extract features from different modalities of heterogeneous data respectively.Then the extracted features extracted from multimodal data are seamlessly integrated into a hidden layer for fusion. Finally, the fused feature is fed to a classification layer for identifying the diagnostic results.

3.1 Preprocessing of Multimodal Heterogeneous Data

In order to maintain the consistency on the time nodes between the different modals of heterogeneous data during the training process of DMLF,data with different signal modal needs to be preprocessed as the following methods.

To process audio data used by CNN, which is more suitable for learning from images, we convert the audio data into a continuous waveform signal in the time domain, and draw the waveforms in a figure. In order to maintain the consistency of time nodes between the audio data fed to CNN and structured data used by SDAE, the waveforms in a continuous time domain need to be converted into its counterparts in a discrete time domain. As the rule shown in Fig.4, suppose thattis the initial time point, Δtis the time interval,λis the moving step length, andkis an integer indicating the sampling sequence. If a time-domain waveform during (t+kλ,t+kλ+Δt) is taken as a sample and is used to train a CNN, the corresponding structured data at timet+kλ+Δt/2 will be applied for training a SDAE. For convenience, the time period (t+kλ,t+kλ+Δt) and the time pointt+kλ+Δt/2 are c ollectively referred to astk.

Fig.4 Conversion rules for unstructured data

3.2 Establishing a Diagnostic Model

Our DMLF network structure consists of two stages, as shown in Fig.5. One is for feature extraction and the other for feature fusion and classification. Feature extraction is divided into two components. The first is to process image data and obtain the fault features through the convolution and pooling operations of CNN. Suppose that in CNN, a fully-connected layer has an output dimension ofNC, its resulting output feature can be denoted asvC∈RNC. The second is to process structured data with the same time stamp as image data, and to learn fault knowledge hidden in structured data by using SDAE. Similarly,suppose that the output dimension of the last hidden layer in SDAE isND, the outputvD∈RNDof SDAE is taken as the final resulting feature.

Fig.5 Network structure of the DMLF model

Fig.6 Training process of DMLF

Feature vectorvinis fused and mapped as follows:

3.3 Two-Stage Training Algorithm

DMLF has a deep structure involving different deep learning methods, which leads to a more complex solution space, with a large number of saddle points and local minima. If we directly use a Back Propagation (BP) algorithm to train the whole DMLF model, it will face some problems,such as a long training time and difficulties to find a global minimum. Because of its deep structure, it is prone to encounter gradient disappearance. As a result, the parameters of later layers in the model cannot be adjusted during the training process, which may cause the trained model basically useless. To avoid such issues, a supervised pre-training and fine-tuning based twostage training method is proposed in this paper.The training process is shown in Fig.6.

In the stage of supervised pre-training, the BP algorithm is used to train a complete CNN and SDAE model respectively. For CNN, its parameters are initialized, and its prediction results in the forward propagation process are calculated according to the formulas in Section 2.1.For SDAE, it is formed by using the unsupervised training method to generate several denoising autoencoders according to Section 2.2. In the both back propagation procedures of CNN and SDAE, we use the following loss function:

where 1(yk) denotes the indicator function, and is 1, ifykis true, and is 0, otherwise.akandykare the output and label of thek-th neuron in the classification layer, respectively.

For both models, the updating process of each parameter in the BP procedure can be given via the chain rule as follows:

whereZiis the input of thei-th neuron at the classification layer, andajis the output of thejth neuron. ΔWlis the weight increment in thelth layer.ηis the learning rate. Its corresponding weights are updated as Eq. (11). Repeat this iteration until the accuracy of both models reaches to its respective threshold.

During the training process of our DMLF network for fault diagnosis, we first convert the waveform data in the continuous-time domain into the waveform images in the discrete-time domain. In order to keep the data synchronized,corresponding timestamps are added to the sound data used by CNN and structured data used by SDAE. After training the CNN and SDAE models, their parameters will be saved, respectively,including convolution kernel parameters and weights at the fully-connected layer in the trained CNN and all the weights and bias between hidden layers in the trained SDAE.Then, according to the features extracted from the CNN and SDAE, i.e., the outputs of fully connected layer in CNN and the last hidden layer in SDAE, the feature fusion and classification layers are trained subsequently. Similarly, all the obtained weights and biases will be saved and transferred to the DMLF as well as the parameters of CNN and SDAE. In the DMLF, all parameters are fine-tuned using a supervised learning method. Finally, the performance of our DMLF is verified by testing samples. The general process of implementing DMLF-based fault diagnosis is shown in Fig.7.

Fig.7 General process for implementing DMLF-based fault diagnosis

4 Experimental Results and Analysis

To validate the effectiveness of our DMLF, a gear fault simulation system is built. The experimental system adopts a drive shaft matching structure. Drive shaft, gear and bearing are used as three components. Several fault structures are then constructed. We interchange different fault structures by quick dismantling different components to collect data as much as possible, so as to cover many different types of faults. In our experiments, DMLF is built with Aanconda 3 and Tensorflow 1.3.0 [34].

4.1 Multimodal Datasets

There are four types of gear faults considered in our experiments, which are crack in tooth root(A), worn tooth (B), chipped tooth (C), and missing tooth (D), as shown in Fig.8. For structured data, the gear signals of known fault types are collected, and 47 characteristic parameters are obtained from time domain and amplitude domain analysis, which are current, voltage,speed, torque, peak index, kurtosis index, impulsion index, tolerance index, rotational frequency,temperature, vibration frequency, meshing frequency, rotating frequency, amplitude, etc. While structured data being collected, microphones are used to collect acoustic signals to acquire unstructured data synchronously. The sampling rate is 12.8 kHz, and sampling time is 60 s. According to the conversion rules in Section III, the data collected from different modalities is processed, where Δt=0.1,λ=0.1, and all samples in the structured data set are normalized. Each of the two datasets contains 560 samples, where the structured dataset isD1and unstructured one isD2. The datasets are divided so that a 7-fold cross validation method can be used. We divideD1andD2into 7 mutually exclusive subsets with the same size respectively by stratified sampling,and each subset has 80 samples. In each experiment, five subsets are randomly selected fromD1andD2respectively, as training sets, and the remaining subsets are used as testing and validation sets. It is worth noting that we use the same sampling sequence during a subset division and subset selection process, to make the samples of different data types with the same timestamp and guarantee their temporal consistency.

Fig.8 Four fault types of gear

4.2 Configuration

Our CNN’s structure is shown in Tab. 1, with max-pooling used, and our SDAE’s structure is 47-35-23, where the feature fusion is implemented by two hidden layers, and its structure is 239-80. The output layer of our DLMF employs a softmax function for final fault classification.Other hyper-parameter settings include: maximum iteration count is set to 800, learning rate is 0.001, dropout rate is set to 0.2, batch size is 20,and the optimizer is RMSprop in Tensorflow.

Tab. 1 Structure and parameters of CNN part

For Local Binary Pattern (LBP), the neighborhood radius is 1. For SVM, the penalty coefficient and radius of its kernel function are limited within [1, 2] and [1, 4], respectively. For eXtreme Gradient Boosting (XGB) [35], the maximum depth is [8, 9] and learning rate is [0.11,0.14]. For the Nearest Neighbor (NN) algorithm,the number of leaves is 30, and Euclidean Distance is used as the distance metric. For the Oriented FAST and Rotated BRIEF (ORB), we take the default parameters according to [36].

4.3 Results and Analysis

To verify and better reflect the performance of the proposed DMLF, we select several existing fault diagnosis methods as its peers, and conduct the following two experiments.

1) Accuracy Comparison between Single Modal Data Based Methods with DMLF: In this subsection, we present a comparative study of DMLF with the methods that only use different single modal data. For structured data, we use XGB and SDAE as peers. For unstructured data,we compare with CNN, LBP_SVM and ORB_NN, where LBP_SVM means that features are extracted by LBP and are classified by SVM. While ORB_NN focuses to convert features extracted by ORB into fisher vectors and classifies them by NN.

Ten trials are carried out for each method.The detailed statistical results of each method are shown in Fig.9, where the symbol ‘×’ and horizontal line ‘—’ in the box represent the average accuracy and the median of 10 results, respectively. It is obvious that the proposed DMLF model obtains the best average accuracy which is 97.14%. Since other methods only use one modal of data for training and diagnosis, this result not only proves that a single type of data cannot represent comprehensive equipment information, but also testifies the effectiveness of our DMLF in fault diagnosis by combining different types of data. When comparing the peers that only use structured data, it can be observed that SDAE performs better than XGB by 1.25%in both median and mean accuracy of testing sets, from Fig.9.

Fig.9 Accuracy results on single modal data

This may be because that the neural network-based model structure is more suitable for processing structured data than the tree-based model structure, and its special neurons and hierarchical structure can better explore the nonlinear mapping relationships between structured data and its corresponding fault labels. While for the CNN, LBP_SVM and ORB_NN that only use unstructured data, it can be seen that CNN outperforms LBP_SVM and ORB_NN by 1.66%, 1. 07% in the median of 10 results, and 1.62%, 0.80% in the average accuracy, respectively. It perhaps results from that in CNN, all convolution kernel parameters at the same layer are shared to process high-dimensional data without too much calculation, and the feature extraction can be carried out automatically. Thus CNN has competitive advantages in image data processing. Through the above experiments, we can find that CNN and SDAE, as representative deep learning methods, have certain advantages in dealing with unstructured and structured data,respectively.

2) Accuracy Comparison Between Multimodal Data Based Methods with DMLF: From the above experiments, we can see that CNN and SDAE performs better on the testing sets. In order to further illustrate the performance of our proposed method, DMLF is compared with SDAE_SDAE_weighting, CNN_CNN_weighting and SDAE_CNN_weighting under multimodal data, i.e., the structured and unstructured data. Specifically, SDAE_CNN_weighting utilizes the structured and unstructured data to train SDAE and CNN, respectively, then uses a validation set to verify both the trained models,and selects a model with higher validation accuracy to calculate the weights of SDAE and CNN.Finally, the test samples are fed into the trained SDAE and CNN, respectively to output probability vectors of SDAE and CNN, and then their resulting vectors are multiplied by the corresponding weights and added together to obtain a new output vector. The indexes of the largest elements of its output vectors are taken as the category labels of the test samples. For example, if the validation accuracy of SDAE and CNN is 92.50% and 97.50% respectively, the weight for SDAE will be 92.50%/(92.50% + 97.50%), and that for CNN will be 97.50%/(92.50% +97.50%). Similarly, SDAE_SDAE_weighting and CNN_CNN_weighting utilize two SDAE and CNN networks respectively and output their results by weighting.

Similarly, we conduct 10 trials for SDAE_SDAE_weighting, CNN_CNN_weighting, SDAE_CNN_weighting and our proposed DMLF, and their detailed statistical results are shown in Fig.10. For DMLF, its average, median and highest testing accuracy are 97.14%,96.88% and 98.75%, respectively. For SDAE_CNN_weighting, its average, median and highest testing accuracy are 96.41%, 96.25% and 97.68%,respectively. For CNN_CNN_weighting, its average, median and highest testing accuracy are 96.13%, 96.25% and 97.50%, respectively. For SDAE_SDAE_weighting, its average, median and highest testing accuracy are 93.63%, 95.00%and 96.25%, respectively. These results demonstrate that DMLF achieves higher performance than SDAE_SDAE_weighting, CNN_CNN_weighting and SDAE_CNN_weighting. This primarily comes from that DMLF can make better and full use of data with different modalities,and adaptively extract more comprehensive fault features and discover more fault knowledge to improve diagnostic performance.

Fig.10 Accuracy results on multimodal data

3) Running Time Comparison between Multimodal Data Based Methods: We also roughly evaluate our proposed algorithm from the computational efficiency perspective. The comparisons in running time among different methods on multimodal data are shown in Tab.2. It can be observed that our proposed method costs a little bit more running time than SDAE_SDAE_weighting, CNN_CNN_weighting and SDAE_CNN_weighting. This primarily comes from that our feature fusion and classification layers which result in more computations compared with those simple weighting methods. Besides, SDAE_SDAE_weighting saves more running time by 0.0292s than CNN_CNN_weighting, which is presumably owing to that CNN’s convolution and pooling operations are more time-consuming.In general, our DMLF achieves the highest precision but does not run fastest, i.e., it sacrifices a certain amount of computation time to obtain an accuracy increase.

Tab. 2 Running time of four methods on multimodal data

5 Conclusion

In this paper, a novel deep multimodal learning and fusion-based fault diagnosis method is proposed to process multimodal heterogeneous data for recognizing faults. A CNN and SDAE based deep multimodal learning and fusion (DMLF)network is designed to extract features from multimodal heterogeneous data and integrate the resulting different features seamlessly into a fusion layer for identifying faults more accurately.While fully considering the characteristics of different modal data and taking the advantages of different deep learning methods in dealing with multimodal heterogeneous data, our DMLF integrates feature learning and feature fusion into a single process to realize end-to-end learning, i.e.,from original multimodal heterogeneous data to fault classification. Experimental results show that our proposed method can make full use of the complementary information from different data and nonlinear relationships among features,and characterize equipment operation states more comprehensively then achieve better diagnostic performance than its peers. It should be indicated that, combining multimodal IoT data with appropriate feature mining strategies to discover more comprehensive fault knowledge would further facilitate the development of IoT. In future work, we plan to use more multimodal heterogeneous IoT data to get more and deeper knowledge, and develop new feature fusion strategies to address more types of multimodal data critical to complex fault diagnosis.