Yudong Zhang,Xin Zhangand Weiguo Zhu

1Jiangsu Key Laboratory of Advanced Manufacturing Technology,Huaiyin Institute of Technology,Huai’an,223003,China

2Department of Medical Imaging,The Fourth People’s Hospital of Huai’an,Huai’an,223002,China

3School of Informatics,University of Leicester,Leicester,LE1 7RH,UK

ABSTRACT Aim:To diagnose COVID-19 more efficiently and more correctly,this study proposed a novel attention network for COVID-19(ANC).Methods:Two datasets were used in this study.An 18-way data augmentation was proposed to avoid overfitting.Then,convolutional block attention module(CBAM)was integrated to our model,the structure of which is fine-tuned.Finally, Grad-CAM was used to provide an explainable diagnosis.Results:The accuracy of our ANC methods on two datasets are 96.32%±1.06%,and 96.00%±1.03%,respectively.Conclusions:This proposed ANC method is superior to 9 state-of-the-art approaches.

KEYWORDS Deep learning; convolutional block attention module; attention mechanism; COVID-19; explainable diagnosis

1 Introduction

COVID-19 (also known as coronavirus) pandemic is an ongoing infectious disease [1] caused by severe acute respiratory syndrome (SARS) coronavirus 2 [2].As of 7/Feb/2021, there are over 106.22 m confirmed cases and over 2.31 m deaths attributed to COVID-19 (See Fig.1).The main symptoms of COVID-19 are a low fever, a new and ongoing cough, a loss or change to taste and smell [3].

In UK, the vaccines approved were developed by Pfizer/BioNTech, Oxford/AstraZeneca, and Moderna [4].The joint committee on vaccination and immunization (JCVI) [5] determines the order in which people will be offered the vaccine.Currently, people aged over 80, people living or working in the care homes, and health care providers are being offered.

Two COVID-19 diagnosis methods are available.The first method is viral testing to test the existance of viral RNA fragments [6].The shortcomings are two folds: (i) the swab may be contaminated and (ii) it needs to wait from several hours to several days to get the results.The other method is chest imaging.The chest computed tomography (CCT) [7] is one of the best chest imaging techniques.CCT is operator independent.Besides, it provides the highest sensitivity compared to ultrasound and X-ray [8].It provides real 3D volumetric image of the chest region [9].

Figure 1: The cumulative data till 7/Feb/2021.(a) Cumulative cases, (b) cumulative deaths

Nevertheless, the manual labelling by human experts are time-consuming, tedious, laborintensive, and easily influenced by human factors (fatiguing or lethargic emotions).In contrast to manual labelling, computer vision techniques [10] are now gaining promising results on automatic labelling of COVID-19 and other medical images with the help of artificial intelligence (AI) [11].

For non-COVID-19 images, Lu [12] proposed a bat algorithm-based extreme learning machine(BA-ELM) approach, and applied it to pathological brain detection.Lu [13] presented a radial basis function (RBFNN) to recognize pathological brain types.Fulton et al.[14] employed ResNet-50 for identifying Alzheimer’s disease.Guo et al.[15] utilized ResNet-18 to identify Thyroid ultrasound plane images.Although those above methods were not directly developed for COVID-19 diagnosis, they are chosen and used as comparison methods in this study.

For COVID-19 images, Yu [16] proposed GoogleNet-COD model.The authors first replaced the last two layers with four new layers, which included the dropout layer, two fully connected layers and the output layer.Satapathy [17] proposed a five-layer deep convolutional neural network with stochastic pooling (DCNN-SP).Yao [18] combined wavelet entropy (WE) and biogeographybased optimization (BBO) techniques.Wu [19] used wavelet Renyi entropy (WRE) to extract features from chest CT images.Li et al.[20] presented a COVID-19 detection neural network(COVNet).Akram et al.[21] proposed a four-step procedure to handle COVID-19: (i) data collection & normalization; (ii) feature extraction; (iii) feature selection;, and (iv) feature classification.Khan et al.[22] used one class kernel extreme learning machine to predict COVID-19 pneumonia.Khan et al.[23] used DenseNet and firefly algorithm for classification of positive COVID-19 CT scans.

To further improve the COVID-19 diagnosis performance, this paper proved a novel AI model, attention network for COVID (ANC), for providing an explainable diagnosis.Here “attention” means it can tell neural network which region should focus.Compared to ordinary neural networks, the advantages of ANC are four folds:

(i) A novel 18-way data augmentation was proposed to avoid overfitting;

(ii) Convolutional block attention module was integrated so our model can infer attention maps;

(iii) Our model was fine-tuned, and its performances were better than 9 state-of-theart approaches;

(iv) Grad-CAM was used to provide an explainable heatmap so the users can understand our model.

2 Dataset

Two datasets are used in this study.The first dataset contains 148 COVID-19 images and 148 healthy control (HC) images [19].The second dataset contains 320 COVID-19 images and 320 HC images [24].Tab.1 provides the descriptions of these two COVID-19 CCT datasets, wherea+bstands foraCOVID-19 subject/images andbHC subjects/images.

Table 1: Descriptions of two COVID-19 CCT datasets

Preprocessing were applied to all the images.LetR0andRstand for the raw dataset(Dataset-1 or Dataset-2) and the final preprocessed set, andR1,R2,R3denote three temporary sets.The flowchart of our pre-processing is displayed in Fig.2.

Th raw dataset setR0={r0(k),k=1,2,...,|R|}, where |R|means the number of images.The size of each image issize[r0(k)]=1024×1024×3.Although the raw images appear grayscale,but they are stored in RGB format at hospitals’servers.

Figure 2: Flowchart of preprocessing

First, we need to grayscale all those raw images.The grayscale transformation is described as

which utilized the standard RGB to grayscale transformation [25].

Second, the histogram stretching (HS) was used to improve the contrast of all grayscaled imagesR1={r1(k)}.For thek-th imager1(k), suppose its upper bound (UB) and lower bound(LB) grayscale values ofr1(k)areand.They can be obtained as

where(x,y)are the indexes of width and height dimension, respectively.(W1,H1)stand for the width and height of imager1, respectively.

The new HS-enhanced imager2(k)is calculated as

whereis the grayscale range of the imager1(k).It is defined as

The HS-enhanced image will occupy the full grayscale range as [rmin,rmax], whererminandrmaxstand for the minimum and maximum gray values, respectively.

Third, we cropped the texts at the right region and the check-up bed at the bottom region.The crop values(e1,e2,e3,e4)are pixels to be cropped from four directions: top, left, bottom,and right, respectively.The cropped imager3(k)can be written as

where

where(W3,H3)stand for the weight and height of any imager3, respectively.The rangea:bstands for the range from the integerato the integerb.

Finally, downsampling was carried out to further reduce the image size and remove redundant information.Suppose the final size is(W,H), the final image,

wherefdsis the downsampling function defined as

Figs.3a and 3b show the raw and pre-processed images of a COVID-19 case, while Figs.3c and 3d illustrates the raw and pre-processed images of a HC case.As can be observed from Fig.3,the pre-processed images have better contrast, remove the irrelevant information, down-sample to a smaller size, and take less storage than the raw images.Tab.2 itemizes the abbreviation list,which will help readers to understand the following sections.

Figure 3: Raw images and pre-processed images selected from our datasets.(a) Raw COVID-19 image, (b) pre-processed image of (a), (c) raw HC image, (d) pre-processed image of (c)

3 Methodology

3.1 18-Way Data Augmentation

Data augmentation (DA) is an important tool over the training set to avoid overfitting of classifiers, and to overcome the small-size dataset problem.Recently, Wang [26] proposed a novel 14-way data augmentation (DA), which used seven different DA techniques to the preprocessed imager(x,y|k),x= 1,...,W,y= 1,...,H,k= 1,...,|R| and its horizontal mirrored imagerM(x,y|k), respectively.

This study enhances the 14-way DA method [26] to 18-way DA, by adding two new DA methods: salt-and-pepper noise (SAPN) and speckle noise (SN) on bothr(x,y|k)andrM(x,y|k).In our future studies, we will research on including more types of DAs.Fig.4 displays the illustration example of using two new DAs.

For a given imager(x,y), the SAPN altered image [27] is defined asrSAPN(x,y)with its values are set as

wherestands for noise density, and P the probability function.rminandrmaxcorrespond to black and white colors, respectively.The definition ofrminandrmaxcan be seen at Section 2.

Table 2: Abbreviation list

Figure 4: Illustration of two new DAs.(a) Raw image, (b) SAPN altered, (c) SN altered

On the other side, the SN altered image [28] is defined as

where N is uniformly distributed random noise, of which the mean and variance are symbolized asandrespectively.

LetNastands for the number of DA techniques to the preprocessed imager(x,y|k), andNbstands for the number of new generated images for each DA.Thus, our(2×Na)-way DA algorithm is a four-step algorithm depicted below:

First,Nageometric/photometric/noise-injection DA transforms are utilized on preprocessed imager(x,y), as shown in Fig.5.We usem=1,...,Nato denote each DA operation.Note each DA operationswill yieldNbnew images.So, for a given imager(x,y), we will yieldNadifferent data set[r(x,y)],m=1,...,Na, and each dataset hasNbnew images.

Figure 5: Schematic of proposed 18-way DA

Second, horizontal mirror image is generated as

wherehMmeans horizontal mirror function.

Third, all theNadifferent DA methods are carried out on the mirror imageqM(x,y), and generateNadifferent datasetm=1,...,Na.

The final combined dataset is defined asΓ

Therefore, one imager(x,y)will generate

images (including the original imager(x,y)).Algorithm 1 shows the pseudocode of this proposed 18-way data augmentation on one imager(x,y).

Algorithm 1: Pseudocode of proposed 18-way DA on one training image r(x,y)Inputs Import raw training image r(x,y).Step 1 Na geometric or photometric or noise-injection DA transforms are utilized on raw image r(x,y),We obtain hDA(m)[r(x,y)], m=1,...,Na.Step 2 Horizontal mirror image is generated as rM(x,y)=hM[r(x,y)].See Eq.(11).Step 3 Other Na-way data augmentation methods are carried out on rM(x,y),we obtain hDA(m)images/BZ_224_713_2134_731_2179.pngrM(x,y)images/BZ_224_901_2134_919_2180.png, m=1,...,Na.Step 4 The raw image, the mirrored image, all the above Na-way DA results of raw image, and Na-way DA results of horizontal mirrored image are combined together.See Eq.(12).Output A new dataset Γ is generated with |Γ|=2×Na×Nb+2.See Eq.(13).

3.2 Convolutional Block Attention Module

Deep learning (DL) has gained successes in prediction/classification tasks.There are many DL structures, such as deep neural network [29], deep belief network, convolutional neural network(CNN) [30], recurrent neural network [31], graph neural network, etc.Among all those DL structures, CNN is particularly suitable for analyzing visual images.

To further improve the performance of CNN, a lot of researches are done with respects to either depth, or width, or cardinality of CNN.In recent times, Woo et al.[32] proposed a novel convolutional block attention module (CBAM), which improves the traditional convolutional block (CB) by integrating attention mechanism.There are many successful applications of CBAM.For example, Mo et al.[33] combined self-attention and CBAM.They proposed a light-weight dual-path attention block.Chen et al.[34] proposed a 3D spatiotemporal convolutional neural network with CBAM for micro-expression recognition.

Fig.6a displays the structure of a traditional CB.The output of previous block was sent ton-repetitions of convolution layer, batch normalization (BN), rectified linear unit (ReLU) layer,and followed by a pooling layer.The output is called activation map (AM), symbolized as D ∈RC×H×W, where(C,H,W)stands for the sizes of channel, height, and width, respectively.

Figure 6: Structural comparison with CB against CBAM.(a) Traditional CB, (b) our CBAM

In contrast to Figs.6a and 6b displays the structure of CBAM, in which two modules:channel attention module (CAM) and spatial attention module (SAM) are added to refine the activation map D.Suppose the CBAM applies a 1D CAM NCAM∈RC×1×1and a 2D SAM NSAM∈R1×H×Win sequence to the input D.Thus, we have the channel-refined activation map

And the final refined AM

where ⊗stands for the element-wise multiplication.This refined AM F will be sent to the next block.

Note if the two operands are not with the same dimension, then the values are broadcasted(copied) in such ways the spatial attentional values are broadcasted along the channel dimension,and the channel attention values are broadcasted along the spatial dimension.

3.3 Channel Attention Module

We define the CAM here.Both max pooling (MP) [35]zmpand average pooling (AP) [36]zmpare utilized, generating two features Gapand Gmpas shown in Fig.7.

Both Gapand Gmpare then forwarded to a shared multi-layer perceptron (MLP) to generate the output features, which are then merged using element-wise summation ⊕.The merged sum is finally sent to the sigmoid functionσ.Mathematically

To reduce the parameter resources, the hidden size of MLP is set to RC/r×1×1, whereris defined as the reduction ratio.Suppose W0∈RC/r×Cand W1∈RC×C/rstand for the MLP weights,respectively, we can rephrase Eq.(17) as

Note W0and W1are shared by both Gapand Gmp.Fig.7 shows the flowchart of CAM.Note that squeeze-and-excitation (SE) [37] method is similar to CAM.

Figure 7: Flowchart of CAM

3.4 Spatial Attention Module

Now we define the spatial attention module (SAM) as shown in Fig.8.The spatial attention module NSAMis a complementary step to the previous channel attention module NCAM.The average poolingzapand max poolingzmpare applied again to the channel-refined activation map E, and we get

Both Hapand Hmpare two dimensional AMs: Hap∈R1×H×W∧Hmp∈R1×H×W.They are concatenated together along the channel dimension as

The concatenated activation map is then passed into a standard 7×7 convolutionzconvand followed by a sigmoid functionσ.In all, we have

The NSAM(E)is then element-wisely multiplied by E to get the final refined AE F.See Eq.(15).The flowchart of SAM is drawn in Fig.8.

Figure 8: Flowchart of SAM

3.5 Proposed Attention Network for COVID-19

In this study, we proposed a novel Attention Network for COVID-19 (ANC) based on CBAM(See Fig.6b).The structure of this proposed ANC is determined by trial-and-error method.The variablenin each block varies, and we found the best values are chosen in the range from [1,3].We tested values larger than 3, which increase the computation burden, but the performances do not increase.

The structure of proposed shown below in Tab.3, which is a 15-layer deep CNN.The InputIis a 256×256×1-size preprocessed image.The first block “CBAM-1” contains 3 repetitions of 32 filters, each filter is with size of 3×3.The AM after CBAM-1 is symbolized asT1 with size of 128×128×32.We can observe the size of AMs after CBAM-2, CBAM-3, CBAM-4,and CBAM-5 aresize(T2)=64×64×32,size(T3)=32×32×64,size(T4)=16×16×64, andsize(T5)=8×8×128,respectively.The output of CBAM-5T5 is flattened to a vector symbolized asFLwith size of 1×1×8192.

Table 3: Structure of proposed ANC

HereT5 is used to generate heatmap by the gradient-weighted class activation mapping(Grad-CAM) [38] method, which shows explanations (i.e., heatmaps) which region the ANC model pays more attention to and how our ANC model makes prediction.

The variableFLis then sent to three consecutive fully-connected layers (FCLs).The neurons of those three FCLs are set to 200, 50, and 2, respectively.The output of those three FCLs are symbolized asF1,F2, andF3, respectively, as shown in Fig.9.Finally,F3 is sent through a softmax layer to output the probabilities of all classes.

Figure 9: Variables and sizes of AMs of proposed ANC

3.6 Implementation and Evaluation

F-fold cross validation is employed on both datasets.Suppose confusion matrix L overr-th run andf-th fold is defined as

where(l11,l12,l21,l22)stand for TP, FN, FP, and TN, respectively.frepresents the index of trial,andrstands for the index of run(1 ≤r≤R).Atf-th trial, thef-th fold is used as test, and all the left folds are used as training, 1 ≤f≤F.

Note that L(r,f)is calculated based on each test fold, and are then summarized across allFtrials, as shown in Fig.10.Afterwards, we get the confusion matrix atr-th run L(r)

Figure 10: Diagram of one run of F-fold cross validation

where first four indicators are:τ1sensitivity,τ2specificity,τ3precision, andτ4accuracy.We have

τ5is F1 score.

τ6is Matthews correlation coefficient (MCC)

andτ7is the Fowlkes-Mallows index (FMI).

There are two indicatorsτ4andτ6utilizing all the four basic measures(l11,l12,l21,l22).Considering the range ofτ4is 0 ≤τ4≤1, and the range ofτ6is -1 ≤τ6≤+1, we finally chooseτ6as the most important indicator.

Above procedure is one run ofF-fold cross validation.In the experiment, we run theF-fold cross validationRruns.The mean and standard deviation (MSD) of all seven indicatorsτm(m=1,...,7)are calculated over allRruns.

wherev1stands for the mean value, andv2stands for the standard deviation.The MSDs are reported in the format ofv1±v2.The pseudocode of our evaluation is displayed in Algorithm 2.In the experiment, the evaluations of two datasets will be reported.

Algorithm 2: Pseudocode of evaluation Input: prediction labels and actual labels.for r=1: R for f =1: F Calculate the confusion matrix L(r,f).See Eq.(22).end Summarize the r-th run confusion L(r)=∑Ff=1 L(r,f).See Eq.(23).Evaluate seven indicators over r-th run and obtain [τ1(r),τ2(r),...,τ7(r)].See Eq.(24).end Output: MSD v1±v2.See Eq.(29).

4 Experiments,Results,and Discussions

4.1 Parameter Setting

Tab.4 shows the parameter setting.Here the crop values of top, bottom, left, and right are all set to 200 pixels.The size of preprocessed image is set to(256,256).The minimum and maximum gray values are set to 0 and 255, respectively.The noise density of SAPN is set to=0.05.The mean and variance of uniformly distributed random noise in SN are set to 0 and 0.05, respectively.Both the number of folds and the number of runs are set to 10.

Table 4: Parameter setting

4.2 Results of Proposed 18-Way DA

Take Fig.3b as an example, Fig.11 displays theNaresults of this original image.Due to the page limit, the horizontal mirrored image and its correspondingNaresults are not displayed in Fig.11.As it is calculated, one image will generate 2×Na×Nb+2=542 images.

Figure 11: Na DA results on original preprocessed image.(a) Gaussian noise, (b) SAPN, (c) SN,(d) horizontal shear, (e) vertical shear, (f) rotation, (g) gamma correction, (h) random translation,(i) scaling

4.3 Statistics of Proposed ANC

First, the statistical results on the first dataset and the second dataset are shown in Tabs.5 and 6 respectively.We can observe that for Dataset-1, the sensitivity is 96.49±1.18, the specificity is 96.15±1.56, the precision is 96.18±1.50, and accuracy is 96.32±1.06, the F1 score is 96.33±1.05, MCC is 92.65±2.11, and FMI is 96.33±1.05.For dataset-2, the performances are slightly lower than those of dataset-1.The sensitivity is 95.88±1.28, the specificity is 96.13±1.16, the precision is 96.12±1.15 , the accuracy is 96.00±1.03, F1 score is 95.99±1.04, MCC is 92.01±2.07, and FMI is 96.00±1.04.Fig.12 display the receiver operating characteristic (ROC) curves on datasets D1 and D2.The area under curve (AUC) of proposed ANC model on two datasets are 0.9951 and 0.9911, respectively.

Table 5: Statistical results of proposed ANC model on D1

Table 6: Statistical results of proposed ANC model on D2

Figure 12: ROC curves of proposed ANC model (10 runs).(a) Dataset D1 (AUC=0.9951), (b)dataset D2 (AUC=0.9911)

4.4 Effect of Attention Mechanism

This ablation study removed attention blocks (CAM and SAM) from our ANC model.Here NA in Tab.7 means no attention.Tab.7 shows that removing attention blocks will reduce the performance significantly.For example, the MCCs, i.e.,τ6values, of ANC-NA and ANC on the first dataset are 86.37%±1.92% and 92.65%±2.11%, respectively.For the second dataset, the MCC values of ANC-NA and ANC are 84.54%±1.93%, and 92.01%±2.07%, respectively.

Fig.13 shows the error bars of different models, where D1 and D2 stand for the 1st and 2nd dataset, respectively, as introduced in Section 2.It clearly indicates that using attention mechanism can help improve the classification performances on both datasets.

Table 7: Comparison of models with attention against without attention

Figure 13: Effect of attention mechanism on two datasets.(D1: the first dataset; D2: the second dataset)

4.5 Comparison to State-of-the-Art Approaches

We compare our method “ANC”with 9 state-of-the-art methods: ELM-BA [12], RBFNN [13],ResNet-50 [14], ResNet-18 [15], GoogleNet-COD [16], DCNN-SP [17], WE-BBO [18], WRE [19],and COVNet [20].All the methods were evaluated via 10 runs of 10-fold cross validation.The MSD results →τon ten runs of 10-fold cross validation are displayed in Tab.8, where the definitions abbreviations can be found in Section 1.This proposed ANC outperforms all the other 9 comparison baseline methods in terms of all indicators.

Table 8: Comparison with SOTA approaches in two datasets (unit: %)

Fig.14 shows the 3D bar plot of our ANC method and 9 state-of-the-art algorithms.All the algorithms are sorted in terms ofτ6, i.e., MCC indicator.As is shown here, the best is our ANC.The second best and the third best are DCNN-SP [17] and COVNet [20], respectively.The reason why our ANC model gives the best results are three folds: (i) The proposed 18-way DA help resist overfitting; (ii) The CBAM help build our ANC model integrate attention mechanism; and (iii)The structure of ANC is fine-tuned (See Tab.3).

Figure 14: Comparison to state-of-the-art approaches over two datasets (D1 and D2).(a) D1,(b) D2

Figure 15: Manual delineation and heatmap results of COVID-19 sample image.(a) Manual delineation of COVID-19, (b) heatmap of (a), (c) HC image, (d) heatmap of (c)

4.6 Explainable Heatmap

Fig.15 shows the manual delineation and heatmap results of Fig.3.Figs.15a-15c show the delineation, where the healthy control image has no lesion (Fig.15c), so the radiologist does not delineate the lesion regions in Fig.15c.Figs.15b-15d show the corresponding heatmaps, where we can observe that this proposed ANC model can find the COVID-19 related regions correctly.

5 Conclusion

This paper proposed a novel attention network for COVID-19 (ANC) model.The classification results of ANC model are better than 9 state-of-the-art approaches in terms of seven indicators: sensitivity, specificity, precision, accuracy, F1 score, MCC, and FMI.

There are two disadvantages of our proposed method: (i) The dataset is relatively small.We expect to expand the dataset to have more than 1k images.(ii) Graph neural network technologies will be attempted to integrate to our model.

Funding Statement:This paper is partially supported by Open Fund for Jiangsu Key Laboratory of Advanced Manufacturing Technology (HGAMTL-1703); Guangxi Key Laboratory of Trusted Software (kx201901); Fundamental Research Funds for the Central Universities (CDLS-2020-03);Key Laboratory of Child Development and Learning Science (Southeast University), Ministry of Education; Royal Society International Exchanges Cost Share Award, UK (RP202G0230); Medical Research Council Confidence in Concept Award, UK (MC_PC_17171); Hope Foundation for Cancer Research, UK (RM60G0680); British Heart Foundation Accelerator Award, UK.

Conflicts of Interest:The authors declare that they have no conflicts of interest to report regarding the present study.