Hongjun Wang,Xiaoyu Ji,Hui Zhao,2,Youjun Yue

(1.College of Electrical and Electronic Engineering,Tianjin University of Technology,Tianjin,300384,China; 2.College of Engineering and Technology,Tianjin Agricultural University,Tianjin,300392,China)

Abstract: The rapid identification and precise positioning of the freshwater fish’s head,abdomen,and fins are prerequisites for the robot to achieve fast capturing,accurate segmentation,and the core of working efficiency improvement.The freshwater fish body’s semantic segmentation algorithm conducts many invalid feature channels under the encoding feature extraction stage,and the constant network sample collection and pooling operation lead to the loss of detailed information of some fish bodies,decreasing the performance and poor the performance effect of edge cutting.A semantic segmentation algorithm base on SENet optimized Deeplabv3+ freshwater fish head,abdomen,and fins is proposed.Using dilated/atrous convolutions to enlarge the receptive field,prevent detail information loss,and actualize the precise positioning.Meanwhile,the SENet optimization enables Deeplabv3+ to improve the useful feature of freshwater fish and restrains its ineffectual feature.Eventually,the semantic segmentation MIoU of each part of the freshwater fish reached about 93% on the self-established freshwater fish data set.Its feature improved significantly and reached the current advanced segmentation level.

Keywords: recognition;location;deep learning;characteristic channel;receptive field;semantic segmentation

0 Introduction

Semantic segmentation can actualize every collected image pixel value;therefore,the semantic label can be put on each granular pixel value.With the continuous development,semantic segmentation had become one of the most important regions of computer vision research and had been applied to the fast-identification and detection field of the targeted freshwater fish[1].

The traditional target detection method included several steps:region selection,feature extraction,classifier classification,and target location[2].The feature extraction method based on the learning method mainly included edge detection and threshold segmentation,such as Otsu’s threshold segmentation based on maximum between-class variance[3]and Canny’s edge detection methods based on double threshold processing and edge tracking[4].The features of manual extraction included texture,color,shape,etc.And then,the classifier is trained through SVM,AdaBoost,neural network,and other methods to achieve the target detection[5-6].In the past,although the number of parameters of these methods was very small and the difficulty of detection was very low,it is difficult to convey the deep pixel information that extracted feature contained,which caused a big difference between the target detection result and the actual task requirement.

With the progress of artificial neural network (ANN)[7]and computer technologies,the continuous improvement of algorithm optimization,and deep learning of the neural network potential,deep learning methods in target pixel-level semantic segmentation technology had achieved a huge improvement[8].Considering this,this paper proposed a semantic segmentation model of Deeplabv3+ freshwater fish body based on SENet[9-10]optimization.This paper utilized the dilated convolution in the Deeplabv3+ to extract image semantic feature,realized the expansion of the receptive field,and diminished the loss information of freshwater fish edge and other details while pooling and sampling.In addition,this paper added atrous spatial pyramid pooling (ASPP)[11]structure to process multi-scale feature fusion,captured context from the image,gradually accurate the segmentation results,introduced SENet to the network structure,perfected the network’s function to select information channels for the feature dimensions,lowered the identification error rate and enhanced network performance and accomplished a more refined semantic segmentation result for freshwater fish head,abdomen,and fins.

1 Segmentation algorithm flow and Deeplabv3+ network

1.1 Fish body segmentation algorithm flow

The freshwater fish semantic segmentation method proposed in this paper mainly took advantage of the SENet network,which can properly use the feature channels and achieved a further optimization of Deeplabv3+ network.Firstly,images were collected and data were expanded.Then,Labelme software was used to calibrate sample labels.The freshwater fish data set was completed and inputted into the improved fish body segmentation network to finish the task.When the loss value closed to a convergence point,training would be stopped.In the final segmentation process,the verification set was used to evaluate the semantic segmentation effect.The process is shown in Figure 1.

1.2 Deeplabv3+ network

In 2014,Long et al.replaced the fully connected layer in the traditional convolution neural network (CNN)with a convolution layer to build a fully convolution neural network (FCN).Achieved the classification to each pixel level of the images,therefore,made the history of implementing the convolutional neural network into semantic segmentation.At present,the semantic segmentation method based on deep learning includes (1)use FCN to modify the original prototype internet with a fully connected layer (i.e.AlexNet,VGG,GoogleNet)to conduct a refined semantic network[12];(2)contracting path in the U-Net structure uses pooling layer to minimize the space,capture information from the context through continuously down-sampling.The expansive path achieves precise positioning by recovering the object details and space dimensions with deconvolution.Meanwhile,the same expansion path and shrinking path feature map the way of connecting information horizontally.The expansion path should restore the object’s detail features further[13].(3)The Deeplabv3+ semantic segmentation algorithm is promoted and evolved from the former Deeplabv1-v3.Firstly,the convolution operation in Deeplabv1[14]had holes,achieved the decrease of down-sampling and pooling,and enlarged the network receptive field to make the feature map denser.The use of fully connected conditional random field (CRF)operation increased the accuracy of object segmentation.However,there were some limitations when dealing with multi-scale segmentation objects.In order to overcome the above difficulties,Deeplabv2[15]adopted ASPP operation in v2 structure,used different sample rates of dilated convolution on the input feature maps to complete the relative convolution process,realized the context information obtained in various dimensions.But when the 3×3 convolution kernel expansion rate increased in ASPP operation,3×3 convolution would form into 1×1 convolution.For getting over this issue and achieving the combination of the whole map,Deeplabv3[16]modified the 3×3 convolution kernel quantity to 3,and the expansion rate of each convolution kernel was {6,12,18}.At the same time,a global average pooling was introduced to overcome the loss of detail information and the decrease of image resolution caused by constant down-sampling.Meanwhile,it enlarged the information range of each convolution output without increasing the amount of calculation,amplified the receptive field,enhanced the target semantic segmentation speed,improved identification accuracy,and preserved the notable semantic segmentation effect on the targeted smaller and thinner fish fins that showed on the images.Since ASPP contained shallow features,which was the object’s position,v3 deleted the CRF full connection.Deeplabv3+ summarized some of the advantages of the above semantic segmentation algorithms.With the continuous deepening of the network,the backbone network changed from ResNet to Xception[17].The number of parameters that required calculation by model reduced,the content of encoding and decoding increased so that the feature map gathered the low-level feature while up-sampling.It’s beneficial to the optimization of the object’s boundary information.In the meantime,restoring the feature content using linear interpolation achieved further advancement in network segmentation accuracy.The Deeplabv3+ model structure is shown in Figure 2.

Fig.2 Deeplabv3+ network structure

1.3 SENet module

The SENet network structure took advantage of the dependence between different channel characteristics,which could improve feature selection and deletion performance.It consisted of Squeeze,Exception,and Reweight,as shown in Figure 3.The whole process is (1)Squeeze operation:used global average pooling to pool the feature layer of each channel into a specific number;(2)Excitation operation:by assignment operation,each of the two fully connected layers was given a weight value to build a mutual dependence among different channels;and (3)Reweight operation:by normalized the weight of sigmoid function,the operation would normalize to the range of 0 to 1.Finally,each channel was multiplied to make the weight value to its original feature at the end.The mapping relationship is shown in formula (1)to formula (3).

s=Fex(z,W)

=σ(g(z,W))=σ(W2δ(W1,z)) (2)

xc=Fscale(uc,s)=sc·uc(3)

Whereucrepresents each feature channel;W1andW2the weight value for full connection operation.δ(ReLu)was used to activate the full connection for non-linear processing.The weight vector was outputted throughσ(Sigmoid)to perform multiplication corresponding to the original feature map.

Fig.3 SENet module structure

1.4 Optimization of Deeplabv3+

The backbone network DCNN of Deeplabv3+ used Xception as its basic network,as shown in Figure 4.The network continuously downsampling and pooling to gradually enlarged the network feature dimension channels from 64 to 128,256,728,1 024,1 536,and 2 048.With the increase of feature channel numbers,the amount of feature calculation for each layer rose sharply.However,these feature channels contained a large amount of invalid feature dimension information,which led to a decrease in accuracy,performance,and parameter utilization in the freshwater fish recognition network during a large calculation.

Fig.4 Base network in semantic segmentation of freshwater fish

To solvethis issue,the SENet structure took advantage of the dependence of different channels to urge the segmentation network to improve its selecting and deleting performance in freshwater fish features.To achieve the optimized semantic segmentation result,it enhanced the accuracy of each part and improved the utilization of parameters.The structure of the convolution module in SENet,as shown in Figure 5.Since the Xception included skip-connections module,re-calibrating the Xception branch feature needs to be realized before addition.Suppose the operation was to re-calibrate the feature on the main branch after the addition because the 0-1 Scale performance was on the backbone of the process.In that case,it’s easy to appear vanishing gradient problem near the input layer while BP optimizing in a deep network structure.This vanishing gradient problem made it difficult to optimize the model.

Fig.5 Convolution module embedded in SENet

2 Analysis of test result

The following is a conclusive verification of the optimized Deeplabv3+ semantic segmentation algorithm.The first is the composition of the operation platform and the setting of neural network super parameters.Then there is the data set that is used in this experiment.After the experiment,theoretical analysis on the proposed freshwater fish segmentation algorithm was performed,compared,and analyzed with respect to the unimproved semantic segmentation algorithm.

2.1 Test platform and related parameter setting

This paper used Microsoft open-source data set (COCO)on the internet to train the Xception network structure model optimized by SENet and used the single self-built background freshwater fish data set to realize the change of network parameter and match convolution with different expansion rates.It also satisfied the dense pixel feature requirement for extracting each part of the freshwater fish body.The network training process was achieved with the stochastic gradient descent (SGD)method.In terms of sample date expansion[18],input sample set with a random scale from 0.5 to 2 were used,and the images were rotated in the process of network training.Adopted the GeForce GTX1080Ti×2GPU of Intel Xeon(R)cpue5-2650 v4@2.20 Hz × 48、12 GB and ran 64 GB storage as a test hardware platform,Ubuntu18.04 as a test operation system,CUDA9.0 as a version number of parallel computing framework,CUDNN7.6.0 as DNN acceleration library.Used the programming language Python to achieve the SENet optimized Deeplabv3+ semantic segmentation model on deep learning framework Tensorflow,and completed the related model training process.

2.2 Test data set of freshwater fish

The source ofthe training sample data set consisted of two parts.The first one was Microsoft open-source data set (COCO).It contained more than 80 kinds of objects and was used in the pre-training process of the model.The other part was a self-built data set for training and evaluating the semantic segmentation performance of the head,abdomen,and fins of freshwater fish.There were 2 258 images of freshwater fish data set for training,1 464 verification images,and 1 448 images for testing.The background of all images was pure white in order to reduce the interference of extra variables,and the imaging quality of samples was slightly different.The image contained five kinds of targets:carp,silver carp,crucian carp,grass carp,and bream.All targets presented different posture features in the image.There are four semantic categories:head,fins,and body,and three foreground categories,and one background category.

2.3 Qualitative analysis of fish segmentation

According to the SENet optimized Deeplabv3+ semantic segmentation algorithm proposed in this paper,the experiment is carried out on the self-built freshwater fish data set.This paper used the SENet optimized Deeplav3+ semantic segmentation algorithm and the original Deeplav3+ semantic segmentation algorithm,the experiment results shown in Figure 6.The origin freshwater fish image is shown in the first column,and the unoptimized Deeplabv3+ semantic segmentation result is shown in the second column.The third column shows the result of Deeplabv3+ semantic segmentation after using the SENet optimization.The fourth column is ground-truth,which used the labelme software to conduct the real semantic label and the fifth column shown the visualized result of freshwater fish image segmentation.

(a)Origin image

From Figure 6,it can be seen that compared with the traditional Deeplabv3+ semantic segmentation algorithm,the algorithm used in this paper fully made use of information channels of feature dimensions and had better accuracy in processing the edge detail of freshwater fish,realizing the improvement of semantic segmentation performance on the freshwater fish body.Furthermore,it helps to actualize the mechanic process on freshwater fish,reduce the pressure of manual operation on aquatic products,and overcome the problem of low working efficiency and poor accuracy on segmenting freshwater fish’s head,abdomen,and fins et al.

After Deeplabv3+ semantic segmentation model and the SENet optimized Deeplabv3+ semantic segmentation model trained over 50 000 times,their loss value on function is shown in Figure 7.Through a comparison chart of the loss function,it can be seen that the loss value of the optimized SENet network was lower than the traditional Deeplabv3+ semantic segmentation model for nearly 0.15.The lower of loss value function image,the better fitting effect on the semantic segmentation model.Meanwhile,the robustness of the Deeplabv3+ semantic segmentation model was more superior.

Fig.7 Loss function diagram for implementing semantic segmentation of freshwater fish body under different algorithms

2.4 Quantitative analysis of fish segmentation

This paper adoptedMIoUas an important evaluation index for freshwater fish semantic segmentation ability.It can also be interpreted as an averageIoU,which calculates the average value ofIoUvalue on each part of the freshwater fish body.The definitions ofIoUandMIoUare shown in the following formula (4)and formula (5).

(4)

(5)

Wherek+1 is the total number of semantic categories,including background,iis the true value,jis the predictive value,andPijis the categoryiis predicted as the number of pixels of categoryj.

Based on the above evaluation standards,used the SENet optimized Deeplabv3+ algorithm and the origin Deeplabv3+ semantic segmentation algorithm that purposed in this paper to compare the experimental results based on the self-built freshwater fish data set,as showed in Table 1 and Table 2.Compared with Deeplabv3+,the optimized algorithm obtained the various characteristics of freshwater fish images through autonomous learning,then enhanced the details according to the importance level,and also suppressed currently useless features of freshwater fish body semantic segmentation tasks.As a result,it better optimized the original features of Deeplabv3+ network and had better performance on the processing of edge details.From Table 2,the semantic segmentationMIoUvalue in this paper reached about 93%.At the same time,the SENet network optimization reduced the FPS from the original Deeplabv3+ model.It saved time,improved the processing efficiency,enhanced the performance significantly,and reached the current advanced freshwater fish segmenting level.Thus,it provides a solid software system support for the subsequent realization of fully automated freshwater fish processing.

Tab.1 Comparison of IoU values of freshwater fish characteristics under different network structures

Tab.2 Performance comparison under different network structures

3 Conclusion

This paper presents a Deeplabv3+ based on the optimization of SENet.The following conclusions are obtained by the semantic segmentation algorithm of freshwater fish.

1)By fully exploring the advantages of Deeplabv3+ and applying the feature map generated by the expansion/cavity convolution to the backbone network,it overcomes the problem of continuous downsampling that causes the loss of resolution and information on detail parts.In addition,without increasing the amount of calculation,each output convolution contains a larger range of freshwater fish information,which increases the receptive field,improves the speed of semantic segmentation of various parts of the fish body,and reduces the size of the image.It also has a significant semantic segmentation effect on the smaller and thinner fin targets in the images.

2)SENet structure used the autonomous learning method to obtain the importance of every feature channel in freshwater fish images,and then improved the useful feature according to the importance and restrain useless feature for the freshwater fish semantic segmentation task at present.Compared with the traditional Deeplabv3+ method,the optimized network improved to 93% on theMIoU,the detection speed reduced to 42 ms,and the loss value of the loss function lower to 0.15.In addition,it achieved a better identification and detection of each part of the freshwater fish body.

3)Based on semantic segmentation,it can also be applied to the 3D modeling of fish bodies to obtain the semantic point cloud of the spatial position of freshwater fish bodies,which is beneficial for the further refine result of semantic segmentation of freshwater fish bodies.

(Translator:Mingxin Li,University of Illinois at Urbana-Champaign.Translated from:JournalofChineseAgriculturalMechanization,2021,42(2),slightly abridged.)