Wenxia Bao， Yaping Yang， Dong Liang and Ming Zhu

(Key Laboratory Intelligent Computing and Signal Processing of the Ministry of Education, Anhui University, Hefei 230039, China)

Abstract: A multi-residual module stacked hourglass network (MRSH) was proposed to improve the accuracy and robustness of human body pose estimation. The network uses multiple hourglass sub-networks and three new residual modules. In the hourglass sub-network, the large receptive field residual module (LRFRM) and the multi-scale residual module (MSRM) are first used to learn the spatial relationship between features and body parts at various scales. Only the improved residual module (IRM) is used when the resolution is minimized. The final network uses four stacked hourglass sub-networks, with intermediate supervision at the end of each hourglass, repeating high-low (from high resolution to low resolution) and low-high (from low resolution to high resolution) learning. The network was tested on the public datasets of Leeds sports poses (LSP) and MPII human pose. The experimental results show that the proposed network has better performance in human pose estimation.

Key words: human pose estimation; residual learning; image feature; hourglass network

Human pose estimation aims to determine the precise pixel position of human body joints from an input image. It is a fundamental yet challenging task in computer vision and an important basis for high-level vision tasks such as behavior recognition, human-computer interaction, clothing parsing, re-identification and motion capture. An effective human pose estimation must be able to deal with large pose changes, changes in clothing and lighting conditions, severe body deformation and body occlusion problems. One of the key issues in solving these problems is how to extract powerful low-level and mid-level appearance features to capture relevant contextual information on each scale, and how to establish complex part relationships to achieve effective pose inference.

Traditional methods for human pose estimation are based on pictorial structure models, which use tree structures to model the spatial relationship of rigid body parts, usually using a predefined human body model assuming that each body part is independent of all other body parts except those connected to it. These models can describe pose accurately, but their main disadvantage is that they need to hand-design the structure of the model to capture specific dependencies between different output variables while allowing for easy inference.

Methods based on a pictorial structure model are limited by feature expression ability. In practical applications, the flexible movement of the human body and the constant changes in vision and clothing lead to a large change in the visual information of body parts, thus bringing great challenges to human pose estimation. Human pose estimation has recently showed significant progress with the widespread use of convolutional neural networks (ConvNets) and many auxiliary methods, such as resnet, inception design and batch normalization.

Human pose estimation based on ConvNets can be classified as detection-based or regression-based. Detection-based methods use ConvNets as part detectors, then combine the graphical model to estimate pose, which requires the manual design of the energy function or the heuristic spatial probability prior initialization to remove outliers on the regressive score map. Some of these methods also use private networks for precise correction. For example, Tompson et al.[1]used multiple branches of ConvNets to fuse features of image pyramids and used Markov random fields (MRF) for post processing. This architecture can improve the performance of pose estimation using the geometric relationship between the part positions. Chu et al.[2]proposed a geometric transformation kernel for the convolutional layer of ConvNets, which can effectively characterize the dependence between keypoints, and a bidirectional tree structure model, which can make each keypoint receive information from all relevant joints and optimize their features. Bulat et al.[3]designed a detection-regression ConvNet cascade network structure, which first uses the detection sub-network to output the score maps of each part and then uses the regression sub-network for joint regression. The regression-based method mainly uses ConvNets to realize the nonlinear mapping from the image to human pose. Pfister et al.[4]proposed a spatial fusion layer and combined the optical flow information from multiple frames in the video to capture the dependencies between joints, so as to estimate the human pose in the video. Carreira et al.[5]use an iterative error feedback network that introduces top-down feedback on the standard model and iteratively feeds the learned pose information back to the first layer of the network to gradually correct the initial prediction.

Since human pose estimation based on ConvNets have proven to have significant advantages, how to design a dedicated network structure with higher accuracy and robustness for human pose estimation has become a new research direction. Therefore, based on the regression-based approach, a human pose estimation method based on multi-residual module stacked hourglass networks (MRSH) is proposed in this paper. Three different residual modules are used in the network as the building blocks for an hourglass sub-network. When the resolution is high, the large receptive field residual module (LRFRM) and the multi-scale residual module (MSRM) are used to capture information in a larger range and on each scale. When the resolution is reduced to the lowest level, only the improved residual module (IRM) is used, and intermediate supervision is performed at the end of each hourglass sub-network, repeating the high-low, low-high learning. The final network in this paper uses four multi-residual module hourglass sub-networks and verifies network performance on two standard public datasets.

1 Hourglass Sub-Network

1.1 Common hourglass sub-network

Fig.1 is a common hourglass sub-network consisting of pool, upsample and residual modules. The hourglass network is designed to capture information at every scale in a feed-forward fashion. Local features are essential for learning smaller parts such as arms and legs, and the final pose estimation requires a coherent understanding of the whole body. For example, the orientation of the person, the position of the limbs, and the relationship between adjacent joints are some of the most recognizable cues at different scales in the image. The hourglass network captures these features better and combines them to ultimately output pixel predictions.

Fig.1 Schematic diagram of the common hourglass sub-networks

During the learning process of the feature map, the size of the feature map of the input hourglass sub-network is 64×64. First, the feature map is downsampled to perform high-low processing. The network achieves the lowest resolution at 4 pixels and then performs low-high processing by upsampling the features from the bottom layer combined with the higher resolution features.

1.2 Residual module

The building block used by the common hourglass sub-network is the residual module shown in Fig.2. The module consists of two branches: the first branch is the identity map branch and the second branch is composed of three convolution layers (the size of the convolution kernel is 1×1, 3×3, 1×1), batch normalization, and relu. The use of this residual module not only improves the performance of the ConvNets, but also reduces the training burden on the network.

Fig.2 Schematic diagram of residual module

2 Multi-Residual Module Stacked Hourglass Network

2.1 Improved residual module

Residual learning[6-7]uses identity mapping to enable us to train very deep neural networks, but it is also the root cause of residual learning: identity mapping increases the response change when the network is deeper, thereby increasing optimization difficulty.

Fig.3 Schematic diagram of IRM

Therefore, we use batch normalization, rrelu and 1×1 convolution to replace the identity mapping in the previous method (the branch(1) in Fig.3). The advantage of this substitution is that when the resolution of the feature map is reduced, or the dimension of the feature channel is increased, the identity mapping is replaced by the convolution layer, and the network can reset the response variance to a smaller value, thereby effectively avoiding larger variance response.

In addition, we introduce a spatial dropout layer[8]and use rrelu activation to the IRM before the convolution in the middle of the branch(2) of the original residual module. The role of the dropout layer is to improve the generalization ability of the model by preventing the activation from becoming strong and preventing over-fitting. Due to the full-convolution network we use, there is a strong spatial correlation between the joints of human body on the training image, and the activation of the feature map is also has strong correlation. As a result, the ordinary dropout layer cannot regularize its output; otherwise it will lead to a significant decline in learning rate. In this case, the spatial dropout proposed in Ref.[8] can help the network learn the correlation between adjacent pixels on the feature map.

2.2 Large receptive field residual module

We focus on human pose estimation, in which larger receptive fields[9-10]have been shown to be important for locating local body parts, and the context area of neurons is its corresponding receptive field. Although the IRM proposed has a certain improvement on the performance of the model, the receptive field of this module is small. In order to better learn the correlation between the joints of the human body, a large receptive field residual module is designed on the basis of the IRM.

Fig.4 shows a schematic diagram of the LRFRM, which is mainly composed of convolution, batch normalization, activation layer, pool, upsample, and spatial dropout layers. This module adds a branch to the IRM, that is, the branch(3) in Fig.4, which includes two 3×3 convolution layers, each convolution layer has a batch normalization and an activation layer before it, and the spatial dropout layer is added before the second 3×3 convolution layer. The addition of this convolution branch can effectively expand the receptive field of the output layer.

Fig.4 Schematic diagram of LRFRM

Deep residual networks have made great breakthroughs in image classification tasks by introducing stacked residual modules, which can be represented as

(1)

wherepiandpi+1are the input and output of thei-th residual module,Fis the stacked convolution, batch normalization and rrelu nonlinearity, andh(pi)=piis the identity mapping.

The expression of LRFRM designed in this paper is

(2)

2.3 Multi-scale residual module

Based on the LRFRM, a MSRM is proposed, which consists of convolutional, batch normalization, activation layer, pool, upsample and spatialdropout layers. As shown in Fig.5, the module adds branch(4) to the LRFRM, including a 1×1 convolution and two 3×3 convolutions, in which the first two convolution layers have a downsampling, and the last 3×3 convolution layer is preceded by a spatial dropout layer. In front of each convolution layer there is a batch normalization and an activation layer. The representation of this module is

(3)

Fig.5 Schematic diagram of MSRM

The MSRM consists of four branches. The branches(1)(2)(3) are consistent with the LRFRM, and branch(4) is the new design in this paper. Its structure is pool2×2+conv3×3+pool2×2+conv3×3+upsample2×2+upsample2×2. Branch (4) is downsampled twice, so the feature map is reduced to one-fourth of the input feature map and then merged with the other three branches after being upsampled to the size of the input feature map, and then sent to the next module. Branch(3) reduces the feature map to one-half of the input feature map. Branches(1) and (2) do not change the size of the feature map. Therefore, the learning of images at different scales is achieved.

2.4 Multi-residual module stacked hourglass network

2.4.1Multi-residual module hourglass sub-network

In the previous method, the hourglass sub-network (shown in Fig.1) only use one residual module, so it can only capture a single scale visual mode or semantic information, and the effective field of the network is small.

In this paper, three different functional residual modules are used as the building blocks of the hourglass sub-network. Our hourglass sub-network is named as multi-residual module hourglass sub-network, the schematic diagram shown in Fig.6 which can better capture the multi-scale visual mode and richer feature information. It consists of pool, upsample, IRM, LRFRM, and MSRM. The size of the input image of the hourglass sub-network is 64×64. The LRFRM and the MSRM are used to capture information on a lager range and on various scales, while the IRM is only used when the resolution is reduced to a minimum.

Fig.6 Schematic diagram of multi-residual module hourglass sub-network

2.4.2Multi-residual module stacked hourglass network

Fig.7 shows the overall framework of our MRSH proposed in this paper, which is composed of two parts. The first half of the network uses convolution, LRFRM, and pool to capture the information on the feature map and learn the human pose estimation. The second half is composed of multiple hourglass sub-networks, and intermediate supervision is performed at the end of each hourglass. This repeated high-low, low-high processing creates a stacked hourglass network, that is, our multi-residual module stacked hourglass network. The input picture size of the network is 256×256; after a convolution layer (the size of the convolution kernel is 7×7, the step length is 2, the feature map size becomes 128×128. Then we use LRFRM to obtain the feature information, and the feature map size is changed to 64×64 after pool. Finally, the feature map passes through two LRFRMs and enter the stacked hourglass sub-network for further feature learning. The number of hourglass sub-networks finally used in this paper is 4, achieving high precision on two common datasets.

Fig.7 Schematic diagram of MRSH

3 Joint Prediction

Sk(t)～N(vk,Σ)

(4)

(5)

whereMis the number of samples. During inference, we obtain the predicted joint positionkfrom the predicted score maps generated by the last stack of hourglasses by obtaining the position with the largest score as

(6)

4 Datasets and Criteria

4.1 Datasets

We tested the performance of network models on two widely used public datasets: Leeds Sports Poses (LSP) and MPII Human Pose.

① LSP datasets: We represent the LSP dataset[11]and its extended dataset[12]as LSP datasets. The LSP dataset is a sample of images collected from Flickr, each image annotating 14 whole body joints and containing universally recognized poses that are difficult to estimate. The data set consists of 11 000 training image samples and 1 000 test image samples, and the experiment uses person-centered (PC) annotations for training. Adding MPII data significantly improves our performance, due to its labeling quality being much better than LSP. Because of the noisy labels in the LSP dataset, we use the LSP dataset to train with the MPII Human Pose dataset and test on the LSP test set.

② MPII datasets: The MPII Human Pose dataset[13]includes approximately 25 000 images, including over 40 000 annotated poses. These images were collected from YouTube videos, each of which annotated 16 whole body joints, covering daily human activities with highly articulated human poses. Since the test set of the MPII dataset is not disclosed, the test results for the MPII dataset are all from the official website.

4.2 Criteria

Two criteria are used in the experiment to evaluate the performance of the human pose estimation method: percentage of corrected parts (PCP)[14-15], percentage of corrected keypoints (PCK)[14].

① PCP: A widely-used criterion for human pose estimation which evaluates the position accuracy of body parts. If the predicted part end points are within half of the part length from the ground truth part end points, then the body part is determined to be correct.

② PCK: Although PCP is the preferred criterion for pose evaluation, it has the drawback of penalizing shorter limbs. Therefore, PCK is introduced to measure the detection rate of body joints. PCK evaluates the position accuracy of the human joints, where a joint is considered as detected if the distance between the detected joint and the real joint is less than a fraction of the torso diameter. The torso diameter is defined as the distance between the left shoulder and the right hip of each real pose.

In our experiments, we follow the official benchmark evaluation protocols. The official benchmark on the MPII dataset adopts PCKh (using portion of head length as reference) at 0.5, while the official benchmark on the LSP dataset adopts PCP at 0.5 and PCK at 0.2.

5 Experiments and Analysis

To reduce over-fitting, we augment the training data by rotating, scaling and flipping, so that the performance of the model is more robust. All the models are trained using Torch. The initial learning rate is 2.5×10-4and the parameters are optimized by the RMSProp algorithm. We use a GeForce GTX Titan X GPU to train the model on the LSP dataset and the MPII dataset.

5.1 Performance analysis of MRSH

In order to evaluate the influence of the hourglass sub-network in MRSH on human pose estimation, we use different numbers of hourglass sub-networks on the LSP dataset under the same conditions. Tab.1 and Tab.2 respectively show the experimental results of the network training model composed of different numbers of hourglass sub-networks under the two evaluation criteria of PCP and PCK. Fig.8 and Fig.9 are PCK curves and PCP curves, where the abscissa indicates different thresholds, and the ordinate indicates the accuracy of each joint.

Tab.1 PCK results on LSP data

Tab.2 PCP results on LSP data

Fig.8 PCK curve

Fig.9 PCP curve

5.2 Comparative experiment

① Comparative experiment on LSP

Our method is compared with other pose estimation methods in recent years. The comparative experiment results under the two criteria of PCK and PCP on the LSP data set are given in Tab.3 and Tab.4. Fig.10 shows the PCK plots for all methods in Tab.3, and Fig.11 shows the PCP plots for all of the methods in Tab.4.

② Comparative experiment on MPII

In the experiment, several pose estimation methods related to the MPII dataset in recent years were selected and compared with our method. The comparison results under the PCKh standard are given in Tab.5.

Tab.3 PCK comparison results on LSP data

Tab.4 PCP comparison results on LSP data

Fig.10 PCK curves on LSP data

The results in Tab.5 are all from the MPII official website. Ref. [10] and Ref. [22] also use the hourglass network but the final result is 8 stacked hourglasses, and the final result of this paper is 4 stacked hourglasses.

Fig.11 PCP curves on LSP data

Tab.5 PCKh comparison results on MPII data

5.3 Analysis of results

According to the results in Tab.1 and Tab.2, it can be seen that an increase in the number of hourglass sub-networks can effectively improve the performance of the network. This is because the characteristics learned by a single hourglass sub-network are limited and the network’s receptive field is small, while the stacked hourglass network can learn more feature information. From the results in Tab.3, Tab.4, and Tab.5, we can know that the method in our paper is competitive. We can achieve such good results because our MRSH can fully learn the feature information of the image at various scales during the training process, and the correlation between the joint points is learned in a large enough receptive field, which greatly improves the accuracy of the human body posture estimation.

6 Conclusion

This paper proposes a multi-residual module stacked hourglass network for human pose estimation. The network uses three different residual modules to effectively capture image feature information and visual information of images at various scales and more accurately predict joint coordinates. Our approach was tested on two challenging datasets and achieved the highest accuracy under both widely used criteria. However, some failures were observed in the experiment, including multiple people or rare poses in the picture. Handling multiple people in a single end-to-end architecture is also a challenging issue, and our future work will explore the multitude of images and the estimation of human poses for joint occlusion.