Wenqing Sun ,Long Xu ,Yin Zhang ,Dong Zhao ,and Fengzhen Zhang

1 State Key Laboratory of Space Weather,National Space Science Center,Chinese Academy of Sciences,Beijing 100190,China;lxu@nao.cas.cn

2 University of Chinese Academy of Sciences,Beijing 100049,China

3 Peng Cheng National Laboratory,Shenzhen 518000,China

4 National Astronomical Observatories,Chinese Academy of Sciences,Beijing 100101,China

5 State Key Laboratory of Communication Content Cognition,People’s Daily Online,Beijing 100733,China

Abstract Learning the mapping of magnetograms and EUV images is important for understanding the solar eruption mechanism and space weather forecasting.Previous works are mainly based on the pix2pix model for full-disk magnetograms generation and obtain good performance.However,in general,we are more concerned with the magnetic field distribution in the active regions where various solar storms such as the solar flare and coronal mass ejection happen.In this paper,we fuse the self-attention mechanism with the pix2pix model which allows more computation resource and greater weight for strong magnetic regions.In addition,the attention features are concatenated by the Residual Hadamard Production (RHP) with the abstracted features after the encoder.We named our model as RHP-attention pix2pix.From the experiments,we can find that the proposed model can generate magnetograms with finer strong magnetic structures,such as sunspots.In addition,the polarity distribution of generated magnetograms at strong magnetic regions is more consistent with observed ones.

Key words: techniques: image processing–Sun: magnetic fields–Sun: general

1.Introduction

With the development of satellite and electronic technology,more and more space telescopes and equipment have been built to obtain different kinds of observations of the Sun.Solar Dynamics Observatory (SDO) (Pesnell et al.2012),a near-Earth space satellite,was launched in 2010 February.The telescope Atmospheric Imaging Assembly(AIA)(Lemen et al.2012)on board SDO can observe solar UV and EUV emissions from the transition region to the corona.As a part of the SDO,the Helioseismic and Magnetic Imager (HMI) (Scherrer et al.2012) can provide components of the photospheric magnetic field.NASA’s twin Solar TErrestrial RElationship Observatory(STEREO) (Kaiser et al.2008) space craft can only provide EUV observations to monitor the far hemisphere from the farside of the Earth’s orbit without magnetograms.However,the magnetic field manipulates the variations of the Sun,especially solar eruptions such as solar flares and CMEs.These eruptions can cause disastrous space weather.Thus,Kim et al.(2019) apply pix2pix (Isola et al.2017) to generate the farside magnetograms from STEREO EUVI 304 Å images.In this way,we can capture the global observation of magnetic fields for space weather forecasting and the evolution of active regions for the rotating of the Sun.

Conventionally,farside magnetograms can be constructed from local helioseismology without any farside data,but their quality is lower than that of typical frontside magnetograms(Lindsey and Braun 2000).Kim et al.(2019) generate solar farside magnetograms from the STEREO EUVI 304 Å using Generative Adversarial Networks(GAN) (Isola et al.2017).This model defines the task as an image translation problem and learns the mapping between AIA EUV 304 Å and HMI LOS magnetograms.Then the trained model can generate magnetograms from STEREO EUVI 304 Å images.Liu et al.(2021) analyze the reliability of this model and points out that this method has problems of data leakage on the division of the data set.Jeong et al.(2020) apply the pix2pixHD (Wang et al.2018)for magnetogram generation with±3000 Gauss dynamic range and multi-channel AIA EUV images of 171,195 and 304 Å.Alshehhi (2020) modifies the pix2pix model with pyramid architecture for magnetograms generation from AIA EUV 304 Å images to magnetograms.This method can capture internal distributions at different scales with a higher quality of the magnetic structure.Recently,Jeong et al.(2022) have upgraded the pix2pix.They apply a correlation coefficientbased loss function.In addition,the input data applied to not only farside STEREO EUV images but also frontside data pairs of SDO/AIA EUV images and the HMI magnetograms as reference information.It is worth noting that these existing works all concentrated on full-disk magnetogram generation.

From the above observation,we can find that the key point of magnetogram generation is to learn a mapping between magnetograms and EUV images.The full-disk magnetogram is composed of invalid pixels around the disk and quiet region,which is more vulnerable to noise.It is particularly noteworthy that solar extreme events on the Sun,such as flares and coronal mass ejections (CMEs),occur at the active regions (ARs)with a stronger magnetic field relative to the quiet regions.Thus,in this paper,our efforts mainly focus on magnetogram generation of the active region.The baseline of our model is a pix2pix model,which is a conditional generative adversarial network(cGAN) (Mirza &Osindero 2014;Goodfellow et al.2014),consisting of a generator and a discriminator.The generator is a U-Net(Ronneberger et al.2015)which is a fully convolutional network,so it can only calculate the correlation of the spatially local points and treat each pixel equally.However,the stronger magnetic field of the active region should be allocated more computation resources and greater weights.In addition,the magnetic field at a distance can be coupled,and an attention mechanism can explore the correlation in distant positions.For this purpose,the self-attention mechanism (Xu et al.2017;Wang et al.2017;Zhang et al.2018) should be employed in a U-Net for our concerned task.

In this paper,we fuse a self-attention module with the pix2pix for magnetogram generation of the active region from local AIA 304 Å images with its correspondent local HMI LOS magnetograms as shown in Figure 3.The attention features are concatenated by the Residual Hadamard Production (RHP)(Mejjati et al.2018) with the abstracted features after the encoder.In Section 2,we introduce the proposed model in detail.In Section 3,we give the data set processing and the experimental analysis.In Section 4,we draw the conclusion.

Figure 1.Overall Architecture.

2.Methods

Many computer vision problems,such as denoising (Park et al.2020) and super-resolution (Rahman et al.2020),can be viewed as image-to-image translation problem (Isola et al.2017).Pix2pix was the first unified framework for this task.However,the pix2pix is a fully convolutional neural network.Convolution processes the information in a local neighborhood,thus using the convolutional layer alone cannot explore the non-local dependencies in an image.For AR magnetogram generation,the strong magnetic field region as the most informative region should be allocated more computation resource.Thus,we fuse the self-attention module with pix2pix for the AR magnetogram generation,named RHP-attention pix2pix.

2.1.Pix2pix Baseline

Pix2pix is a conditional GAN which is consists of the generator (G) and discriminator (D).The architecture of the generator is a U-Net (Ronneberger et al.2015).The discriminator is a multi-layer convolutional network.The generator is trained to fool the discriminator,and the discriminator is trained to distinguish the generated image from real ones.During the adversarial process,the generator can generate real-like images.The adversarial loss is defined as:

wherexis the real observed magnetogram whileyis the generated one,zis the input AIA 304 Å image.We employ a U-Net as the backbone of our proposed model.A U-Net is an encoder-decoder architecture overall.The attention block is inserted between the encoder and the decoder,as shown in Figure 1.In this way,the abstracted features by the encoder are weighted by the attention block,and the decoder uses these processed features for magnetogram generation.

2.2.Attention Block

The detailed framework of the attention block can be found in Figure 2 (Zhang et al.2018).The features which are abstracted from the previous layers,x∊Rc×N,are first transformed into two feature spaces byf(x)=Wfx,g(x)=Wgx,where.Then,a weighted sum of the features is calculated for each location by

Figure 2.Attention block.

where βj,iindicate the extent of the model attends to theith location when synthesizing thejth location.The output of the attention layer is computed by

where γ is a learnable parameter.

The self-attention feature maps should concatenate with the abstracted feature maps of the encoder.We concatenate these features by the Residual Hadmard Product(RHP)(Mejjati et al.2018).The RHP is formulated by :

whereFrepresents the concatenated features,xrepresents the encoded features,andyis the attention feature maps.

Besides,we apply the spectral normalization method(Miyato et al.2018) to the learned weights,such asWg,Wf,WhandWv,which can stabilize the training process of the GAN.The spectral normalization can be formulated as follows:

whereWis the learned weight.σ(W)is the spectral norm of the matrixWand it is equivalent to the largest singular value ofW.

3.Experiments

For evaluating the proposed model,namely RHP-attention pix2pix,we compare it with the pix2pix backbone.The configuration of the pix2pix and the RHP-attention pix2pix are the same.We implement our model using the deep learning platform of PyTorch (Paszke et al.2019) and train our model on a single NVIDIA Tesla P100 GPU.The conditions of experiments are as follows.The model can converge after training epoch of 200.The learning rate is 0.0002.Batch size is 32.Slope of LeakyReLu is 0.2 and Dropout (Srivastava et al.2014) probability is 0.5.

3.1.Dataset

In this work,the data set consists of magnetograms from the SDO/HMI and EUV 304 Å images from SDO/AIA.We downloaded these data with Sunpy (SunPy Community et al.2015) from Virtual Solar Observatory (VSO).The observation time of the data set is from 2012 January 1 to 2012 December 31 with a cadence of 12 minute.The data processing process is as follows.First,the HMI LOS magnetograms and AIA 304 Å images are processed by calibrating,rotating and centering.In addition,they are aligned.The saturation limits of magnetograms are ±200 G,and the AIA 304 Å images are normalized by their corresponding exposure times.We extract a patch of 180° in east–west direction and 160° in north–south direction centered on the equator from full-disk solar image.Then,we interpolate the patch to the size of 1800×1600 pixels by the CUBIC method (Fritsch &Carlson 1980) and crop the center 900×800 pixels to reduce the projection effect.For model training,we randomly crop a bunch of image patch of 512×512 from each image patch of 900×800 mentioned above for data augmentation.Thus,a data set with 20,099 samples in total is established.A sample of this data set is shown in Figure 3.Finally,we split the data set into the training,validation and testing sets.The training set contains samples from 2012 January 1 to 2012 August 30,validation set from 2012 September 1 to 2012 September 30,and testing set from 2012 October 1 to 2012 December 31.There is a time interval of one month between training set and testing set.

Figure 3.Local AIA 304 Å image with its correspondent local magnetogram.

Figure 4.Comparison of generated local magnetograms among different models.(2012-Dec-14 19:36UT).

Figure 5.Comparison of generated local magnetograms among different models.(2012 October 17 08:00UT).

3.2.Results and Analysis

In this paper,we propose a model based on the self-attention mechanism called RHP-attention pix2pix,using pix2pix as a benchmark network.The features provided by the attention mechanism need to be combined with the features extracted by the encoder to generate the corresponding magnetogram through the decoder.We compare two schemes of feature combination.The first oneF1is to add the attention featuresFattento the featuresxlearned by the encoder,and then concatenate them withxby channel.The second oneF2is to compute the RHP product of the attention featuresFattenwith the encoder featuresx,and then concatenate them withxby channel.These two feature concatenations are named catattention pix2pix and RHP-attention pix2pix respectively.

For objective evaluation,two image quality measurements,SSIM and PSNR,and pixel-to-pixel correlation coefficient are employed.The PSNR and SSIM can be formulated as:

wherexandyrepresent observed magnetogram and generated one respectively,μxand μyare the means ofxandyrespectively,andare the variances ofxandyrespectively,σxyis the covariance ofxandy,c1andc2aretwo constant parameters to stabilize the division with weak denominator.Following the setting of original authors,c1andc2are 2.55 and 7.65,respectively Wang et al.(2004).SSIM is more likely to reflect the structural similarity of two images.PSNR is defined by the mean squared error (MSE) and maximum pixel value (MaxValue) of an image.It qualifies pixel-to-pixel difference between generated image and groundtruth.

In Table 1,peak signal-to-noise ratio (PSNR),feature similarity measure (SSIM) and correlation coefficient (CC) of magnetic field strength are calculated for the three models.It can be observed that all of the metrics are improved by the two proposed attention augmented models,especially the CC is about 20% larger than that of the baseline.We also give the standard deviation of PSNR,SSIM and CC in Table 1.It can be observed that the standard deviations (STDs) are comparable among the three models.Relative to pix2pix baseline,RHPattention pix2pix achieves PSNR,SSIM and CC improvements of 6.21%,10.17% and 20.41% respectively,proving the effectiveness of RHP-attention module.Comparing the two attention models,they are almost comparable in terms of objective quality.However,in terms of subjective quality,RHP-attention can generate more realistic and clearly structured results than cat-attention in strong magnetic fields.

Table 1 Objective Comparisons between the Pix2pix Baseline and the Proposed Mode Pix2pix-attention

In Figures 4–5,the observed magnetograms and the magnetograms generated by the different models and the corresponding color maps are shown.Observing Figure 4 with the observed time of 2012 December 14 19:36 UT,we can notice the presence of a sunspot at region (a) for the HMI observed magnetogram.Comparing three generated magnetograms,the pix2pix and cat-attention pix2pix fail to generate the corresponding sunspot structure,while RHP-attention pix2pix can successfully generate the corresponding sunspot structure.Observing Figure 5 with the observed time of 2012 October 17 8:00UT,there is a clear distinction between the positive and negative polarity of the magnetic field at region (c).The RHPattention pix2pix model can successfully generate this distribution.While the pix2pix and cat-attention pix2pix generate the positive polarity structure mixed with negative polarity structure.For the small scale magnetic region (b) and region (d),none of models can generate the right magnetic polarity distribution.

It is still a big challenge to generate right magnetic polarities at a fine scale for magnetogram generation currently.In fact,we cannot guarantee absolutely consistent polarities between generated magnetogram and observed one at small scale.They are only morphologically similar on large scale magnetic structures.This result can mostly satisfy qualitative analysis,e.g.,event analysis of solar burst.

From the above two experiments,we can further conclude that:

1.Compared to cat-attention,RHP-attention is not only more sensitive to the strong magnetic field structure but also more faithful to the magnetic polarities.This phenomenon indicates that the RHP concatenation is more efficient than channel concatenation when fusing attention maps and feature maps;

2.By augmenting attention module to the pix2pix model,more computational resources are allocated to stronger magnetic field,so the generated magnetic field around strong magnetic field has higher fidelity.

4.Conclusion

This paper proposes an RHP-attention pix2pix model for AR magnetogram generation.The RHP-attention pix2pix model augments the self-attention mechanism to the pix2pix.In addition,it concatenates attention features through the RHP.From the experiments,the proposed model can generate magnetograms with finer strong magnetic structures,such as sunspots.In addition,the proposed model can generate more faithful strong magnetic polarity distribution.

Acknowledgments

This work was funded by the National Key R&D Program of China (Nos.2021YFA1600504 and 2022YFE0133700),and the National Natural Science Foundation of China (NSFC)(Nos.11790305,11963003,12273007 and 61902371).