Chenzhong Gao, Wei Li

Abstract: This paper aims at providing multi-source remote sensing images registered in geometric space for image fusion. Focusing on the characteristics and differences of multi-source remote sensing images, a feature-based registration algorithm is implemented. The key technologies include image scale-space for implementing multi-scale properties, Harris corner detection for keypoints extraction, and partial intensity invariant feature descriptor (PIIFD) for keypoints description. Eventually, a multi-scale Harris-PIIFD image registration algorithm framework is proposed. The experimental results of fifteen sets of representative real data show that the algorithm has excellent,stable performance in multi-source remote sensing image registration, and can achieve accurate spatial alignment, which has strong practical application value and certain generalization ability.

Keywords: image registration; multi-source; remote sensing; scale-space; Harris corner; partial intensity invariant feature descriptor (PIIFD)

1 Introduction

Image registration has always been one of the most challenging works in image processing, especially for multi-source images. In order to achieve in-depth analysis, it is necessary to take advantage of the images obtained by different sensors[1−4], and the accurate registration of these images is the fundamental premise. Multi-source image registration is an essential step to implement multi-source data fusion, multi-source collaborative classification, and joint analysis.

In general, satellite remote sensing data provide spatial calibration information (spatial reference), such as latitude and longitude, map grid, etc. But in practice, it is found that there can be problems of spatial information loss. Besides, the spatial reference of different satellites or sensors often has the problem of inconformity,with a difference of a few pixels or even large deviations. It leads to inaccurate ground object correspondence, which can cause a decline in the accuracy of image processing results in data fusion,collaborative classification, change detection, image mosaic, and other applications.

In practical work, remote sensing image registration is usually carried out manually, by using ground control points, or through observation of the human eye, which is time-consuming,laborious, and inefficient. The existing automatic registration algorithm cannot meet the requirements of multi-source remote sensing image registration in accuracy, stability, efficiency, and adaptability. Many mature algorithms are dependent on the single-mode properties of images [5−9].However, multi-source remote sensing image registration is a more complex problem in which image pairs are generally obtained from different sensors, times, and viewpoints. Thus, differences between the imaging principle, resolution, condition, and many other factors can cause a multisource remote sensing image pair to contain changes in scale, blur, rotation, geometric and radiance distortion, content details, and noise, etc.,which invalidate the algorithms.

Multi-source remote sensing images are rich in types and have different characteristics, bringing great difficulty to realize universal registration algorithm. For example, hyperspectral image (HSI) and multispectral image (MSI), which provide wealthy spectral information, and light detection and ranging (LiDAR) data, which provide elevation information, are used together for joint classification [1−3]. Correspondence analysis with optical and synthetic aperture radar(SAR) images is investigated in depth [4]. In addition, the fusion of infrared and optical images is also one of the most important and commonly used technical means [10]. Panchromatic (PAN)images with high spatial resolution and texture information are often used for fusion with HSI and MSI. Therefore, it is in great need of effective, universal, and robust algorithms for multisource remote sensing image registration.

In this paper, focusing on the characteristics and differences of multi-source remote sensing images, a robust feature-based image registration algorithm is implemented, in which we use Harris corner detector for keypoints extraction,scale-space to implement multi-scale properties,and partial intensity invariant feature descriptor(PIIFD) for key points description. Eventually, a multi-scale PIIFD (MS-PIIFD) image registration algorithm framework is proposed, which can solve the main problems and difficulties: 1) different scales and blur, 2) radiance distortion, 3)geometric distortion, 4) different content details.The experimental results of 15 scenes of representative data show that the algorithm has excellent, stable performance in multi-source remote sensing image registration, which can achieve accurate spatial alignment, and has good universality and strong practical application value.

2 Related Works

Existing automatic image registration algorithms can be generally divided into two categories, the traditional algorithms and the algorithms based on deep learning. Traditional registration algorithms are systematically summarized and classified into area-based methods and feature-based methods according to technical means [11].

2.1 Area-Based Methods

The area-based registration methods, also called intensity-based methods, register images by directly using correlation of pixel values, mutual information, or transform domain of the image pairs. Usually, the global image or a predefined local window is used to search for matching [12].By establishing the similarity relationship of the original pixel values [13, 14], statistical properties [15−17], or transform domain, such as Fourier transformation [18, 19], and wavelet transformation [20], the optimal spatial transformation model between the image pairs is found through parameter optimization methods. The advantage of the area-based methods lies in high registration accuracy, but the drawbacks are particularly obvious. The drawbacks of the areabased methods mainly lie in three aspects.

The first drawback is that this type of method only works for simple spatial changes, such as translation and rotation, which are called rigid transformations. If the image pairs contain more complex spatial changes, such as scale difference,affine or perspective transformation, viewpoint changes, and various nonlinear geographic distortions, it is almost impossible to calculate the transformation parameters by comparing the similarity of the information in a window, which will make the methods fail.

The second drawback, which is the most serious, is that measuring similarity based on pixel values makes such algorithms too dependent on the modality of the images. In other words, areabased methods cannot handle intensity differences. For multi-source images, especially multisensor images, because of the different purpose and principle of image collection, their pixel values represent different physical information, so that there are multi-modal properties between them. Take optical and thermal infrared image registration as an example. The former not only has higher spatial resolution and clearer edge of the objects but also represents a lot of texture information. The latter has a lower resolution,blurry edges, and no complex content. The pixel values between the image pairs may have nonlinear differences or be totally different. For multi-temporal images, the image content is inconsistent, resulting in the similarity measurement being meaningless. Furthermore, noise also has a great influence on these methods.

Another drawback is that, when using a local window to search for matching, the content in the window should be weakly textured. If the area contained in the window is relatively smooth, there will be a lack of obvious features for matching. This problem also leads to a lot of redundant computations, making the algorithms less efficient.

The area-based methods are not good choices for multi-source remote sensing image registration.

2.2 Feature-Based Methods

The feature-based registration methods detect images’ salient structure to extract features as sparse representation, which can greatly reduce the computational complexity. Moreover, the feature-based methods are more robust to the scene difference and noise interference of the images,being relatively efficient.

The commonly used features in image registration include point, line, edge profile, and area features. Among them, compared with other features, the coordinate position of point features can be directly used to calculate the parameters of the transformation model, while other features often need to be transformed into control points. For example, the line feature is represented by the endpoints. Therefore, the point feature is an ideal matching primitive [21]. The feature-based, especially point feature-based algorithm has always been the focus of image registration research and is relatively mature at present. The point feature-based registration algorithm involves the following steps: 1) feature point detection, 2) feature description, 3) feature matching, 4) mismatch removal, 5) parameter estimation and transformation.

Scale-invariant feature transform (SIFT)[5, 22, 23] is one of the most classic, effective,and commonly used feature point matching methods. It first constructs a Gaussian scale space, supported by scale-space theory [24], and extracts local extreme points in a difference of Gaussian (DoG) pyramid as the feature points.Then, feature points are described by statistical gradient histograms. SIFT has a good scale and rotation invariance. Many improved algorithms[6, 25−30] based on SIFT were proposed in recent years.

Harris et al. [31] proposed a feature point detection method, which detects the corner structure that changes sharply in multiple directions in an image by calculating corner response according to the gradient. It is an effective and stable feature point detection algorithm, which is widely applied in feature-based image registration.

The oriented FAST and rotated BRIEF(ORB) [7] uses improved accelerated segment test (FAST) [32] and directional binary robust independent elementary features (BRIEF) [33]respectively for feature points detection and description. The algorithm has low time complexity, which is suitable for real-time registration.

Binary robust invariant scalable keypoints(BRISK) [8] and fast retina keypoint (FREAK)[9] are the improved algorithms of BRIEF, which are scale invariant, rotation invariant, and robust to noise. KAZE feature [34] uses the nonlinear diffusion filtering method to establish the scale-space without blurring the image, different from SIFT.

However, most algorithms are only applicable to single-modal image registration. With the increasing requirements of multi-source image registration, several algorithms for multi-modal image registration were proposed. Based on SIFT algorithm, Chen et al. [35] proposed PIIFD for multi-source retinal image registration, which has intensity and rotation invariance but is unable to deal with scale differences. Du et al. [36] later improved PIIFD by adding certain scale robustness,and proposed scale-invariant PIIFD (SI-PIIFD)for infrared and optical image registration. PSOSIFT [37] introduces a new gradient definition and an enhanced feature matching method for the registration of remote sensing images with intensity differences. Radiation-variation insensitive feature transform (RIFT) [38] proposes a maximum index map (MIM) for feature description, which has high intensity invariance, but low rotation invariance, and is not scale invariant.

Most of the research on multimodal image registration is focused on medical images.However, except for some differences in image characteristics and details, the multi-source remote sensing image registration has little difference in essence with medical image registration,so the algorithms also have universality.

2.3 Deep Learning-Based Methods

In recent years, research has attempted to achieve image registration by using deep learning methods. In general, relevant methods can be divided into two categories [39]: using deep learning networks to estimate the similarity measure of two images to drive iterative optimization, and directly using deep regression networks to predict transformation parameters. Deep learningbased methods first achieved good results in medical image registration. More and more research on remote sensing image registration began to shift its focus to the neural network and deep learning. Although the algorithms based on deep learning are relatively advanced, they often need plenty of annotated samples for model training,in which the model is highly targeted. So the deep learning-based methods cannot be widely applied to all kinds of multi-source data, and are not very mature at present.

3 Proposed Registration Method

The process of the MS-PIIFD registration algorithm is shown in Fig.1. The input is the image pair to be registered. The preprocessing is carried out first, including simple denoising and data normalization. The images used to obtain features are single-band grayscale images. For multi-band images, such as HSI and MSI, since the proposed algorithm is robust to radiation distortion, it is optional to select one of the bands or sum the multi-band data to generate singleband images as the input. Harris corner point detection is carried out on the pre-processed image.Then the scale-space of each image is created, in which a Gaussian pyramid is built. The PIIFD descriptor of each detected Harris corner point is established in the scale space. The feature points are matched according to the descriptors, and then the mismatch removal is carried out.

Fig.1 The proposed multi-source remote sensing image registration framework of MS-PIIFD

Finally, the spatial correspondence of the matching pairs is used to calculate (estimate)the parameters of the transformation model between the image pair.

3.1 Feature Point Extraction

To ensure the accuracy of registration, the feature points should be highly stable and repetitive between the image pair. For multi-source remote sensing images, the difference between scale, blur, intensity, or content details can lead to instability of local features. The feature point detection in SIFT and FAST algorithms is too sensitive to the intensity. Harris corner detection[31] focuses on locating obvious “corner” structure, which is distinct and stable even in multimodel images. Fig.2 shows the feature point detection results of the three methods in an opticalinfrared image pair. By comparison, the feature points obtained by the Harris corner detector are the most highly repetitive. Harris corner detector also has the advantages of strong robustness to local noise and high computation efficiency.Therefore, the Harris corner detector is adopted to extract feature points from multi-source remote sensing images.

Fig.2 Results of three feature point detection methods in an optical-infrared image pair: (a) SIFT; (b) FAST; (c) Harris

One key point worth noting is that multisource remote sensing images may have serious intensity distortion or scale difference, which results in the nonuniform distribution and low repeatability of feature points in the image pair.This problem is visually illustrated in Fig.3,which is the Harris corner detection result of an optical-infrared image pair with a large scale difference.

Fig.3 Harris corner detection results with and without LNMS in an optical-infrared image pair: (a)optical image without LNMS;(b)infrared image; (c)optical image with LNMS

The result of directly filtering with a threshold of the optical image is shown in Fig.3(a), in which most feature points are concentrated in the left part of the image. However, the approximate position of the infrared image in the visible image is indicated by the red box, and only a small number of feature points are distributed in it, which are very nonuniform. The proposed algorithm adopts local non-maximum suppression (LNMS) to solve this problem. From Fig.3(c), it can be seen that, with LNMS, the feature points are far more uniformly distributed,and the repeatability is higher.

3.2 Multi-Scale PIIFD

PIIFD [35] is an improved algorithm based on SIFT [5] to solve the problem of the full inverse(light and dark flipping) and nonlinear intensity difference in multi-source retinal images. Similar problems are evident in multi-source remote sensing images, which can be seen through the image pairs in Fig.2 and Fig.3. So in this paper, the PIIFD algorithm is used to describe the feature points.

In the original PIIFD [35], at first, each feature point is given a main orientation (0-π), by calculating the local second-order gradient, which ensures rotation invariance. Then the neighborhood of each feature point, the size of which is fixed at 40×40 pixels, is used for description.Taking the main orientation as the datum (0°),the gradient within the neighborhood is counted.Finally, a descriptor vector of length 128 is generated for feature point description. However, for multi-source remote sensing images with different scales, the information in the local neighborhood can have a large difference, which will lead to different descriptors of the same point.

SI-PIIFD [36] improved PIIFD by using a set of local windows with decreasing size (40×40,36×36, 32×32, 28×28, 24×24, and 20×20) for feature description on both images instead of fixed size. The final matching result is obtained by a majority vote strategy. For multi-source images with scale differences, SI-PIIFD can roughly obtain the same receptive field around the feature points in the two images, which has certain robustness but limited scale invariance. The scale difference mainly lies into two aspects, one is different resolution, and the other is different blur.The former can be roughly regarded as the object of the same width covering different numbers of pixels in the images, which is the main reason for the failure of the fixed-size window.The latter is usually related to the ability of sensors, which can also greatly affect the neighborhood information of feature points. Moreover,the feature points in a multi-source image pair may not all match best at a certain scale ratio,and the continuity of scale is not high in SIPIIFD.

In order to realize multi-scale PIIFD, local information of feature points needs to be extracted in the scale space of the images. The method of building image’s Gaussian pyramids based on the scale-space theory [24] is adopted. The schematic diagram of establishing Gaussian pyramid of image in the proposed algorithm is shown in Fig.4. The original image is sampled down step by step to obtain a series of images with different scales, that is, the first layer in each octave. Then a series of Gaussian blurs are performed on each octave:

Fig.4 Structure of the scale space (Gaussian pyramid) used in MS-PIIFD

whereIis the original image,Gis a Gaussian kernel with standard deviation ofσandLis the Gaussian blur image.

After the scale-space of the images is established, for each Harris corner point, the multiscale PIIFD descriptor is calculated by obtaining the neighborhood information at the corresponding location in the scale space. The algorithm flow is shown in Algorithm 1, whereGI(x,y,O,L)is the Gaussian scale space of imageIestablished through down-sampling and Gaussian blur,Ois the number of octave with different scale,Lis the number of layers with different blurring degree. The algorithm outputs the feature point descriptor setDP, which contains 128-dimensional descriptors for each feature point at each scale.The feature described in the scale-space can reduce the influence of scale difference of multisource images and obtain the best matching result of feature points.

3.3 Matching and Transformation

After all the descriptors of each feature point are created, the best-bin-first (BBF) [40] matching method is adopted to obtain the initial matches of the feature points, when the cosine similarity is used as the measurement. For multi-source images, there is likely to be a large number of mismatches in initial matches, which causes common outlier removal algorithms to fail, such as RANSAC (random sample consensus). In the proposed algorithm, the consistency of the main orientation and spatial distribution of the feature points [35] are used for mismatch removal.This is because although there may be spatial distortion in the multi-source remote sensing image pairs, there will not be too much deformation on the whole. Then, assuming that there is only overall rotation or scaling between the image pair. So the angle between the main orientation of each pair of correctly matched feature points should be consistent, and their spatial relationship with other correctly matched points in the two images should also be consistent. Therefore, this method is suitable for multi-source remote sensing images.

Finally, three transformation models, similarity, affine, and projective transformation, are selected for parameter estimation and image transformation to deal with the different spatial distortion that may exist between the image pairs, where the least square method is adopted to estimate parameters.

Algorithm 1 Proposed multi-scale PIIFD for registration Input: feature point set , image scale space PI GI(x,y,O,S L), PIIFD window scale DP Output: feature descriptor set GI(x,y,O,L)1 In each layer of :2 Calculate the second-order gradient of the image 3 Calculate the first-order gradient of the image P PI 4 For each feature point in :5 Calculate the corresponding position 6 Calculate the main orientation using second-order gradient 7 Taking the main orientation as the reference direction (0°), establish a square window with size (side length) of S, and divide it into 16 small squares 8 Statistics the first-order gradient direction within each small square to obtain feature descriptor D(P, O, L)9 Output feature descriptor set DPI

4 Experimental Results and Discussion

To validate the effectiveness of the proposed MSPIIFD, a set of distinct multi-source image data are used to test the robustness and universality of the proposed algorithm, and five state-of-theart multi-source registration algorithms including SIFT [5], PIIFD [35], SI-PIIFD [36], PSOSIFT [37] and RIFT+[38] are selected for comparison. Except for the feature point detection and description method, all processing is the same, including image normalization, noise reduction, feature point matching, and transformation,etc. The feature points used in PIIFD and SIPIIFD are the same as the proposed algorithm.The experiments are all implemented using MATLAB2021a on a platform with Intel Core I7-8 700 CPU.

4.1 Experimental Data

The data used in experiments are multi-source image pairs of fifteen scenes, labeled a-o, in which there are both multi-sensor and multi-temporal images, including HSI, MSI, LiDAR-derived DSM(digital surface model), SAR, optical images, infrared images, depth map, and artificially produced rasterized map. These selected experimental data basically contain all the problems in multi-source image registration, including large resolution differences, serious radiance distortions, spatial distortion, rotation, content details differences, and image noise. To further test the universality of the proposed algorithm, not only satellite data but also UAV data and ground scene data are selected.

4.2 Parameter Setting

The proposed MS-PIIFD algorithm contains four main parameters that can greatly affect the registration, which are the size of the LNMS window and the threshold of the number of feature points in the Harris corner detector, and the total number of octaves and layers of the Gaussian pyramid.

In Harris corner points detection, since the positional relationship and scale difference of the image pairs are unknown in advance, we hope that the feature points are distributed as uniformly as possible in the image pair. So, the ratio of the size of the LNMS window depends on the ratio of the pixel number of the two images:

whereM,Nandm,nare the length and width of the two images. The minimum size of the LNMS window is set according to the size of the images but at least 3×3 to prevent poor separability of the descriptors due to the feature points being excessively close. High repetition of feature points is required to provide sufficient matching candidates. But too many feature points not only increase the computational burden but also bring a lot of outliers. Through experiments, the threshold of the Harris corner points’ number is set between 500 and 1 000. The specific number is determined according to the size of the image.In this experiment, the maximum number of points was set at 1 000.

To ensure the efficiency of registration, it is necessary to set the suitable total number of octaves and layers, when too large scale space will increases the computational burden a lot.Through experiments, the total numbers of octaves and layers are set to 3 and 6, which can basically handle most multi-source image pairs.

4.3 Visual Results

The visual registration results of the proposed algorithm are shown in Fig.5. The results of each scene are respectively displayed in three forms:matching results of feature points, fusion form,and checkerboard form of the image pairs. Although the characteristics of scene contents of multi-source images are not consistent, the alignment of the images can be evaluated by roughly comparing the coincidence of objects’ edges.

Fig.5 Registration results of the proposed MS-Harris-PIIFD in three forms, including matching results (left), fusion form (middle),and checkerboard form (right): (a)PAN-HSI;(b)SAR-SAR;(c)MSI-HSI;(d)MSI-HSI;(e)optical-infrared-1;(f)optical-infrared-2;(g)optical-optical-1;(h)optical-optical-2;(i)HSI-LiDAR-1;(j)HSI-LiDAR-2;(k)HSI-LiDAR-3;(l)optical-infrared-3;(m)opticalinfrared-4;(n)depth-optical;(o)map-optical

By artificial comparison, there is no obvious deviation or “virtual shadow” in the images, except for scene f. This is because the imaging quality of the infrared image is low, and the image has a large spatial distortion, especially at the edge of the image. Simply using similarity,affine, or projection transformations will not align the image in all parts. In general, some functional interpolation and fitting methods will be applied in this case to achieve a better alignment effect, but it is not the focus of this paper,and further research will pay attention to these issues. It can be found that the spatial difference of images is within 1-2 pixels. Apart from the edge positions of scene f, there is no obvious registration failure in any of the scenes, and the registration algorithm is effective.

4.4 Numerical Results

The registration accuracy is usually measured by root mean squared error (RMSE) or methods through comparing the similarity of the image pair after registration. However, since the intensity of multi-source, especially multi-modal images, is different, the methods of measuring similarity do not have physical significance. In addition, without reference of geographic coordinates,the correspondence of pixels in multi-source images in real space is unknown, and RMSE is of little significance without groundtruth. Therefore,the registration effect is evaluated by directly comparing the number of correct matches of the feature points, because the number of correct control points is positively correlated with the accuracy of the transformation model. When the number of matched feature points is less than 3 pairs, the parameters of the spatial transformation model cannot be solved, and the registration is considered to be failed.

The numbers of correct matching pairs of the six registration methods are shown in Tab. 1.In scene g, i, j, n, o, the proposed algorithm does not get the most matches, and in contrast, the RIFT+is doing better. It is because there is basically no scale difference between the selected images of these scenes. In particular, the scene i,j, n, o are image pairs that have been artificially modified to the same scale. However, the proposed MS-PIIFD still obtains a considerable number of correct matches. Among all the competitive methods, the proposed registration algorithm is the only one that does not fail in any scene and has significant advantages in most scenes.

Tab. 1 Number of correct matching pairs comparison by six registration methods

5 Conclusion

In this paper, an effective image registration algorithm based on MS-PIIFD was proposed for multi-source remote sensing image registration.Through experiments and analysis, the algorithm can well achieve the registration of multi-source remote sensing images and performs well in practical work. The image registration error can reach within 1-2 pixels, and the objects in the scene can be aligned accurately.Through tests of various scene data registration,it is concluded that the algorithm has good robustness, stability, and universality, and has strong practical application value.