Jinfu CHEN ,Xiaoli WANG ,Saihua CAI ,Jiaping XU ,Jingyi CHEN ,Haibo CHEN

1School of Computer Science and Communication Engineering,Jiangsu University,Zhenjiang 212013,China

2Jiangsu Key Laboratory of Security Technology for Industrial Cyberspace,Jiangsu University,Zhenjiang 212013,China

†E-mail:caisaih@ujs.edu.cn

Abstract: Cross-project software defect prediction solves the problem of insufficient training data for traditional defect prediction,and overcomes the challenge of applying models learned from multiple different source projects to target project.At the same time,two new problems emerge:(1) too many irrelevant and redundant features in the model training process will affect the training efficiency and thus decrease the prediction accuracy of the model;(2) the distribution of metric values will vary greatly from project to project due to the development environment and other factors,resulting in lower prediction accuracy when the model achieves cross-project prediction.In the proposed method,the Pearson feature selection method is introduced to address data redundancy,and the metric compensation based transfer learning technique is used to address the problem of large differences in data distribution between the source project and target project.In this paper,we propose a software defect prediction method with metric compensation based on feature selection and transfer learning.The experimental results show that the model constructed with this method achieves better results on area under the receiver operating characteristic curve(AUC)value and F1-measure metric.

Key words: Defect prediction;Feature selection;Transfer learning;Metric compensation

1 Introduction

With the rapid development of information technology,software has been widely used in applications(Malhotra,2015;Wahono,2015).However,due to coding errors and incorrect understanding of software requirements,problems often occur in software(Chen X et al.,2018),and the defective software can cause incalculable losses once deployed.Software defect prediction is an important technique to ensure software quality and safety(Hall et al.,2012),which improves the quality of software products by identifying classes or modules with defects in advance to effectively allocate testing resources(Tsuda et al.,2019).

In the past two decades,many software defect prediction methods have been proposed to predict defects in a single project (called within-project defect prediction,or WPDP) (Wu et al.,2017),which trains the defect prediction model using a historical labeled dataset from the same project.The high similarity and high coupling between modules in the same project allow the defect prediction model to achieve a good prediction performance.However,it is difficult to obtain enough historical data from the same project to train the model in practical applications,and because of the lack of historical data,the constructed model cannot achieve a high performance (Wan et al.,2020).A simple solution to this problem is to build a defect prediction model for the target project using high-quality datasets collected from other projects;this is called crossproject defect prediction (CPDP) (Ryu et al.,2017;Tabassum et al.,2020).CPDP is less accurate than WPDP (Liu et al.,2019) due to the distribution of metric values between the source and target project datasets (Iqbal et al.,2020).This problem has already attracted wide attention from researchers.For example,Watanabe et al.(2008) proposed a metric compensation method to improve the similarity between the source project and target project,but this method focuses only on using the source project to unilaterally adapt to the data distribution of the target project and do not consider the effect of redundant features on the efficiency of model training.In contrast,we would like to achieve bidirectional adaptation of the data distribution between the source project and target project.In addition,when the dimensionality of the project’s features is too high,the spatial and temporal efficiency of model training affects the prediction accuracy of the model.If these two challenges can be solved,the defect prediction performance can be greatly improved.Our approach solves the bidirectional adaptation problem by combining metric compensation and transfer learning,and solves the dimensionality problem using the Pearson feature selection method.These solutions are the main contributions of this paper.

Based on the metric compensation technique,in this paper we propose a new method called pe-UpMeCom,that is,metric compensation based on transfer learning (Li et al.,2020b) and Pearson feature selection (Wang et al.,2010),to construct an efficient model for CPDP.peUpMeCom consists of two main components:feature selection(Saidi et al.,2019) and bidirectional metric compensation based on transfer learning (Rahman and Devanbu,2013).In the model training phase,redundant features that are not related to the defect category are filtered out according to the Pearson coefficients (Yu et al.,2015;Shippey et al.,2019).The distribution difference of the features between the source project and target project is then reduced using a combination of transfer component analysis (TCA) and bidirectional metric compensation(Madeyski and Jureczko,2015;Chen JY et al.,2019).The main contributions of this paper are as follows:

1.We propose a CPDP method (peUpMeCom)that combines Pearson feature selection and bidirectional metric compensation,which improves prediction accuracy of the model.

2.We introduce the Pearson feature selection method to filter redundant features that are unrelated to the defect category to improve the training efficiency of the model.

3.We use a combination of TCA and bidirectional metric compensation to reduce variability in the feature distribution of the source project and target project,thus improving the similarity between projects.

2 Related works

In recent years,many studies have been conducted in the field of software defect prediction(Chen JY et al.,2020).Due to limitations such as insufficient historical data,CPDP is also emerging as a popular research direction(Guo et al.,2018).CPDP trains prediction models using enough training data from existing source projects and then applies the model to the target project to obtain prediction results.Existing research focuses mainly on the following aspects of CPDP methods:reducing the differences in data distribution between the source project and target project,removing features that are irrelevant and redundant for model construction,and solving the class imbalance problem.These methods have also shown strong feasibility in experimental studies.

2.1 Reducing data distribution differences

The main study of CPDP was carried out by Briand et al.(2002),in which a prediction model was developed using linear regression.The localization of defect-prone classes was accurately predicted and such a model outperforms most irregular models in terms of class size.However,the predicted failure probabilities were not representative because of the variation between different projects.

To increase the distribution similarity of the features between the source project and target project,Nam et al.(2013) proposed a combination of normalization and TCA and then proposed the TCA+method.This method first selects an optimal normalization method according to the target data.Then,TCA is applied to determine the mapping of the training and target data to the homogeneous metric space.The method can automatically select the best data normalization method after analyzing the characteristics of the source project and target project with the assistance of a set of pre-learned rules,so that the source project and target project have similar feature distributions.

In addition,Watanabe et al.(2008) proposed a metric compensation method.This method attempts to effectively improve the similarity between the source project and target project using the target set to adjust the data distribution of the source set(Li et al.,2020a).They analyzed the effect of CPDP between a Java project and C++project.The metric compensation method was found to be effective in improving the accuracy and recall of CPDP,with a large increase in recall.Another improvement of the method was proposed;Herbold et al.(2018)’s idea was that it is better to let the training data adapt to the data distribution of the target set than to let the target set adapt to the data distribution of the source set during training.However,these two methods adjust only the data distribution in one direction,which lacks some flexibility.In addition,for projects with a large volume of data,these approaches consume too much time for model training,which reduces the model’s prediction efficiency.

2.2 Deletion of redundant features

It has been shown that too many irrelevant and redundant features in a dataset can lead to degradation in the performance of the defect prediction model.He et al.(2015) proposed a solution from the viewpoint of feature selection that first analyzed different datasets and selected different feature subsets.Finally,the correlation between the features in the subsets was further analyzed,redundant features were removed,and minimized feature subsets were constructed in each dataset.

Similarly,with the transformation of the above idea,Amasaki et al.(2015) started with the target project and tried to remove irrelevant feature instances.Because the instances in the target projects were not labeled,they used unsupervised learning methods to identify these irrelevant feature instances to improve the prediction accuracy of the model.

2.3 Resolving class imbalances

Class imbalance problems are common in software defect datasets (Siers and Islam,2015).There are more classes without defects than classes with defects,which results in low prediction accuracy of prediction models for defective classes.The currently available class imbalance methods(Peng et al.,2019)can be briefly divided into two categories:(1) costsensitive learning methods (McBride et al.,2019),a type of method that achieves class balance by setting different costs for different types of prediction errors;(2) sampling methods,including the random under sampling,random oversampling,and synthetic minority oversampling technique (SMOTE) methods(Purnami and Trapsilasiwi,2017).Fukushima et al.(2014) and Kamei et al.(2016) reduced the number of majority class instances in the training set using the random under sampling method,so the number of majority class instances was consistent with the number of minority class instances to balance the defective classes.However,this method discards a large amount of defect-free data,which may cause deviation in the model prediction results.To solve this problem,Ryu et al.(2014,2016) proposed the value-cognitive boosting with support vector machine(VCB-SVM)method.The method first calculates the similarity weight of each instance in the source project based on the dataset of the target project,and then constructs the defect prediction model using a boosting method based on an SVM.

After a comprehensive and holistic analysis of the advantages and disadvantages of the above proposed methods,we propose our method,i.e.,peUp-MeCom.Compared with traditional methods mentioned above,our method achieves better results,using metric compensation based on feature selection and transfer learning to resolve the differences between the source project and target project.

3 The proposed method

In this section,we propose a metric compensation method called peUpMeCom.peUpMeCom consists of two main phases:(1) Features are filtered according to Pearson coefficients.Those features with a strong correlation with the defect category are selected,whereas redundant features with low or no correlation are removed.(2) Upgrade metric compensation (upMeCom) is used to reduce variance distribution of the source project and target project.upMeCom is a bidirectional metric compensation method based on transfer learning.

Fig.1 shows the overall process of defect prediction of the proposed peUpMeCom method.As shown in Fig.1,the first step in building the model is to collect feature instances from the source project.Because the datasets used in the experimental application are abstract numerical representations of projects,which include defect prediction metrics such as complexity metrics,module size metrics,program length,and the number of conditional statements,the values of these collected initial metrics are represented as initial feature instances in the study.

Fig.1 Overall process of the proposed method for building a model

After the initial features of the source project have been collected,all instances of the feature set are traversed.The correlation coefficient is calculated between each feature and the defect category;then,feature correlation analysis is performed and feature redundancy is determined based on the magnitude of the correlation coefficient between the feature and the category.Too few features cannot fully represent the defective features of a project,and too many features can cause inefficiencies in model training.Before testing the test set,we select some data from each dataset to verify the appropriate number of features.Based on the Pearson correlation coeffi-cient array,we record the area under the receiver operating characteristic curve(AUC)of the model with 10,12,15,18,and 20 features.The experimental results are shown in Table 1,where we can see that the AUC for the test set peaks when the model includes 18 or more features.Therefore,the threshold value for the number of features is set to 18;that is,after obtaining the array of correlation coefficients in descending order,the features corresponding to the top 18 coefficients are selected to form the feature subset.Then,a bidirectional metric compensation method based on the transfer learning technique is applied to the projects.Initially,the project data is mapped to a latent feature space using TCA.Then,based on the low-dimensional features obtained from the prior process,the metric compensation technique is used to enhance the similarity between the source and target data distributions.

Table 1 AUC values for different number of features

The algorithmic flow of the peUpMeCom method is shown in Algorithm 1,in which the feature selection phase is described at steps 1 and 2,the data transformation phase is described from steps 3 to 12,and the prediction phase ends from steps 13 to 15.First,the correlation coefficient between each feature and the defect category is calculated by Pearson correlation analysis,and then the final subset of features for training is selected based on the coeffi-cient arrayR.After that,TCA is used to select the common characteristics of the source project and target project and map them to latent spaceφ.Then,the instance data in the transformed source project is traversed using a for loop.For each metric,a weight is assigned to the original metric value using the metric compensation method,so the compensated data value becomes more similar to the learned data,thus enhancing the resemblance between projects.The process will be described in detail next.

For selection of machine learning classifiers,we use a decision tree classification model.Currently,there are several machine learning classification methods,and different classification techniques are applicable to different domains.The commonly used classifiers include naive Bayes (Habibi et al.,2018),SVM (Shuai et al.,2013),linear discriminant analysis(Lv,2019),and decision tree(Marian et al.,2016).According to an experimental comparison of the defect prediction performances of these four classifiers based on six datasets,the decision tree classifier outperforms the other three classifiers in most cases,both in terms of PofB20 and F1-score metrics(Yang et al.,2017).In light of experimental experience,we choose the decision tree as the classifier for peUpMeCom in this study.In addition,the classification criterion for the defect category is determined by matching the features contained in the data to be tested with the path of the feature nodes of the constructed decision tree.The information represented by the leaf nodes in the matching path is the defect category information of the data to be tested.For entity data with a given unknown label,the prediction information of the entity category is obtained by traversing from the root node to the leaf nodes.

3.1 Pearson-based feature selection

Defect prediction datasets suffer commonly from the problem of dimensional catastrophe;the redundant and irrelevant features contained in the dataset can lead to decreased performance,excessive complexity,and extensive defect prediction model training time.Because feature selection can effectively alleviate this problem,it is necessary to select features that have a significant impact on the model before training it.There are two main types of feature selection methods:wrapper method and filter method (Thejas et al.,2021).The wrapper method treats the selection of subsets as a search and optimization process.By constantly comparing the combined effect of various feature subsets,the optimal feature subset is finally selected,but this method is prone to overfitting.

The filter method filters features based on weight order.For the defect prediction domain knowledge studied in this study,an initial indicator system needs to be established for relevance screening before predicting the software module,and the completed filtered indicator system is put into a machine learning classifier for training to produce results.The main relevance-based feature selection methods are Pearson coefficient,Spearman coeffi-cient,and Kendall coefficient,which were described in detail in Grimm and Nesselroade (2018).Correlation coefficients are generally used to determine whether two variables are correlated and how closely they are related to each other.The Spearman and Kendall coefficients are generally appropriate for variables of constant order.The Pearson coefficient is generally used to analyze the degree of correlation between characteristics and response variables.The module defect category is a constant-ratio variable that exhibits non-linear trend fluctuations,and there is some correlation between software module features and defect category predictions (i.e.,there is variation in the extent to which software module features contribute,to defect category predictions).Cai et al.(2020) used Pearson coefficient to measure the correlation between module measures and module category linear correlation to filter out the impact.Therefore,in this study,Pearson correlation coefficients are used to filter the features;this method selects features that are strongly correlated with the defect class to form a subset of features for training the model.The main process of Pearson feature selection is shown in Fig.2.A Pearson correlation coefficient is a linear correlation coefficient,and is used to reflect the degree of linear correlation between two variables,with the correlation coeffi-cientrranging from-1 to 1.The larger the absolute value ofr,the stronger the correlation.Given a project datasetS,the number of samples and the feature dimensions are denoted bynandm,respectively.Thejthfeature of theithsample is denoted as.The class label of this sample is represented asyi,and the correlation coefficient of theithfeature with the defect class is calculated using Eq.(1),which is denoted by the correlation coefficientr:

Fig.2 Overall process of Pearson feature selection

wheresirepresents the independent variable (e.g.,a metric) andyirepresents the dependent variable(e.g.,clean or buggy).

Through the calculation of the correlation coeffi-cients ofmfeatures and defect categories,an array of coefficients,denoted byR{r1,r2,...,rm},is constituted.Then,an ordered array of correlation coeffi-cients is obtained,denoted byR′which is sorted by the values of elements inRfrom the largest to the smallest.The subscripts of the coefficients inRrepresent the subscripts of the features in the dataset.Then,the model trained bymfeature subsets is experimented on the validation set,and the impact ofmfeatures on the prediction performance of the model is evaluated with the AUC,where the feature with the largest AUC is selected as the best feature subset for training the defect prediction model.

3.2 Metric compensation based on transfer learning

After obtaining the best subset of the features through feature selection,we combine TCA and bidirectional metric compensation techniques to further reduce the data distribution difference between the source project and target project.The overall process of metric compensation based on transfer learning is shown in Fig.3.First,we use TCA to explore the potential common factors between the source project and target project,and then project them into the learned feature space to reduce the distribution differences of the data between the source domain and target domain,while preserving the original data structure.Assume that the data matrix of the source project isS,and that the data matrix of the target project after feature selection isT,wheren1andn2denote the numbers of samples of the source project and target project respectively,denotes data instance information in the target dataset,is the sample data information in the source project,anddenotes the defect information corresponding to this sample.The purpose of TCA is to attempt to learn a transformationφthat maps the original data from the source domain and target domain into a space where the differences between the domains are small and the variance is large.Assuming that transformationφ:Pm →Pnmaps the originalm-dimensional feature vector into ann-dimensional subspace,the objective of TCA can be expressed as

Fig.3 Overall process of metric compensation based on transfer learning

where Dist(φ(S),φ(T)) is measured by the maximum mean difference (MMD),P(φ) is a regularization term to prevent overfitting,andλ ≥0 is a tradeoffparameter to control the effects of regularization.MMD is calculated as follows:

Transformation expressions can be expressed asφ(x)xΩ,whereΩ ∈Pm×nis a matrix that mapsm-dimensional feature vectors ton-dimensional feature vectors.After learning the transformation using TCA,the transformed data can be represented byφ(S)SΩandφ(T)T Ω.

In the space mapped by TCA,the source project data is represented asS′{s1,s2,...,sm},and the target project is represented asT′{t1,t2,...,tn}.Subsequently,we apply a bidirectional metric compensation technique to the processed source and target project data.Denoting thejthmetric in theithsample data of the source project as,we assign a weight to each metric in the source project to make the transformed metric more consistent with the distribution of the target data.Thejthmetric mentioned here is not sorted according to a certain method,but refers to the default sort order of the metric elements in the downloaded project dataset.Similarly,we adopt the idea of weighting for the target project metrics to make the transformed target data fit the source project data after training.The weight is set to the ratio of the mean value of the specified metric of the source project to that of the target project.The two-way conversion can improve the training efficiency and accuracy of the model.The transformed source project metric is denoted byand the target project metric is denoted byThe specific formulas used in the metric compensation process are as follows:

where mean(sj) denotes the mean value of thejthmetric in all samples of the source project,and mean(tj) denotes the mean value of thejthmetric in all sample data of the target project.Finally,the processed source project data is fed into a decision tree classifier for training the model,and the trained model is then used to predict the defects of the target data.

4 Experiments

We conducted several experiments to verify that the combination of feature selection and bidirectional metric compensation based on transfer learning can improve the performance of the defect prediction model.We used the NASA metric data program(MDP),Relink,and AEEEM datasets in these experiments.Each instance in the dataset represents a module,and the granularity of the module representation is different for different datasets;for the NASA MDP dataset,the module granularity is method,while for AEEEM,the module granularity is class,and for the Relink dataset,the module granularity is file.Dataset details are presented in Table 2.

Table 2 Details of the selected datasets

4.1 Dataset description

The NASA MDP repository has been widely used for software defect prediction.We selected five projects from this repository that have the same metrics in our experiments.Each project represents a NASA software system that contains the corresponding defect marking data and various static code metrics.The static code metrics of the NASA MDP repository include size,complexity,etc.,which are closely related to software quality.Because our approach was carried out in homogeneous CPDP,we chose five datasets that share the same 21 metric structures in the NASA MDP dataset for the experiments.This dataset is large;therefore,to verify the validity of the method,for selecting the source and target project pairs for subsequent experiments,we used random matching to select 10 data pairs for experiments.

AEEEM is a collection of apache lucene (LC),equinox(EQ),eclipse JDT core(JDT),eclipse PDE UI (PDE),and Mylyn (ML) projects.Each dataset consists of 61 metrics:17 source code metrics,5 prior defect metrics,5 change entropy metrics,17 source code entropy metrics,and 17 source code churn metrics.

The Relink dataset includes 26 metrics,all of which focus on code complexity.The items analyzed were Apache HTTP Server,Safe,and Zxing,which are widely used in defect prediction.We can see that with only 56 instances in the Safe dataset,it is difficult to support the model training.We chose this dataset because it could be compared with the experimental data obtained from the TCA method.In addition,to address the problem of small sample of data instances in the Safe dataset,the SMOTE oversampling method was used to expand the sample in our experiments to mitigate the problem appropriately.

4.2 Baseline comparison

To evaluate the performance of the proposed pe-UpMeCom method,three methods,that is,meCom(the original metric compensation)(Watanabe et al.,2008),meCom17 (the improved metric compensation) (Herbold et al.,2018),and the TCA method(Nam et al.,2013),were used as the baselines.

1.meCom (original metric compensation)(Watanabe et al.,2008)

meCom is an effective CPDP method that improves the homogeneity between the training data and target data by adjusting the target data to ensure that it fits the distribution of the source data.Because the averages of each metric value are different between different projects,the authors thought that these values needed to be normalized.They considered the following compensation method:Supposing that error-prone files of projectAare predicted using the training data of projectB,then the test data will be compensated as follows:

2.meCom17 (improved metric compensation)(Herbold et al.,2018)

meCom17 is an outstanding CPDP method that improves the homogeneity between the training data and target data by standardizing the training data instead of the target data.The authors’idea is that instead of using the target set to adapt to the data distribution of the training set,it is better to adapt to the data distribution of the target set at the time of training.The target data will be compensated as follows:

3.TCA

The main motivation for TCA is that there may be some common underlying factors between the source domain and target domain,although the characteristics of the observed domains are different.To reveal the underlying factors,the domains are projected onto a new space,which is called the potential space.In this way,the differences between domains can be reduced and the original data structure can be preserved.In other words,TCA attempts to learn a transformationφto map the original data in the source domain and target domain to a potential space,where the differences between different domains are small and the variance of the transformed data is large.Before applying TCA,we apply minmax normalization to pre-process the data which is commonly used in machine learning literature.Normalization is performed so that all features of the dataset have the same weight and it is useful for classification.

4.3 Evaluation of prediction models

In this study,AUC and F1-measure are used as the evaluation criteria.A good prediction model should have high precision and high recall,but high recall is often achieved at the cost of low precision.Consequently,we use F1-measure to evaluate the performance,which is used to measure the harmonic average of the precision and recall.F1-measure and AUC are defined as

where precision is defined as TP/(TP+FN) and recall is defined as FP/(FP+TN).Herein,TP,TN,FP,and FN are defined in Table 3.rankirepresents the ordinal number of theithsample (the probability score is ranked from the smallest to the largest,in the rank position) andMandNare the numbers of positive and negative samples respectively;rankiindicates that only the ordinal numbers of positive samples are summed.

Table 3 Defect prediction results

4.4 Experimental results

4.4.1 RQ1:How well does our approach perform?

Scheme 1To verify the effectiveness of the proposed peUpMeCom method,we compared it with three baseline methods based on the NASA MDP,Relink,and AEEEM datasets.Taking the AEEEM dataset as an example,when EQ is selected as the source project and JDT is selected as the target project,it can be expressed as EQ＞JDT,where the left side of“＞”indicates the source project and the right side of“＞”indicates the target project.In addition,to verify that the advantages of the pe-UpMeCom method are fully demonstrated,for the same set of source dataset and target dataset,we conducted 500 experiments for all methods.The average values of all experiments based on the AEEEM,Relink,and NASA MDP datasets were calculated as the final experimental results,which are shown in Tables 4,5,and 6,respectively.To statistically analyze the detailed prediction results of the dataset,we also used a statistical test,that is,the Wilcoxon rank sum test.The confidence level in the experiments was selected as 95%.The specific results in terms of AUC and F1-measure of the statistical tests are reported in Tables 7 and 8,respectively.In addition,we drew the experimental result of each experiment in box plots to analyze the stability of the proposed model.

Finding 1Tables 4-6 show the experimental results of meCom,meCom17,TCA,and peUpMe-Com based on the selected datasets.These tables show that in most cases,our method presents a clear advantage based on the experimental datasets,in terms of both AUC and F1-measure,and outperforms the meCom,meCom17,and TCA methods.Taking the AUC metric as an example,for EQ＞JDT based on the AEEEM dataset,peUp-MeCom improves the AUC value by 0.023 compared to meCom,by 0.025 compared to meCom17,and by 0.114 compared to TCA.For Apache＞Safe based on the Relink dataset,there is an improvement of 0.131 over meCom,0.125 over meCom17,and 0.391 over TCA.For the NASA MDP dataset with CM1＞KC2,our method represents an improvement of 0.041 compared to meCom,0.035 compared to meCom17,and 0.178 compared to TCA.These results show that our method constitutes a better prediction model for defect prediction when dealing with differences in data distribution across different domains compared to the baseline methods.This is attributed to the feature selection and TCA dimensionality reduction techniques.Pearson feature selection is used to eliminate features that are not relevant to the defect category prediction,while the metric compensation technique based on transfer learning ensures the similarity of the data distribution between the source project and target project in a low-dimensional space,which leads to an increase in the prediction accuracy of the model.

Table 4 Comparison results among different methods based on the AEEEM dataset

Table 5 Comparison results among different methods based on the Relink dataset

Table 6 Comparison results among different methods based on the NASA MDP dataset

Table 7 Statistical test results between meCom and peUpMeCom in terms of AUC

Table 8 Statistical test results between meCom and peUpMeCom in terms of F1-measure

From the perspective of statistical test results,we can see from Tables 7 and 8 that all thep-values are less than 0.05 and that all the effect-size values are more than 0.5,confirming the significant difference between our method and the baseline method.Figs.4,5,and 6 show the prediction performances in terms of AUC and F1-measure metrics of different methods based on the NASA MDP,AEEEM,and Relink datasets,respectively.Most of the predicted values of the peUpMeCom method are higher than those of the baseline methods,and there are almost no outliers in the experimental data;most of the data resides in the box of the box-line plot,which implies high stability of the proposed method for model prediction performance.

4.4.2 RQ2:How does Pearson feature selection affect the prediction performance of the model?

Scheme 2To evaluate the impact of the Pearson feature selection method on the model prediction performance,we compared a model that was trained using full metrics with a model that was trained using the optimal subset of features selected using the feature selection method,and conducted experiments based on the NASA MDP,Relink,and AEEEM datasets.The specific results of the prediction examples based on the AEEEM,NASA MDP,and Relink datasets are shown in Tables 9,10,and 11,respectively.For all projects,to make the experimental data more convincing,experiments using the same dataset maintained the same experimental environment.In addition,we plotted the results for each set of experiments as box plots to verify the stability of the method for model prediction.

Fig.4 Performance plots of different techniques based on the NASA MDP dataset:(a) AUC;(b) F1-measure

Fig.5 Performance plots of different techniques based on the AEEEM dataset:(a) AUC;(b) F1-measure

Fig.6 Performance plots of different techniques based on the Relink dataset:(a) AUC;(b) F1-measure

Finding 2Figs.7,8,and 9 show the prediction results of the feature selection method based on the NASA MDP,AEEEM,and Relink datasets respectively.Tables 9,10,and 11 show the AUC and F1-measure values that are specific to the predicted cases,based on the AEEEM,NASA MDP,and Relink datasets respectively,with larger F1-measure and AUC values in bold in the tables.For the NASA MDP,Relink,and AEEEM datasets,in most cases(not generalized to 100%cases),the experimental results after feature selection are better than the original results.Taking EQ=>JDT in Table 9 for example,the original AUC value without feature selection is 0.639,and the AUC value after using the Pearsonfeature selection method is 0.686;the experimental result is 7.35%better than the original one,and similarly,the F1-measure is increased by 4.75%.It is obvious that the Pearson feature selection method has improved the prediction efficiency of the model.The main reason is that deleting redundant features that are unrelated to the defect category effectively reduces the model training time and weakens the interference of redundant features on the model prediction,thus improving the prediction performance and accuracy of the model.

Table 9 Results of the original data and Pearson feature selection based on the AEEEM dataset

Table 10 Results of the original data and Pearson feature selection based on the NASA MDP dataset

Table 11 Results of the original data and Pearson feature selection based on the Relink dataset

4.4.3 RQ3:Does the transfer learning technique improve metric compensation?

Scheme 3To validate the contribution of transfer learning to the bidirectional metric compensation technique in model prediction performance,we conducted experiments based on these three datasets using different metric compensation methods.To demonstrate the effectiveness of the experiments,two baseline methods were chosen for comparison:meCom and meCom17.One project was then selected as the target project,and the remaining projects were used as the source datasets for training the model.As an example,when CM1 is used as the test set,JM1,KC1,KC2,and PC1 are used together as the training set for model training.Five hundred experiments were conducted on the same project of experimental data,and the average of the experimental values was taken as the final result for this target dataset;the results are shown in Table 12.In addition,to confirm the stability of the method,we plotted the prediction results of each set of experiments as box plots,and the comparison of the experimental results of the same set under different methods is shown in Fig.10.

Fig.7 Performance plots of feature selection based on the NASA MDP dataset:(a) AUC;(b) F1-measure

Fig.8 Performance plots of feature selection based on the AEEEM dataset:(a) AUC;(b) F1-measure

Fig.9 Performance plots of feature selection based on the Relink dataset:(a) AUC;(b) F1-measure

Finding 3Fig.10 shows the AUC values for CPDP based on the three datasets for the method combining transfer learning and bidirectional metric compensation compared to the traditional metric compensation methods.Table 12 shows the average multiple-to-one CPDP results for several project prediction cases.From Table 12 and Fig.10,we can see that the model combining transfer learning and bidirectional metric compensation achieves a higher AUC than the baselines in most cases.Taking the KC1 target set as an example((CM1+JM1+KC2+PC1)＞KC1),upMeCom can improve the average AUC value by 0.05 (0.628-0.578) compared to meCom and by 0.085 (0.628-0.543) compared to meCom17.This indicates that the combination of transfer learning and metric compensation can effectively improve the model prediction performance.One reason for this performance improvement is that the fusion of the TCA technique allows the training and testing samples to retain the maximum attribute features of the data in the low-dimensional feature space.Another reason is that traditional metric compensation methods tend to transform the data in one direction,whereas our bidirectional metric compensation combines the idea of bidirectional metric value conversion with a further reduction of the variability of the source and target project feature distributions.In addition,it can be seen from the box-line plots in Fig.10 thatthe experimental data derived from our method is basically outlier-free,which indicates that the model constructed by this method is relatively stable.

Table 12 AUC obtained from the upMeCom and other metric compensation methods

Fig.10 AUC values under different metric compensation methods

5 Discussions

5.1 Insights learned from our empirical study

Our empirical study reveals some important facts that provide new insights to further advance CPDP research.Our comprehensive experimental results show that the metric compensation technique combining feature selection and transfer learning can improve CPDP performance.In particular,this technique can shorten model training time and effectively improve prediction accuracy of the model.Such findings provide important insight and guidance for future research:when predicting defects,if the data dimensionality is too large,the ideas of feature dimensionality reduction and enhanced data distribution similarity can be fused to provide new research directions for the development of CPDP techniques.

5.2 Threats to validity

Two potential threats to the effectiveness of the experiment are presented here.Threats to internal validity come mainly from experimental deviations due to parameter settings and the replication of the comparison methods.Most of the compared methods in the experiment are not available with their source codes,so some specific information like parameter settings is unknown and there is probably some variation in the baselines.Although we experimented exactly as mentioned in the literature,our reproducible results do not exactly match the original outcome.We overcame this threat by adjusting the parameter settings to choose the best combination of parameters.

The main external threat to validity is dataset bias.In this study,we selected 13 datasets from NASA MDP,Relink,and AEEEM for experimental validation of the proposed defect prediction model.However,it is still unknown how well the models that are constructed using this method will perform when applied to other applications.

6 Conclusions

To address the inefficiency of defect prediction models caused by the large variability of data distribution between the source project and target project,we propose a metric compensation method based on transfer learning and feature selection.First,peUp-MeCom uses the Pearson feature selection method to filter out redundant features that are not relevant to the defect category prediction.The correlation coefficient between each feature and the defect category is calculated,and the irrelevant features are removed based on the correlation coeffi-cient ranking to reduce training time of the model.After that,TCA is introduced to map the data features to a low-dimensional space,while preserving the original data characteristics.Then,using the bidirectional metric compensation technique,we enhance the feature distribution similarity between the source project and target project,thereby improving the performance of CPDP.Extensive experimental results verify that our peUpMeCom method effectively improves the prediction accuracy in both AUC value and F1-measure,and outperforms the traditional metric compensation techniques.

Contributors

Jinfu CHEN and Saihua CAI designed the research.Xiaoli WANG,Saihua CAI,and Jiaping XU processed the data.Jinfu CHEN,Xiaoli WANG,and Saihua CAI drafted the paper.Xiaoli WANG,Jiaping XU,Jingyi CHEN,and Haibo CHEN finished the experiments.Jingyi CHEN and Haibo CHEN helped organize the paper.Jinfu CHEN,Xiaoli WANG,and Saihua CAI revised and finalized the paper.

Compliance with ethics guidelines

Jinfu CHEN,Xiaoli WANG,Saihua CAI,Jiaping XU,Jingyi CHEN,and Haibo CHEN declare that they have no conflict of interest.