云环境下软件错误报告自动分类算法改进
2016-05-14黄伟林劼江育娥
黄伟 林劼 江育娥



摘要:用户提交的软件错误报告随意性大、主观性强且内容少导致自动分类正确率不高,需要花费大量人工干预时间。随着互联网的快速发展用户提交的错误报告数量也不断增加,如何在海量数据下提高其自动分类的精确度越来越受到关注。通过改进词频逆文档频率(TFIDF),考虑到词条在类间和类内出现情况对文本分类的影响,提出一种基于软件错误报告数据集的改进多项式朴素贝叶斯算法,同时在Hadoop平台下使用MapReduce计算模型实现该算法的分布式版本。实验结果表明,改进的多项式朴素贝叶斯算法将F1值提高到71%,比原算法提高了27个百分点,同时在海量数据下可以通过拓展节点的方式缩短运行时间,有较好的执行效率。
关键词:多项式朴素贝叶斯;错误报告;文本自动分类;词频逆文档频率;云计算
中图分类号:TP311 文献标志码:A
Abstract:Usersubmitted bug reports are arbitrary and subjective. The accuracy of automatic classification of bug reports is not ideal. Hence it requires many human labors to intervention. With the bug reports database growing bigger and bigger, the problem of improving the accuracy of automatic classification of these reports is becoming urgent. A TFIDF (Term FrequencyInverse Document Freqency) based Naive Bayes (NB) algorithm was proposed. It not only considered the relationship of a term in different classes but also the relationship of a term inside a class. It was also implemented in distributed parallel environment of MapReduce model in Hadoop platform. The experimental results show that the proposed Naive Bayes algorithm improves the performance of F1 measument to 71%, which is 27 percentage points higher than the stateoftheart method. And it is able to deal with massive amounts of data in distributed way by addding computational node to offer shorter running time and has better effective performance.
Key words:Naive Bayes of polynomials; bug report; text automatic classification; Term FrequencyInverse Document Frequency (TFIDF); cloud computing
0 引言
随着大数据时代的到来,海量数据的处理速度越来越受到重视,传统的单机处理已经呈现出其弊端,如何在大量的数据情况下提高处理速度受到广泛的关注。Hadoop作为一个分布式的框架,其在超大数据集下的表现令人满意。开源软件的错误报告随着版本的更新收到用户越来越多的反馈,如何在短时间内将用户的反馈分门别类更快地进行修复已经成为各企业提升自我软件竞争力的重点。用户提交软件错误报告有着很大的随意性,即使事先给出类别也无法保证用户能够正确地选对,因此将错误报告进行自动分类能够节省时间并提高效率。……
