面向大数据处理的Hadoop与MongoDB整合技术研究
2016-03-17曾强缪力秦拯
曾 强 缪 力 秦 拯
(湖南大学信息科学与工程学院 湖南 长沙 410086)
面向大数据处理的Hadoop与MongoDB整合技术研究
曾强缪力秦拯
(湖南大学信息科学与工程学院湖南 长沙 410086)
摘要随着数据种类的增多和数据规模的增大,NoSQL技术与MapReduce并行处理思想越来越受到重视。MongoDB作为NoSQL数据库的典型代表,支持对海量数据进行索引和查询,但MongoDB提供的MapReduce还不能满足复杂的数据分析和计算。而Hadoop虽然提供了强大的MapReduce并行计算框架,却在实时服务方面存在较高延时。针对这种情况,综合考虑扩展性,数据本地化,I/O性能等因素,提出并实现Hadoop与MongoDB四种不同的整合方案。通过设计三种具有代表性的应用作为性能的测量基准,对不同的整合方案进行性能对比实验,得出不同应用场景下的最优整合方案。实验表明,在MongoDB与Hadoop的折衷使用过程中,若对不同的应用采用合理的方案,性能最多可以提高3倍。
关键词整合MongoDBHadoop大数据
ON MONGODB AND HADOOP INTEGRATION TECHNOLOGY FOR BIG DATA PROCESSING
Zeng QiangMiao LiQin Zheng
(College of Computer Science and Electronic Engineering,Hunan University,Changsha 410086,Hunan,China)
AbstractWith the exponential growth in data variety and data volumes, NoSQL technology and MapReduce for scalable parallel analysis have garnered a lot of attentions. MongoDB, as a typical representative of NoSQL database, supports both scalable index and flexible query for massive data, but the MapReduce provided by MongoDB cannot meet the need of complex data analysis and computation. While Hadoop offers a powerful MapReduce framework for parallel computing, it performs high latency in real-time services. In view of this, we propose and implement four different integration schemes of Hadoop and MongoDB by considering comprehensively the factors of scalability, data locality and I/O performance. The optimal integration schemes under different scenarios are derived by designing three kinds of representative applications as the measuring benchmarks of performances and by performance contrastive experiments on different integration schemes. Experiments show that in the process of trade-off use of MongoDB and Hadoop, if reasonable integration schemes are applied to different applications, the performance can be improved up to 3 times.
KeywordsIntegrationMongoDBHadoopBig data
0引言
随着互联网的发展和科技的进步,不论企业,政府,科研机构还是个人,所产生的数据量越来越大。面对种类繁多的半结构化和非结构化海量数据,显然,传统的数据分析和存储技术已力不从心。在大数据时代的背景下,海量数据的存储和分析计算技术应运而生。对于海量数据的处理,应包括三个部分:存储、计算和查询。而要同时实现存储和计算的可扩展性以及查询的高效性成为大数据处理面临的一个挑战。
现今涌现出的很多云计算和云存储技术在不同程度上满足了大数据的处理要求。……
