Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

Computer Scientists from Saarbrücken Search Through Large Datasets using "Good Trojans"

13.09.2010
Social networks, search engines, digital archives, and several global-scale internet companies host very large data collections.

In order to search through this data companies like Facebook, Ebay, Yahoo, and Twitter use the freely available Hadoop software - a variant of MapReduce originally proposed by Google. Several database experts, however, criticize Hadoop for being inefficient.

Computer scientists from Saarland University (Germany) are now proposing a new system coined Hadoop++. It allows users to search through very large datasets much faster than before. Hadoop++ improves over Hadoop by up to a factor of 20.

Internet companies need to process data volumes on the order of millions of Gigabytes (Petabytes) on a daily basis. In order to search through this data effectively, Google proposed the MapReduce programming model. MapReduce divides the input data into smaller chunks that are then distributed over a large network of machines and processed in parallel. The open-source counterpart of MapReduce is called Hadoop. Even though Google holds a patent on MapReduce, it granted a license to Hadoop. Therefore, Hadoop may still be used by companies free of charge. However, "Database experts who are fluent in SQL consider MapReduce a major step backwards towards the database stone age", explains Jens Dittrich, Professor of Information Systems at Saarland University. He adds, "MapReduce disregards considerable wisdom from database research. As a consequence MapReduce is often slow and inefficient."

Despite this criticism, over the past years Hadoop has gained considerable attention from both industry and academia. Hadoop is very popular among programmers throughout the world. Professor Dittrich explains why, "The reason is its ease of use: the user neither has to learn a complex database language nor a data model. Furthermore, Hadoop is relatively easy to administrate. In summary, Hadoop allows even database-illiterate people to search through billions of records on very large computer clusters."

However, this comes at a price: "When compared to modern relational database management systems, Hadoop is just too slow." Therefore the researcher and his team at Saarland University have developed a new system, coined Hadoop++. It aims to eliminate the performance deficiencies encountered in Hadoop. The creativity of the new approach lies in how the problem is tackled: Hadoop++ works similarly to a "trojan", i.e. a computer virus which infects a computer system clandestinely and may then cause considerable harm. In contrast to these bad trojans, Hadoop++ injects hidden code into a system in order to "heal" it, ie. dynamically accelerating the underlying Hadoop. Professor Dittrich emphasizes, "One could say that Hadoop++ is a good trojan".

The Saarbrücken approach has the considerable advantage that Hadoop's tested code base does not have to be modified and retested. Thus, Hadoop++ avoids complex changes to a working system and unforeseen consequences. Hadoop++ will be presented at this year's International Conference on Very Large Databases (VLDB) - one of the world's most prestigious database conferences to be held in Singapore from September 13-17.

Background

There have been heated discussions about the pros and cons of MapReduce/Hadoop when compared to traditional database management systems. This discussion was led by database professors in the US. Please refer to the links below for details. Recently there have been attempts to improve the runtime efficiency of MapReduce. However, as Professor Dittrich explains, "those attempts could not really marry MapReduce with database technology."

Press Pictures: www.uni-saarland.de/pressefotos

For questions, contact:
Jens Dittrich
Professor of Information Systems
Saarland University
Tel. (+49) 681 302 70141
Weitere Informationen:
http://infosys.cs.uni-saarland.de/hadoop++.php
http://hadoop.apache.org/
http://www.mapreduce.org/
http://www.vldb2010.org

Friederike Meyer zu Tittingdorf | idw
Further information:
http://www.uni-saarland.de

More articles from Information Technology:

nachricht Satellite data for agriculture
28.07.2017 | Julius-Maximilians-Universität Würzburg

nachricht Magnetic Quantum Objects in a "Nano Egg-Box"
25.07.2017 | Universität Wien

All articles from Information Technology >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: Abrupt motion sharpens x-ray pulses

Spectrally narrow x-ray pulses may be “sharpened” by purely mechanical means. This sounds surprisingly, but a team of theoretical and experimental physicists developed and realized such a method. It is based on fast motions, precisely synchronized with the pulses, of a target interacting with the x-ray light. Thereby, photons are redistributed within the x-ray pulse to the desired spectral region.

A team of theoretical physicists from the MPI for Nuclear Physics (MPIK) in Heidelberg has developed a novel method to intensify the spectrally broad x-ray...

Im Focus: Physicists Design Ultrafocused Pulses

Physicists working with researcher Oriol Romero-Isart devised a new simple scheme to theoretically generate arbitrarily short and focused electromagnetic fields. This new tool could be used for precise sensing and in microscopy.

Microwaves, heat radiation, light and X-radiation are examples for electromagnetic waves. Many applications require to focus the electromagnetic fields to...

Im Focus: Carbon Nanotubes Turn Electrical Current into Light-emitting Quasi-particles

Strong light-matter coupling in these semiconducting tubes may hold the key to electrically pumped lasers

Light-matter quasi-particles can be generated electrically in semiconducting carbon nanotubes. Material scientists and physicists from Heidelberg University...

Im Focus: Flexible proximity sensor creates smart surfaces

Fraunhofer IPA has developed a proximity sensor made from silicone and carbon nanotubes (CNT) which detects objects and determines their position. The materials and printing process used mean that the sensor is extremely flexible, economical and can be used for large surfaces. Industry and research partners can use and further develop this innovation straight away.

At first glance, the proximity sensor appears to be nothing special: a thin, elastic layer of silicone onto which black square surfaces are printed, but these...

Im Focus: 3-D scanning with water

3-D shape acquisition using water displacement as the shape sensor for the reconstruction of complex objects

A global team of computer scientists and engineers have developed an innovative technique that more completely reconstructs challenging 3D objects. An ancient...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

Event News

Clash of Realities 2017: Registration now open. International Conference at TH Köln

26.07.2017 | Event News

Closing the Sustainability Circle: Protection of Food with Biobased Materials

21.07.2017 | Event News

»We are bringing Additive Manufacturing to SMEs«

19.07.2017 | Event News

 
Latest News

New 3-D imaging reveals how human cell nucleus organizes DNA and chromatin of its genome

28.07.2017 | Health and Medicine

Heavy metals in water meet their match

28.07.2017 | Power and Electrical Engineering

Oestrogen regulates pathological changes of bones via bone lining cells

28.07.2017 | Life Sciences

VideoLinks
B2B-VideoLinks
More VideoLinks >>>