Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

Storage system for 'big data' dramatically speeds access to information

30.01.2014
Using multiple nodes allows the same bandwidth and performance from a storage network as far more expensive machines

As computers enter ever more areas of our daily lives, the amount of data they produce has grown enormously.

But for this "big data" to be useful it must first be analyzed, meaning it needs to be stored in such a way that it can be accessed quickly when required.

Previously, any data that needed to be accessed in a hurry would be stored in a computer's main memory, or dynamic random access memory (DRAM) — but the size of the datasets now being produced makes this impossible.

So instead, information tends to be stored on multiple hard disks on a number of machines across an Ethernet network. However, this storage architecture considerably increases the time it takes to access the information, according to Sang-Woo Jun, a graduate student in the Computer Science and Artificial Intelligence Laboratory (CSAIL) at MIT.

"Storing data over a network is slow because there is a significant additional time delay in managing data access across multiple machines in both software and hardware," Jun says. "And if the data does not fit in DRAM, you have to go to secondary storage — hard disks, possibly connected over a network — which is very slow indeed."

Now Jun, fellow CSAIL graduate student Ming Liu, and Arvind, the Charles W. and Jennifer C. Johnson Professor of Electrical Engineering and Computer Science, have developed a storage system for big-data analytics that can dramatically speed up the time it takes to access information.

The system, which will be presented in February at the International Symposium on Field-Programmable Gate Arrays in Monterey, Calif., is based on a network of flash storage devices.

Flash storage systems perform better at tasks that involve finding random pieces of information from within a large dataset than other technologies. They can typically be randomly accessed in microseconds. This compares to the data "seek time" of hard disks, which is typically four to 12 milliseconds when accessing data from unpredictable locations on demand.

Flash systems also are nonvolatile, meaning they do not lose any of the information they hold if the computer is switched off.

In the storage system, known as BlueDBM — or Blue Database Machine — each flash device is connected to a field-programmable gate array (FPGA) chip to create an individual node. The FPGAs are used not only to control the flash device, but are also capable of performing processing operations on the data itself, Jun says.

"This means we can do some processing close to where the data is [being stored], so we don't always have to move all of the data to the machine to work on it," he says.

What's more, FPGA chips can be linked together using a high-performance serial network, which has a very low latency, or time delay, meaning information from any of the nodes can be accessed within a few nanoseconds. "So if we connect all of our machines using this network, it means any node can access data from any other node with very little performance degradation, [and] it will feel as if the remote data were sitting here locally," Jun says.

Using multiple nodes allows the team to get the same bandwidth and performance from their storage network as far more expensive machines, he adds.

The team has already built a four-node prototype network. However, this was built using 5-year-old parts, and as a result is quite slow.

So they are now building a much faster 16-node prototype network, in which each node will operate at 3 gigabytes per second. The network will have a capacity of 16 to 32 terabytes.

Using the new hardware, Liu is also building a database system designed for use in big-data analytics. The system will use the FPGA chips to perform computation on the data as it is accessed by the host computer, to speed up the process of analyzing the information, Liu says.

"If we're fast enough, if we add the right number of nodes to give us enough bandwidth, we can analyze high-volume scientific data at around 30 frames per second, allowing us to answer user queries at very low latencies, making the system seem real-time," he says. "That would give us an interactive database."

As an example of the type of information the system could be used on, the team has been working with data from a simulation of the universe generated by researchers at the University of Washington. The simulation contains data on all the particles in the universe, across different points in time.

"Scientists need to query this rather enormous dataset to track which particles are interacting with which other particles, but running those kind of queries is time-consuming," Jun says. "We hope to provide a real-time interface that scientists can use to look at the information more easily."

Helen Knight | EurekAlert!
Further information:
http://www.mit.edu

More articles from Information Technology:

nachricht The app for frequent fliers and those who are radiation-conscious
15.04.2015 | Physikalisch-Technische Bundesanstalt (PTB)

nachricht Passenger-focused air conditioning
14.04.2015 | Technische Universitaet Muenchen

All articles from Information Technology >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: On the trail of a trace gas

Max Planck researcher Buhalqem Mamtimin determines how much nitrogen oxide is released into the atmosphere from agriculturally used oases.

In order to make statements about current and future air pollution, scientists use models which simulate the Earth’s atmosphere. A lot of information such as...

Im Focus: Advances in Molecular Electronics: Lights On – Molecule On

Scientists at the Helmholtz-Zentrum Dresden-Rossendorf (HZDR) and the University of Konstanz are working on storing and processing information on the level of single molecules to create the smallest possible components that will combine autonomously to form a circuit. As recently reported in the academic journal Advanced Science, the researchers can switch on the current flow through a single molecule for the first time with the help of light.

Dr. Artur Erbe, physicist at the HZDR, is convinced that in the future molecular electronics will open the door for novel and increasingly smaller – while also...

Im Focus: Pruning of Blood Vessels: Cells Can Fuse With Themselves

Cells of the vascular system of vertebrates can fuse with themselves. This process, which occurs when a blood vessel is no longer necessary and pruned, has now been described on the cellular level by Prof. Markus Affolter from the Biozentrum of the University of Basel. The findings of this study have been published in the journal “PLoS Biology”.

The vascular system is the supply network of the human organism and delivers oxygen and nutrients to the last corners of the body. So far, research on the...

Im Focus: Astronomers reveal supermassive black hole's intense magnetic field

Astronomers from Chalmers University of Technology have used the giant telescope Alma to reveal an extremely powerful magnetic field very close to a supermassive black hole in a distant galaxy

Astronomers from Chalmers University of Technology have used the giant telescope Alma to reveal an extremely powerful magnetic field very close to a...

Im Focus: A “pin ball machine” for atoms and photons

A team of physicists from MPQ, Caltech, and ICFO proposes the combination of nano-photonics with ultracold atoms for simulating quantum many-body systems and creating new states of matter.

Ultracold atoms in the so-called optical lattices, that are generated by crosswise superposition of laser beams, have been proven to be one of the most...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

Event News

HHL's Entrepreneurship Conference on FinTech

13.04.2015 | Event News

World Conference On Regenerative Medicine 2015: Registration And Abstract Submission Now Open

25.03.2015 | Event News

University presidents from all over the world meet in Hamburg

19.03.2015 | Event News

 
Latest News

Better battery imaging paves way for renewable energy future

21.04.2015 | Materials Sciences

Extending climate predictability beyond El Niño

21.04.2015 | Earth Sciences

Risk Perception: Social Exchange Can Amplify Subjective Fears

21.04.2015 | Studies and Analyses

VideoLinks
B2B-VideoLinks
More VideoLinks >>>