Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

Managing the data on the worldwide Grid

07.01.2004


Did you just load that interesting new software on your PC? Then wondered where all your storage space has gone, while at the same time your PC seems to have ground to a halt? Think of the same predicament but magnified, not thousands but millions of times. Now you begin to see the problems confronting the world’s research community in cutting-edge research projects. And in particular, the challenges facing the DATAGRID project.



Most challenging processing applications on Earth

The project partners in DATAGRID deal with what are probably the most demanding data processing applications on Earth. Some 21 partners, among the major research institutions in Europe, are confronted with increasingly data-hungry applications of the type that even super-computers cannot handle.


The EC launched already in 2000 the Grid series of projects as a solution. A means of interconnecting all the national research networks within and without Europe, in order to share data processing and storage capacity for these most challenging data processing applications of all.

DATAGRID is part of that solution. The project focuses on the middleware that is essential to handling the enormous amounts of data involved. The DATAGRID middleware ensures that the right processing resources are used, wherever they may be on the Grid, in an efficient and reliable way. Its task is to organise and integrate the disparate computational resources of the Grid into a coherent whole.

The partners in DATAGRID include world-leading European research institutions such as CERN, CNRS in France, INFN in Italy and PPARC in the UK. They have been joined in their endeavours by industrial giants such as IBM and other European IT companies such as Datamat in Italy and CSSI in France.

Quasi-production processing capability

Although project completion is due not until the end of March 2004, the partners have already largely achieved their objectives. This is no mean feat according to the DATAGRID project leader Fabrizio Gagliardi from the European Organisation for Nuclear Research. "Our most important single achievement is that we have successfully put together a real testbed, on which we can demonstrate data processing applications at near-production quality."

"We have applications in three domains where we show a quasi-production capability - particle physics, bio-information and earth observation," he continues. "All require processing and analysis of very large amounts of data. What do I mean by this? I mean making available fifteen terabytes of online data storage, working, at any time and offering the processing capability of over two thousand CPUs."

Particle physics provides a good illustration of the data needs of such applications. To explore fundamental particles and the forces between them, high-energy physics applications are expected soon to be producing as much as 10 Petabytes of data per year. Dozens of universities around the world will need to analyse this data.

In the bio-sciences, 10 years ago biologists were happy if they could simulate a single small molecule on a computer. Now they expect to simulate thousands of molecular drug candidates to see how they interact with specific proteins. And unlocking the secrets of the human genome would be impossible without the computerised analysis of massive amounts of data, including the sequence of the three billion chemical units that comprise human DNA, the genetic blueprint for our species.

In space research, scientists keep track of the level of atmospheric ozone with satellite observations. For this task alone they download, from space to ground, about 100 Gigabytes of raw images per day. And in disaster response scenarios, governments and international organisations such as the UN and the World Bank will be greatly aided by the Grid’s potential for sharing archives and coordinating responses to the devastation caused by floods, fires and earthquakes.

A worldwide data capability

Like the Grid itself, the DATAGRID testbed is not limited to Europe alone. Its capabilities have been successfully tested around the world, at some 40 sites from Europe to North America, to Russia, to the Far East and to the Antipodes. Its sister project DATATAG has broken the world record for data transmission capacity, reaching the hitherto unheard of speed of 5.44 Gigabits per second.

DATAGRID middleware is based on an Open Source software policy. Once the project is completed the source code will be made available for anyone to download, to develop for themselves and share their results with the world software community.

As part of their software policy the partners have also invested considerable resources in education. Training in the use of the software, practical tutorials in the use of the testbed and road-shows have been given in locations as diverse as Pakistan, Taiwan and Poland, as well as within the EU. So successful was the Polish road-show, for example, that the project has been included into the curriculum of the University of Krakow computer sciences faculty.

Real challenge still to come

Challenging as its tasks may be now, the real test for the Grid is still to come. Starting in 2007, the new particle accelerator at CERN, the Large Hadron Collider (LHC), will come on stream. The LHC will be the largest scientific instrument in the world and will require all the abilities of the Grid to support the huge amounts of data it will generate.

Scientists around the world will be clamouring for access to the data from the LHC. The 10 Petabytes of data per year it is expected to generate is equivalent to more than 1,000 times the amount of information printed in book form every year around the world, and nearly 10 per cent of all information that humans produce on the planet each year.

The Grid network is the only resource capable of processing this immense quantity of information. And the partners in DATAGRID are already involved in forward planning for a successor project EGEE (Enabling Grids for E-science in Europe, www.eu-egee.org), to develop the 24/7 capabilities and call-centre backup that will be necessary to support it.

Tara Morris | alfa
Further information:
http://istresults.cordis.lu/

More articles from Information Technology:

nachricht The Flexible Grid Involves its Users
27.09.2016 | Fraunhofer-Institut für Angewandte Informationstechnik FIT

nachricht Optical fiber transmits one terabit per second – Novel modulation approach
16.09.2016 | Technische Universität München

All articles from Information Technology >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: First-Ever 3D Printed Excavator Project Advances Large-Scale Additive Manufacturing R&D

Heavy construction machinery is the focus of Oak Ridge National Laboratory’s latest advance in additive manufacturing research. With industry partners and university students, ORNL researchers are designing and producing the world’s first 3D printed excavator, a prototype that will leverage large-scale AM technologies and explore the feasibility of printing with metal alloys.

Increasing the size and speed of metal-based 3D printing techniques, using low-cost alloys like steel and aluminum, could create new industrial applications...

Im Focus: New welding process joins dissimilar sheets better

Friction stir welding is a still-young and thus often unfamiliar pressure welding process for joining flat components and semi-finished components made of light metals.
Scientists at the University of Stuttgart have now developed two new process variants that will considerably expand the areas of application for friction stir welding.
Technologie-Lizenz-Büro (TLB) GmbH supports the University of Stuttgart in patenting and marketing its innovations.

Friction stir welding is a still-young and thus often unfamiliar pressure welding process for joining flat components and semi-finished components made of...

Im Focus: First quantum photonic circuit with electrically driven light source

Optical quantum computers can revolutionize computer technology. A team of researchers led by scientists from Münster University and KIT now succeeded in putting a quantum optical experimental set-up onto a chip. In doing so, they have met one of the requirements for making it possible to use photonic circuits for optical quantum computers.

Optical quantum computers are what people are pinning their hopes on for tomorrow’s computer technology – whether for tap-proof data encryption, ultrafast...

Im Focus: OLED microdisplays in data glasses for improved human-machine interaction

The Fraunhofer Institute for Organic Electronics, Electron Beam and Plasma Technology FEP has been developing various applications for OLED microdisplays based on organic semiconductors. By integrating the capabilities of an image sensor directly into the microdisplay, eye movements can be recorded by the smart glasses and utilized for guidance and control functions, as one example. The new design will be debuted at Augmented World Expo Europe (AWE) in Berlin at Booth B25, October 18th – 19th.

“Augmented-reality” and “wearables” have become terms we encounter almost daily. Both can make daily life a little simpler and provide valuable assistance for...

Im Focus: Artificial Intelligence Helps in the Discovery of New Materials

With the help of artificial intelligence, chemists from the University of Basel in Switzerland have computed the characteristics of about two million crystals made up of four chemical elements. The researchers were able to identify 90 previously unknown thermodynamically stable crystals that can be regarded as new materials. They report on their findings in the scientific journal Physical Review Letters.

Elpasolite is a glassy, transparent, shiny and soft mineral with a cubic crystal structure. First discovered in El Paso County (Colorado, USA), it can also be...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

Event News

Call for Paper – Panacea Green Infrastructure?

30.09.2016 | Event News

HLF: From an experiment to an establishment

29.09.2016 | Event News

European Health Forum Gastein 2016 kicks off today

28.09.2016 | Event News

 
Latest News

First-Ever 3D Printed Excavator Project Advances Large-Scale Additive Manufacturing R&D

30.09.2016 | Materials Sciences

New Technique for Finding Weakness in Earth’s Crust

30.09.2016 | Earth Sciences

Cells migrate collectively by intermittent bursts of activity

30.09.2016 | Life Sciences

VideoLinks
B2B-VideoLinks
More VideoLinks >>>