Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

25 years of conventional evaluation of data analysis proves worthless in practice

05.09.2008
So-called ‘intelligent’ computer-based methods for classifying patient samples, for example, have been evaluated with the help of two methods that have completely dominated research for 25 years.

Now Swedish researchers at Uppsala University are revealing that this methodology is worthless when it comes to practical problems. The article is published in the journal Pattern Recognition Letters.

Today there is rapidly growing interest in ‘intelligent’ computer-based methods that use various classes of measurement signals, from different patient samples, for instance, to create a model for classifying new observations. This type of method is the basis for many technical applications, such as recognition of human speech, images, and fingerprints, and is now also beginning to attract new fields such as health care.

“Especially in applications in which faulty classification decisions can lead to catastrophic consequences, such as choosing the wrong form of therapy for treating cancer, it is extremely important to be able to make a reliable estimate of the performance of the classification model,” explains Mats Gustafsson, Professor of signal processing and medical bioinformatics at Uppsala University, who co-directed the new study together with Associate Professor Anders Isaksson.

To evaluate the performance of a classification model, one normally tests it on a number of trial examples that have never been involved in the design of the model. Unfortunately there are seldom tens of thousands of test examples available for this type of evaluation. In biomedicine, for instance, it is often expensive and difficult to collect the patient samples needed, especially if one wishes to analyze a rare disease. To solve this problem, many different methods have been proposed. Since the 1980s two methods have completely dominated research, namely, cross validation and resampling/bootstrapping.

“This has entailed that the performance assessment of virtually all new methods and applications reported in the scientific literature in the last 25 years has been carried out using one of these two methods,” says Mats Gustafsson.

In the new study, the Uppsala researchers use both theory and convincing computer simulations to show that this methodology is worthless in practice when the total number of examples is small in relation to the natural variation that exists among different observations. What is considered a small number depends in turn on what problem is being studied-­in other words, it is impossible to determine whether the number of examples is sufficient.

“Our main conclusion is that this methodology cannot be depended on at all, and that it therefore needs to be immediately replaces by Bayesian methods, for example, which can deliver reliable measures of the uncertainty that exists. Only then will multivariate analyses be in any position to be adopted in such critical applications as health care,” says Mats Gustafsson.

Mats Gustafsson | alfa
Further information:
http://www.uu.se

More articles from Information Technology:

nachricht Researchers create new technique for manipulating polarization of terahertz radiation
20.07.2017 | Brown University

nachricht Holograms taken to new dimension
19.07.2017 | University of Utah

All articles from Information Technology >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: Manipulating Electron Spins Without Loss of Information

Physicists have developed a new technique that uses electrical voltages to control the electron spin on a chip. The newly-developed method provides protection from spin decay, meaning that the contained information can be maintained and transmitted over comparatively large distances, as has been demonstrated by a team from the University of Basel’s Department of Physics and the Swiss Nanoscience Institute. The results have been published in Physical Review X.

For several years, researchers have been trying to use the spin of an electron to store and transmit information. The spin of each electron is always coupled...

Im Focus: The proton precisely weighted

What is the mass of a proton? Scientists from Germany and Japan successfully did an important step towards the most exact knowledge of this fundamental constant. By means of precision measurements on a single proton, they could improve the precision by a factor of three and also correct the existing value.

To determine the mass of a single proton still more accurate – a group of physicists led by Klaus Blaum and Sven Sturm of the Max Planck Institute for Nuclear...

Im Focus: On the way to a biological alternative

A bacterial enzyme enables reactions that open up alternatives to key industrial chemical processes

The research team of Prof. Dr. Oliver Einsle at the University of Freiburg's Institute of Biochemistry has long been exploring the functioning of nitrogenase....

Im Focus: The 1 trillion tonne iceberg

Larsen C Ice Shelf rift finally breaks through

A one trillion tonne iceberg - one of the biggest ever recorded -- has calved away from the Larsen C Ice Shelf in Antarctica, after a rift in the ice,...

Im Focus: Laser-cooled ions contribute to better understanding of friction

Physics supports biology: Researchers from PTB have developed a model system to investigate friction phenomena with atomic precision

Friction: what you want from car brakes, otherwise rather a nuisance. In any case, it is useful to know as precisely as possible how friction phenomena arise –...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

Event News

»We are bringing Additive Manufacturing to SMEs«

19.07.2017 | Event News

The technology with a feel for feelings

12.07.2017 | Event News

Leipzig HTP-Forum discusses "hydrothermal processes" as a key technology for a biobased economy

12.07.2017 | Event News

 
Latest News

Researchers create new technique for manipulating polarization of terahertz radiation

20.07.2017 | Information Technology

High-tech sensing illuminates concrete stress testing

20.07.2017 | Materials Sciences

First direct observation and measurement of ultra-fast moving vortices in superconductors

20.07.2017 | Physics and Astronomy

VideoLinks
B2B-VideoLinks
More VideoLinks >>>