Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

25 years of conventional evaluation of data analysis proves worthless in practice

05.09.2008
So-called ‘intelligent’ computer-based methods for classifying patient samples, for example, have been evaluated with the help of two methods that have completely dominated research for 25 years.

Now Swedish researchers at Uppsala University are revealing that this methodology is worthless when it comes to practical problems. The article is published in the journal Pattern Recognition Letters.

Today there is rapidly growing interest in ‘intelligent’ computer-based methods that use various classes of measurement signals, from different patient samples, for instance, to create a model for classifying new observations. This type of method is the basis for many technical applications, such as recognition of human speech, images, and fingerprints, and is now also beginning to attract new fields such as health care.

“Especially in applications in which faulty classification decisions can lead to catastrophic consequences, such as choosing the wrong form of therapy for treating cancer, it is extremely important to be able to make a reliable estimate of the performance of the classification model,” explains Mats Gustafsson, Professor of signal processing and medical bioinformatics at Uppsala University, who co-directed the new study together with Associate Professor Anders Isaksson.

To evaluate the performance of a classification model, one normally tests it on a number of trial examples that have never been involved in the design of the model. Unfortunately there are seldom tens of thousands of test examples available for this type of evaluation. In biomedicine, for instance, it is often expensive and difficult to collect the patient samples needed, especially if one wishes to analyze a rare disease. To solve this problem, many different methods have been proposed. Since the 1980s two methods have completely dominated research, namely, cross validation and resampling/bootstrapping.

“This has entailed that the performance assessment of virtually all new methods and applications reported in the scientific literature in the last 25 years has been carried out using one of these two methods,” says Mats Gustafsson.

In the new study, the Uppsala researchers use both theory and convincing computer simulations to show that this methodology is worthless in practice when the total number of examples is small in relation to the natural variation that exists among different observations. What is considered a small number depends in turn on what problem is being studied-­in other words, it is impossible to determine whether the number of examples is sufficient.

“Our main conclusion is that this methodology cannot be depended on at all, and that it therefore needs to be immediately replaces by Bayesian methods, for example, which can deliver reliable measures of the uncertainty that exists. Only then will multivariate analyses be in any position to be adopted in such critical applications as health care,” says Mats Gustafsson.

Mats Gustafsson | alfa
Further information:
http://www.uu.se

More articles from Information Technology:

nachricht CubeSats prove their worth for scientific missions
17.04.2019 | American Physical Society

nachricht Largest, fastest array of microscopic 'traffic cops' for optical communications
12.04.2019 | University of California - Berkeley

All articles from Information Technology >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: Explosion on Jupiter-sized star 10 times more powerful than ever seen on our sun

A stellar flare 10 times more powerful than anything seen on our sun has burst from an ultracool star almost the same size as Jupiter

  • Coolest and smallest star to produce a superflare found
  • Star is a tenth of the radius of our Sun
  • Researchers led by University of Warwick could only see...

Im Focus: Quantum simulation more stable than expected

A localization phenomenon boosts the accuracy of solving quantum many-body problems with quantum computers which are otherwise challenging for conventional computers. This brings such digital quantum simulation within reach on quantum devices available today.

Quantum computers promise to solve certain computational problems exponentially faster than any classical machine. “A particularly promising application is the...

Im Focus: Largest, fastest array of microscopic 'traffic cops' for optical communications

The technology could revolutionize how information travels through data centers and artificial intelligence networks

Engineers at the University of California, Berkeley have built a new photonic switch that can control the direction of light passing through optical fibers...

Im Focus: A long-distance relationship in femtoseconds

Physicists observe how electron-hole pairs drift apart at ultrafast speed, but still remain strongly bound.

Modern electronics relies on ultrafast charge motion on ever shorter length scales. Physicists from Regensburg and Gothenburg have now succeeded in resolving a...

Im Focus: Researchers 3D print metamaterials with novel optical properties

Engineers create novel optical devices, including a moth eye-inspired omnidirectional microwave antenna

A team of engineers at Tufts University has developed a series of 3D printed metamaterials with unique microwave or optical properties that go beyond what is...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

VideoLinks
Industry & Economy
Event News

Revered mathematicians and computer scientists converge with 200 young researchers in Heidelberg!

17.04.2019 | Event News

First dust conference in the Central Asian part of the earth’s dust belt

15.04.2019 | Event News

Fraunhofer FHR at the IEEE Radar Conference 2019 in Boston, USA

09.04.2019 | Event News

 
Latest News

New automated biological-sample analysis systems to accelerate disease detection

18.04.2019 | Life Sciences

Explosion on Jupiter-sized star 10 times more powerful than ever seen on our sun

18.04.2019 | Physics and Astronomy

New eDNA technology used to quickly assess coral reefs

18.04.2019 | Life Sciences

VideoLinks
Science & Research
Overview of more VideoLinks >>>