Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

One in 5 materials chemistry papers may be wrong, study suggests

15.12.2017

Can companies rely on the results of one or two scientific studies to design a new industrial process or launch a new product? In at least one area of materials chemistry, the answer may be yes -- but only 80 percent of the time.

The replicability of results from scientific studies has become a major source of concern in the research community, particularly in the social sciences and biomedical sciences. But many researchers in the fields of engineering and the hard sciences haven't felt the same level of concern for independent validation of their results.


A new study that compared the results reported in thousands of papers published about the properties of metal organic framework (MOF) materials -- which are prominent candidates for carbon dioxide adsorption and other separations -- suggests the replicability problem should be a concern for materials researchers.

Credit: Georgia Tech

A new study that compared the results reported in thousands of papers published about the properties of metal organic framework (MOF) materials - which are prominent candidates for carbon dioxide adsorption and other separations - suggests the replicability problem should be a concern for materials researchers, too.

One in five studies of MOF materials examined by researchers at the Georgia Institute of Technology were judged to be "outliers," with results far beyond the error bars normally used to evaluate study results. The thousands of research papers yielded just nine MOF compounds for which four or more independent studies allowed appropriate comparison of results.

"At a fundamental level, I think people in materials chemistry feel that things are reproducible and that they can count on the results of a single study," said David Sholl, a professor and John F. Brock III School Chair in the Georgia Tech School of Chemical and Biomolecular Engineering. "But what we found is that if you pull out any experiment at random, there's a one in five chance that the results are completely wrong - not just slightly off, but not even close."

Whether the results can be more broadly applied to other areas of materials science awaits additional studies, Sholl said. The results of the study, which was supported by the U.S. Department of Energy, were published November 28 in the ACS journal Chemistry of Materials.

Sholl chose MOFs because they're an area of interest to his lab - he develops models for the materials - and because the National Institute of Standards and Technology (NIST) and the Advanced Research Projects Agency-Energy (ARPA-E) had already assembled a database summarizing the properties of MOFs. Co-authors Jongwoo Park and Joshua Howe used meta-analysis techniques to compare the results of single-component adsorption isotherm testing - how much CO2 can be removed at room temperature.

That measurement is straightforward and there are commercial instruments available for doing the tests. "People in the community would consider this to be an almost foolproof experiment," said Sholl, who is also a Georgia Research Alliance Eminent Scholar in Energy Sustainability.

The researchers considered the results definitive when they had four or more studies of a given MOF at comparable conditions.

The implications for errors in materials science may be less than in other research fields. But companies could use the results of a just one or two studies to choose a material that appear to be more efficient, and in other cases, researchers unable to replicate an experiment may simply move on to another material.

"The net result is non-optimal use of resources at the very least," Sholl said. "And any report using one experiment to conclude a material is 15 or 20 percent better than another material should be viewed with great skepticism, as we cannot be very precise on these measurements in most cases."

Why the variability in results? Some MOFs can be finicky, quickly absorbing moisture that affect adsorption, for instance. The one-in-five "outliers" may be a result of materials contamination.

"One of the materials we studied is relatively simple to make, but it's unstable in an ambient atmosphere," Sholl explained. "Exactly what you do between making it in the lab and testing it will affect the properties you measure. That could account for some of what we saw, and if a material is that sensitive, we know it's going to be a problem in practical use."

Other factors that may prevent replication include details that were inadvertently left out of a methods description - or that the original scientists didn't realize were relevant. That could be as simple as the precise atmosphere in which the material is maintained, or the materials used in the apparatus producing the MOFs.

Sholl hopes the paper will lead to more replication of experiments so scientists and engineers can know if their results really are significant.

"As a result of this, I think my group will look at all reported data in a more nuanced way, not necessarily suspecting it is wrong, but thinking about how reliable that data might be," he said. "Instead of thinking about data as a number, we need to always think about it as a number plus a range."

Sholl suggests that more reporting of second, third or fourth efforts to replicate an experiment would help raise the confidence of data on MOF materials properties. The scientific publishing system doesn't currently provide much incentive for reporting validation, though Sholl hopes that will change.

He also feels the issue needs to be discussed within all parts of the scientific community, though he admits that can lead to "uncomfortable" conversations.

"We have presented this study a few times at conferences, and people can get pretty defensive about it," Sholl said. "Everybody in the field knows everybody else, so it's always easier to just not bring up this issue."

And, of course, Sholl would like to see others replicate the work he and his research team did. "It will be interesting to see if this one-in-five number holds up for other types of experiments and materials," he added. "There are other certainly other areas of materials chemistry where this kind of comparison could be done."

###

This research was supported by the U.S. Department of Energy through grant DE-FE0026433 and by the Center for Understanding and Control of Acid Gas-Induced Evolution of Materials for Energy (UNCAGE-ME), an Energy Frontier Research Center funded by U.S. Department of Energy, Office of Science, Basic Energy Sciences under Award #DE-SC0012577. Any opinions, findings, conclusions or recommendations expressed herein are those of the author(s) and do not necessarily reflect the views of sponsors.

Media Contact

John Toon
jtoon@gatech.edu
404-894-6986

 @GeorgiaTech

http://www.gatech.edu 

John Toon | EurekAlert!

More articles from Materials Sciences:

nachricht Epoxy compound gets a graphene bump
14.11.2018 | Rice University

nachricht Automated adhesive film placement and stringer integration for aircraft manufacture
14.11.2018 | Fraunhofer IFAM

All articles from Materials Sciences >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: A Chip with Blood Vessels

Biochips have been developed at TU Wien (Vienna), on which tissue can be produced and examined. This allows supplying the tissue with different substances in a very controlled way.

Cultivating human cells in the Petri dish is not a big challenge today. Producing artificial tissue, however, permeated by fine blood vessels, is a much more...

Im Focus: A Leap Into Quantum Technology

Faster and secure data communication: This is the goal of a new joint project involving physicists from the University of Würzburg. The German Federal Ministry of Education and Research funds the project with 14.8 million euro.

In our digital world data security and secure communication are becoming more and more important. Quantum communication is a promising approach to achieve...

Im Focus: Research icebreaker Polarstern begins the Antarctic season

What does it look like below the ice shelf of the calved massive iceberg A68?

On Saturday, 10 November 2018, the research icebreaker Polarstern will leave its homeport of Bremerhaven, bound for Cape Town, South Africa.

Im Focus: Penn engineers develop ultrathin, ultralight 'nanocardboard'

When choosing materials to make something, trade-offs need to be made between a host of properties, such as thickness, stiffness and weight. Depending on the application in question, finding just the right balance is the difference between success and failure

Now, a team of Penn Engineers has demonstrated a new material they call "nanocardboard," an ultrathin equivalent of corrugated paper cardboard. A square...

Im Focus: Coping with errors in the quantum age

Physicists at ETH Zurich demonstrate how errors that occur during the manipulation of quantum system can be monitored and corrected on the fly

The field of quantum computation has seen tremendous progress in recent years. Bit by bit, quantum devices start to challenge conventional computers, at least...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

VideoLinks
Industry & Economy
Event News

“3rd Conference on Laser Polishing – LaP 2018” Attracts International Experts and Users

09.11.2018 | Event News

On the brain’s ability to find the right direction

06.11.2018 | Event News

European Space Talks: Weltraumschrott – eine Gefahr für die Gesellschaft?

23.10.2018 | Event News

 
Latest News

Epoxy compound gets a graphene bump

14.11.2018 | Materials Sciences

Microgel powder fights infection and helps wounds heal

14.11.2018 | Health and Medicine

How algae and carbon fibers could sustainably reduce the athmospheric carbon dioxide concentration

14.11.2018 | Life Sciences

VideoLinks
Science & Research
Overview of more VideoLinks >>>