Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:


One in 5 materials chemistry papers may be wrong, study suggests


Can companies rely on the results of one or two scientific studies to design a new industrial process or launch a new product? In at least one area of materials chemistry, the answer may be yes -- but only 80 percent of the time.

The replicability of results from scientific studies has become a major source of concern in the research community, particularly in the social sciences and biomedical sciences. But many researchers in the fields of engineering and the hard sciences haven't felt the same level of concern for independent validation of their results.

A new study that compared the results reported in thousands of papers published about the properties of metal organic framework (MOF) materials -- which are prominent candidates for carbon dioxide adsorption and other separations -- suggests the replicability problem should be a concern for materials researchers.

Credit: Georgia Tech

A new study that compared the results reported in thousands of papers published about the properties of metal organic framework (MOF) materials - which are prominent candidates for carbon dioxide adsorption and other separations - suggests the replicability problem should be a concern for materials researchers, too.

One in five studies of MOF materials examined by researchers at the Georgia Institute of Technology were judged to be "outliers," with results far beyond the error bars normally used to evaluate study results. The thousands of research papers yielded just nine MOF compounds for which four or more independent studies allowed appropriate comparison of results.

"At a fundamental level, I think people in materials chemistry feel that things are reproducible and that they can count on the results of a single study," said David Sholl, a professor and John F. Brock III School Chair in the Georgia Tech School of Chemical and Biomolecular Engineering. "But what we found is that if you pull out any experiment at random, there's a one in five chance that the results are completely wrong - not just slightly off, but not even close."

Whether the results can be more broadly applied to other areas of materials science awaits additional studies, Sholl said. The results of the study, which was supported by the U.S. Department of Energy, were published November 28 in the ACS journal Chemistry of Materials.

Sholl chose MOFs because they're an area of interest to his lab - he develops models for the materials - and because the National Institute of Standards and Technology (NIST) and the Advanced Research Projects Agency-Energy (ARPA-E) had already assembled a database summarizing the properties of MOFs. Co-authors Jongwoo Park and Joshua Howe used meta-analysis techniques to compare the results of single-component adsorption isotherm testing - how much CO2 can be removed at room temperature.

That measurement is straightforward and there are commercial instruments available for doing the tests. "People in the community would consider this to be an almost foolproof experiment," said Sholl, who is also a Georgia Research Alliance Eminent Scholar in Energy Sustainability.

The researchers considered the results definitive when they had four or more studies of a given MOF at comparable conditions.

The implications for errors in materials science may be less than in other research fields. But companies could use the results of a just one or two studies to choose a material that appear to be more efficient, and in other cases, researchers unable to replicate an experiment may simply move on to another material.

"The net result is non-optimal use of resources at the very least," Sholl said. "And any report using one experiment to conclude a material is 15 or 20 percent better than another material should be viewed with great skepticism, as we cannot be very precise on these measurements in most cases."

Why the variability in results? Some MOFs can be finicky, quickly absorbing moisture that affect adsorption, for instance. The one-in-five "outliers" may be a result of materials contamination.

"One of the materials we studied is relatively simple to make, but it's unstable in an ambient atmosphere," Sholl explained. "Exactly what you do between making it in the lab and testing it will affect the properties you measure. That could account for some of what we saw, and if a material is that sensitive, we know it's going to be a problem in practical use."

Other factors that may prevent replication include details that were inadvertently left out of a methods description - or that the original scientists didn't realize were relevant. That could be as simple as the precise atmosphere in which the material is maintained, or the materials used in the apparatus producing the MOFs.

Sholl hopes the paper will lead to more replication of experiments so scientists and engineers can know if their results really are significant.

"As a result of this, I think my group will look at all reported data in a more nuanced way, not necessarily suspecting it is wrong, but thinking about how reliable that data might be," he said. "Instead of thinking about data as a number, we need to always think about it as a number plus a range."

Sholl suggests that more reporting of second, third or fourth efforts to replicate an experiment would help raise the confidence of data on MOF materials properties. The scientific publishing system doesn't currently provide much incentive for reporting validation, though Sholl hopes that will change.

He also feels the issue needs to be discussed within all parts of the scientific community, though he admits that can lead to "uncomfortable" conversations.

"We have presented this study a few times at conferences, and people can get pretty defensive about it," Sholl said. "Everybody in the field knows everybody else, so it's always easier to just not bring up this issue."

And, of course, Sholl would like to see others replicate the work he and his research team did. "It will be interesting to see if this one-in-five number holds up for other types of experiments and materials," he added. "There are other certainly other areas of materials chemistry where this kind of comparison could be done."


This research was supported by the U.S. Department of Energy through grant DE-FE0026433 and by the Center for Understanding and Control of Acid Gas-Induced Evolution of Materials for Energy (UNCAGE-ME), an Energy Frontier Research Center funded by U.S. Department of Energy, Office of Science, Basic Energy Sciences under Award #DE-SC0012577. Any opinions, findings, conclusions or recommendations expressed herein are those of the author(s) and do not necessarily reflect the views of sponsors.

Media Contact

John Toon


John Toon | EurekAlert!

More articles from Materials Sciences:

nachricht Researchers demonstrate existence of new form of electronic matter
15.03.2018 | University of Illinois at Urbana-Champaign

nachricht Boron can form a purely honeycomb, graphene-like 2-D structure
15.03.2018 | Science China Press

All articles from Materials Sciences >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: Locomotion control with photopigments

Researchers from Göttingen University discover additional function of opsins

Animal photoreceptors capture light with photopigments. Researchers from the University of Göttingen have now discovered that these photopigments fulfill an...

Im Focus: Surveying the Arctic: Tracking down carbon particles

Researchers embark on aerial campaign over Northeast Greenland

On 15 March, the AWI research aeroplane Polar 5 will depart for Greenland. Concentrating on the furthest northeast region of the island, an international team...

Im Focus: Unique Insights into the Antarctic Ice Shelf System

Data collected on ocean-ice interactions in the little-researched regions of the far south

The world’s second-largest ice shelf was the destination for a Polarstern expedition that ended in Punta Arenas, Chile on 14th March 2018. Oceanographers from...

Im Focus: ILA 2018: Laser alternative to hexavalent chromium coating

At the 2018 ILA Berlin Air Show from April 25–29, the Fraunhofer Institute for Laser Technology ILT is showcasing extreme high-speed Laser Material Deposition (EHLA): A video documents how for metal components that are highly loaded, EHLA has already proved itself as an alternative to hard chrome plating, which is now allowed only under special conditions.

When the EU restricted the use of hexavalent chromium compounds to special applications requiring authorization, the move prompted a rethink in the surface...

Im Focus: Radar for navigation support from autonomous flying drones

At the ILA Berlin, hall 4, booth 202, Fraunhofer FHR will present two radar sensors for navigation support of drones. The sensors are valuable components in the implementation of autonomous flying drones: they function as obstacle detectors to prevent collisions. Radar sensors also operate reliably in restricted visibility, e.g. in foggy or dusty conditions. Due to their ability to measure distances with high precision, the radar sensors can also be used as altimeters when other sources of information such as barometers or GPS are not available or cannot operate optimally.

Drones play an increasingly important role in the area of logistics and services. Well-known logistic companies place great hope in these compact, aerial...

All Focus news of the innovation-report >>>



Industry & Economy
Event News

Ultrafast Wireless and Chip Design at the DATE Conference in Dresden

16.03.2018 | Event News

International Tinnitus Conference of the Tinnitus Research Initiative in Regensburg

13.03.2018 | Event News

International Virtual Reality Conference “IEEE VR 2018” comes to Reutlingen, Germany

08.03.2018 | Event News

Latest News

Wandering greenhouse gas

16.03.2018 | Earth Sciences

'Frequency combs' ID chemicals within the mid-infrared spectral region

16.03.2018 | Physics and Astronomy

Biologists unravel another mystery of what makes DNA go 'loopy'

16.03.2018 | Life Sciences

Science & Research
Overview of more VideoLinks >>>