A gold standard for computational materials science codes

For the past few decades, physicists and materials scientists around the world have been busy developing computer codes that simulate the key properties of materials. They can now choose from a whole family of such tools, using them to publish tens of thousands of scientific articles per year.

These codes are typically based on density-functional theory (DFT), a modelling method that uses several approximations to reduce the otherwise mind-boggling complexity of calculating the behaviour of each individual electron according to the laws of quantum mechanics. The differences between the results obtained with various codes come down to the numerical approximations being made, and the choice of the numerical parameters behind those approximations, often tailored to study specific classes of materials, or to calculate properties that are key for specific applications – say, conductivity for potential battery materials.

Given the complexity of these codes, it is difficult to make sure that all of them are free of any possible coding error, or do not suffer from numerical approximations that are too coarse. But it is crucial for the community to verify that the results from different codes are comparable, consistent with each other and reproducible. 

In a new article just published in Nature Review Physics, a group of scientists has carried out the most comprehensive verification effort so far on solid-state DFT codes and provided their colleagues with the tools and a set of guidelines for assessing and improving existing and future codes. 

Do different codes give different answers?

The work builds on a previous study published in Science in 2016 that had compared 40 computational approaches by using each one of them to calculate the energies of a test set of 71 crystals, each one corresponding to an element on the periodic table, and concluded that the mainstream codes were in very good agreement with each other. 

“That work was reassuring, but it did not really explore enough chemical diversity,” says Giovanni Pizzi, leader of the Materials Software and Data Group at the Paul Scherrer Institute PSI, who led the study. “In this study we considered 96 elements, and for each of them we simulated ten possible crystal structures”.

In particular, for each of the first 96 elements of the periodic table, they studied four different unaries - crystals made only with atoms of the element itself - and six different oxides, which also include oxygen atoms. The result is a dataset of 960 materials and their properties, calculated by two independent, state-of-the-art DFT codes called FLEUR and WIEN2k. Both are “all-electron” (AE) codes, meaning that they consider explicitly all the electrons in the atoms under consideration. 

This dataset can now be used by anyone as a benchmark to test the precision of other codes, in particular, those based on pseudopotentials where, unlike in all-electron (AE) codes, the electrons that do not participate in chemical bonds are treated in a simplified way to make the computation lighter.  “We actually have already started to improve nine such codes in our paper, comparing their results to those in our dataset, measuring the discrepancies and adjusting their numerical parameters (such as the pseudopotentials) accordingly” explains Pizzi. 

“A reference for the field for years to come”

The study also includes a series of recommendations for users of DFT codes, to make sure that computational studies are reproducible, on how to use the reference dataset to conduct future verification studies, and on how to expand it to include other families of codes and other materials properties. 

“We hope our dataset will be a reference for the field for years to come,” says Pizzi.  

The study also provides an environment for future verification studies through AiiDA, the open-access computational framework developed by the National Centre for Competence in Research (NCCR) MARVEL, that supported the work and in which Pizzi is a project leader, and by the European Centre of Excellence MaX. “AiiDA allows us to write the same instruction in the same way for 11 different codes, for example the request to compute a specific structure. It can then run the calculation for you and select the right numerical parameters for each”. 

In addition to expanding the reference dataset with more structures, Pizzi says that in the future he hopes to take into account not only how accurate the different codes are, but also how expensive they are in terms of time and computational power, so helping scientist find the most cost-efficient parameters for their calculations. 

Text: Based on a news article by NCCR MARVEL / Nicola Nosengo with modifications from Paul Scherrer Institute / Miriam Arrell