Skip to content

matbench_v0.1 matbench_perovskites

Individual Task Leaderboard for matbench_perovskites

Leaderboard for an individual task. Algorithms shown here may include both general purpose and specialized algorithms (i.e., algorithms which are only valid for a subset of tasks in the benchmark.

Leaderboard

algorithm mean mae std mae mean rmse max max_error
CGCNN v2019 0.0452 0.0007 0.0722 0.9923
AMMExpress v2020 0.2005 0.0085 0.2954 3.3116
RF-SCM/Magpie 0.2355 0.0034 0.3346 2.8870
Dummy 0.5660 0.0048 0.7424 3.6873

Dataset info

Description

Matbench v0.1 test dataset for predicting formation energy from crystal structure. Adapted from an original dataset generated by Castelli et al. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

Number of samples: 18928

Task type: regression

Input type: structure

Dataset columns
  • e_form: Target variable. Heat of formation of the entire 5-atom perovskite cell, in eV as calculated by RPBE GGA-DFT. Note the reference state for oxygen was computed from oxygen's chemical potential in water vapor, not as oxygen molecules, to reflect the application which these perovskites were studied for.
  • structure: Pymatgen Structure of the material.
Dataset reference

Ivano E. Castelli, David D. Landis, Kristian S. Thygesen, Søren Dahl, Ib Chorkendorff, Thomas F. Jaramillo and Karsten W. Jacobsen (2012) New cubic perovskites for one- and two-photon water splitting using the computational materials repository. Energy Environ. Sci., 2012,5, 9034-9043 https://doi.org/10.1039/C2EE22341D

Metadata

{'bibtex_refs': ['@Article{Dunn2020,\n'
                 'author={Dunn, Alexander\n'
                 'and Wang, Qi\n'
                 'and Ganose, Alex\n'
                 'and Dopp, Daniel\n'
                 'and Jain, Anubhav},\n'
                 'title={Benchmarking materials property prediction methods: '
                 'the Matbench test set and Automatminer reference '
                 'algorithm},\n'
                 'journal={npj Computational Materials},\n'
                 'year={2020},\n'
                 'month={Sep},\n'
                 'day={15},\n'
                 'volume={6},\n'
                 'number={1},\n'
                 'pages={138},\n'
                 'abstract={We present a benchmark test suite and an automated '
                 'machine learning procedure for evaluating supervised machine '
                 'learning (ML) models for predicting properties of inorganic '
                 'bulk materials. The test suite, Matbench, is a set of '
                 '13{\\thinspace}ML tasks that range in size from 312 to 132k '
                 'samples and contain data from 10 density functional '
                 'theory-derived and experimental sources. Tasks include '
                 'predicting optical, thermal, electronic, thermodynamic, '
                 "tensile, and elastic properties given a material's "
                 'composition and/or crystal structure. The reference '
                 'algorithm, Automatminer, is a highly-extensible, fully '
                 'automated ML pipeline for predicting materials properties '
                 'from materials primitives (such as composition and crystal '
                 'structure) without user intervention or hyperparameter '
                 'tuning. We test Automatminer on the Matbench test suite and '
                 'compare its predictive power with state-of-the-art crystal '
                 'graph neural networks and a traditional descriptor-based '
                 'Random Forest model. We find Automatminer achieves the best '
                 'performance on 8 of 13 tasks in the benchmark. We also show '
                 'our test suite is capable of exposing predictive advantages '
                 'of each algorithm---namely, that crystal graph methods '
                 'appear to outperform traditional machine learning methods '
                 'given {\\textasciitilde}104 or greater data points. We '
                 'encourage evaluating materials ML algorithms on the Matbench '
                 'benchmark and comparing them against the latest version of '
                 'Automatminer.},\n'
                 'issn={2057-3960},\n'
                 'doi={10.1038/s41524-020-00406-3},\n'
                 'url={https://doi.org/10.1038/s41524-020-00406-3}\n'
                 '}\n',
                 '@Article{C2EE22341D,\n'
                 'author ="Castelli, Ivano E. and Landis, David D. and '
                 'Thygesen, Kristian S. and Dahl, Søren and Chorkendorff, Ib '
                 'and Jaramillo, Thomas F. and Jacobsen, Karsten W.",\n'
                 'title  ="New cubic perovskites for one- and two-photon water '
                 'splitting using the computational materials repository",\n'
                 'journal  ="Energy Environ. Sci.",\n'
                 'year  ="2012",\n'
                 'volume  ="5",\n'
                 'issue  ="10",\n'
                 'pages  ="9034-9043",\n'
                 'publisher  ="The Royal Society of Chemistry",\n'
                 'doi  ="10.1039/C2EE22341D",\n'
                 'url  ="http://dx.doi.org/10.1039/C2EE22341D",\n'
                 'abstract  ="A new efficient photoelectrochemical cell (PEC) '
                 'is one of the possible solutions to the energy and climate '
                 'problems of our time. Such a device requires development of '
                 'new semiconducting materials with tailored properties with '
                 'respect to stability and light absorption. Here we perform '
                 'computational screening of around 19\u2009000 oxides{,} '
                 'oxynitrides{,} oxysulfides{,} oxyfluorides{,} and '
                 'oxyfluoronitrides in the cubic perovskite structure with PEC '
                 'applications in mind. We address three main applications: '
                 'light absorbers for one- and two-photon water splitting and '
                 'high-stability transparent shields to protect against '
                 'corrosion. We end up with 20{,} 12{,} and 15 different '
                 'combinations of oxides{,} oxynitrides and oxyfluorides{,} '
                 'respectively{,} inviting further experimental '
                 'investigation."}'],
 'columns': {'e_form': 'Target variable. Heat of formation of the entire '
                       '5-atom perovskite cell, in eV as calculated by RPBE '
                       'GGA-DFT. Note the reference state for oxygen was '
                       "computed from oxygen's chemical potential in water "
                       'vapor, not as oxygen molecules, to reflect the '
                       'application which these perovskites were studied for.',
             'structure': 'Pymatgen Structure of the material.'},
 'description': 'Matbench v0.1 test dataset for predicting formation energy '
                'from crystal structure. Adapted from an original dataset '
                'generated by Castelli et al. For benchmarking w/ nested cross '
                'validation, the order of the dataset must be identical to the '
                'retrieved data; refer to the Automatminer/Matbench '
                'publication for more details.',
 'file_type': 'json.gz',
 'hash': '4641e2417f8ec8b50096d2230864468dfa08278dc9d257c327f65d0305278483',
 'input_type': 'structure',
 'mad': 0.5659924184827462,
 'n_samples': 18928,
 'num_entries': 18928,
 'reference': 'Ivano E. Castelli, David D. Landis, Kristian S. Thygesen, Søren '
              'Dahl, Ib Chorkendorff, Thomas F. Jaramillo and Karsten W. '
              'Jacobsen (2012) New cubic perovskites for one- and two-photon '
              'water splitting using the computational materials repository. '
              'Energy Environ. Sci., 2012,5, 9034-9043 '
              'https://doi.org/10.1039/C2EE22341D',
 'target': 'e_form',
 'task_type': 'regression',
 'unit': 'eV/unit cell',
 'url': 'https://ml.materialsproject.org/projects/matbench_perovskites.json.gz'}