Leaderboard: General Purpose Algorithms on
Find more information about this benchmark on the benchmark info page
|Task name||Samples||Algorithm||Verified MAE (unit) or ROCAUC||Notes|
|matbench_steels||312||AMMExpress v2020||97.4929 (MPa)|
|matbench_jdft2d||636||AMMExpress v2020||39.8497 (meV/atom)|
|matbench_phonons||1,265||AMMExpress v2020||56.1706 (cm^-1)|
|matbench_expt_gap||4,604||AMMExpress v2020||0.4161 (eV)|
|matbench_dielectric||4,764||AMMExpress v2020||0.3150 (unitless)|
|matbench_log_gvrh||10,987||AMMExpress v2020||0.0874 (log10(GPa))|
|matbench_log_kvrh||10,987||AMMExpress v2020||0.0647 (log10(GPa))|
|matbench_perovskites||18,928||CGCNN v2019||0.0452 (eV/unit cell)||structure required|
|matbench_mp_gap||106,113||AMMExpress v2020||0.2824 (eV)|
|matbench_mp_is_metal||106,113||CGCNN v2019||0.9520||structure required|
|matbench_mp_e_form||132,752||CGCNN v2019||0.0337 (eV/atom)||structure required|
Matbench is an ImageNet for materials science; a curated set of 13 supervised, pre-cleaned, ready-to-use ML tasks for benchmarking and fair comparison. The tasks span a wide domain of inorganic materials science applications including electronic, thermodynamic, mechanical, and thermal properties among crystals, 2D materials, disordered metals, and more.
The Matbench python package provides everything needed to use Matbench with your ML algorithm in ~10 lines of code or less.
What can Matbench offer?
- Leaderboard of results for state-of-the-art materials ML algorithms on standardized test problems
- Interactively explore and download the tasks on MPContribs-ML, a platform hosted by The Materials Project. See Benchmark Info for links to each dataset.
- Each and every result is backed by a peer-reviewed publication and a jupyter notebook (similar to Papers With Code) - i.e., how were these results were obtained?
- Glossary of all algorithms' results on the Matbench problems
The Matbench Python package
- Probe ML algorithms strengths and weaknesses across a wide range of materials property prediction tasks
- Run a full benchmark in ~10 lines of code
- Submit results as a PR to the Matbench repo to compare with other algorithms and appear on the leaderboard
- Benchmark both general purpose ML models as well as algorithms specialized for particular domains
Summary of Matbench's Tasks
Matbench's 13 tasks can be broken down into various categories; it includes both the small - less than 10,000 samples - datasets that characterize experimental materials data as well as larger datasets from computer modelling methods like density functional theory (DFT).
Each task in Matbench consists of a three things:
- A set of inputs: crystal structures or chemical compositions.
- A set of outputs: target properties, such as formation energy.
- A test procedure: a way to get a score for your algorithm
The Matbench Python package provides functions for getting the first two (packaged together for each task as a dataset) as well as running the test procedure. See the How to use documentation page to get started.
You can find details and results on the benchmark in our paper Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference . Please consider citing this paper if you use Matbench v0.1 for benchmarking, comparison, or prototyping.
You can cite Matbench using this reference:
Dunn, A., Wang, Q., Ganose, A., Dopp, D., Jain, A. Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm. npj Computational Materials 6, 138 (2020). https://doi.org/10.1038/s41524-020-00406-3