Skip to content

matbench_v0.1: Dummy

Algorithm description:

Dummy regressor, using strategy 'mean', and Dummy regressor, using strategy 'stratified'.

No notes.

Raw data download and example notebook available on the matbench repo.

References (in bibtex format):

('@article{Dunn2020,\n'
 '  doi = {10.1038/s41524-020-00406-3},\n'
 '  url = {https://doi.org/10.1038/s41524-020-00406-3},\n'
 '  year = {2020},\n'
 '  month = sep,\n'
 '  publisher = {Springer Science and Business Media {LLC}},\n'
 '  volume = {6},\n'
 '  number = {1},\n'
 '  author = {Alexander Dunn and Qi Wang and Alex Ganose and Daniel Dopp and '
 'Anubhav Jain},\n'
 '  title = {Benchmarking materials property prediction methods: the Matbench '
 'test set and Automatminer reference algorithm},\n'
 '  journal = {npj Computational Materials}\n'
 '}')

User metadata:

{'algorithm': 'dummy',
 'classification_strategy': 'stratified',
 'regression_strategy': 'mean'}

Metadata:

Tasks recorded: 13 of 13 total

Benchmark is complete? True

Software Requirements

{'python': ['scikit-learn==0.24.1', 'numpy==1.20.1', 'matbench==0.1.0']}

Task data:

matbench_dielectric

Fold scores
fold mae rmse mape* max_error
fold_0 0.7026 1.0677 0.3201 14.9501
fold_1 0.7811 1.4374 0.3142 20.3552
fold_2 0.9218 3.1055 0.3155 59.6653
fold_3 0.8382 2.4438 0.3266 53.4563
fold_4 0.8004 1.8094 0.3222 28.5706
Fold score stats
metric mean max min std
mae 0.8088 0.9218 0.7026 0.0718
rmse 1.9728 3.1055 1.0677 0.7263
mape* 0.3197 0.3266 0.3142 0.0045
max_error 35.3995 59.6653 14.9501 17.9221
Fold parameters
fold params dict
fold_0 {'constant_': [[2.4569770107667264]]}
fold_1 {'constant_': [[2.4254682439737882]]}
fold_2 {'constant_': [[2.397736448888908]]}
fold_3 {'constant_': [[2.429725254851624]]}
fold_4 {'constant_': [[2.4316628587393003]]}

matbench_expt_gap

Fold scores
fold mae rmse mape* max_error
fold_0 1.0965 1.3397 0.8589 7.0119
fold_1 1.1922 1.5156 0.7802 8.3754
fold_2 1.1527 1.5268 1.0398 10.7354
fold_3 1.1445 1.4389 0.8373 9.5190
fold_4 1.1317 1.3979 1.2418 9.0085
Fold score stats
metric mean max min std
mae 1.1435 1.1922 1.0965 0.0310
rmse 1.4438 1.5268 1.3397 0.0707
mape* 0.9516 1.2418 0.7802 0.1692
max_error 8.9300 10.7354 7.0119 1.2328
Fold parameters
fold params dict
fold_0 {'constant_': [[0.9881156665761609]]}
fold_1 {'constant_': [[0.9545533532446375]]}
fold_2 {'constant_': [[0.9645506380667934]]}
fold_3 {'constant_': [[0.9810371979364648]]}
fold_4 {'constant_': [[0.9914956568946797]]}

matbench_expt_is_metal

Fold scores
fold accuracy balanced_accuracy f1 rocauc
fold_0 0.4701 0.4700 0.4540 0.4700
fold_1 0.5000 0.5001 0.5080 0.5001
fold_2 0.4878 0.4878 0.4878 0.4878
fold_3 0.5071 0.5072 0.5126 0.5072
fold_4 0.4970 0.4969 0.4944 0.4969
Fold score stats
metric mean max min std
accuracy 0.4924 0.5071 0.4701 0.0128
balanced_accuracy 0.4924 0.5072 0.4700 0.0128
f1 0.4913 0.5126 0.4540 0.0207
rocauc 0.4924 0.5072 0.4700 0.0128
Fold parameters
fold params dict
fold_0 {'class_prior_': [0.5020325203252033, 0.49796747967479676]}
fold_1 {'class_prior_': [0.5019050038100076, 0.49809499618999237]}
fold_2 {'class_prior_': [0.5019050038100076, 0.49809499618999237]}
fold_3 {'class_prior_': [0.5019050038100076, 0.49809499618999237]}
fold_4 {'class_prior_': [0.5019050038100076, 0.49809499618999237]}

matbench_glass

Fold scores
fold accuracy balanced_accuracy f1 rocauc
fold_0 0.6127 0.5212 0.7304 0.5212
fold_1 0.6083 0.5217 0.7251 0.5217
fold_2 0.5775 0.4848 0.7033 0.4848
fold_3 0.5731 0.4799 0.7001 0.4799
fold_4 0.5819 0.4951 0.7044 0.4951
Fold score stats
metric mean max min std
accuracy 0.5907 0.6127 0.5731 0.0165
balanced_accuracy 0.5005 0.5217 0.4799 0.0178
f1 0.7127 0.7304 0.7001 0.0125
rocauc 0.5005 0.5217 0.4799 0.0178
Fold parameters
fold params dict
fold_0 {'class_prior_': [0.289612676056338, 0.710387323943662]}
fold_1 {'class_prior_': [0.289612676056338, 0.710387323943662]}
fold_2 {'class_prior_': [0.289612676056338, 0.710387323943662]}
fold_3 {'class_prior_': [0.289612676056338, 0.710387323943662]}
fold_4 {'class_prior_': [0.289612676056338, 0.710387323943662]}

matbench_jdft2d

Fold scores
fold mae rmse mape* max_error
fold_0 53.1447 74.1060 35.3098 509.7791
fold_1 72.8118 118.0523 0.8129 642.7424
fold_2 83.1220 192.2365 0.9798 1025.0199
fold_3 61.3174 85.6603 0.7921 468.0412
fold_4 66.0295 164.1680 1.1452 1491.7993
Fold score stats
metric mean max min std
mae 67.2851 83.1220 53.1447 10.1832
rmse 126.8446 192.2365 74.1060 45.2193
mape* 7.8079 35.3098 0.7921 13.7515
max_error 827.4764 1491.7993 468.0412 385.9016
Fold parameters
fold params dict
fold_0 {'constant_': [[117.03965287603667]]}
fold_1 {'constant_': [[112.91320041366653]]}
fold_2 {'constant_': [[106.46511492350562]]}
fold_3 {'constant_': [[114.84311227394852]]}
fold_4 {'constant_': [[112.23899155170879]]}

matbench_log_gvrh

Fold scores
fold mae rmse mape* max_error
fold_0 0.2943 0.3749 0.2368 1.5529
fold_1 0.2933 0.3743 0.2359 1.5544
fold_2 0.2969 0.3736 0.2353 1.5533
fold_3 0.2875 0.3646 0.2251 1.5524
fold_4 0.2937 0.3706 0.2334 1.5552
Fold score stats
metric mean max min std
mae 0.2931 0.2969 0.2875 0.0031
rmse 0.3716 0.3749 0.3646 0.0038
mape* 0.2333 0.2368 0.2251 0.0042
max_error 1.5536 1.5552 1.5524 0.0010
Fold parameters
fold params dict
fold_0 {'constant_': [[1.5529289714161707]]}
fold_1 {'constant_': [[1.554355173237515]]}
fold_2 {'constant_': [[1.5532719705303168]]}
fold_3 {'constant_': [[1.5523993668681186]]}
fold_4 {'constant_': [[1.5552370733167413]]}

matbench_log_kvrh

Fold scores
fold mae rmse mape* max_error
fold_0 0.2935 0.3774 0.1877 1.8800
fold_1 0.2875 0.3669 0.1858 1.8809
fold_2 0.2889 0.3634 0.1825 1.8801
fold_3 0.2833 0.3635 0.1926 1.8790
fold_4 0.2953 0.3752 0.1900 1.8822
Fold score stats
metric mean max min std
mae 0.2897 0.2953 0.2833 0.0043
rmse 0.3693 0.3774 0.3634 0.0059
mape* 0.1877 0.1926 0.1825 0.0035
max_error 1.8804 1.8822 1.8790 0.0011
Fold parameters
fold params dict
fold_0 {'constant_': [[1.8800295900875317]]}
fold_1 {'constant_': [[1.880914404358644]]}
fold_2 {'constant_': [[1.8800659898099186]]}
fold_3 {'constant_': [[1.8789962707394416]]}
fold_4 {'constant_': [[1.8822230471663404]]}

matbench_mp_e_form

Fold scores
fold mae rmse mape* max_error
fold_0 1.0063 1.1626 11.6409 3.8987
fold_1 1.0036 1.1597 7.2868 3.8782
fold_2 1.0062 1.1662 8.5651 3.9096
fold_3 1.0024 1.1597 10.9729 3.8934
fold_4 1.0111 1.1675 11.2779 3.9051
Fold score stats
metric mean max min std
mae 1.0059 1.0111 1.0024 0.0030
rmse 1.1631 1.1675 1.1597 0.0032
mape* 9.9487 11.6409 7.2868 1.7134
max_error 3.8970 3.9096 3.8782 0.0109
Fold parameters
fold params dict
fold_0 {'constant_': [[-1.4071424641223964]]}
fold_1 {'constant_': [[-1.4079146341783042]]}
fold_2 {'constant_': [[-1.4100821676758766]]}
fold_3 {'constant_': [[-1.406498235557698]]}
fold_4 {'constant_': [[-1.4079540738106724]]}

matbench_mp_gap

Fold scores
fold mae rmse mape* max_error
fold_0 1.3199 1.5863 13.8283 7.1079
fold_1 1.3224 1.5888 12.1282 8.1096
fold_2 1.3252 1.5964 14.5509 7.6322
fold_3 1.3335 1.6113 19.3774 7.4334
fold_4 1.3348 1.6118 18.0392 8.5092
Fold score stats
metric mean max min std
mae 1.3272 1.3348 1.3199 0.0060
rmse 1.5989 1.6118 1.5863 0.0108
mape* 15.5848 19.3774 12.1282 2.7022
max_error 7.7585 8.5092 7.1079 0.4963
Fold parameters
fold params dict
fold_0 {'constant_': [[1.216204829779715]]}
fold_1 {'constant_': [[1.2168485710920014]]}
fold_2 {'constant_': [[1.2161007256449523]]}
fold_3 {'constant_': [[1.2119634071927532]]}
fold_4 {'constant_': [[1.2120157684560202]]}

matbench_mp_is_metal

Fold scores
fold accuracy balanced_accuracy f1 rocauc
fold_0 0.5158 0.5069 0.4405 0.5069
fold_1 0.5032 0.4944 0.4277 0.4944
fold_2 0.5069 0.4986 0.4342 0.4986
fold_3 0.5119 0.5032 0.4376 0.5032
fold_4 0.5118 0.5030 0.4367 0.5030
Fold score stats
metric mean max min std
accuracy 0.5099 0.5158 0.5032 0.0044
balanced_accuracy 0.5012 0.5069 0.4944 0.0043
f1 0.4353 0.4405 0.4277 0.0043
rocauc 0.5012 0.5069 0.4944 0.0043
Fold parameters
fold params dict
fold_0 {'class_prior_': [0.5650724466957239, 0.4349275533042761]}
fold_1 {'class_prior_': [0.5650724466957239, 0.4349275533042761]}
fold_2 {'class_prior_': [0.5650842266462481, 0.4349157733537519]}
fold_3 {'class_prior_': [0.5650775700604305, 0.4349224299395696]}
fold_4 {'class_prior_': [0.5650775700604305, 0.4349224299395696]}

matbench_perovskites

Fold scores
fold mae rmse mape* max_error
fold_0 0.5672 0.7361 0.7398 3.4868
fold_1 0.5742 0.7618 0.8046 3.3123
fold_2 0.5660 0.7438 0.7674 3.6873
fold_3 0.5614 0.7342 0.7738 3.3906
fold_4 0.5612 0.7362 0.7058 3.5084
Fold score stats
metric mean max min std
mae 0.5660 0.5742 0.5612 0.0048
rmse 0.7424 0.7618 0.7342 0.0102
mape* 0.7583 0.8046 0.7058 0.0334
max_error 3.4771 3.6873 3.3123 0.1264
Fold parameters
fold params dict
fold_0 {'constant_': [[1.4731871615374454]]}
fold_1 {'constant_': [[1.4677308149517898]]}
fold_2 {'constant_': [[1.4726720380398888]]}
fold_3 {'constant_': [[1.4694433071386122]]}
fold_4 {'constant_': [[1.4716264940896784]]}

matbench_phonons

Fold scores
fold mae rmse mape* max_error
fold_0 337.1003 542.7449 0.8225 3020.7169
fold_1 299.1209 452.2982 0.7977 2702.0312
fold_2 348.2576 545.4772 0.9223 3062.3450
fold_3 325.2402 480.9296 1.0268 3048.7920
fold_4 310.1921 439.3166 0.8936 1970.0884
Fold score stats
metric mean max min std
mae 323.9822 348.2576 299.1209 17.7269
rmse 492.1533 545.4772 439.3166 44.5176
mape* 0.8926 1.0268 0.7977 0.0810
max_error 2760.7947 3062.3450 1970.0884 417.1581
Fold parameters
fold params dict
fold_0 {'constant_': [[571.8686083004105]]}
fold_1 {'constant_': [[583.1997247898747]]}
fold_2 {'constant_': [[581.3984519265839]]}
fold_3 {'constant_': [[588.7935123141577]]}
fold_4 {'constant_': [[581.4972239423439]]}

matbench_steels

Fold scores
fold mae rmse mape* max_error
fold_0 241.4591 293.7245 0.1647 941.0643
fold_1 219.3770 289.2253 0.1550 1064.2831
fold_2 225.7932 291.5410 0.1600 1084.8760
fold_3 241.2035 343.9346 0.1567 1088.0568
fold_4 220.8898 287.6803 0.1576 983.3424
Fold score stats
metric mean max min std
mae 229.7445 241.4591 219.3770 9.6958
rmse 301.2211 343.9346 287.6803 21.4551
mape* 0.1588 0.1647 0.1550 0.0034
max_error 1032.3245 1088.0568 941.0643 59.3579
Fold parameters
fold params dict
fold_0 {'constant_': [[1415.3357429718874]]}
fold_1 {'constant_': [[1423.0168674698796]]}
fold_2 {'constant_': [[1425.424]]}
fold_3 {'constant_': [[1413.0431999999998]]}
fold_4 {'constant_': [[1428.1575999999998]]}