BioASQ Participants Area
Test Results for Task 10a
The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.
+ Test batch 1, week 1
Annotated articles:9628/9659
Flat Measures
System |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
MTI First Line Index |
0.6264 |
0.6799 |
0.6146 |
0.6264 |
0.6489 |
0.5773 |
0.5546 |
0.6676 |
0.5900 |
0.4729 |
Default MTI |
0.6306 |
0.6435 |
0.6516 |
0.6289 |
0.6228 |
0.6111 |
0.5689 |
0.6351 |
0.6263 |
0.4740 |
Dexstr system |
0.1673 |
0.3644 |
0.1206 |
0.1689 |
0.5833 |
0.1609 |
0.1768 |
0.3430 |
0.1107 |
0.0981 |
NLM System 2 |
0.6961 |
0.7275 |
0.6890 |
0.6906 |
0.7101 |
0.5678 |
0.5673 |
0.7241 |
0.6701 |
0.5426 |
deepmesh_dmiip_fdu |
0.6621 |
0.8166 |
0.5840 |
0.6630 |
0.7919 |
0.4937 |
0.5227 |
0.8182 |
0.5560 |
0.5120 |
attention_dmiip_fdu |
0.6896 |
0.7281 |
0.6763 |
0.6841 |
0.6899 |
0.5830 |
0.5747 |
0.7229 |
0.6593 |
0.5350 |
NLM CNN |
0.6481 |
0.6726 |
0.6431 |
0.6395 |
0.6202 |
0.4783 |
0.4764 |
0.6680 |
0.6292 |
0.4850 |
NLM System 1 |
0.6960 |
0.7281 |
0.6869 |
0.6901 |
0.7132 |
0.5658 |
0.5663 |
0.7251 |
0.6690 |
0.5421 |
Plain dict match |
0.2333 |
0.3171 |
0.2042 |
0.2311 |
0.3714 |
0.3498 |
0.2861 |
0.2926 |
0.1940 |
0.1369 |
XLinear model |
0.5496 |
0.7419 |
0.4526 |
0.5412 |
0.6963 |
0.3142 |
0.3478 |
0.7379 |
0.4378 |
0.3874 |
Hierarchical Measures
System |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |
MTI First Line Index |
0.5082 |
0.7902 |
0.7272 |
0.7392 |
0.5533 |
0.4966 |
Default MTI |
0.5106 |
0.7598 |
0.7609 |
0.7429 |
0.5286 |
0.5223 |
Dexstr system |
0.2228 |
0.6757 |
0.2540 |
0.3433 |
0.4164 |
0.1651 |
NLM System 2 |
0.5539 |
0.8211 |
0.7688 |
0.7777 |
0.5886 |
0.5486 |
deepmesh_dmiip_fdu |
0.5140 |
0.8874 |
0.6667 |
0.7433 |
0.6246 |
0.4585 |
attention_dmiip_fdu |
0.5473 |
0.8203 |
0.7595 |
0.7722 |
0.5857 |
0.5389 |
NLM CNN |
0.5150 |
0.7894 |
0.7291 |
0.7402 |
0.5519 |
0.5085 |
NLM System 1 |
0.5536 |
0.8221 |
0.7672 |
0.7775 |
0.5891 |
0.5474 |
Plain dict match |
0.2528 |
0.5484 |
0.4092 |
0.4427 |
0.3065 |
0.2395 |
XLinear model |
0.4227 |
0.8544 |
0.5395 |
0.6388 |
0.5718 |
0.3553 |
+ Test batch 1, week 2
Annotated articles:4528/4531
Flat Measures
System |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
MTI First Line Index |
0.6350 |
0.6778 |
0.6412 |
0.6388 |
0.6235 |
0.5905 |
0.5603 |
0.6614 |
0.6106 |
0.4890 |
Default MTI |
0.6323 |
0.6264 |
0.6779 |
0.6303 |
0.5860 |
0.6268 |
0.5746 |
0.6170 |
0.6483 |
0.4761 |
Dexstr system |
0.3343 |
0.3166 |
0.4231 |
0.3294 |
0.3503 |
0.4367 |
0.3510 |
0.2858 |
0.4025 |
0.2043 |
NLM System 2 |
0.6971 |
0.7103 |
0.7131 |
0.6911 |
0.6856 |
0.5854 |
0.5740 |
0.7058 |
0.6887 |
0.5446 |
deepmesh_dmiip_fdu |
0.6756 |
0.7988 |
0.6168 |
0.6759 |
0.7720 |
0.5097 |
0.5284 |
0.8000 |
0.5847 |
0.5280 |
attention_dmiip_fdu |
0.6951 |
0.7155 |
0.7017 |
0.6885 |
0.6769 |
0.5984 |
0.5836 |
0.7095 |
0.6813 |
0.5413 |
NLM CNN |
0.6498 |
0.6616 |
0.6622 |
0.6406 |
0.6032 |
0.4967 |
0.4862 |
0.6553 |
0.6444 |
0.4871 |
NLM System 1 |
0.6986 |
0.7136 |
0.7090 |
0.6917 |
0.6946 |
0.5786 |
0.5701 |
0.7102 |
0.6873 |
0.5453 |
Plain dict match |
0.2294 |
0.3030 |
0.2060 |
0.2276 |
0.3490 |
0.3545 |
0.2913 |
0.2789 |
0.1948 |
0.1349 |
XLinear model |
0.5577 |
0.7324 |
0.4683 |
0.5474 |
0.6826 |
0.3261 |
0.3514 |
0.7270 |
0.4523 |
0.3950 |
combination system |
0.2466 |
0.2931 |
0.2408 |
0.2464 |
0.3159 |
0.3975 |
0.3143 |
0.2679 |
0.2285 |
0.1465 |
consensus system |
0.1161 |
0.5388 |
0.0702 |
0.1184 |
0.4564 |
0.0997 |
0.0876 |
0.4693 |
0.0663 |
0.0666 |
Hierarchical Measures
System |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |
MTI First Line Index |
0.5178 |
0.7807 |
0.7468 |
0.7445 |
0.5503 |
0.5169 |
Default MTI |
0.5088 |
0.7404 |
0.7813 |
0.7415 |
0.5117 |
0.5374 |
Dexstr system |
0.3108 |
0.4814 |
0.6398 |
0.5137 |
0.2855 |
0.3917 |
NLM System 2 |
0.5533 |
0.8027 |
0.7895 |
0.7777 |
0.5751 |
0.5627 |
deepmesh_dmiip_fdu |
0.5257 |
0.8727 |
0.6977 |
0.7566 |
0.6154 |
0.4826 |
attention_dmiip_fdu |
0.5496 |
0.8076 |
0.7819 |
0.7765 |
0.5752 |
0.5543 |
NLM CNN |
0.5133 |
0.7764 |
0.7454 |
0.7404 |
0.5406 |
0.5187 |
NLM System 1 |
0.5540 |
0.8067 |
0.7865 |
0.7788 |
0.5775 |
0.5611 |
Plain dict match |
0.2511 |
0.5314 |
0.4116 |
0.4384 |
0.2960 |
0.2433 |
XLinear model |
0.4267 |
0.8447 |
0.5549 |
0.6459 |
0.5632 |
0.3650 |
combination system |
0.2542 |
0.5092 |
0.4493 |
0.4532 |
0.2800 |
0.2608 |
consensus system |
0.1176 |
0.6858 |
0.1226 |
0.1965 |
0.3408 |
0.0743 |
+ Test batch 1, week 3
Annotated articles:4291/4291
Flat Measures
System |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
MTI First Line Index |
0.6324 |
0.6953 |
0.6186 |
0.6360 |
0.6637 |
0.5850 |
0.5636 |
0.6807 |
0.5905 |
0.4848 |
Default MTI |
0.6293 |
0.6431 |
0.6540 |
0.6283 |
0.6172 |
0.6125 |
0.5717 |
0.6324 |
0.6262 |
0.4728 |
Dexstr system |
0.3492 |
0.3251 |
0.4083 |
0.3477 |
0.3826 |
0.4293 |
0.3578 |
0.3168 |
0.3890 |
0.2169 |
NLM System 2 |
0.6840 |
0.7133 |
0.6844 |
0.6795 |
0.6943 |
0.5770 |
0.5693 |
0.7051 |
0.6642 |
0.5291 |
deepmesh_dmiip_fdu |
0.6797 |
0.7172 |
0.6714 |
0.6744 |
0.6846 |
0.5823 |
0.5735 |
0.7075 |
0.6540 |
0.5240 |
attention_dmiip_fdu |
0.6556 |
0.8088 |
0.5790 |
0.6567 |
0.7878 |
0.4944 |
0.5180 |
0.8067 |
0.5522 |
0.5043 |
NLM CNN |
0.6364 |
0.6686 |
0.6326 |
0.6298 |
0.6209 |
0.4831 |
0.4792 |
0.6572 |
0.6168 |
0.4744 |
NLM System 1 |
0.6814 |
0.7121 |
0.6792 |
0.6765 |
0.7007 |
0.5706 |
0.5657 |
0.7032 |
0.6609 |
0.5260 |
XLinear model |
0.5424 |
0.7341 |
0.4495 |
0.5357 |
0.6901 |
0.3237 |
0.3504 |
0.7275 |
0.4324 |
0.3818 |
Hierarchical Measures
System |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |