BioASQ Participants Area
Test Results for Task 6a
The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.
+ Dry Run 1
Annotated articles:27480/28563
Flat Measures
System |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
deep learning only |
0.5614 |
0.5704 |
0.5609 |
0.5439 |
0.5710 |
0.3180 |
0.3336 |
0.5683 |
0.5547 |
0.3881 |
Hierarchical Measures
System |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |
deep learning only |
0.4616 |
0.7414 |
0.6591 |
0.6746 |
0.5056 |
0.4525 |
+ Dry Run 2
Annotated articles:198/255
Flat Measures
System |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
MTI First Line Index |
0.6844 |
0.7010 |
0.6873 |
0.6751 |
0.6294 |
0.6275 |
0.6109 |
0.6898 |
0.6791 |
0.5348 |
Default MTI |
0.6797 |
0.6454 |
0.7501 |
0.6762 |
0.5706 |
0.6779 |
0.6493 |
0.6334 |
0.7334 |
0.5298 |
Hierarchical Measures
System |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |
MTI First Line Index |
0.5559 |
0.8129 |
0.7777 |
0.7778 |
0.5882 |
0.5573 |
Default MTI |
0.5545 |
0.7574 |
0.8298 |
0.7763 |
0.5449 |
0.5927 |
+ Test batch 1, week 1
Annotated articles:6639/7240
Flat Measures
System |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
MTI First Line Index |
0.6134 |
0.6654 |
0.5868 |
0.6022 |
0.6204 |
0.5367 |
0.5159 |
0.6553 |
0.5766 |
0.4520 |
Default MTI |
0.6231 |
0.6264 |
0.6424 |
0.6151 |
0.5881 |
0.5828 |
0.5414 |
0.6222 |
0.6240 |
0.4640 |
iria-1 |
0.4992 |
0.4331 |
0.5873 |
0.4846 |
0.4053 |
0.4603 |
0.4060 |
0.4402 |
0.5765 |
0.3340 |
iria-2 |
0.4961 |
0.4310 |
0.5830 |
0.4816 |
0.3992 |
0.4556 |
0.4008 |
0.4380 |
0.5721 |
0.3313 |
DeepMeSH1 |
0.6360 |
0.6652 |
0.6180 |
0.6228 |
0.6369 |
0.4867 |
0.4851 |
0.6698 |
0.6054 |
0.4697 |
DeepMeSH2 |
0.6565 |
0.6858 |
0.6426 |
0.6459 |
0.6588 |
0.5344 |
0.5274 |
0.6894 |
0.6266 |
0.4954 |
DeepMeSH3 |
0.6434 |
0.6758 |
0.6255 |
0.6316 |
0.6406 |
0.5040 |
0.4997 |
0.6793 |
0.6111 |
0.4789 |
DeepMeSH4 |
0.6573 |
0.6888 |
0.6414 |
0.6465 |
0.6592 |
0.5357 |
0.5281 |
0.6917 |
0.6261 |
0.4962 |
xgx |
0.5348 |
0.6013 |
0.4860 |
0.5184 |
0.5415 |
0.3318 |
0.3462 |
0.5997 |
0.4826 |
0.3684 |
AUTH-Atypon 1 |
0.5593 |
0.6199 |
0.5109 |
0.5363 |
0.5918 |
0.3467 |
0.3534 |
0.6183 |
0.5106 |
0.3843 |
Hierarchical Measures
System |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |