BioASQ Participants Area
Test Results for Task 8a
The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.
+ Dry Run 1
Annotated articles:9676/9823
Flat Measures
System |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
BioASQ_Baseline |
0.0003 |
0.0006 |
0.0002 |
0.0002 |
0.0004 |
0.0003 |
0.0002 |
0.0004 |
0.0002 |
0.0001 |
Hierarchical Measures
System |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |
BioASQ_Baseline |
0.0991 |
0.1488 |
0.0898 |
0.0994 |
0.1486 |
0.0867 |
+ Test batch 1, week 1
Annotated articles:6487/6510
Flat Measures
System |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
MTI First Line Index |
0.6495 |
0.6808 |
0.6468 |
0.6451 |
0.6360 |
0.5899 |
0.5624 |
0.6684 |
0.6316 |
0.4953 |
Default MTI |
0.6501 |
0.6436 |
0.6810 |
0.6448 |
0.6039 |
0.6207 |
0.5738 |
0.6361 |
0.6647 |
0.4925 |
Transformer-inspired |
0.3669 |
0.3611 |
0.3586 |
0.3440 |
0.2285 |
0.0128 |
0.0078 |
0.3592 |
0.3750 |
0.2179 |
deepmesh_dmiip_fdu |
0.7049 |
0.7848 |
0.6567 |
0.7011 |
0.7460 |
0.5653 |
0.5753 |
0.7869 |
0.6383 |
0.5559 |
deepmesh_dmiip_fdu_ |
0.3998 |
0.7839 |
0.2691 |
0.3802 |
0.7218 |
0.0457 |
0.0613 |
0.7856 |
0.2682 |
0.2493 |
attention_dmiip_fdu |
0.6963 |
0.7132 |
0.6957 |
0.6896 |
0.6730 |
0.5709 |
0.5564 |
0.7067 |
0.6862 |
0.5419 |
Hierarchical Measures
System |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |
MTI First Line Index |
0.5295 |
0.7897 |
0.7421 |
0.7476 |
0.5628 |
0.5256 |
Default MTI |
0.5291 |
0.7596 |
0.7747 |
0.7509 |
0.5369 |
0.5483 |
Transformer-inspired |
0.3305 |
0.5461 |
0.4438 |
0.4676 |
0.3817 |
0.3125 |
deepmesh_dmiip_fdu |
0.5646 |
0.8638 |
0.7383 |
0.7817 |
0.6294 |
0.5316 |
deepmesh_dmiip_fdu_ |
0.2851 |
0.8818 |
0.2964 |
0.4182 |
0.5736 |
0.2017 |
attention_dmiip_fdu |
0.5642 |
0.8120 |
0.7793 |
0.7807 |
0.5879 |
0.5651 |
+ Test batch 1, week 2
Annotated articles:7074/7126
Flat Measures
System |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
MTI First Line Index |
0.6584 |
0.6824 |
0.6536 |
0.6495 |
0.6351 |
0.5925 |
0.5623 |
0.6749 |
0.6426 |
0.5009 |
Default MTI |
0.6593 |
0.6490 |
0.6881 |
0.6506 |
0.6073 |
0.6259 |
0.5779 |
0.6440 |
0.6753 |
0.5000 |
deepmesh_dmiip_fdu |
0.7059 |
0.7816 |
0.6555 |
0.6984 |
0.7337 |
0.5590 |
0.5668 |
0.7854 |
0.6409 |
0.5538 |
deepmesh_dmiip_fdu_ |
0.6684 |
0.7475 |
0.6174 |
0.6606 |
0.6969 |
0.5047 |
0.5155 |
0.7515 |
0.6018 |
0.5099 |
attention_dmiip_fdu |
0.6957 |
0.7092 |
0.6933 |
0.6863 |
0.6591 |
0.5711 |
0.5519 |
0.7051 |
0.6867 |
0.5390 |
Hierarchical Measures
System |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |