BioASQ Participants Area
Evaluation Overview
In the following table you can see the number of the articles for each test set that has been released. In parentheses, there is the number of the articles that have
been annotated by the curators for each test set.
| Batch 1 | Batch 2 | Batch 3 |
Week 1 | 4440 (3319) | 4085 (3422) | 4342 (3009) |
Week 2 | 4721 (3734) | 3496 (2788) | 8840 (5883) |
Week 3 | 4802 (3884) | 4524 (3274) | 3702 (2860) |
Week 4 | 3579 (2431) | 5407 (3923) | 4726 (3252) |
Week 5 | 5299 (3693) | 5454 (3666) | 4533 (3252) |
Test Results for Task 2a
The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.
+ Dry-run test
Annotated articles:2703/3186
Flat Measures
System Name |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
BioASQ_Baseline |
0.2510 |
0.2516 |
0.2893 |
0.2499 |
0.3592 |
0.3845 |
0.3182 |
0.2296 |
0.2767 |
0.1484 |
MTI First Line Index |
0.5624 |
0.6087 |
0.5473 |
0.5535 |
0.5675 |
0.4858 |
0.4633 |
0.6023 |
0.5274 |
0.3998 |
MeSH Indexing |
0.5675 |
0.6155 |
0.5629 |
0.5671 |
0.5722 |
0.4671 |
0.4551 |
0.6132 |
0.5281 |
0.4116 |
Default MTI |
0.5673 |
0.5721 |
0.5908 |
0.5595 |
0.5381 |
0.5273 |
0.4889 |
0.5647 |
0.5699 |
0.4040 |
Macro |
0.1033 |
0.6722 |
0.0639 |
0.1149 |
0.6722 |
0.0001 |
0.0001 |
0.6722 |
0.0560 |
0.0639 |
Micro |
0.1033 |
0.6722 |
0.0639 |
0.1149 |
0.6722 |
0.0001 |
0.0001 |
0.6722 |
0.0560 |
0.0639 |
PerExample |
0.1033 |
0.6722 |
0.0639 |
0.1149 |
0.6722 |
0.0001 |
0.0001 |
0.6722 |
0.0560 |
0.0639 |
spoon-spoon |
0.3678 |
0.3562 |
0.4103 |
0.3557 |
0.4423 |
0.3241 |
0.3045 |
0.3402 |
0.4002 |
0.2271 |
fork-fork |
0.3755 |
0.4382 |
0.3531 |
0.3609 |
0.5331 |
0.2736 |
0.2750 |
0.4167 |
0.3416 |
0.2328 |
spork-spork |
0.3716 |
0.3891 |
0.3837 |
0.3586 |
0.4775 |
0.2996 |
0.2899 |
0.3703 |
0.3729 |
0.2301 |
EO_Sys1 |
0.4538 |
0.5471 |
0.5060 |
0.4963 |
0.3787 |
0.2869 |
0.2420 |
0.4118 |
0.5054 |
0.3469 |
Limited sample |
0.4506 |
0.5467 |
0.4992 |
0.4923 |
0.3701 |
0.2806 |
0.2370 |
0.4104 |
0.4997 |
0.3433 |
Mixing with MTI |
0.5060 |
0.4937 |
0.6908 |
0.5550 |
0.4305 |
0.5841 |
0.4773 |
0.4041 |
0.6767 |
0.3978 |
Hierarchical Measures
System Name |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |
BioASQ_Baseline |
0.3044 |
0.5320 |
0.5566 |
0.5120 |
0.3261 |
0.3181 |
MTI First Line Index |
0.4694 |
0.7532 |
0.6736 |
0.6829 |
0.5203 |
0.4603 |
MeSH Indexing |
0.4784 |
0.7601 |
0.6833 |
0.6946 |
0.5313 |
0.4656 |
Default MTI |
0.4782 |
0.7215 |
0.7210 |
0.6954 |
0.4989 |
0.4938 |
Macro |
0.0845 |
0.6955 |
0.0521 |
0.0951 |
0.2803 |
0.0512 |
Micro |
0.0845 |
0.6955 |
0.0521 |
0.0951 |
0.2803 |
0.0512 |
PerExample |
0.0845 |
0.6955 |
0.0521 |
0.0951 |
0.2803 |
0.0512 |
spoon-spoon |
0.3393 |
0.5673 |
0.5425 |
0.5198 |
0.3524 |
0.3694 |
fork-fork |
0.3337 |
0.6555 |
0.4522 |
0.4967 |
0.4135 |
0.3141 |
spork-spork |
0.3362 |
0.5994 |
0.5009 |
0.5098 |
0.3742 |
0.3438 |
EO_Sys1 |
0.4247 |
0.7087 |
0.6104 |
0.6173 |
0.4797 |
0.4214 |
Limited sample |
0.4209 |
0.7075 |
0.6058 |
0.6141 |
0.4783 |
0.4162 |
Mixing with MTI |
0.4791 |
0.6548 |
0.7983 |
0.6946 |
0.4454 |
0.5636 |
+ Test batch 1, week 1
Annotated articles:3319/4440
Flat Measures
System Name |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
BioASQ_Baseline |
0.2549 |
0.2576 |
0.2864 |
0.2530 |
0.3769 |
0.3786 |
0.3158 |
0.2375 |
0.2750 |
0.1501 |
MTI First Line Index |
0.5605 |
0.6213 |
0.5368 |
0.5557 |
0.5803 |
0.4745 |
0.4558 |
0.6132 |
0.5162 |
0.3994 |
Default MTI |
0.5704 |
0.5895 |
0.5797 |
0.5659 |
0.5490 |
0.5166 |
0.4813 |
0.5828 |
0.5585 |
0.4082 |
FU System |
0.4499 |
0.3444 |
0.6714 |
0.4398 |
0.3393 |
0.4932 |
0.3889 |
0.3441 |
0.6496 |
0.2913 |
FU_System_t25 |
0.4529 |
0.3467 |
0.6763 |
0.4428 |
0.3428 |
0.5008 |
0.3961 |
0.3464 |
0.6538 |
0.2937 |
FU_System_t15 |
0.5056 |
0.4753 |
0.5668 |
0.4980 |
0.4725 |
0.3778 |
0.3533 |
0.4757 |
0.5395 |
0.3444 |
Macro |
0.1101 |
0.7798 |
0.0690 |
0.1248 |
0.7798 |
0.0001 |
0.0001 |
0.7798 |
0.0592 |
0.0690 |
Micro |
0.1101 |
0.7798 |
0.0690 |
0.1248 |
0.7798 |
0.0001 |
0.0001 |
0.7798 |
0.0592 |
0.0690 |
PerExample |
0.1101 |
0.7798 |
0.0690 |
0.1248 |
0.7798 |
0.0001 |
0.0001 |
0.7798 |
0.0592 |
0.0690 |
spoon-spoon |
0.3847 |
0.3680 |
0.4449 |
0.3769 |
0.4469 |
0.3480 |
0.3264 |
0.3478 |
0.4304 |
0.2418 |
fork-fork |
0.3944 |
0.4500 |
0.3846 |
0.3849 |
0.5395 |
0.2947 |
0.2987 |
0.4234 |
0.3690 |
0.2495 |
spork-spork |
0.3894 |
0.4001 |
0.4174 |
0.3804 |
0.4833 |
0.3245 |
0.3152 |
0.3769 |
0.4027 |
0.2451 |
EO_Sys1 |
0.5270 |
0.5647 |
0.5104 |
0.5099 |
0.5234 |
0.2790 |
0.2655 |
0.5473 |
0.5083 |
0.3567 |
FU_System_k15 |
0.5198 |
0.4886 |
0.5845 |
0.5127 |
0.4473 |
0.4314 |
0.3963 |
0.4890 |
0.5546 |
0.3574 |
FU_System_k25 |
0.4645 |
0.3555 |
0.6951 |
0.4544 |
0.3119 |
0.5611 |
0.4307 |
0.3553 |
0.6707 |
0.3033 |
Limited sample |
0.5214 |
0.5629 |
0.5008 |
0.5030 |
0.5111 |
0.2701 |
0.2562 |
0.5450 |
0.4997 |
0.3507 |
Mixing with MTI |
0.5726 |
0.5089 |
0.6872 |
0.5665 |
0.4978 |
0.5728 |
0.5038 |
0.4989 |
0.6720 |
0.4070 |
Asclepius |
0.5890 |
0.5882 |
0.6060 |
0.5814 |
0.5961 |
0.4434 |
0.4323 |
0.5904 |
0.5876 |
0.4241 |
Antinomyra SYS3 |
0.5346 |
0.5427 |
0.5409 |
0.5257 |
0.6215 |
0.2530 |
0.2652 |
0.5421 |
0.5272 |
0.3709 |
Antinomyra SYS4 |
0.5345 |
0.5401 |
0.5447 |
0.5262 |
0.5838 |
0.2791 |
0.2885 |
0.5399 |
0.5292 |
0.3712 |
L2R-n1 |
0.5671 |
0.6490 |
0.5422 |
0.5702 |
0.6019 |
0.4328 |
0.4324 |
0.6470 |
0.5047 |
0.4129 |
L2R-n2 |
0.5718 |
0.6361 |
0.5576 |
0.5733 |
0.5891 |
0.4482 |
0.4422 |
0.6322 |
0.5219 |
0.4159 |
L2R-n3 |
0.5659 |
0.6642 |
0.5312 |
0.5692 |
0.6144 |
0.4233 |
0.4259 |
0.6588 |
0.4960 |
0.4122 |
L2R-n4 |
0.0799 |
0.0904 |
0.0733 |
0.0778 |
0.0620 |
0.0655 |
0.0598 |
0.0895 |
0.0722 |
0.0425 |
L2R-n5 |
0.0918 |
0.0994 |
0.0830 |
0.0871 |
0.0621 |
0.0657 |
0.0600 |
0.1017 |
0.0836 |
0.0481 |
Hierarchical Measures
System Name |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |
BioASQ_Baseline |
0.3074 |
0.5517 |
0.5617 |
0.5278 |
0.3341 |
0.3140 |
MTI First Line Index |
0.4729 |
0.7692 |
0.6634 |
0.6882 |
0.5345 |
0.4529 |
Default MTI |
0.4840 |
0.7425 |
0.7077 |
0.7029 |
0.5151 |
0.4863 |
FU System |
0.4094 |
0.5320 |
0.7947 |
0.6153 |
0.3445 |
0.5468 |
FU_System_t25 |
0.4120 |
0.5368 |
0.7974 |
0.6197 |
0.3468 |
0.5493 |
FU_System_t15 |
0.4394 |
0.6637 |
0.6813 |
0.6487 |
0.4430 |
0.4688 |
Macro |
0.0847 |
0.7994 |
0.0546 |
0.1007 |
0.2976 |
0.0505 |
Micro |
0.0847 |
0.7994 |
0.0546 |
0.1007 |
0.2976 |
0.0505 |
PerExample |
0.0847 |
0.7994 |
0.0546 |
0.1007 |
0.2976 |
0.0505 |
spoon-spoon |
0.3582 |
0.5821 |
0.6017 |
0.5598 |
0.3607 |
0.3983 |
fork-fork |
0.3541 |
0.6695 |
0.5122 |
0.5449 |
0.4201 |
0.3409 |
spork-spork |
0.3557 |
0.6154 |
0.5623 |
0.5537 |
0.3821 |
0.3728 |
EO_Sys1 |
0.4364 |
0.7310 |
0.6122 |
0.6336 |
0.4946 |
0.4253 |
FU_System_k15 |
0.4504 |
0.6727 |
0.6983 |
0.6615 |
0.4535 |
0.4810 |
FU_System_k25 |
0.4224 |
0.5474 |
0.8125 |
0.6322 |
0.3555 |
0.5625 |
Limited sample |
0.4314 |
0.7306 |
0.6046 |
0.6274 |
0.4955 |
0.4175 |
Mixing with MTI |
0.4903 |
0.6772 |
0.7954 |
0.7108 |
0.4602 |
0.5622 |
Asclepius |
0.4951 |
0.7384 |
0.7298 |
0.7152 |
0.5118 |
0.5066 |
Antinomyra SYS3 |
0.4474 |
0.7281 |
0.6385 |
0.6592 |
0.4894 |
0.4363 |
Antinomyra SYS4 |
0.4504 |
0.7216 |
0.6454 |
0.6603 |
0.4887 |
0.4423 |
L2R-n1 |
0.4787 |
0.7945 |
0.6473 |
0.6887 |
0.5578 |
0.4476 |
L2R-n2 |
0.4833 |
0.7841 |
0.6650 |
0.6951 |
0.5504 |
0.4605 |
L2R-n3 |
0.4762 |
0.8064 |
0.6350 |
0.6850 |
0.5670 |
0.4384 |
L2R-n4 |
0.2235 |
0.3887 |
0.3956 |
0.3755 |
0.2385 |
0.2233 |
L2R-n5 |
0.2268 |
0.3899 |
0.3983 |
0.3774 |
0.2407 |
0.2274 |
+ Test batch 1, week 2
Annotated articles:3734/4721
Flat Measures
System Name |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
BioASQ_Baseline |
0.2382 |
0.4054 |
0.2253 |
0.2197 |
0.3747 |
0.2919 |
0.2572 |
0.2561 |
0.2227 |
0.1288 |
MTI First Line Index |
0.5596 |
0.6268 |
0.5267 |
0.5521 |
0.5824 |
0.4809 |
0.4605 |
0.6185 |
0.5109 |
0.3961 |
Default MTI |
0.5709 |
0.5924 |
0.5755 |
0.5651 |
0.5441 |
0.5233 |
0.4836 |
0.5848 |
0.5576 |
0.4084 |
FU System |
0.4541 |
0.3461 |
0.6788 |
0.4445 |
0.3438 |
0.5043 |
0.3948 |
0.3461 |
0.6598 |
0.2954 |
FU_System_t25 |
0.4569 |
0.3483 |
0.6837 |
0.4474 |
0.3470 |
0.5145 |
0.4031 |
0.3483 |
0.6639 |
0.2976 |
FU_System_t15 |
0.5232 |
0.4903 |
0.5879 |
0.5159 |
0.4388 |
0.4535 |
0.4117 |
0.4903 |
0.5608 |
0.3605 |
Macro |
0.1018 |
0.7185 |
0.0610 |
0.1109 |
0.7185 |
0.0001 |
0.0001 |
0.7185 |
0.0548 |
0.0610 |
Micro |
0.1018 |
0.7185 |
0.0610 |
0.1109 |
0.7185 |
0.0001 |
0.0001 |
0.7185 |
0.0548 |
0.0610 |
PerExample |
0.1018 |
0.7185 |
0.0610 |
0.1109 |
0.7185 |
0.0001 |
0.0001 |
0.7185 |
0.0548 |
0.0610 |
spoon-spoon |
0.3817 |
0.3582 |
0.4340 |
0.3684 |
0.4468 |
0.3556 |
0.3319 |
0.3453 |
0.4267 |
0.2352 |
fork-fork |
0.3524 |
0.4961 |
0.2834 |
0.3381 |
0.4400 |
0.1225 |
0.1359 |
0.4836 |
0.2772 |
0.2175 |
spork-spork |
0.1018 |
0.7185 |
0.0610 |
0.1109 |
0.7185 |
0.0001 |
0.0001 |
0.7185 |
0.0548 |
0.0610 |
EO_Sys1 |
0.5325 |
0.5814 |
0.5043 |
0.5140 |
0.5341 |
0.2864 |
0.2730 |
0.5619 |
0.5060 |
0.3607 |
FU_System_k15 |
0.5226 |
0.4898 |
0.5873 |
0.5153 |
0.4433 |
0.4499 |
0.4095 |
0.4898 |
0.5602 |
0.3600 |
FU_System_k25 |
0.4657 |
0.3550 |
0.6987 |
0.4564 |
0.3042 |
0.5730 |
0.4341 |
0.3550 |
0.6768 |
0.3049 |
Limited sample |
0.5251 |
0.5788 |
0.4921 |
0.5050 |
0.5211 |
0.2779 |
0.2636 |
0.5598 |
0.4945 |
0.3528 |
Mixing with MTI |
0.5778 |
0.5194 |
0.6807 |
0.5711 |
0.4971 |
0.5812 |
0.5083 |
0.5079 |
0.6702 |
0.4124 |
Asclepius |
0.0040 |
0.0037 |
0.0035 |
0.0035 |
0.0019 |
0.0010 |
0.0007 |
0.0041 |
0.0040 |
0.0018 |
Antinomyra SYS3 |
0.5473 |
0.5985 |
0.5143 |
0.5353 |
0.6170 |
0.2607 |
0.2733 |
0.5983 |
0.5042 |
0.3801 |
Antinomyra SYS4 |
0.5351 |
0.5464 |
0.5357 |
0.5249 |
0.5868 |
0.2860 |
0.2957 |
0.5461 |
0.5246 |
0.3695 |
L2R-n1 |
0.5844 |
0.6231 |
0.5793 |
0.5808 |
0.5719 |
0.4918 |
0.4726 |
0.6156 |
0.5562 |
0.4238 |
L2R-n2 |
0.5857 |
0.5842 |
0.6151 |
0.5805 |
0.5369 |
0.5286 |
0.4905 |
0.5766 |
0.5951 |
0.4231 |
L2R-n3 |
0.5812 |
0.6417 |
0.5619 |
0.5784 |
0.5889 |
0.4748 |
0.4628 |
0.6317 |
0.5381 |
0.4216 |
L2R-n4 |
0.5770 |
0.6390 |
0.5591 |
0.5767 |
0.5918 |
0.4610 |
0.4531 |
0.6350 |
0.5287 |
0.4199 |
L2R-n5 |
0.5750 |
0.6483 |
0.5520 |
0.5749 |
0.6008 |
0.4527 |
0.4492 |
0.6412 |
0.5213 |
0.4177 |
Antinomyra SYS2 |
0.5378 |
0.5532 |
0.5345 |
0.5277 |
0.6269 |
0.2650 |
0.2786 |
0.5521 |
0.5242 |
0.3720 |
FDU_MeSHIndexing_1 |
0.5429 |
0.5453 |
0.5715 |
0.5379 |
0.5203 |
0.4441 |
0.4242 |
0.5453 |
0.5406 |
0.3811 |
FDU_MeSHIndexing_2 |
0.5430 |
0.5454 |
0.5716 |
0.5381 |
0.5162 |
0.4431 |
0.4230 |
0.5454 |
0.5406 |
0.3813 |
FDU_MeSHIndexing_3 |
0.5426 |
0.5450 |
0.5705 |
0.5374 |
0.5250 |
0.4290 |
0.4103 |
0.5450 |
0.5402 |
0.3808 |
FDU_MeSHIndexing_4 |
0.5428 |
0.5452 |
0.5706 |
0.5376 |
0.5270 |
0.4313 |
0.4126 |
0.5452 |
0.5405 |
0.3809 |
FDU_MeSHIndexing_5 |
0.5397 |
0.5420 |
0.5680 |
0.5347 |
0.5175 |
0.4355 |
0.4147 |
0.5420 |
0.5373 |
0.3783 |
Hierarchical Measures
System Name |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |