BioASQ Participants Area
Evaluation Overview
In the following table you can see the number of the articles for each test set that has been released. In parentheses, there is the number of the articles that have
been annotated by the curators for each test set.
| Batch 1 | Batch 2 | Batch 3 |
Week 1 | 4440 (3319) | 4085 (3422) | 4342 (3009) |
Week 2 | 4721 (3734) | 3496 (2788) | 8840 (5883) |
Week 3 | 4802 (3884) | 4524 (3274) | 3702 (2860) |
Week 4 | 3579 (2431) | 5407 (3923) | 4726 (3252) |
Week 5 | 5299 (3693) | 5454 (3666) | 4533 (3252) |
Test Results for Task 2a
The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.
+ Dry-run test
Annotated articles:2703/3186
Flat Measures
System Name |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
BioASQ_Baseline |
0.2510 |
0.2516 |
0.2893 |
0.2499 |
0.3592 |
0.3845 |
0.3182 |
0.2296 |
0.2767 |
0.1484 |
MTI First Line Index |
0.5624 |
0.6087 |
0.5473 |
0.5535 |
0.5675 |
0.4858 |
0.4633 |
0.6023 |
0.5274 |
0.3998 |
MeSH Indexing |
0.5675 |
0.6155 |
0.5629 |
0.5671 |
0.5722 |
0.4671 |
0.4551 |
0.6132 |
0.5281 |
0.4116 |
Default MTI |
0.5673 |
0.5721 |
0.5908 |
0.5595 |
0.5381 |
0.5273 |
0.4889 |
0.5647 |
0.5699 |
0.4040 |
Macro |
0.1033 |
0.6722 |
0.0639 |
0.1149 |
0.6722 |
0.0001 |
0.0001 |
0.6722 |
0.0560 |
0.0639 |
Micro |
0.1033 |
0.6722 |
0.0639 |
0.1149 |
0.6722 |
0.0001 |
0.0001 |
0.6722 |
0.0560 |
0.0639 |
PerExample |
0.1033 |
0.6722 |
0.0639 |
0.1149 |
0.6722 |
0.0001 |
0.0001 |
0.6722 |
0.0560 |
0.0639 |
spoon-spoon |
0.3678 |
0.3562 |
0.4103 |
0.3557 |
0.4423 |
0.3241 |
0.3045 |
0.3402 |
0.4002 |
0.2271 |
fork-fork |
0.3755 |
0.4382 |
0.3531 |
0.3609 |
0.5331 |
0.2736 |
0.2750 |
0.4167 |
0.3416 |
0.2328 |
spork-spork |
0.3716 |
0.3891 |
0.3837 |
0.3586 |
0.4775 |
0.2996 |
0.2899 |
0.3703 |
0.3729 |
0.2301 |
EO_Sys1 |
0.4538 |
0.5471 |
0.5060 |
0.4963 |
0.3787 |
0.2869 |
0.2420 |
0.4118 |
0.5054 |
0.3469 |
Limited sample |
0.4506 |
0.5467 |
0.4992 |
0.4923 |
0.3701 |
0.2806 |
0.2370 |
0.4104 |
0.4997 |
0.3433 |
Mixing with MTI |
0.5060 |
0.4937 |
0.6908 |
0.5550 |
0.4305 |
0.5841 |
0.4773 |
0.4041 |
0.6767 |
0.3978 |
Hierarchical Measures
System Name |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |
BioASQ_Baseline |
0.3044 |
0.5320 |
0.5566 |
0.5120 |
0.3261 |
0.3181 |
MTI First Line Index |
0.4694 |
0.7532 |
0.6736 |
0.6829 |
0.5203 |
0.4603 |
MeSH Indexing |
0.4784 |
0.7601 |
0.6833 |
0.6946 |
0.5313 |
0.4656 |
Default MTI |
0.4782 |
0.7215 |
0.7210 |
0.6954 |
0.4989 |
0.4938 |
Macro |
0.0845 |
0.6955 |
0.0521 |
0.0951 |
0.2803 |
0.0512 |
Micro |
0.0845 |
0.6955 |
0.0521 |
0.0951 |
0.2803 |
0.0512 |
PerExample |
0.0845 |
0.6955 |
0.0521 |
0.0951 |
0.2803 |
0.0512 |
spoon-spoon |
0.3393 |
0.5673 |
0.5425 |
0.5198 |
0.3524 |
0.3694 |
fork-fork |
0.3337 |
0.6555 |
0.4522 |
0.4967 |
0.4135 |
0.3141 |
spork-spork |
0.3362 |
0.5994 |
0.5009 |
0.5098 |
0.3742 |
0.3438 |
EO_Sys1 |
0.4247 |
0.7087 |
0.6104 |
0.6173 |
0.4797 |
0.4214 |
Limited sample |
0.4209 |
0.7075 |
0.6058 |
0.6141 |
0.4783 |
0.4162 |
Mixing with MTI |
0.4791 |
0.6548 |
0.7983 |
0.6946 |
0.4454 |
0.5636 |
+ Test batch 1, week 1
Annotated articles:3319/4440
Flat Measures
System Name |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
BioASQ_Baseline |
0.2549 |
0.2576 |
0.2864 |
0.2530 |
0.3769 |
0.3786 |
0.3158 |
0.2375 |
0.2750 |
0.1501 |
MTI First Line Index |
0.5605 |
0.6213 |
0.5368 |
0.5557 |
0.5803 |
0.4745 |
0.4558 |
0.6132 |
0.5162 |
0.3994 |
Default MTI |
0.5704 |
0.5895 |
0.5797 |
0.5659 |
0.5490 |
0.5166 |
0.4813 |
0.5828 |
0.5585 |
0.4082 |
FU System |
0.4499 |
0.3444 |
0.6714 |
0.4398 |
0.3393 |
0.4932 |
0.3889 |
0.3441 |
0.6496 |
0.2913 |
FU_System_t25 |
0.4529 |
0.3467 |
0.6763 |
0.4428 |
0.3428 |
0.5008 |
0.3961 |
0.3464 |
0.6538 |
0.2937 |
FU_System_t15 |
0.5056 |
0.4753 |
0.5668 |
0.4980 |
0.4725 |
0.3778 |
0.3533 |
0.4757 |
0.5395 |
0.3444 |
Macro |
0.1101 |
0.7798 |
0.0690 |
0.1248 |
0.7798 |
0.0001 |
0.0001 |
0.7798 |
0.0592 |
0.0690 |
Micro |
0.1101 |
0.7798 |
0.0690 |
0.1248 |
0.7798 |
0.0001 |
0.0001 |
0.7798 |
0.0592 |
0.0690 |
PerExample |
0.1101 |
0.7798 |
0.0690 |
0.1248 |
0.7798 |
0.0001 |
0.0001 |
0.7798 |
0.0592 |
0.0690 |
spoon-spoon |
0.3847 |
0.3680 |
0.4449 |
0.3769 |
0.4469 |
0.3480 |
0.3264 |
0.3478 |
0.4304 |
0.2418 |
fork-fork |
0.3944 |
0.4500 |
0.3846 |
0.3849 |
0.5395 |
0.2947 |
0.2987 |
0.4234 |
0.3690 |
0.2495 |
spork-spork |
0.3894 |
0.4001 |
0.4174 |
0.3804 |
0.4833 |
0.3245 |
0.3152 |
0.3769 |
0.4027 |
0.2451 |
EO_Sys1 |
0.5270 |
0.5647 |
0.5104 |
0.5099 |
0.5234 |
0.2790 |
0.2655 |
0.5473 |
0.5083 |
0.3567 |
FU_System_k15 |
0.5198 |
0.4886 |
0.5845 |
0.5127 |
0.4473 |
0.4314 |
0.3963 |
0.4890 |
0.5546 |
0.3574 |
FU_System_k25 |
0.4645 |
0.3555 |
0.6951 |
0.4544 |
0.3119 |
0.5611 |
0.4307 |
0.3553 |
0.6707 |
0.3033 |
Limited sample |
0.5214 |
0.5629 |
0.5008 |
0.5030 |
0.5111 |
0.2701 |
0.2562 |
0.5450 |
0.4997 |
0.3507 |
Mixing with MTI |
0.5726 |
0.5089 |
0.6872 |
0.5665 |
0.4978 |
0.5728 |
0.5038 |
0.4989 |
0.6720 |
0.4070 |
Asclepius |
0.5890 |
0.5882 |
0.6060 |
0.5814 |
0.5961 |
0.4434 |
0.4323 |
0.5904 |
0.5876 |
0.4241 |
Antinomyra SYS3 |
0.5346 |
0.5427 |
0.5409 |
0.5257 |
0.6215 |
0.2530 |
0.2652 |
0.5421 |
0.5272 |
0.3709 |
Antinomyra SYS4 |
0.5345 |
0.5401 |
0.5447 |
0.5262 |
0.5838 |
0.2791 |
0.2885 |
0.5399 |
0.5292 |
0.3712 |
L2R-n1 |
0.5671 |
0.6490 |
0.5422 |
0.5702 |
0.6019 |
0.4328 |
0.4324 |
0.6470 |
0.5047 |
0.4129 |
L2R-n2 |
0.5718 |
0.6361 |
0.5576 |
0.5733 |
0.5891 |
0.4482 |
0.4422 |
0.6322 |
0.5219 |
0.4159 |
L2R-n3 |
0.5659 |
0.6642 |
0.5312 |
0.5692 |
0.6144 |
0.4233 |
0.4259 |
0.6588 |
0.4960 |
0.4122 |
L2R-n4 |
0.0799 |
0.0904 |
0.0733 |
0.0778 |
0.0620 |
0.0655 |
0.0598 |
0.0895 |
0.0722 |
0.0425 |
L2R-n5 |
0.0918 |
0.0994 |
0.0830 |
0.0871 |
0.0621 |
0.0657 |
0.0600 |
0.1017 |
0.0836 |
0.0481 |
Hierarchical Measures
System Name |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |
BioASQ_Baseline |
0.3074 |
0.5517 |
0.5617 |
0.5278 |
0.3341 |
0.3140 |
MTI First Line Index |
0.4729 |
0.7692 |
0.6634 |
0.6882 |
0.5345 |
0.4529 |
Default MTI |
0.4840 |
0.7425 |
0.7077 |
0.7029 |
0.5151 |
0.4863 |
FU System |
0.4094 |
0.5320 |
0.7947 |
0.6153 |
0.3445 |
0.5468 |
FU_System_t25 |
0.4120 |
0.5368 |
0.7974 |
0.6197 |
0.3468 |
0.5493 |
FU_System_t15 |
0.4394 |
0.6637 |
0.6813 |
0.6487 |
0.4430 |
0.4688 |
Macro |
0.0847 |
0.7994 |
0.0546 |
0.1007 |
0.2976 |
0.0505 |
Micro |
0.0847 |
0.7994 |
0.0546 |
0.1007 |
0.2976 |
0.0505 |
PerExample |
0.0847 |
0.7994 |
0.0546 |
0.1007 |
0.2976 |
0.0505 |
spoon-spoon |
0.3582 |
0.5821 |
0.6017 |
0.5598 |
0.3607 |
0.3983 |
fork-fork |
0.3541 |
0.6695 |
0.5122 |
0.5449 |
0.4201 |
0.3409 |
spork-spork |
0.3557 |
0.6154 |
0.5623 |
0.5537 |
0.3821 |
0.3728 |
EO_Sys1 |
0.4364 |
0.7310 |
0.6122 |
0.6336 |
0.4946 |
0.4253 |
FU_System_k15 |
0.4504 |
0.6727 |
0.6983 |
0.6615 |
0.4535 |
0.4810 |
FU_System_k25 |
0.4224 |
0.5474 |
0.8125 |
0.6322 |
0.3555 |
0.5625 |
Limited sample |
0.4314 |
0.7306 |
0.6046 |
0.6274 |
0.4955 |
0.4175 |
Mixing with MTI |
0.4903 |
0.6772 |
0.7954 |
0.7108 |
0.4602 |
0.5622 |
Asclepius |
0.4951 |
0.7384 |
0.7298 |
0.7152 |
0.5118 |
0.5066 |
Antinomyra SYS3 |
0.4474 |
0.7281 |
0.6385 |
0.6592 |
0.4894 |
0.4363 |
Antinomyra SYS4 |
0.4504 |
0.7216 |
0.6454 |
0.6603 |
0.4887 |
0.4423 |
L2R-n1 |
0.4787 |
0.7945 |
0.6473 |
0.6887 |
0.5578 |
0.4476 |
L2R-n2 |
0.4833 |
0.7841 |
0.6650 |
0.6951 |
0.5504 |
0.4605 |
L2R-n3 |
0.4762 |
0.8064 |
0.6350 |
0.6850 |
0.5670 |
0.4384 |
L2R-n4 |
0.2235 |
0.3887 |
0.3956 |
0.3755 |
0.2385 |
0.2233 |
L2R-n5 |
0.2268 |
0.3899 |
0.3983 |
0.3774 |
0.2407 |
0.2274 |
+ Test batch 1, week 2
Annotated articles:3734/4721
Flat Measures
System Name |
MiF |
EBP |
EBR |
EBF |
MaP |
MaR |
MaF |
MiP |
MiR |
Acc. |
BioASQ_Baseline |
0.2382 |
0.4054 |
0.2253 |
0.2197 |
0.3747 |
0.2919 |
0.2572 |
0.2561 |
0.2227 |
0.1288 |
MTI First Line Index |
0.5596 |
0.6268 |
0.5267 |
0.5521 |
0.5824 |
0.4809 |
0.4605 |
0.6185 |
0.5109 |
0.3961 |
Default MTI |
0.5709 |
0.5924 |
0.5755 |
0.5651 |
0.5441 |
0.5233 |
0.4836 |
0.5848 |
0.5576 |
0.4084 |
FU System |
0.4541 |
0.3461 |
0.6788 |
0.4445 |
0.3438 |
0.5043 |
0.3948 |
0.3461 |
0.6598 |
0.2954 |
FU_System_t25 |
0.4569 |
0.3483 |
0.6837 |
0.4474 |
0.3470 |
0.5145 |
0.4031 |
0.3483 |
0.6639 |
0.2976 |
FU_System_t15 |
0.5232 |
0.4903 |
0.5879 |
0.5159 |
0.4388 |
0.4535 |
0.4117 |
0.4903 |
0.5608 |
0.3605 |
Macro |
0.1018 |
0.7185 |
0.0610 |
0.1109 |
0.7185 |
0.0001 |
0.0001 |
0.7185 |
0.0548 |
0.0610 |
Micro |
0.1018 |
0.7185 |
0.0610 |
0.1109 |
0.7185 |
0.0001 |
0.0001 |
0.7185 |
0.0548 |
0.0610 |
PerExample |
0.1018 |
0.7185 |
0.0610 |
0.1109 |
0.7185 |
0.0001 |
0.0001 |
0.7185 |
0.0548 |
0.0610 |
spoon-spoon |
0.3817 |
0.3582 |
0.4340 |
0.3684 |
0.4468 |
0.3556 |
0.3319 |
0.3453 |
0.4267 |
0.2352 |
fork-fork |
0.3524 |
0.4961 |
0.2834 |
0.3381 |
0.4400 |
0.1225 |
0.1359 |
0.4836 |
0.2772 |
0.2175 |
spork-spork |
0.1018 |
0.7185 |
0.0610 |
0.1109 |
0.7185 |
0.0001 |
0.0001 |
0.7185 |
0.0548 |
0.0610 |
EO_Sys1 |
0.5325 |
0.5814 |
0.5043 |
0.5140 |
0.5341 |
0.2864 |
0.2730 |
0.5619 |
0.5060 |
0.3607 |
FU_System_k15 |
0.5226 |
0.4898 |
0.5873 |
0.5153 |
0.4433 |
0.4499 |
0.4095 |
0.4898 |
0.5602 |
0.3600 |
FU_System_k25 |
0.4657 |
0.3550 |
0.6987 |
0.4564 |
0.3042 |
0.5730 |
0.4341 |
0.3550 |
0.6768 |
0.3049 |
Limited sample |
0.5251 |
0.5788 |
0.4921 |
0.5050 |
0.5211 |
0.2779 |
0.2636 |
0.5598 |
0.4945 |
0.3528 |
Mixing with MTI |
0.5778 |
0.5194 |
0.6807 |
0.5711 |
0.4971 |
0.5812 |
0.5083 |
0.5079 |
0.6702 |
0.4124 |
Asclepius |
0.0040 |
0.0037 |
0.0035 |
0.0035 |
0.0019 |
0.0010 |
0.0007 |
0.0041 |
0.0040 |
0.0018 |
Antinomyra SYS3 |
0.5473 |
0.5985 |
0.5143 |
0.5353 |
0.6170 |
0.2607 |
0.2733 |
0.5983 |
0.5042 |
0.3801 |
Antinomyra SYS4 |
0.5351 |
0.5464 |
0.5357 |
0.5249 |
0.5868 |
0.2860 |
0.2957 |
0.5461 |
0.5246 |
0.3695 |
L2R-n1 |
0.5844 |
0.6231 |
0.5793 |
0.5808 |
0.5719 |
0.4918 |
0.4726 |
0.6156 |
0.5562 |
0.4238 |
L2R-n2 |
0.5857 |
0.5842 |
0.6151 |
0.5805 |
0.5369 |
0.5286 |
0.4905 |
0.5766 |
0.5951 |
0.4231 |
L2R-n3 |
0.5812 |
0.6417 |
0.5619 |
0.5784 |
0.5889 |
0.4748 |
0.4628 |
0.6317 |
0.5381 |
0.4216 |
L2R-n4 |
0.5770 |
0.6390 |
0.5591 |
0.5767 |
0.5918 |
0.4610 |
0.4531 |
0.6350 |
0.5287 |
0.4199 |
L2R-n5 |
0.5750 |
0.6483 |
0.5520 |
0.5749 |
0.6008 |
0.4527 |
0.4492 |
0.6412 |
0.5213 |
0.4177 |
Antinomyra SYS2 |
0.5378 |
0.5532 |
0.5345 |
0.5277 |
0.6269 |
0.2650 |
0.2786 |
0.5521 |
0.5242 |
0.3720 |
FDU_MeSHIndexing_1 |
0.5429 |
0.5453 |
0.5715 |
0.5379 |
0.5203 |
0.4441 |
0.4242 |
0.5453 |
0.5406 |
0.3811 |
FDU_MeSHIndexing_2 |
0.5430 |
0.5454 |
0.5716 |
0.5381 |
0.5162 |
0.4431 |
0.4230 |
0.5454 |
0.5406 |
0.3813 |
FDU_MeSHIndexing_3 |
0.5426 |
0.5450 |
0.5705 |
0.5374 |
0.5250 |
0.4290 |
0.4103 |
0.5450 |
0.5402 |
0.3808 |
FDU_MeSHIndexing_4 |
0.5428 |
0.5452 |
0.5706 |
0.5376 |
0.5270 |
0.4313 |
0.4126 |
0.5452 |
0.5405 |
0.3809 |
FDU_MeSHIndexing_5 |
0.5397 |
0.5420 |
0.5680 |
0.5347 |
0.5175 |
0.4355 |
0.4147 |
0.5420 |
0.5373 |
0.3783 |
Hierarchical Measures
System Name |
LCA-F |
HiP |
HiR |
HiF |
LCA-P |
LCA-R |
BioASQ_Baseline |
0.2498 |
0.6247 |
0.4168 |
0.4150 |
0.3262 |
0.2429 |
MTI First Line Index |
0.4725 |
0.7789 |
0.6576 |
0.6901 |
0.5371 |
0.4494 |
Default MTI |
0.4867 |
0.7494 |
0.7092 |
0.7074 |
0.5190 |
0.4876 |
FU System |
0.4128 |
0.5371 |
0.8084 |
0.6264 |
0.3448 |
0.5515 |
FU_System_t25 |
0.4146 |
0.5414 |
0.8095 |
0.6296 |
0.3464 |
0.5539 |
FU_System_t15 |
0.4527 |
0.6765 |
0.7110 |
0.6717 |
0.4535 |
0.4836 |
Macro |
0.0810 |
0.7389 |
0.0496 |
0.0915 |
0.2879 |
0.0482 |
Micro |
0.0810 |
0.7389 |
0.0496 |
0.0915 |
0.2879 |
0.0482 |
PerExample |
0.0810 |
0.7389 |
0.0496 |
0.0915 |
0.2879 |
0.0482 |
spoon-spoon |
0.3511 |
0.5742 |
0.5858 |
0.5493 |
0.3537 |
0.3895 |
fork-fork |
0.3005 |
0.7087 |
0.3565 |
0.4392 |
0.4639 |
0.2437 |
spork-spork |
0.0810 |
0.7389 |
0.0496 |
0.0915 |
0.2879 |
0.0482 |
EO_Sys1 |
0.4366 |
0.7507 |
0.6080 |
0.6393 |
0.5062 |
0.4178 |
FU_System_k15 |
0.4522 |
0.6765 |
0.7101 |
0.6710 |
0.4532 |
0.4831 |
FU_System_k25 |
0.4244 |
0.5482 |
0.8224 |
0.6388 |
0.3547 |
0.5664 |
Limited sample |
0.4301 |
0.7457 |
0.5973 |
0.6294 |
0.5043 |
0.4090 |
Mixing with MTI |
0.4956 |
0.6914 |
0.7934 |
0.7185 |
0.4696 |
0.5604 |
Asclepius |
0.1064 |
0.1082 |
0.1775 |
0.1303 |
0.0914 |
0.1345 |
Antinomyra SYS3 |
0.4532 |
0.7614 |
0.6256 |
0.6654 |
0.5241 |
0.4228 |
Antinomyra SYS4 |
0.4497 |
0.7300 |
0.6425 |
0.6632 |
0.4922 |
0.4380 |
L2R-n1 |
0.4914 |
0.7763 |
0.6962 |
0.7116 |
0.5408 |
0.4797 |
L2R-n2 |
0.4977 |
0.7473 |
0.7341 |
0.7200 |
0.5181 |
0.5097 |
L2R-n3 |
0.4871 |
0.7904 |
0.6771 |
0.7054 |
0.5522 |
0.4655 |
L2R-n4 |
0.4856 |
0.7865 |
0.6728 |
0.7026 |
0.5498 |
0.4626 |
L2R-n5 |
0.4814 |
0.7964 |
0.6599 |
0.6975 |
0.5562 |
0.4536 |
Antinomyra SYS2 |
0.4488 |
0.7424 |
0.6355 |
0.6644 |
0.4971 |
0.4330 |
FDU_MeSHIndexing_1 |
0.4652 |
0.7323 |
0.6905 |
0.6879 |
0.4941 |
0.4709 |
FDU_MeSHIndexing_2 |
0.4651 |
0.7316 |
0.6905 |
0.6875 |
0.4934 |
0.4714 |
FDU_MeSHIndexing_3 |
0.4638 |
0.7317 |
0.6876 |
0.6864 |
0.4934 |
0.4681 |
FDU_MeSHIndexing_4 |
0.4645 |
0.7323 |
0.6878 |
0.6868 |
0.4943 |
0.4687 |
FDU_MeSHIndexing_5 |
0.4608 |
0.7291 |
0.6827 |
0.6821 |
0.4897 |
0.4660 |