BioASQ Participants Area

Evaluation Overview

In the following table you can see the number of the articles for each test set that has been released. In parentheses, there is the number of the articles that have been annotated by the curators for each test set.

	Batch 1	Batch 2	Batch 3
Week 1	4440 (3319)	4085 (3422)	4342 (3009)
Week 2	4721 (3734)	3496 (2788)	8840 (5883)
Week 3	4802 (3884)	4524 (3274)	3702 (2860)
Week 4	3579 (2431)	5407 (3923)	4726 (3252)
Week 5	5299 (3693)	5454 (3666)	4533 (3252)

Test Results for Task 2a

The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.

+ Dry-run test

Annotated articles:2703/3186

Flat Measures

System Name	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
BioASQ_Baseline	0.2510	0.2516	0.2893	0.2499	0.3592	0.3845	0.3182	0.2296	0.2767	0.1484
MTI First Line Index	0.5624	0.6087	0.5473	0.5535	0.5675	0.4858	0.4633	0.6023	0.5274	0.3998
MeSH Indexing	0.5675	0.6155	0.5629	0.5671	0.5722	0.4671	0.4551	0.6132	0.5281	0.4116
Default MTI	0.5673	0.5721	0.5908	0.5595	0.5381	0.5273	0.4889	0.5647	0.5699	0.4040
Macro	0.1033	0.6722	0.0639	0.1149	0.6722	0.0001	0.0001	0.6722	0.0560	0.0639
Micro	0.1033	0.6722	0.0639	0.1149	0.6722	0.0001	0.0001	0.6722	0.0560	0.0639
PerExample	0.1033	0.6722	0.0639	0.1149	0.6722	0.0001	0.0001	0.6722	0.0560	0.0639
spoon-spoon	0.3678	0.3562	0.4103	0.3557	0.4423	0.3241	0.3045	0.3402	0.4002	0.2271
fork-fork	0.3755	0.4382	0.3531	0.3609	0.5331	0.2736	0.2750	0.4167	0.3416	0.2328
spork-spork	0.3716	0.3891	0.3837	0.3586	0.4775	0.2996	0.2899	0.3703	0.3729	0.2301
EO_Sys1	0.4538	0.5471	0.5060	0.4963	0.3787	0.2869	0.2420	0.4118	0.5054	0.3469
Limited sample	0.4506	0.5467	0.4992	0.4923	0.3701	0.2806	0.2370	0.4104	0.4997	0.3433
Mixing with MTI	0.5060	0.4937	0.6908	0.5550	0.4305	0.5841	0.4773	0.4041	0.6767	0.3978

Hierarchical Measures

System Name	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
BioASQ_Baseline	0.3044	0.5320	0.5566	0.5120	0.3261	0.3181
MTI First Line Index	0.4694	0.7532	0.6736	0.6829	0.5203	0.4603
MeSH Indexing	0.4784	0.7601	0.6833	0.6946	0.5313	0.4656
Default MTI	0.4782	0.7215	0.7210	0.6954	0.4989	0.4938
Macro	0.0845	0.6955	0.0521	0.0951	0.2803	0.0512
Micro	0.0845	0.6955	0.0521	0.0951	0.2803	0.0512
PerExample	0.0845	0.6955	0.0521	0.0951	0.2803	0.0512
spoon-spoon	0.3393	0.5673	0.5425	0.5198	0.3524	0.3694
fork-fork	0.3337	0.6555	0.4522	0.4967	0.4135	0.3141
spork-spork	0.3362	0.5994	0.5009	0.5098	0.3742	0.3438
EO_Sys1	0.4247	0.7087	0.6104	0.6173	0.4797	0.4214
Limited sample	0.4209	0.7075	0.6058	0.6141	0.4783	0.4162
Mixing with MTI	0.4791	0.6548	0.7983	0.6946	0.4454	0.5636

+ Test batch 1, week 1

Annotated articles:3319/4440

Flat Measures

System Name	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
BioASQ_Baseline	0.2549	0.2576	0.2864	0.2530	0.3769	0.3786	0.3158	0.2375	0.2750	0.1501
MTI First Line Index	0.5605	0.6213	0.5368	0.5557	0.5803	0.4745	0.4558	0.6132	0.5162	0.3994
Default MTI	0.5704	0.5895	0.5797	0.5659	0.5490	0.5166	0.4813	0.5828	0.5585	0.4082
FU System	0.4499	0.3444	0.6714	0.4398	0.3393	0.4932	0.3889	0.3441	0.6496	0.2913
FU_System_t25	0.4529	0.3467	0.6763	0.4428	0.3428	0.5008	0.3961	0.3464	0.6538	0.2937
FU_System_t15	0.5056	0.4753	0.5668	0.4980	0.4725	0.3778	0.3533	0.4757	0.5395	0.3444
Macro	0.1101	0.7798	0.0690	0.1248	0.7798	0.0001	0.0001	0.7798	0.0592	0.0690
Micro	0.1101	0.7798	0.0690	0.1248	0.7798	0.0001	0.0001	0.7798	0.0592	0.0690
PerExample	0.1101	0.7798	0.0690	0.1248	0.7798	0.0001	0.0001	0.7798	0.0592	0.0690
spoon-spoon	0.3847	0.3680	0.4449	0.3769	0.4469	0.3480	0.3264	0.3478	0.4304	0.2418
fork-fork	0.3944	0.4500	0.3846	0.3849	0.5395	0.2947	0.2987	0.4234	0.3690	0.2495
spork-spork	0.3894	0.4001	0.4174	0.3804	0.4833	0.3245	0.3152	0.3769	0.4027	0.2451
EO_Sys1	0.5270	0.5647	0.5104	0.5099	0.5234	0.2790	0.2655	0.5473	0.5083	0.3567
FU_System_k15	0.5198	0.4886	0.5845	0.5127	0.4473	0.4314	0.3963	0.4890	0.5546	0.3574
FU_System_k25	0.4645	0.3555	0.6951	0.4544	0.3119	0.5611	0.4307	0.3553	0.6707	0.3033
Limited sample	0.5214	0.5629	0.5008	0.5030	0.5111	0.2701	0.2562	0.5450	0.4997	0.3507
Mixing with MTI	0.5726	0.5089	0.6872	0.5665	0.4978	0.5728	0.5038	0.4989	0.6720	0.4070
Asclepius	0.5890	0.5882	0.6060	0.5814	0.5961	0.4434	0.4323	0.5904	0.5876	0.4241
Antinomyra SYS3	0.5346	0.5427	0.5409	0.5257	0.6215	0.2530	0.2652	0.5421	0.5272	0.3709
Antinomyra SYS4	0.5345	0.5401	0.5447	0.5262	0.5838	0.2791	0.2885	0.5399	0.5292	0.3712
L2R-n1	0.5671	0.6490	0.5422	0.5702	0.6019	0.4328	0.4324	0.6470	0.5047	0.4129
L2R-n2	0.5718	0.6361	0.5576	0.5733	0.5891	0.4482	0.4422	0.6322	0.5219	0.4159
L2R-n3	0.5659	0.6642	0.5312	0.5692	0.6144	0.4233	0.4259	0.6588	0.4960	0.4122
L2R-n4	0.0799	0.0904	0.0733	0.0778	0.0620	0.0655	0.0598	0.0895	0.0722	0.0425
L2R-n5	0.0918	0.0994	0.0830	0.0871	0.0621	0.0657	0.0600	0.1017	0.0836	0.0481

Hierarchical Measures

System Name	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
BioASQ_Baseline	0.3074	0.5517	0.5617	0.5278	0.3341	0.3140
MTI First Line Index	0.4729	0.7692	0.6634	0.6882	0.5345	0.4529
Default MTI	0.4840	0.7425	0.7077	0.7029	0.5151	0.4863
FU System	0.4094	0.5320	0.7947	0.6153	0.3445	0.5468
FU_System_t25	0.4120	0.5368	0.7974	0.6197	0.3468	0.5493
FU_System_t15	0.4394	0.6637	0.6813	0.6487	0.4430	0.4688
Macro	0.0847	0.7994	0.0546	0.1007	0.2976	0.0505
Micro	0.0847	0.7994	0.0546	0.1007	0.2976	0.0505
PerExample	0.0847	0.7994	0.0546	0.1007	0.2976	0.0505
spoon-spoon	0.3582	0.5821	0.6017	0.5598	0.3607	0.3983
fork-fork	0.3541	0.6695	0.5122	0.5449	0.4201	0.3409
spork-spork	0.3557	0.6154	0.5623	0.5537	0.3821	0.3728
EO_Sys1	0.4364	0.7310	0.6122	0.6336	0.4946	0.4253
FU_System_k15	0.4504	0.6727	0.6983	0.6615	0.4535	0.4810
FU_System_k25	0.4224	0.5474	0.8125	0.6322	0.3555	0.5625
Limited sample	0.4314	0.7306	0.6046	0.6274	0.4955	0.4175
Mixing with MTI	0.4903	0.6772	0.7954	0.7108	0.4602	0.5622
Asclepius	0.4951	0.7384	0.7298	0.7152	0.5118	0.5066
Antinomyra SYS3	0.4474	0.7281	0.6385	0.6592	0.4894	0.4363
Antinomyra SYS4	0.4504	0.7216	0.6454	0.6603	0.4887	0.4423
L2R-n1	0.4787	0.7945	0.6473	0.6887	0.5578	0.4476
L2R-n2	0.4833	0.7841	0.6650	0.6951	0.5504	0.4605
L2R-n3	0.4762	0.8064	0.6350	0.6850	0.5670	0.4384
L2R-n4	0.2235	0.3887	0.3956	0.3755	0.2385	0.2233
L2R-n5	0.2268	0.3899	0.3983	0.3774	0.2407	0.2274

+ Test batch 1, week 2

Annotated articles:3734/4721

Flat Measures

System Name	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
BioASQ_Baseline	0.2382	0.4054	0.2253	0.2197	0.3747	0.2919	0.2572	0.2561	0.2227	0.1288
MTI First Line Index	0.5596	0.6268	0.5267	0.5521	0.5824	0.4809	0.4605	0.6185	0.5109	0.3961
Default MTI	0.5709	0.5924	0.5755	0.5651	0.5441	0.5233	0.4836	0.5848	0.5576	0.4084
FU System	0.4541	0.3461	0.6788	0.4445	0.3438	0.5043	0.3948	0.3461	0.6598	0.2954
FU_System_t25	0.4569	0.3483	0.6837	0.4474	0.3470	0.5145	0.4031	0.3483	0.6639	0.2976
FU_System_t15	0.5232	0.4903	0.5879	0.5159	0.4388	0.4535	0.4117	0.4903	0.5608	0.3605
Macro	0.1018	0.7185	0.0610	0.1109	0.7185	0.0001	0.0001	0.7185	0.0548	0.0610
Micro	0.1018	0.7185	0.0610	0.1109	0.7185	0.0001	0.0001	0.7185	0.0548	0.0610
PerExample	0.1018	0.7185	0.0610	0.1109	0.7185	0.0001	0.0001	0.7185	0.0548	0.0610
spoon-spoon	0.3817	0.3582	0.4340	0.3684	0.4468	0.3556	0.3319	0.3453	0.4267	0.2352
fork-fork	0.3524	0.4961	0.2834	0.3381	0.4400	0.1225	0.1359	0.4836	0.2772	0.2175
spork-spork	0.1018	0.7185	0.0610	0.1109	0.7185	0.0001	0.0001	0.7185	0.0548	0.0610
EO_Sys1	0.5325	0.5814	0.5043	0.5140	0.5341	0.2864	0.2730	0.5619	0.5060	0.3607
FU_System_k15	0.5226	0.4898	0.5873	0.5153	0.4433	0.4499	0.4095	0.4898	0.5602	0.3600
FU_System_k25	0.4657	0.3550	0.6987	0.4564	0.3042	0.5730	0.4341	0.3550	0.6768	0.3049
Limited sample	0.5251	0.5788	0.4921	0.5050	0.5211	0.2779	0.2636	0.5598	0.4945	0.3528
Mixing with MTI	0.5778	0.5194	0.6807	0.5711	0.4971	0.5812	0.5083	0.5079	0.6702	0.4124
Asclepius	0.0040	0.0037	0.0035	0.0035	0.0019	0.0010	0.0007	0.0041	0.0040	0.0018
Antinomyra SYS3	0.5473	0.5985	0.5143	0.5353	0.6170	0.2607	0.2733	0.5983	0.5042	0.3801
Antinomyra SYS4	0.5351	0.5464	0.5357	0.5249	0.5868	0.2860	0.2957	0.5461	0.5246	0.3695
L2R-n1	0.5844	0.6231	0.5793	0.5808	0.5719	0.4918	0.4726	0.6156	0.5562	0.4238
L2R-n2	0.5857	0.5842	0.6151	0.5805	0.5369	0.5286	0.4905	0.5766	0.5951	0.4231
L2R-n3	0.5812	0.6417	0.5619	0.5784	0.5889	0.4748	0.4628	0.6317	0.5381	0.4216
L2R-n4	0.5770	0.6390	0.5591	0.5767	0.5918	0.4610	0.4531	0.6350	0.5287	0.4199
L2R-n5	0.5750	0.6483	0.5520	0.5749	0.6008	0.4527	0.4492	0.6412	0.5213	0.4177
Antinomyra SYS2	0.5378	0.5532	0.5345	0.5277	0.6269	0.2650	0.2786	0.5521	0.5242	0.3720
FDU_MeSHIndexing_1	0.5429	0.5453	0.5715	0.5379	0.5203	0.4441	0.4242	0.5453	0.5406	0.3811
FDU_MeSHIndexing_2	0.5430	0.5454	0.5716	0.5381	0.5162	0.4431	0.4230	0.5454	0.5406	0.3813
FDU_MeSHIndexing_3	0.5426	0.5450	0.5705	0.5374	0.5250	0.4290	0.4103	0.5450	0.5402	0.3808
FDU_MeSHIndexing_4	0.5428	0.5452	0.5706	0.5376	0.5270	0.4313	0.4126	0.5452	0.5405	0.3809
FDU_MeSHIndexing_5	0.5397	0.5420	0.5680	0.5347	0.5175	0.4355	0.4147	0.5420	0.5373	0.3783

Hierarchical Measures

System Name	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
BioASQ_Baseline	0.2498	0.6247	0.4168	0.4150	0.3262	0.2429
MTI First Line Index	0.4725	0.7789	0.6576	0.6901	0.5371	0.4494
Default MTI	0.4867	0.7494	0.7092	0.7074	0.5190	0.4876
FU System	0.4128	0.5371	0.8084	0.6264	0.3448	0.5515
FU_System_t25	0.4146	0.5414	0.8095	0.6296	0.3464	0.5539
FU_System_t15	0.4527	0.6765	0.7110	0.6717	0.4535	0.4836
Macro	0.0810	0.7389	0.0496	0.0915	0.2879	0.0482
Micro	0.0810	0.7389	0.0496	0.0915	0.2879	0.0482
PerExample	0.0810	0.7389	0.0496	0.0915	0.2879	0.0482
spoon-spoon	0.3511	0.5742	0.5858	0.5493	0.3537	0.3895
fork-fork	0.3005	0.7087	0.3565	0.4392	0.4639	0.2437
spork-spork	0.0810	0.7389	0.0496	0.0915	0.2879	0.0482
EO_Sys1	0.4366	0.7507	0.6080	0.6393	0.5062	0.4178
FU_System_k15	0.4522	0.6765	0.7101	0.6710	0.4532	0.4831
FU_System_k25	0.4244	0.5482	0.8224	0.6388	0.3547	0.5664
Limited sample	0.4301	0.7457	0.5973	0.6294	0.5043	0.4090
Mixing with MTI	0.4956	0.6914	0.7934	0.7185	0.4696	0.5604
Asclepius	0.1064	0.1082	0.1775	0.1303	0.0914	0.1345
Antinomyra SYS3	0.4532	0.7614	0.6256	0.6654	0.5241	0.4228
Antinomyra SYS4	0.4497	0.7300	0.6425	0.6632	0.4922	0.4380
L2R-n1	0.4914	0.7763	0.6962	0.7116	0.5408	0.4797
L2R-n2	0.4977	0.7473	0.7341	0.7200	0.5181	0.5097
L2R-n3	0.4871	0.7904	0.6771	0.7054	0.5522	0.4655
L2R-n4	0.4856	0.7865	0.6728	0.7026	0.5498	0.4626
L2R-n5	0.4814	0.7964	0.6599	0.6975	0.5562	0.4536
Antinomyra SYS2	0.4488	0.7424	0.6355	0.6644	0.4971	0.4330
FDU_MeSHIndexing_1	0.4652	0.7323	0.6905	0.6879	0.4941	0.4709
FDU_MeSHIndexing_2	0.4651	0.7316	0.6905	0.6875	0.4934	0.4714
FDU_MeSHIndexing_3	0.4638	0.7317	0.6876	0.6864	0.4934	0.4681
FDU_MeSHIndexing_4	0.4645	0.7323	0.6878	0.6868	0.4943	0.4687
FDU_MeSHIndexing_5	0.4608	0.7291	0.6827	0.6821	0.4897	0.4660