BioASQ Participants Area

Test Results for Task 6a

The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.

+ Dry Run 1

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
deep learning only	0.5614	0.5704	0.5609	0.5439	0.5710	0.3180	0.3336	0.5683	0.5547	0.3881

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
deep learning only	0.4616	0.7414	0.6591	0.6746	0.5056	0.4525

+ Dry Run 2

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6844	0.7010	0.6873	0.6751	0.6294	0.6275	0.6109	0.6898	0.6791	0.5348
Default MTI	0.6797	0.6454	0.7501	0.6762	0.5706	0.6779	0.6493	0.6334	0.7334	0.5298

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5559	0.8129	0.7777	0.7778	0.5882	0.5573
Default MTI	0.5545	0.7574	0.8298	0.7763	0.5449	0.5927

+ Test batch 1, week 1

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6134	0.6654	0.5868	0.6022	0.6204	0.5367	0.5159	0.6553	0.5766	0.4520
Default MTI	0.6231	0.6264	0.6424	0.6151	0.5881	0.5828	0.5414	0.6222	0.6240	0.4640
iria-1	0.4992	0.4331	0.5873	0.4846	0.4053	0.4603	0.4060	0.4402	0.5765	0.3340
iria-2	0.4961	0.4310	0.5830	0.4816	0.3992	0.4556	0.4008	0.4380	0.5721	0.3313
DeepMeSH1	0.6360	0.6652	0.6180	0.6228	0.6369	0.4867	0.4851	0.6698	0.6054	0.4697
DeepMeSH2	0.6565	0.6858	0.6426	0.6459	0.6588	0.5344	0.5274	0.6894	0.6266	0.4954
DeepMeSH3	0.6434	0.6758	0.6255	0.6316	0.6406	0.5040	0.4997	0.6793	0.6111	0.4789
DeepMeSH4	0.6573	0.6888	0.6414	0.6465	0.6592	0.5357	0.5281	0.6917	0.6261	0.4962
xgx	0.5348	0.6013	0.4860	0.5184	0.5415	0.3318	0.3462	0.5997	0.4826	0.3684
AUTH-Atypon 1	0.5593	0.6199	0.5109	0.5363	0.5918	0.3467	0.3534	0.6183	0.5106	0.3843

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.4965	0.7902	0.6984	0.7202	0.5463	0.4818
Default MTI	0.5110	0.7567	0.7467	0.7329	0.5281	0.5223
iria-1	0.4304	0.5983	0.7289	0.6404	0.4050	0.4859
iria-2	0.4284	0.5976	0.7257	0.6386	0.4033	0.4833
DeepMeSH1	0.5087	0.7910	0.7159	0.7338	0.5507	0.4959
DeepMeSH2	0.5271	0.8042	0.7344	0.7503	0.5670	0.5159
DeepMeSH3	0.5153	0.7983	0.7215	0.7400	0.5583	0.5022
DeepMeSH4	0.5276	0.8064	0.7326	0.7500	0.5690	0.5159
xgx	0.4403	0.7577	0.6072	0.6519	0.5115	0.4095
AUTH-Atypon 1	0.4504	0.7883	0.6119	0.6632	0.5264	0.4197

+ Test batch 1, week 2

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6219	0.6734	0.6013	0.6157	0.6317	0.5533	0.5312	0.6642	0.5847	0.4641
Default MTI	0.6339	0.6422	0.6520	0.6293	0.6018	0.5957	0.5541	0.6366	0.6313	0.4767
iria-1	0.5223	0.4682	0.6015	0.5136	0.4445	0.4743	0.4252	0.4721	0.5845	0.3584
iria-2	0.5100	0.4585	0.5863	0.5015	0.4174	0.4592	0.4091	0.4614	0.5700	0.3477
DeepMeSH1	0.6456	0.6819	0.6292	0.6384	0.6458	0.4871	0.4865	0.6824	0.6126	0.4837
DeepMeSH2	0.6651	0.7026	0.6509	0.6597	0.6684	0.5380	0.5331	0.7017	0.6321	0.5079
DeepMeSH3	0.6515	0.6926	0.6333	0.6451	0.6536	0.5043	0.5017	0.6917	0.6158	0.4911
DeepMeSH4	0.6664	0.7050	0.6509	0.6607	0.6706	0.5414	0.5358	0.7039	0.6327	0.5091
xgx	0.5697	0.6755	0.5064	0.5611	0.6061	0.3504	0.3717	0.6685	0.4963	0.4062
xgx0	0.5683	0.6841	0.5004	0.5601	0.6303	0.3425	0.3663	0.6766	0.4898	0.4051

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5059	0.7913	0.7073	0.7276	0.5535	0.4918
Default MTI	0.5193	0.7639	0.7533	0.7410	0.5375	0.5290
iria-1	0.4459	0.6271	0.7308	0.6597	0.4267	0.4918
iria-2	0.4376	0.6169	0.7201	0.6489	0.4187	0.4835
DeepMeSH1	0.5176	0.8007	0.7177	0.7401	0.5639	0.5009
DeepMeSH2	0.5356	0.8135	0.7356	0.7562	0.5796	0.5208
DeepMeSH3	0.5226	0.8082	0.7211	0.7449	0.5708	0.5048
DeepMeSH4	0.5358	0.8153	0.7351	0.7567	0.5808	0.5206
xgx	0.4603	0.8124	0.6096	0.6762	0.5564	0.4142
xgx0	0.4584	0.8217	0.5969	0.6709	0.5627	0.4081

+ Test batch 1, week 3

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6220	0.6810	0.5938	0.6155	0.6506	0.5452	0.5293	0.6709	0.5797	0.4627
Default MTI	0.6361	0.6503	0.6454	0.6308	0.6196	0.5888	0.5530	0.6442	0.6281	0.4777
iria-1	0.5198	0.4671	0.5970	0.5118	0.4482	0.4688	0.4211	0.4702	0.5811	0.3566
iria-2	0.5257	0.4719	0.6034	0.5174	0.4537	0.4692	0.4228	0.4753	0.5880	0.3615
DeepMeSH1	0.6463	0.6888	0.6236	0.6388	0.6592	0.4757	0.4809	0.6893	0.6082	0.4843
DeepMeSH2	0.6662	0.7092	0.6455	0.6603	0.6843	0.5314	0.5314	0.7083	0.6288	0.5084
DeepMeSH3	0.6535	0.6971	0.6319	0.6470	0.6623	0.5024	0.5030	0.6965	0.6155	0.4932
DeepMeSH4	0.6678	0.7123	0.6461	0.6621	0.6843	0.5330	0.5330	0.7117	0.6290	0.5105
xgx	0.5678	0.6762	0.5045	0.5596	0.6040	0.3458	0.3706	0.6668	0.4944	0.4044
xgx1	0.5601	0.6666	0.4955	0.5518	0.6149	0.3324	0.3613	0.6635	0.4847	0.3976
xgx0	0.5734	0.6646	0.5202	0.5651	0.6061	0.3552	0.3778	0.6557	0.5094	0.4098
AUTH-Atypon 1	0.5739	0.6600	0.5217	0.5634	0.6035	0.3440	0.3565	0.6540	0.5113	0.4076

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5032	0.7972	0.7009	0.7275	0.5558	0.4850
Default MTI	0.5179	0.7693	0.7471	0.7412	0.5399	0.5238
iria-1	0.4435	0.6229	0.7260	0.6558	0.4247	0.4882
iria-2	0.4478	0.6280	0.7326	0.6617	0.4285	0.4931
DeepMeSH1	0.5170	0.8056	0.7164	0.7421	0.5647	0.4995
DeepMeSH2	0.5350	0.8176	0.7341	0.7577	0.5806	0.5188
DeepMeSH3	0.5239	0.8098	0.7235	0.7480	0.5705	0.5072
DeepMeSH4	0.5357	0.8185	0.7342	0.7583	0.5815	0.5193
xgx	0.4588	0.8111	0.6076	0.6737	0.5544	0.4132
xgx1	0.4540	0.8034	0.5988	0.6660	0.5500	0.4076
xgx0	0.4656	0.8048	0.6224	0.6800	0.5529	0.4255
AUTH-Atypon 1	0.4654	0.8078	0.6218	0.6812	0.5516	0.4255

+ Test batch 1, week 4

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6212	0.6681	0.5981	0.6114	0.6184	0.5483	0.5244	0.6598	0.5869	0.4589
Default MTI	0.6328	0.6381	0.6505	0.6258	0.5854	0.5926	0.5475	0.6316	0.6340	0.4723
iria-1	0.5240	0.4632	0.6091	0.5132	0.4409	0.4840	0.4290	0.4678	0.5956	0.3579
iria-3	0.4913	0.4334	0.5691	0.4799	0.4013	0.4191	0.3723	0.4387	0.5583	0.3284
iria-4	0.2692	0.2593	0.3094	0.2678	0.4109	0.3933	0.3334	0.2469	0.2960	0.1609
DeepMeSH1	0.6454	0.6820	0.6245	0.6353	0.6357	0.4871	0.4820	0.6845	0.6105	0.4815
DeepMeSH2	0.6647	0.7000	0.6468	0.6559	0.6603	0.5392	0.5303	0.7017	0.6313	0.5045
DeepMeSH3	0.6522	0.6884	0.6321	0.6426	0.6411	0.5117	0.5030	0.6913	0.6173	0.4894
DeepMeSH4	0.6664	0.7030	0.6478	0.6580	0.6630	0.5428	0.5337	0.7050	0.6318	0.5068
xgx3	0.6003	0.7268	0.5188	0.5838	0.6311	0.3829	0.4011	0.7244	0.5125	0.4312
AUTH-Atypon 1	0.5755	0.6506	0.5291	0.5623	0.5961	0.3638	0.3667	0.6464	0.5186	0.4074

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5028	0.7878	0.7017	0.7221	0.5495	0.4894
Default MTI	0.5170	0.7586	0.7501	0.7358	0.5334	0.5292
iria-1	0.4478	0.6228	0.7393	0.6609	0.4246	0.4983
iria-3	0.4276	0.5970	0.7073	0.6325	0.4063	0.4752
iria-4	0.2941	0.4864	0.5535	0.4972	0.2819	0.3386
DeepMeSH1	0.5159	0.8012	0.7138	0.7376	0.5616	0.5006
DeepMeSH2	0.5343	0.8121	0.7319	0.7530	0.5767	0.5211
DeepMeSH3	0.5230	0.8051	0.7210	0.7436	0.5673	0.5082
DeepMeSH4	0.5358	0.8131	0.7326	0.7538	0.5791	0.5220
xgx3	0.4719	0.8373	0.6126	0.6845	0.5747	0.4233
AUTH-Atypon 1	0.4685	0.7968	0.6290	0.6798	0.5471	0.4343

+ Test batch 1, week 5

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6277	0.6820	0.6024	0.6195	0.6368	0.5402	0.5228	0.6731	0.5881	0.4670
Default MTI	0.6383	0.6515	0.6498	0.6320	0.6064	0.5829	0.5476	0.6456	0.6311	0.4791
Search system 1	0.2311	0.1854	0.3376	0.2286	0.3927	0.4482	0.3640	0.1795	0.3246	0.1334
DeepMeSH1	0.6461	0.6896	0.6241	0.6377	0.6398	0.4672	0.4687	0.6919	0.6061	0.4837
DeepMeSH2	0.6534	0.6970	0.6323	0.6455	0.6466	0.4897	0.4884	0.6992	0.6131	0.4919
DeepMeSH3	0.6534	0.6970	0.6323	0.6455	0.6466	0.4897	0.4884	0.6992	0.6131	0.4919
xgx	0.6193	0.6546	0.6039	0.6069	0.5851	0.4514	0.4508	0.6477	0.5933	0.4515
xgx2	0.6364	0.6897	0.6081	0.6256	0.6385	0.4532	0.4588	0.6832	0.5955	0.4713
xgx3	0.6383	0.6946	0.6062	0.6266	0.6444	0.4442	0.4503	0.6881	0.5953	0.4723
xgx1	0.6342	0.6785	0.6132	0.6235	0.6156	0.4628	0.4633	0.6712	0.6011	0.4686
xgx0	0.6293	0.6583	0.6199	0.6180	0.5885	0.4705	0.4646	0.6512	0.6089	0.4626
TDM KE 1	0.2311	0.1854	0.3376	0.2286	0.3927	0.4482	0.3640	0.1795	0.3246	0.1334
TDM KE 2	0.2312	0.1853	0.3382	0.2287	0.3913	0.4499	0.3654	0.1794	0.3251	0.1335
TDM KE 3	0.1569	0.2192	0.1337	0.1528	0.3583	0.1871	0.1948	0.2169	0.1229	0.0878
TDM KE 4	0.2089	0.1582	0.3534	0.2093	0.2742	0.4764	0.3618	0.1503	0.3422	0.1202
TDM KE 5	0.2036	0.1528	0.3525	0.2043	0.2665	0.4773	0.3604	0.1451	0.3412	0.1169
AttentionMeSH	0.5817	0.5575	0.6185	0.5672	0.4277	0.4204	0.3851	0.5549	0.6112	0.4103

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5066	0.7954	0.7102	0.7307	0.5566	0.4912
Default MTI	0.5211	0.7688	0.7549	0.7434	0.5417	0.5296
Search system 1	0.2597	0.3870	0.5949	0.4527	0.2114	0.3739
DeepMeSH1	0.5151	0.8002	0.7152	0.7374	0.5638	0.4985
DeepMeSH2	0.5219	0.8047	0.7234	0.7440	0.5693	0.5070
DeepMeSH3	0.5219	0.8047	0.7234	0.7440	0.5693	0.5070
xgx	0.4999	0.7816	0.7049	0.7196	0.5457	0.4890
xgx2	0.5110	0.8096	0.6986	0.7292	0.5699	0.4895
xgx3	0.5099	0.8125	0.6980	0.7302	0.5691	0.4881
xgx1	0.5116	0.7983	0.7115	0.7317	0.5614	0.4968
xgx0	0.5084	0.7827	0.7171	0.7280	0.5487	0.5007
TDM KE 1	0.2597	0.3870	0.5949	0.4527	0.2114	0.3739
TDM KE 2	0.2595	0.3866	0.5956	0.4527	0.2110	0.3742
TDM KE 3	0.2279	0.4858	0.3303	0.3677	0.2961	0.2071
TDM KE 4	0.2408	0.3407	0.6346	0.4286	0.1824	0.3998
TDM KE 5	0.2364	0.3319	0.6360	0.4217	0.1772	0.4012
AttentionMeSH	0.4713	0.6989	0.7243	0.6914	0.4757	0.4970

+ Test batch 2, week 1

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6258	0.6668	0.6098	0.6161	0.6108	0.5549	0.5297	0.6577	0.5968	0.4632
Default MTI	0.6361	0.6354	0.6591	0.6284	0.5811	0.5995	0.5540	0.6297	0.6426	0.4750
iria-1	0.5251	0.4646	0.6127	0.5145	0.4435	0.4816	0.4297	0.4689	0.5967	0.3592
DeepMeSH1	0.6475	0.6838	0.6308	0.6386	0.6318	0.4842	0.4819	0.6863	0.6129	0.4843
DeepMeSH2	0.6675	0.7024	0.6543	0.6601	0.6556	0.5385	0.5318	0.7031	0.6353	0.5085
DeepMeSH3	0.6540	0.6921	0.6373	0.6458	0.6382	0.5049	0.5008	0.6939	0.6185	0.4921
DeepMeSH4	0.6682	0.7055	0.6537	0.6610	0.6556	0.5381	0.5320	0.7057	0.6344	0.5094
xgx	0.6275	0.6696	0.6051	0.6152	0.6275	0.4418	0.4456	0.6654	0.5938	0.4599
Semantic NoSQL KE 1	0.2326	0.1839	0.3450	0.2296	0.3885	0.4518	0.3658	0.1782	0.3345	0.1339
Semantic NoSQL KE 2	0.2328	0.1840	0.3459	0.2298	0.3878	0.4544	0.3681	0.1783	0.3353	0.1341
Semantic NoSQL KE 3	0.1710	0.2229	0.1509	0.1661	0.3558	0.2144	0.2175	0.2205	0.1397	0.0959
Semantic NoSQL KE 4	0.2094	0.1575	0.3604	0.2100	0.2706	0.4839	0.3659	0.1490	0.3519	0.1206
Semantic NoSQL KE 5	0.2043	0.1522	0.3601	0.2051	0.2647	0.4872	0.3658	0.1439	0.3518	0.1174
PLM + VP	0.3552	0.3892	0.3295	0.3441	0.4438	0.0746	0.0741	0.3882	0.3274	0.2191

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5056	0.7814	0.7161	0.7256	0.5483	0.4979
Default MTI	0.5187	0.7527	0.7613	0.7378	0.5309	0.5358
iria-1	0.4489	0.6189	0.7423	0.6584	0.4257	0.5019
DeepMeSH1	0.5168	0.7965	0.7196	0.7375	0.5611	0.5042
DeepMeSH2	0.5353	0.8073	0.7395	0.7533	0.5758	0.5253
DeepMeSH3	0.5229	0.8024	0.7266	0.7439	0.5671	0.5101
DeepMeSH4	0.5358	0.8098	0.7394	0.7545	0.5776	0.5247
xgx	0.5053	0.8010	0.6933	0.7216	0.5596	0.4868
Semantic NoSQL KE 1	0.2591	0.3804	0.6003	0.4496	0.2092	0.3797
Semantic NoSQL KE 2	0.2589	0.3800	0.6009	0.4496	0.2088	0.3800
Semantic NoSQL KE 3	0.2361	0.4845	0.3552	0.3830	0.2923	0.2223
Semantic NoSQL KE 4	0.2406	0.3350	0.6386	0.4246	0.1816	0.4048
Semantic NoSQL KE 5	0.2365	0.3260	0.6406	0.4176	0.1766	0.4070
PLM + VP	0.3115	0.5884	0.3899	0.4462	0.3985	0.2744

+ Test batch 2, week 2

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6296	0.6745	0.6053	0.6177	0.6312	0.5511	0.5288	0.6684	0.5949	0.4653
Default MTI	0.6454	0.6524	0.6550	0.6353	0.6054	0.5973	0.5573	0.6493	0.6416	0.4833
iria-1	0.5346	0.4756	0.6146	0.5225	0.4501	0.4849	0.4318	0.4812	0.6012	0.3669
DeepMeSH1	0.6527	0.6904	0.6295	0.6412	0.6458	0.4844	0.4832	0.6952	0.6151	0.4883
DeepMeSH2	0.6732	0.7085	0.6530	0.6627	0.6713	0.5383	0.5327	0.7125	0.6380	0.5126
DeepMeSH3	0.6593	0.6979	0.6373	0.6488	0.6476	0.5074	0.5033	0.7018	0.6218	0.4963
DeepMeSH4	0.6751	0.7128	0.6534	0.6649	0.6755	0.5415	0.5360	0.7167	0.6380	0.5150
xgx	0.6414	0.6804	0.6146	0.6279	0.6325	0.4616	0.4651	0.6823	0.6051	0.4735
xgx2	0.6211	0.6906	0.5718	0.6072	0.6430	0.3783	0.3924	0.6961	0.5606	0.4524
xgx3	0.5925	0.6844	0.5284	0.5777	0.6380	0.2894	0.3095	0.6929	0.5176	0.4229
xgx1	0.6343	0.6872	0.5967	0.6208	0.6443	0.4178	0.4279	0.6911	0.5862	0.4666
xgx0	0.6376	0.6804	0.6088	0.6244	0.6326	0.4418	0.4473	0.6831	0.5978	0.4700
AttentionMeSH	0.6195	0.6657	0.5821	0.6006	0.5881	0.3751	0.3845	0.6650	0.5799	0.4471
Semantic NoSQL KE 1	0.2286	0.1826	0.3355	0.2257	0.3944	0.4533	0.3637	0.1761	0.3258	0.1314
Semantic NoSQL KE 2	0.2286	0.1825	0.3360	0.2257	0.3925	0.4542	0.3644	0.1759	0.3262	0.1314
Semantic NoSQL KE 3	0.1681	0.2218	0.1475	0.1632	0.3631	0.2186	0.2205	0.2199	0.1361	0.0941
Semantic NoSQL KE 4	0.2069	0.1567	0.3542	0.2076	0.2743	0.4860	0.3630	0.1476	0.3456	0.1191
Semantic NoSQL KE 5	0.2018	0.1515	0.3537	0.2028	0.2672	0.4875	0.3624	0.1425	0.3452	0.1160
PLM + VP	0.0979	0.4425	0.0636	0.1036	0.4712	0.0687	0.0693	0.3536	0.0568	0.0580
AttentionMeSH2	0.5656	0.5492	0.5889	0.5498	0.3783	0.4207	0.3737	0.5467	0.5859	0.3952
AttentionMeSH3	0.5976	0.6184	0.5810	0.5794	0.4995	0.3909	0.3806	0.6177	0.5788	0.4247
AttentionMeSH4	0.6022	0.6334	0.5763	0.5832	0.5543	0.3756	0.3734	0.6324	0.5747	0.4288
AttentionMeSH5	0.6030	0.6391	0.5731	0.5836	0.5817	0.3644	0.3658	0.6379	0.5717	0.4294
MeSHmallow-1	0.1027	0.4829	0.0661	0.1083	0.4476	0.0690	0.0704	0.3788	0.0594	0.0606
MeSHmallow-2	0.0979	0.4425	0.0636	0.1036	0.4712	0.0687	0.0693	0.3536	0.0568	0.0580

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5074	0.7913	0.7106	0.7289	0.5551	0.4939
Default MTI	0.5248	0.7689	0.7551	0.7442	0.5457	0.5321
iria-1	0.4535	0.6304	0.7417	0.6659	0.4326	0.5021
DeepMeSH1	0.5202	0.8034	0.7171	0.7404	0.5682	0.5034
DeepMeSH2	0.5385	0.8147	0.7359	0.7568	0.5831	0.5238
DeepMeSH3	0.5268	0.8075	0.7247	0.7467	0.5731	0.5109
DeepMeSH4	0.5399	0.8171	0.7372	0.7586	0.5855	0.5243
xgx	0.5152	0.8054	0.7056	0.7341	0.5670	0.4964
xgx2	0.4949	0.8158	0.6645	0.7130	0.5713	0.4600
xgx3	0.4678	0.8171	0.6171	0.6823	0.5648	0.4224
xgx1	0.5072	0.8125	0.6871	0.7260	0.5695	0.4809
xgx0	0.5124	0.8053	0.7005	0.7308	0.5664	0.4919
AttentionMeSH	0.4866	0.7914	0.6631	0.6999	0.5481	0.4624
Semantic NoSQL KE 1	0.2585	0.3846	0.5947	0.4511	0.2097	0.3756
Semantic NoSQL KE 2	0.2584	0.3843	0.5952	0.4510	0.2095	0.3759
Semantic NoSQL KE 3	0.2341	0.4855	0.3502	0.3808	0.2924	0.2190
Semantic NoSQL KE 4	0.2396	0.3387	0.6354	0.4270	0.1814	0.4012
Semantic NoSQL KE 5	0.2354	0.3298	0.6370	0.4199	0.1763	0.4029
PLM + VP	0.1350	0.6791	0.1238	0.1948	0.3722	0.0880
AttentionMeSH2	0.4623	0.6921	0.6963	0.6749	0.4732	0.4788
AttentionMeSH3	0.4769	0.7506	0.6766	0.6909	0.5182	0.4673
AttentionMeSH4	0.4782	0.7629	0.6692	0.6915	0.5277	0.4628
AttentionMeSH5	0.4777	0.7676	0.6649	0.6907	0.5309	0.4598
MeSHmallow-1	0.1296	0.6934	0.1221	0.1929	0.3643	0.0843
MeSHmallow-2	0.1350	0.6791	0.1238	0.1948	0.3722	0.0880

+ Test batch 2, week 3

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6268	0.6787	0.6044	0.6191	0.6359	0.5595	0.5360	0.6678	0.5907	0.4661
Default MTI	0.6420	0.6534	0.6579	0.6374	0.6072	0.6043	0.5615	0.6447	0.6394	0.4843
iria-1	0.5351	0.4798	0.6127	0.5262	0.4629	0.4863	0.4350	0.4840	0.5983	0.3697
DeepMeSH1	0.6518	0.6970	0.6261	0.6437	0.6525	0.4889	0.4883	0.6983	0.6110	0.4903
DeepMeSH2	0.6719	0.7174	0.6492	0.6656	0.6755	0.5431	0.5376	0.7166	0.6325	0.5148
DeepMeSH3	0.6588	0.7063	0.6332	0.6518	0.6595	0.5129	0.5099	0.7062	0.6175	0.4988
DeepMeSH4	0.6727	0.7203	0.6484	0.6664	0.6782	0.5458	0.5415	0.7195	0.6315	0.5156
xgx	0.6380	0.6898	0.6046	0.6257	0.6462	0.4603	0.4656	0.6851	0.5970	0.4710
xgx1	0.6153	0.7027	0.5574	0.6022	0.6546	0.3713	0.3892	0.7003	0.5487	0.4477
xgx0	0.6344	0.6895	0.5988	0.6221	0.6458	0.4416	0.4496	0.6860	0.5899	0.4675
AttentionMeSH	0.6099	0.6493	0.5815	0.5955	0.5479	0.3841	0.3810	0.6468	0.5771	0.4405
Semantic NoSQL KE 1	0.2292	0.1822	0.3330	0.2261	0.4000	0.4589	0.3704	0.1775	0.3235	0.1315
Semantic NoSQL KE 2	0.2294	0.1823	0.3335	0.2263	0.3985	0.4604	0.3718	0.1775	0.3241	0.1316
Semantic NoSQL KE 3	0.1692	0.2200	0.1484	0.1640	0.3648	0.2182	0.2214	0.2195	0.1377	0.0946
Semantic NoSQL KE 4	0.2090	0.1574	0.3525	0.2094	0.2811	0.4917	0.3700	0.1501	0.3438	0.1200
Semantic NoSQL KE 5	0.2037	0.1522	0.3513	0.2044	0.2750	0.4934	0.3696	0.1449	0.3425	0.1168
PLM + VP	0.3451	0.3828	0.3121	0.3336	0.4609	0.0738	0.0733	0.3812	0.3151	0.2113
AttentionMeSH2	0.6134	0.6586	0.5807	0.5989	0.5819	0.3804	0.3798	0.6554	0.5764	0.4440
AttentionMeSH3	0.6138	0.6601	0.5803	0.5992	0.5900	0.3780	0.3779	0.6568	0.5760	0.4444
AttentionMeSH4	0.6139	0.6607	0.5800	0.5993	0.5938	0.3767	0.3767	0.6574	0.5758	0.4446
AttentionMeSH5	0.6139	0.6609	0.5798	0.5993	0.5956	0.3755	0.3758	0.6576	0.5756	0.4445
MeSHmallow-1	0.1011	0.4636	0.0634	0.1049	0.4786	0.0687	0.0694	0.3872	0.0582	0.0585
MeSHmallow-2	0.0307	0.2809	0.4618	0.2905	0.0117	0.1640	0.0275	0.0159	0.4711	0.1826
MeSHmallow-3	0.0293	0.2755	0.4664	0.2890	0.0109	0.1665	0.0263	0.0151	0.4758	0.1814

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5071	0.7948	0.7044	0.7266	0.5562	0.4922
Default MTI	0.5238	0.7707	0.7520	0.7431	0.5442	0.5317
iria-1	0.4546	0.6353	0.7369	0.6679	0.4352	0.4991
DeepMeSH1	0.5193	0.8081	0.7140	0.7408	0.5685	0.5001
DeepMeSH2	0.5372	0.8196	0.7322	0.7564	0.5841	0.5198
DeepMeSH3	0.5261	0.8133	0.7206	0.7473	0.5749	0.5070
DeepMeSH4	0.5377	0.8215	0.7314	0.7570	0.5857	0.5193
xgx	0.5109	0.8121	0.6945	0.7293	0.5691	0.4875
xgx1	0.4888	0.8240	0.6491	0.7057	0.5746	0.4477
xgx0	0.5084	0.8111	0.6901	0.7264	0.5689	0.4830
AttentionMeSH	0.4828	0.7728	0.6727	0.6995	0.5318	0.4657
Semantic NoSQL KE 1	0.2612	0.3826	0.5860	0.4479	0.2127	0.3738
Semantic NoSQL KE 2	0.2610	0.3823	0.5867	0.4479	0.2123	0.3740
Semantic NoSQL KE 3	0.2371	0.4861	0.3463	0.3781	0.2980	0.2211
Semantic NoSQL KE 4	0.2433	0.3399	0.6299	0.4277	0.1846	0.4010
Semantic NoSQL KE 5	0.2392	0.3313	0.6311	0.4209	0.1797	0.4021
PLM + VP	0.3001	0.5945	0.3701	0.4360	0.4024	0.2541
AttentionMeSH2	0.4841	0.7803	0.6703	0.7011	0.5374	0.4641
AttentionMeSH3	0.4842	0.7817	0.6697	0.7012	0.5384	0.4636
AttentionMeSH4	0.4842	0.7822	0.6693	0.7012	0.5388	0.4633
AttentionMeSH5	0.4842	0.7825	0.6690	0.7011	0.5391	0.4630
MeSHmallow-1	0.1427	0.7223	0.1194	0.1913	0.4300	0.0906
MeSHmallow-2	0.2900	0.4554	0.5580	0.4217	0.3088	0.4088
MeSHmallow-3	0.2901	0.4489	0.5643	0.4217	0.3047	0.4133

+ Test batch 2, week 4

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6098	0.6642	0.5876	0.6023	0.6220	0.5402	0.5189	0.6554	0.5702	0.4485
Default MTI	0.6262	0.6412	0.6391	0.6203	0.5922	0.5829	0.5433	0.6345	0.6181	0.4666
iria-1	0.5222	0.4674	0.6022	0.5121	0.4538	0.4796	0.4286	0.4719	0.5846	0.3567
DeepMeSH1	0.6376	0.6844	0.6145	0.6296	0.6454	0.4718	0.4712	0.6860	0.5956	0.4746
DeepMeSH2	0.6556	0.7018	0.6348	0.6488	0.6642	0.5217	0.5174	0.7027	0.6143	0.4959
DeepMeSH3	0.6443	0.6920	0.6204	0.6362	0.6489	0.4924	0.4894	0.6935	0.6016	0.4817
DeepMeSH4	0.6569	0.7041	0.6352	0.6503	0.6682	0.5236	0.5191	0.7055	0.6145	0.4976
xgx	0.6285	0.6630	0.6112	0.6164	0.6122	0.4577	0.4585	0.6602	0.5998	0.4612
xgx1	0.6192	0.6673	0.5912	0.6069	0.6174	0.4156	0.4230	0.6658	0.5787	0.4512
xgx0	0.6238	0.6618	0.6039	0.6117	0.6125	0.4421	0.4440	0.6603	0.5911	0.4561
AttentionMeSH	0.6069	0.6446	0.5811	0.5917	0.5702	0.3879	0.3868	0.6431	0.5746	0.4361
Semantic NoSQL KE 1	0.2276	0.1796	0.3416	0.2250	0.3894	0.4493	0.3613	0.1747	0.3267	0.1307
Semantic NoSQL KE 2	0.2276	0.1795	0.3420	0.2250	0.3873	0.4505	0.3621	0.1745	0.3271	0.1307
Semantic NoSQL KE 3	0.1666	0.2219	0.1456	0.1621	0.3535	0.2074	0.2108	0.2193	0.1343	0.0934
Semantic NoSQL KE 4	0.2061	0.1540	0.3587	0.2064	0.2703	0.4790	0.3600	0.1468	0.3459	0.1181
Semantic NoSQL KE 5	0.2012	0.1492	0.3577	0.2018	0.2632	0.4818	0.3598	0.1420	0.3451	0.1151
AttentionMeSH2	0.6078	0.6476	0.5807	0.5926	0.5808	0.3859	0.3856	0.6456	0.5742	0.4371
AttentionMeSH3	0.6081	0.6485	0.5805	0.5929	0.5852	0.3854	0.3852	0.6463	0.5741	0.4374
AttentionMeSH4	0.6082	0.6488	0.5805	0.5930	0.5872	0.3850	0.3848	0.6466	0.5741	0.4375
AttentionMeSH5	0.6083	0.6489	0.5804	0.5931	0.5885	0.3848	0.3847	0.6468	0.5740	0.4375
Access Inn MAIstro	0.2527	0.2304	0.3202	0.2525	0.3663	0.4028	0.3341	0.2172	0.3021	0.1496

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.4924	0.7870	0.6916	0.7152	0.5438	0.4775
Default MTI	0.5086	0.7640	0.7380	0.7313	0.5326	0.5154
iria-1	0.4409	0.6253	0.7276	0.6563	0.4239	0.4858
DeepMeSH1	0.5078	0.8034	0.7032	0.7323	0.5600	0.4885
DeepMeSH2	0.5211	0.8133	0.7187	0.7454	0.5710	0.5037
DeepMeSH3	0.5124	0.8070	0.7081	0.7366	0.5647	0.4930
DeepMeSH4	0.5233	0.8147	0.7194	0.7465	0.5732	0.5057
xgx	0.5038	0.7946	0.7019	0.7261	0.5522	0.4893
xgx1	0.4948	0.8007	0.6827	0.7171	0.5538	0.4728
xgx0	0.5008	0.7943	0.6965	0.7226	0.5520	0.4841
AttentionMeSH	0.4777	0.7720	0.6706	0.6982	0.5267	0.4620
Semantic NoSQL KE 1	0.2583	0.3792	0.5971	0.4473	0.2093	0.3770
Semantic NoSQL KE 2	0.2581	0.3786	0.5977	0.4471	0.2089	0.3773
Semantic NoSQL KE 3	0.2346	0.4848	0.3449	0.3761	0.2960	0.2188
Semantic NoSQL KE 4	0.2410	0.3339	0.6386	0.4237	0.1815	0.4056
Semantic NoSQL KE 5	0.2373	0.3258	0.6402	0.4172	0.1769	0.4076
AttentionMeSH2	0.4780	0.7746	0.6699	0.6986	0.5286	0.4613
AttentionMeSH3	0.4781	0.7753	0.6696	0.6988	0.5291	0.4611
AttentionMeSH4	0.4782	0.7755	0.6695	0.6988	0.5293	0.4610
AttentionMeSH5	0.4782	0.7756	0.6695	0.6988	0.5294	0.4610
Access Inn MAIstro	0.2863	0.4553	0.5653	0.4825	0.2659	0.3472

+ Test batch 2, week 5

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6307	0.6830	0.6003	0.6192	0.6420	0.5647	0.5440	0.6761	0.5910	0.4687
Default MTI	0.6450	0.6538	0.6537	0.6360	0.6095	0.6067	0.5668	0.6491	0.6409	0.4850
iria-1	0.5297	0.4715	0.6102	0.5192	0.4618	0.4922	0.4411	0.4759	0.5974	0.3643
DeepMeSH1	0.6463	0.6889	0.6198	0.6362	0.6565	0.4861	0.4867	0.6930	0.6055	0.4833
DeepMeSH2	0.6702	0.7128	0.6453	0.6613	0.6824	0.5436	0.5401	0.7155	0.6302	0.5115
DeepMeSH3	0.6549	0.6995	0.6281	0.6454	0.6613	0.5083	0.5073	0.7027	0.6131	0.4930
DeepMeSH4	0.6719	0.7174	0.6449	0.6629	0.6842	0.5438	0.5419	0.7200	0.6299	0.5132
xgx	0.6335	0.6680	0.6120	0.6203	0.6217	0.4604	0.4622	0.6656	0.6045	0.4665
xgx0	0.6293	0.6684	0.6049	0.6163	0.6257	0.4418	0.4464	0.6665	0.5961	0.4623
AttentionMeSH	0.6295	0.6660	0.6011	0.6138	0.5907	0.4142	0.4126	0.6645	0.5980	0.4610
AttentionMeSH2	0.6305	0.6685	0.6010	0.6148	0.6010	0.4133	0.4122	0.6668	0.5979	0.4620
AttentionMeSH3	0.6307	0.6691	0.6009	0.6150	0.6038	0.4128	0.4117	0.6674	0.5978	0.4623
AttentionMeSH4	0.6308	0.6695	0.6008	0.6151	0.6054	0.4123	0.4113	0.6676	0.5978	0.4624
AttentionMeSH5	0.6309	0.6699	0.6008	0.6152	0.6070	0.4122	0.4112	0.6679	0.5977	0.4625

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5069	0.7970	0.7026	0.7275	0.5587	0.4893
Default MTI	0.5237	0.7706	0.7520	0.7438	0.5442	0.5308
iria-1	0.4504	0.6284	0.7364	0.6633	0.4297	0.4979
DeepMeSH1	0.5159	0.8040	0.7088	0.7366	0.5646	0.4976
DeepMeSH2	0.5361	0.8183	0.7293	0.7551	0.5831	0.5187
DeepMeSH3	0.5234	0.8108	0.7160	0.7437	0.5724	0.5049
DeepMeSH4	0.5362	0.8221	0.7278	0.7558	0.5849	0.5178
xgx	0.5109	0.7961	0.7040	0.7286	0.5584	0.4952
xgx0	0.5071	0.7976	0.6983	0.7258	0.5582	0.4891
AttentionMeSH	0.4970	0.7824	0.6909	0.7151	0.5426	0.4824
AttentionMeSH2	0.4974	0.7843	0.6904	0.7156	0.5439	0.4821
AttentionMeSH3	0.4975	0.7848	0.6901	0.7157	0.5444	0.4820
AttentionMeSH4	0.4976	0.7852	0.6901	0.7157	0.5446	0.4820
AttentionMeSH5	0.4976	0.7855	0.6900	0.7158	0.5447	0.4819

+ Test batch 3, week 1

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6235	0.6806	0.5946	0.6135	0.6447	0.5557	0.5378	0.6713	0.5821	0.4620
Default MTI	0.6391	0.6538	0.6486	0.6322	0.6163	0.5997	0.5631	0.6463	0.6320	0.4807
iria-1	0.5223	0.4657	0.6046	0.5129	0.4562	0.4835	0.4338	0.4697	0.5882	0.3578
xgx	0.6212	0.6540	0.6032	0.6083	0.6044	0.4779	0.4745	0.6506	0.5942	0.4528
xgx1	0.6211	0.6425	0.6133	0.6084	0.6057	0.4596	0.4571	0.6400	0.6032	0.4525
xgx0	0.6241	0.6607	0.6036	0.6111	0.6223	0.4636	0.4653	0.6569	0.5944	0.4554
AttentionMeSH	0.6291	0.6740	0.5993	0.6145	0.6167	0.4216	0.4254	0.6698	0.5931	0.4606
Semantic NoSQL KE 1	0.2363	0.1871	0.3465	0.2331	0.4035	0.4490	0.3654	0.1816	0.3383	0.1361
Semantic NoSQL KE 2	0.2363	0.1870	0.3470	0.2331	0.4017	0.4504	0.3665	0.1815	0.3388	0.1361
Semantic NoSQL KE 3	0.1710	0.2242	0.1497	0.1662	0.3551	0.2027	0.2083	0.2213	0.1393	0.0959
Semantic NoSQL KE 4	0.2126	0.1598	0.3624	0.2129	0.2761	0.4768	0.3624	0.1518	0.3546	0.1225
Semantic NoSQL KE 5	0.2074	0.1546	0.3613	0.2080	0.2693	0.4772	0.3612	0.1467	0.3536	0.1193
AttentionMeSH2	0.6296	0.6753	0.5992	0.6150	0.6229	0.4213	0.4253	0.6710	0.5930	0.4611
AttentionMeSH3	0.6297	0.6757	0.5992	0.6151	0.6246	0.4211	0.4252	0.6712	0.5930	0.4613
AttentionMeSH4	0.6298	0.6759	0.5992	0.6151	0.6257	0.4210	0.4251	0.6714	0.5930	0.4613
AttentionMeSH5	0.6298	0.6760	0.5992	0.6152	0.6268	0.4210	0.4251	0.6715	0.5930	0.4614
MeSHmallow-1	0.0985	0.4542	0.0639	0.1055	0.4741	0.0669	0.0674	0.3736	0.0567	0.0592
MeSHmallow-2	0.0985	0.4542	0.0639	0.1055	0.4741	0.0669	0.0674	0.3736	0.0567	0.0592
MeSHmallow-3	0.0985	0.4542	0.0639	0.1055	0.4741	0.0669	0.0674	0.3736	0.0567	0.0592
UMass Amherst T2T	0.1081	0.7339	0.0653	0.1180	0.7339	0.0001	0.0001	0.7339	0.0584	0.0653
3_runs_ckp4	0.1899	0.1942	0.1872	0.1841	0.1278	0.0011	0.0003	0.1924	0.1876	0.1053
3_runs	0.2006	0.2029	0.1978	0.1938	0.0617	0.0015	0.0005	0.2005	0.2007	0.1130
2_runs_ckp2	0.1978	0.1899	0.2045	0.1906	0.0739	0.0016	0.0005	0.1893	0.2072	0.1101

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5013	0.7937	0.7014	0.7245	0.5542	0.4847
Default MTI	0.5186	0.7685	0.7497	0.7409	0.5413	0.5258
iria-1	0.4460	0.6235	0.7364	0.6595	0.4253	0.4949
xgx	0.5019	0.7847	0.7057	0.7235	0.5456	0.4908
xgx1	0.5042	0.7793	0.7138	0.7256	0.5428	0.4973
xgx0	0.5026	0.7920	0.7026	0.7245	0.5503	0.4890
AttentionMeSH	0.4958	0.7879	0.6948	0.7181	0.5433	0.4820
Semantic NoSQL KE 1	0.2599	0.3812	0.5967	0.4495	0.2103	0.3772
Semantic NoSQL KE 2	0.2597	0.3808	0.5974	0.4494	0.2099	0.3776
Semantic NoSQL KE 3	0.2343	0.4827	0.3470	0.3771	0.2928	0.2193
Semantic NoSQL KE 4	0.2432	0.3362	0.6348	0.4249	0.1841	0.4046
Semantic NoSQL KE 5	0.2389	0.3276	0.6363	0.4181	0.1790	0.4056
AttentionMeSH2	0.4961	0.7891	0.6945	0.7185	0.5442	0.4819
AttentionMeSH3	0.4962	0.7894	0.6944	0.7185	0.5443	0.4819
AttentionMeSH4	0.4962	0.7894	0.6944	0.7185	0.5443	0.4818
AttentionMeSH5	0.4961	0.7895	0.6943	0.7185	0.5444	0.4818
MeSHmallow-1	0.1327	0.6805	0.1203	0.1917	0.3733	0.0854
MeSHmallow-2	0.1327	0.6805	0.1203	0.1917	0.3733	0.0854
MeSHmallow-3	0.1327	0.6805	0.1203	0.1917	0.3733	0.0854
UMass Amherst T2T	0.0786	0.7522	0.0735	0.1320	0.2904	0.0465
3_runs_ckp4	0.2114	0.2800	0.3138	0.2857	0.2160	0.2209
3_runs	0.2241	0.3707	0.3212	0.3320	0.2408	0.2245
2_runs_ckp2	0.2090	0.3376	0.2391	0.2697	0.2353	0.1999

+ Test batch 3, week 2

BioASQ Participants Area

Test Results for Task 6a

+ Dry Run 1

Annotated articles:27480/28563

Flat Measures

Hierarchical Measures

+ Dry Run 2

Annotated articles:198/255

Flat Measures

Hierarchical Measures

+ Test batch 1, week 1

Annotated articles:6639/7240

Flat Measures

Hierarchical Measures

+ Test batch 1, week 2

Annotated articles:7499/7678

Flat Measures

Hierarchical Measures

+ Test batch 1, week 3

Annotated articles:10319/10488

Flat Measures

Hierarchical Measures

+ Test batch 1, week 4

Annotated articles:6073/6225

Flat Measures

Hierarchical Measures

+ Test batch 1, week 5

Annotated articles:6486/6617

Flat Measures

Hierarchical Measures

+ Test batch 2, week 1

Annotated articles:6118/6239

Flat Measures

Hierarchical Measures

+ Test batch 2, week 2

Annotated articles:6803/7152

Flat Measures

Hierarchical Measures

+ Test batch 2, week 3

Annotated articles:6575/7113

Flat Measures

Hierarchical Measures

+ Test batch 2, week 4

Annotated articles:5412/5833

Flat Measures

Hierarchical Measures

+ Test batch 2, week 5

Annotated articles:6606/7379

Flat Measures

Hierarchical Measures

+ Test batch 3, week 1

Annotated articles:5768/6469

Flat Measures

Hierarchical Measures

+ Test batch 3, week 2

Annotated articles:5501/6544

Flat Measures