BioASQ Participants Area

Test Results for Task 8a

The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.

+ Dry Run 1

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
BioASQ_Baseline	0.0003	0.0006	0.0002	0.0002	0.0004	0.0003	0.0002	0.0004	0.0002	0.0001

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
BioASQ_Baseline	0.0991	0.1488	0.0898	0.0994	0.1486	0.0867

+ Test batch 1, week 1

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6495	0.6808	0.6468	0.6451	0.6360	0.5899	0.5624	0.6684	0.6316	0.4953
Default MTI	0.6501	0.6436	0.6810	0.6448	0.6039	0.6207	0.5738	0.6361	0.6647	0.4925
Transformer-inspired	0.3669	0.3611	0.3586	0.3440	0.2285	0.0128	0.0078	0.3592	0.3750	0.2179
deepmesh_dmiip_fdu	0.7049	0.7848	0.6567	0.7011	0.7460	0.5653	0.5753	0.7869	0.6383	0.5559
deepmesh_dmiip_fdu_	0.3998	0.7839	0.2691	0.3802	0.7218	0.0457	0.0613	0.7856	0.2682	0.2493
attention_dmiip_fdu	0.6963	0.7132	0.6957	0.6896	0.6730	0.5709	0.5564	0.7067	0.6862	0.5419

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5295	0.7897	0.7421	0.7476	0.5628	0.5256
Default MTI	0.5291	0.7596	0.7747	0.7509	0.5369	0.5483
Transformer-inspired	0.3305	0.5461	0.4438	0.4676	0.3817	0.3125
deepmesh_dmiip_fdu	0.5646	0.8638	0.7383	0.7817	0.6294	0.5316
deepmesh_dmiip_fdu_	0.2851	0.8818	0.2964	0.4182	0.5736	0.2017
attention_dmiip_fdu	0.5642	0.8120	0.7793	0.7807	0.5879	0.5651

+ Test batch 1, week 2

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6584	0.6824	0.6536	0.6495	0.6351	0.5925	0.5623	0.6749	0.6426	0.5009
Default MTI	0.6593	0.6490	0.6881	0.6506	0.6073	0.6259	0.5779	0.6440	0.6753	0.5000
deepmesh_dmiip_fdu	0.7059	0.7816	0.6555	0.6984	0.7337	0.5590	0.5668	0.7854	0.6409	0.5538
deepmesh_dmiip_fdu_	0.6684	0.7475	0.6174	0.6606	0.6969	0.5047	0.5155	0.7515	0.6018	0.5099
attention_dmiip_fdu	0.6957	0.7092	0.6933	0.6863	0.6591	0.5711	0.5519	0.7051	0.6867	0.5390

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5349	0.7887	0.7495	0.7508	0.5662	0.5327
Default MTI	0.5370	0.7608	0.7800	0.7535	0.5449	0.5564
deepmesh_dmiip_fdu	0.5632	0.8600	0.7372	0.7790	0.6253	0.5319
deepmesh_dmiip_fdu_	0.5288	0.8402	0.7033	0.7494	0.5976	0.4949
attention_dmiip_fdu	0.5636	0.8085	0.7794	0.7786	0.5869	0.5647

+ Test batch 1, week 3

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6408	0.6782	0.6284	0.6339	0.6391	0.5778	0.5527	0.6681	0.6156	0.4821
Default MTI	0.6479	0.6485	0.6679	0.6407	0.6130	0.6107	0.5668	0.6418	0.6540	0.4884
deepmesh_dmiip_fdu	0.6884	0.7815	0.6310	0.6833	0.7442	0.5415	0.5584	0.7840	0.6136	0.5357
deepmesh_dmiip_fdu_	0.6499	0.6736	0.6397	0.6398	0.6233	0.4804	0.4704	0.6701	0.6308	0.4865
attention_dmiip_fdu	0.6499	0.6736	0.6397	0.6398	0.6233	0.4804	0.4704	0.6701	0.6308	0.4865
NLM CNN	0.6520	0.6850	0.6332	0.6403	0.6173	0.4871	0.4865	0.6816	0.6249	0.4870
dmiip_fdu	0.6825	0.7106	0.6716	0.6743	0.6733	0.5461	0.5359	0.7041	0.6623	0.5247

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5186	0.7886	0.7301	0.7398	0.5573	0.5103
Default MTI	0.5251	0.7634	0.7647	0.7473	0.5386	0.5383
deepmesh_dmiip_fdu	0.5479	0.8629	0.7168	0.7677	0.6202	0.5109
deepmesh_dmiip_fdu_	0.5197	0.7863	0.7314	0.7410	0.5531	0.5140
attention_dmiip_fdu	0.5197	0.7863	0.7314	0.7410	0.5531	0.5140
NLM CNN	0.5216	0.8020	0.7227	0.7429	0.5629	0.5100
dmiip_fdu	0.5504	0.8117	0.7599	0.7685	0.5824	0.5458

+ Test batch 1, week 4

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6555	0.6847	0.6521	0.6501	0.6321	0.5876	0.5559	0.6732	0.6388	0.5018
Default MTI	0.6523	0.6442	0.6835	0.6453	0.5950	0.6165	0.5657	0.6372	0.6681	0.4931
deepmesh_dmiip_fdu	0.7040	0.7871	0.6524	0.6991	0.7451	0.5543	0.5639	0.7886	0.6358	0.5540
deepmesh_dmiip_fdu_	0.7087	0.7844	0.6629	0.7036	0.7400	0.5697	0.5762	0.7840	0.6467	0.5609
attention_dmiip_fdu	0.6961	0.7143	0.6928	0.6881	0.6619	0.5675	0.5502	0.7070	0.6855	0.5406
NLM CNN	0.6640	0.6862	0.6573	0.6540	0.6067	0.5090	0.4995	0.6794	0.6492	0.5013
dmiip_fdu	0.7012	0.7228	0.6971	0.6945	0.6721	0.5811	0.5655	0.7141	0.6887	0.5497

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5349	0.7901	0.7465	0.7501	0.5667	0.5317
Default MTI	0.5308	0.7575	0.7770	0.7500	0.5379	0.5515
deepmesh_dmiip_fdu	0.5618	0.8645	0.7334	0.7785	0.6275	0.5284
deepmesh_dmiip_fdu_	0.5689	0.8622	0.7428	0.7827	0.6299	0.5395
attention_dmiip_fdu	0.5637	0.8128	0.7760	0.7787	0.5894	0.5630
NLM CNN	0.5324	0.8002	0.7387	0.7507	0.5652	0.5281
dmiip_fdu	0.5708	0.8186	0.7793	0.7833	0.5979	0.5690

+ Test batch 1, week 5

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6496	0.6917	0.6367	0.6442	0.6440	0.5711	0.5487	0.6794	0.6223	0.4941
Default MTI	0.6524	0.6564	0.6728	0.6467	0.6140	0.6066	0.5662	0.6485	0.6563	0.4950
deepmesh_dmiip_fdu	0.6921	0.7890	0.6312	0.6864	0.7535	0.5344	0.5520	0.7926	0.6142	0.5385
deepmesh_dmiip_fdu_	0.7291	0.8108	0.6776	0.7234	0.7864	0.5902	0.6084	0.8144	0.6599	0.5923
attention_dmiip_fdu	0.6860	0.7127	0.6733	0.6765	0.6772	0.5507	0.5411	0.7085	0.6649	0.5271
NLM CNN	0.6540	0.6867	0.6334	0.6414	0.6202	0.4869	0.4846	0.6842	0.6263	0.4880
dmiip_fdu	0.7220	0.7505	0.7114	0.7164	0.7236	0.6019	0.5975	0.7447	0.7007	0.5842
TransforMeSH_2	0.3216	0.3090	0.2887	0.2882	0.2375	0.0047	0.0023	0.3365	0.3079	0.1797

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5279	0.7992	0.7369	0.7490	0.5676	0.5176
Default MTI	0.5320	0.7703	0.7713	0.7545	0.5451	0.5448
deepmesh_dmiip_fdu	0.5528	0.8703	0.7193	0.7728	0.6284	0.5127
deepmesh_dmiip_fdu_	0.6046	0.8837	0.7545	0.7997	0.6696	0.5709
attention_dmiip_fdu	0.5559	0.8164	0.7646	0.7744	0.5887	0.5489
NLM CNN	0.5251	0.8049	0.7269	0.7468	0.5665	0.5129
dmiip_fdu	0.6082	0.8412	0.7912	0.8019	0.6391	0.6002
TransforMeSH_2	0.2737	0.4511	0.3764	0.3976	0.3176	0.2542

+ Test batch 2, week 1

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6499	0.6805	0.6402	0.6415	0.6339	0.5874	0.5586	0.6729	0.6285	0.4914
Default MTI	0.6511	0.6444	0.6760	0.6429	0.5968	0.6175	0.5697	0.6396	0.6632	0.4909
deepmesh_dmiip_fdu	0.6994	0.7862	0.6419	0.6917	0.7461	0.5498	0.5617	0.7902	0.6274	0.5460
deepmesh_dmiip_fdu_	0.7034	0.7836	0.6508	0.6949	0.7412	0.5649	0.5742	0.7861	0.6364	0.5503
attention_dmiip_fdu	0.6935	0.7116	0.6865	0.6833	0.6676	0.5631	0.5467	0.7072	0.6802	0.5353
NLM CNN	0.6625	0.6885	0.6454	0.6489	0.6121	0.4991	0.4924	0.6854	0.6411	0.4963
dmiip_fdu	0.6994	0.7197	0.6906	0.6894	0.6774	0.5761	0.5626	0.7155	0.6840	0.5433

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5256	0.7899	0.7382	0.7454	0.5605	0.5203
Default MTI	0.5274	0.7597	0.7704	0.7483	0.5369	0.5450
deepmesh_dmiip_fdu	0.5554	0.8651	0.7245	0.7732	0.6259	0.5197
deepmesh_dmiip_fdu_	0.5597	0.8638	0.7307	0.7759	0.6268	0.5272
attention_dmiip_fdu	0.5593	0.8131	0.7733	0.7771	0.5860	0.5586
NLM CNN	0.5278	0.8024	0.7320	0.7486	0.5645	0.5194
dmiip_fdu	0.5656	0.8197	0.7741	0.7810	0.5944	0.5629

+ Test batch 2, week 2

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6631	0.6931	0.6559	0.6567	0.6465	0.5857	0.5609	0.6844	0.6430	0.5083
Default MTI	0.6656	0.6598	0.6924	0.6590	0.6158	0.6222	0.5776	0.6539	0.6777	0.5085
test sys	0.6631	0.6931	0.6559	0.6567	0.6465	0.5857	0.5609	0.6844	0.6430	0.5083
deepmesh_dmiip_fdu	0.7043	0.7873	0.6486	0.6972	0.7443	0.5432	0.5562	0.7924	0.6338	0.5513
deepmesh_dmiip_fdu_	0.7223	0.7920	0.6723	0.7126	0.7513	0.5758	0.5867	0.7974	0.6602	0.5738
attention_dmiip_fdu	0.6970	0.7154	0.6896	0.6874	0.6671	0.5575	0.5443	0.7106	0.6838	0.5393
NLM CNN	0.6686	0.6937	0.6528	0.6558	0.6201	0.4999	0.4950	0.6914	0.6473	0.5036
dmiip_fdu	0.7144	0.7317	0.7060	0.7041	0.6895	0.5883	0.5779	0.7277	0.7015	0.5637
submit mesh index qa	0.6004	0.7496	0.5032	0.5758	0.6492	0.2670	0.2845	0.7377	0.5062	0.4243
attention	0.4003	0.3393	0.5000	0.3922	0.2309	0.2822	0.2439	0.3393	0.4880	0.2530

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5427	0.8026	0.7510	0.7592	0.5770	0.5366
Default MTI	0.5449	0.7744	0.7840	0.7635	0.5544	0.5616
test sys	0.5427	0.8026	0.7510	0.7592	0.5770	0.5366
deepmesh_dmiip_fdu	0.5625	0.8674	0.7315	0.7792	0.6313	0.5268
deepmesh_dmiip_fdu_	0.5837	0.8708	0.7495	0.7910	0.6455	0.5533
attention_dmiip_fdu	0.5654	0.8159	0.7755	0.7806	0.5937	0.5622
NLM CNN	0.5377	0.8101	0.7369	0.7550	0.5762	0.5279
dmiip_fdu	0.5862	0.8271	0.7859	0.7920	0.6139	0.5827
submit mesh index qa	0.4574	0.8450	0.5887	0.6663	0.5767	0.4036
attention	0.3513	0.4695	0.6445	0.5282	0.3137	0.4263

+ Test batch 2, week 3

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6468	0.6806	0.6455	0.6414	0.6381	0.5817	0.5563	0.6704	0.6248	0.4933
Default MTI	0.6461	0.6400	0.6819	0.6394	0.6000	0.6133	0.5672	0.6339	0.6589	0.4877
deepmesh_dmiip_fdu	0.7085	0.7558	0.6848	0.7014	0.7215	0.5907	0.5928	0.7587	0.6645	0.5581
deepmesh_dmiip_fdu_	0.7075	0.7562	0.6842	0.7011	0.7202	0.5943	0.5954	0.7568	0.6642	0.5583
attention_dmiip_fdu	0.7061	0.7540	0.6827	0.6994	0.7160	0.5885	0.5896	0.7564	0.6622	0.5551
NLM CNN	0.6521	0.6770	0.6400	0.6386	0.6133	0.4885	0.4845	0.6756	0.6302	0.4860
dmiip_fdu	0.7069	0.7540	0.6855	0.7007	0.7147	0.5964	0.5953	0.7540	0.6653	0.5575
submit mesh index qa	0.6453	0.6046	0.7220	0.6376	0.5627	0.6383	0.5717	0.5965	0.7029	0.4842
same as one	0.6424	0.6279	0.6886	0.6359	0.5890	0.6224	0.5700	0.6204	0.6662	0.4837
X-BERT BioASQ	0.3166	0.4626	0.2383	0.2943	0.3508	0.0541	0.0667	0.4652	0.2399	0.1848

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5235	0.7862	0.7421	0.7432	0.5571	0.5218
Default MTI	0.5193	0.7526	0.7737	0.7432	0.5266	0.5425
deepmesh_dmiip_fdu	0.5651	0.8397	0.7627	0.7828	0.6085	0.5508
deepmesh_dmiip_fdu_	0.5662	0.8402	0.7627	0.7829	0.6103	0.5515
attention_dmiip_fdu	0.5623	0.8383	0.7611	0.7812	0.6056	0.5482
NLM CNN	0.5160	0.7924	0.7256	0.7382	0.5527	0.5102
dmiip_fdu	0.5657	0.8387	0.7640	0.7828	0.6085	0.5522
submit mesh index qa	0.5202	0.7224	0.8081	0.7434	0.5056	0.5682
same as one	0.5176	0.7419	0.7819	0.7416	0.5191	0.5470
X-BERT BioASQ	0.2714	0.6196	0.3166	0.3894	0.4221	0.2186

+ Test batch 2, week 4

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6500	0.6961	0.6334	0.6452	0.6478	0.5687	0.5477	0.6861	0.6175	0.4966
Default MTI	0.6516	0.6587	0.6678	0.6454	0.6130	0.6016	0.5618	0.6522	0.6511	0.4936
deepmesh_dmiip_fdu	0.7078	0.7683	0.6736	0.7030	0.7326	0.5847	0.5888	0.7686	0.6559	0.5589
deepmesh_dmiip_fdu_	0.7074	0.7678	0.6730	0.7022	0.7300	0.5880	0.5910	0.7668	0.6565	0.5585
attention_dmiip_fdu	0.7054	0.7661	0.6714	0.7008	0.7262	0.5834	0.5867	0.7658	0.6539	0.5557
NLM CNN	0.6527	0.6919	0.6330	0.6431	0.6146	0.4836	0.4815	0.6877	0.6211	0.4894
dmiip_fdu	0.7053	0.7642	0.6730	0.7005	0.7228	0.5892	0.5903	0.7623	0.6562	0.5560
X-BERT BioASQ	0.3279	0.4978	0.2469	0.3102	0.3522	0.0543	0.0675	0.4957	0.2450	0.1953

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5318	0.8062	0.7315	0.7497	0.5746	0.5189
Default MTI	0.5321	0.7752	0.7640	0.7532	0.5497	0.5418
deepmesh_dmiip_fdu	0.5708	0.8558	0.7523	0.7863	0.6242	0.5465
deepmesh_dmiip_fdu_	0.5711	0.8553	0.7521	0.7856	0.6247	0.5470
attention_dmiip_fdu	0.5685	0.8543	0.7513	0.7851	0.6212	0.5448
NLM CNN	0.5251	0.8118	0.7183	0.7448	0.5697	0.5113
dmiip_fdu	0.5692	0.8530	0.7534	0.7855	0.6211	0.5465
X-BERT BioASQ	0.2810	0.6515	0.3192	0.4004	0.4480	0.2214

+ Test batch 2, week 5

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6484	0.6869	0.6319	0.6409	0.6474	0.5714	0.5511	0.6811	0.6187	0.4916
Default MTI	0.6496	0.6502	0.6681	0.6422	0.6101	0.6041	0.5638	0.6462	0.6531	0.4905
deepmesh_dmiip_fdu	0.7086	0.7578	0.6782	0.7014	0.7231	0.5918	0.5937	0.7601	0.6636	0.5573
deepmesh_dmiip_fdu_	0.7111	0.7594	0.6806	0.7038	0.7269	0.5898	0.5928	0.7631	0.6657	0.5603
attention_dmiip_fdu	0.7088	0.7565	0.6792	0.7019	0.7194	0.5899	0.5909	0.7596	0.6644	0.5576
NLM CNN	0.6539	0.6861	0.6345	0.6419	0.6119	0.4860	0.4824	0.6823	0.6277	0.4892
dmiip_fdu	0.7091	0.7555	0.6811	0.7021	0.7167	0.5969	0.5964	0.7573	0.6666	0.5582
X-BERT BioASQ	0.3154	0.4747	0.2353	0.2947	0.3428	0.0532	0.0653	0.4735	0.2365	0.1844

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5233	0.7964	0.7302	0.7447	0.5640	0.5123
Default MTI	0.5248	0.7651	0.7635	0.7480	0.5389	0.5372
deepmesh_dmiip_fdu	0.5668	0.8454	0.7579	0.7847	0.6157	0.5458
deepmesh_dmiip_fdu_	0.5695	0.8462	0.7601	0.7865	0.6176	0.5489
attention_dmiip_fdu	0.5676	0.8442	0.7596	0.7854	0.6149	0.5476
NLM CNN	0.5200	0.8013	0.7239	0.7432	0.5620	0.5073
dmiip_fdu	0.5676	0.8436	0.7608	0.7857	0.6146	0.5482
X-BERT BioASQ	0.2705	0.6360	0.3121	0.3901	0.4311	0.2142

+ Test batch 3, week 1

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6613	0.7083	0.6519	0.6596	0.6488	0.5870	0.5634	0.6948	0.6309	0.5138
Default MTI	0.6586	0.6593	0.6889	0.6540	0.6123	0.6234	0.5786	0.6510	0.6664	0.5028
iria-1	0.5064	0.4477	0.5875	0.4960	0.4105	0.4152	0.3729	0.4529	0.5742	0.3422
iria-mix	0.5396	0.4780	0.6281	0.5298	0.4566	0.4895	0.4374	0.4823	0.6123	0.3727
test sys	0.5879	0.7383	0.4964	0.5679	0.6155	0.2687	0.2807	0.7276	0.4932	0.4163
deepmesh_dmiip_fdu	0.7165	0.7694	0.6905	0.7121	0.7300	0.6002	0.6037	0.7690	0.6707	0.5694
deepmesh_dmiip_fdu_	0.7587	0.8069	0.7370	0.7561	0.7810	0.6630	0.6677	0.8058	0.7168	0.6355
attention_dmiip_fdu	0.7132	0.7656	0.6878	0.7090	0.7232	0.5997	0.6024	0.7651	0.6679	0.5655
NLM CNN	0.6605	0.6905	0.6491	0.6500	0.6139	0.4988	0.4929	0.6855	0.6373	0.4977
dmiip_fdu	0.7580	0.8038	0.7385	0.7555	0.7750	0.6673	0.6702	0.8023	0.7183	0.6349
submit mesh index qa	0.6575	0.6117	0.7417	0.6523	0.5580	0.6577	0.5821	0.6026	0.7233	0.4986
attention	0.6597	0.6142	0.7428	0.6542	0.5612	0.6581	0.5836	0.6054	0.7246	0.5010
same as one	0.6555	0.6476	0.6959	0.6510	0.5999	0.6324	0.5806	0.6382	0.6737	0.4994

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5416	0.8065	0.7456	0.7562	0.5832	0.5315
Default MTI	0.5337	0.7656	0.7796	0.7539	0.5443	0.5531
iria-1	0.4334	0.6001	0.7158	0.6375	0.4103	0.4831
iria-mix	0.4547	0.6273	0.7519	0.6687	0.4299	0.5079
test sys	0.4496	0.8355	0.5852	0.6605	0.5660	0.3972
deepmesh_dmiip_fdu	0.5747	0.8485	0.7662	0.7898	0.6206	0.5576
deepmesh_dmiip_fdu_	0.6398	0.8724	0.8004	0.8208	0.6803	0.6245
attention_dmiip_fdu	0.5724	0.8462	0.7654	0.7886	0.6173	0.5557
NLM CNN	0.5271	0.7999	0.7339	0.7465	0.5630	0.5213
dmiip_fdu	0.6403	0.8707	0.8026	0.8214	0.6790	0.6264
submit mesh index qa	0.5339	0.7261	0.8232	0.7541	0.5137	0.5870
attention	0.5349	0.7270	0.8243	0.7552	0.5147	0.5878
same as one	0.5320	0.7555	0.7878	0.7526	0.5367	0.5578

+ Test batch 3, week 2

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6511	0.6886	0.6420	0.6457	0.6432	0.5782	0.5563	0.6780	0.6263	0.4980
Default MTI	0.6503	0.6472	0.6765	0.6432	0.6037	0.6112	0.5690	0.6412	0.6596	0.4917
iria-1	0.4851	0.4266	0.5622	0.4733	0.3710	0.4009	0.3581	0.4319	0.5532	0.3224
iria-mix	0.5168	0.4552	0.6008	0.5053	0.4161	0.4610	0.4107	0.4601	0.5895	0.3507
deepmesh_dmiip_fdu	0.7130	0.7649	0.6835	0.7069	0.7296	0.5963	0.6000	0.7654	0.6673	0.5646
deepmesh_dmiip_fdu_	0.7606	0.8032	0.7333	0.7537	0.7854	0.6578	0.6647	0.8076	0.7187	0.6344
attention_dmiip_fdu	0.7138	0.7629	0.6861	0.7076	0.7247	0.6002	0.6024	0.7634	0.6702	0.5656
NLM CNN	0.6590	0.6836	0.6487	0.6480	0.5994	0.4885	0.4840	0.6801	0.6393	0.4957
dmiip_fdu	0.7612	0.8015	0.7356	0.7543	0.7804	0.6616	0.6668	0.8057	0.7214	0.6353
attention	0.6464	0.6346	0.6834	0.6398	0.5907	0.6194	0.5704	0.6273	0.6668	0.4877
X-BERT BioASQ	0.3293	0.4946	0.2468	0.3090	0.3437	0.0537	0.0662	0.4943	0.2469	0.1950

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5327	0.7977	0.7446	0.7523	0.5694	0.5261
Default MTI	0.5294	0.7642	0.7759	0.7526	0.5404	0.5468
iria-1	0.4227	0.5853	0.7074	0.6258	0.3991	0.4726
iria-mix	0.4417	0.6108	0.7362	0.6527	0.4177	0.4935
deepmesh_dmiip_fdu	0.5759	0.8499	0.7641	0.7897	0.6249	0.5555
deepmesh_dmiip_fdu_	0.6443	0.8743	0.8010	0.8231	0.6860	0.6266
attention_dmiip_fdu	0.5768	0.8481	0.7661	0.7903	0.6234	0.5581
NLM CNN	0.5291	0.8005	0.7360	0.7490	0.5647	0.5226
dmiip_fdu	0.6453	0.8727	0.8027	0.8234	0.6851	0.6289
attention	0.5272	0.7530	0.7840	0.7507	0.5316	0.5517
X-BERT BioASQ	0.2792	0.6504	0.3197	0.3995	0.4444	0.2209

+ Test batch 3, week 3

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6572	0.6956	0.6452	0.6506	0.6513	0.5741	0.5543	0.6859	0.6308	0.5011
Default MTI	0.6587	0.6592	0.6800	0.6516	0.6158	0.6088	0.5700	0.6534	0.6640	0.4998
iria-1	0.4938	0.4374	0.5729	0.4846	0.3718	0.4022	0.3612	0.4405	0.5616	0.3319
iria-mix	0.5252	0.4660	0.6103	0.5162	0.4171	0.4581	0.4108	0.4686	0.5975	0.3605
deepmesh_dmiip_fdu	0.7149	0.7715	0.6825	0.7090	0.7325	0.5834	0.5901	0.7716	0.6659	0.5662
deepmesh_dmiip_fdu_	0.7685	0.8164	0.7398	0.7631	0.7960	0.6504	0.6596	0.8196	0.7234	0.6471
attention_dmiip_fdu	0.7153	0.7697	0.6853	0.7097	0.7282	0.5877	0.5925	0.7690	0.6686	0.5667
NLM CNN	0.6608	0.6930	0.6494	0.6516	0.6127	0.4920	0.4880	0.6866	0.6369	0.4987
dmiip_fdu	0.7689	0.8150	0.7425	0.7639	0.7922	0.6544	0.6620	0.8172	0.7260	0.6479
X-BERT BioASQ	0.2655	0.7932	0.1630	0.2565	0.6447	0.0156	0.0223	0.8018	0.1591	0.1564

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5360	0.8017	0.7422	0.7529	0.5747	0.5278
Default MTI	0.5375	0.7715	0.7735	0.7556	0.5501	0.5522
iria-1	0.4304	0.5945	0.7129	0.6342	0.4056	0.4819
iria-mix	0.4509	0.6191	0.7417	0.6605	0.4255	0.5041
deepmesh_dmiip_fdu	0.5765	0.8535	0.7612	0.7898	0.6261	0.5557
deepmesh_dmiip_fdu_	0.6568	0.8829	0.8052	0.8297	0.6981	0.6390
attention_dmiip_fdu	0.5771	0.8519	0.7644	0.7909	0.6247	0.5578
NLM CNN	0.5305	0.8030	0.7333	0.7479	0.5682	0.5227
dmiip_fdu	0.6576	0.8815	0.8080	0.8305	0.6972	0.6410
X-BERT BioASQ	0.1959	0.8642	0.1757	0.2764	0.5274	0.1257

+ Test batch 3, week 4

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6585	0.7025	0.6473	0.6555	0.6539	0.5802	0.5588	0.6901	0.6296	0.5089
Default MTI	0.6564	0.6566	0.6815	0.6502	0.6096	0.6139	0.5714	0.6507	0.6621	0.4991
iria-1	0.4831	0.4233	0.5627	0.4716	0.3689	0.4053	0.3617	0.4287	0.5531	0.3210
iria-mix	0.5170	0.4540	0.6038	0.5059	0.4189	0.4655	0.4146	0.4587	0.5921	0.3514
deepmesh_dmiip_fdu	0.7126	0.7645	0.6869	0.7085	0.7219	0.5923	0.5919	0.7655	0.6666	0.5659
deepmesh_dmiip_fdu_	0.7106	0.7648	0.6839	0.7066	0.7212	0.5916	0.5915	0.7642	0.6640	0.5640
attention_dmiip_fdu	0.7108	0.7625	0.6853	0.7069	0.7155	0.5925	0.5916	0.7627	0.6655	0.5635
NLM CNN	0.6556	0.6856	0.6400	0.6439	0.6103	0.4954	0.4885	0.6826	0.6307	0.4910
dmiip_fdu	0.7114	0.7633	0.6868	0.7076	0.7181	0.5961	0.5949	0.7620	0.6670	0.5652
X-BERT BioASQ	0.2543	0.7565	0.1523	0.2406	0.6419	0.0164	0.0233	0.7840	0.1517	0.1463
biobert with sample	0.3072	0.3513	0.3019	0.3027	0.2736	0.2464	0.1525	0.3318	0.2861	0.1861

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5385	0.8051	0.7455	0.7563	0.5772	0.5292
Default MTI	0.5340	0.7679	0.7781	0.7555	0.5457	0.5503
iria-1	0.4215	0.5801	0.7094	0.6235	0.3946	0.4758
iria-mix	0.4420	0.6066	0.7413	0.6524	0.4142	0.4986
deepmesh_dmiip_fdu	0.5767	0.8473	0.7669	0.7899	0.6227	0.5593
deepmesh_dmiip_fdu_	0.5758	0.8479	0.7650	0.7891	0.6228	0.5577
attention_dmiip_fdu	0.5746	0.8456	0.7656	0.7886	0.6203	0.5572
NLM CNN	0.5240	0.8015	0.7273	0.7443	0.5620	0.5153
dmiip_fdu	0.5763	0.8466	0.7675	0.7900	0.6218	0.5593
X-BERT BioASQ	0.1881	0.8296	0.1657	0.2615	0.5077	0.1205
biobert with sample	0.2987	0.5929	0.4859	0.5076	0.3304	0.3023

+ Test batch 3, week 5

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6454	0.6814	0.6351	0.6387	0.6255	0.5763	0.5498	0.6708	0.6218	0.4880
Default MTI	0.6503	0.6502	0.6728	0.6436	0.5957	0.6126	0.5688	0.6426	0.6581	0.4922
iria-1	0.4800	0.4221	0.5571	0.4690	0.3635	0.4029	0.3590	0.4265	0.5490	0.3187
iria-2	0.2718	0.2526	0.3177	0.2689	0.4127	0.4100	0.3470	0.2449	0.3052	0.1617
iria-mix	0.5138	0.4526	0.5976	0.5030	0.4104	0.4648	0.4124	0.4565	0.5877	0.3488
deepmesh_dmiip_fdu	0.7067	0.7568	0.6778	0.7005	0.7143	0.5910	0.5890	0.7584	0.6617	0.5564
deepmesh_dmiip_fdu_	0.7047	0.7569	0.6749	0.6987	0.7133	0.5896	0.5875	0.7570	0.6591	0.5544
attention_dmiip_fdu	0.7044	0.7544	0.6764	0.6985	0.7089	0.5933	0.5898	0.7540	0.6609	0.5540
NLM CNN	0.6490	0.6784	0.6353	0.6378	0.6040	0.4982	0.4881	0.6732	0.6265	0.4844
dmiip_fdu	0.7041	0.7534	0.6760	0.6979	0.7083	0.5896	0.5870	0.7543	0.6602	0.5527
X-BERT BioASQ	0.2601	0.7721	0.1575	0.2489	0.6185	0.0166	0.0230	0.7921	0.1556	0.1518

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5211	0.7895	0.7328	0.7418	0.5568	0.5147
Default MTI	0.5283	0.7642	0.7677	0.7491	0.5398	0.5443
iria-1	0.4191	0.5836	0.7057	0.6241	0.3937	0.4705
iria-2	0.2899	0.4782	0.5633	0.4992	0.2706	0.3434
iria-mix	0.4388	0.6084	0.7353	0.6508	0.4125	0.4925
deepmesh_dmiip_fdu	0.5683	0.8454	0.7586	0.7847	0.6138	0.5502
deepmesh_dmiip_fdu_	0.5667	0.8458	0.7561	0.7834	0.6134	0.5476
attention_dmiip_fdu	0.5663	0.8429	0.7579	0.7831	0.6111	0.5488
NLM CNN	0.5199	0.7969	0.7242	0.7402	0.5571	0.5117
dmiip_fdu	0.5658	0.8421	0.7579	0.7827	0.6103	0.5485
X-BERT BioASQ	0.1893	0.8401	0.1655	0.2628	0.5103	0.1211

Platinum sponsors of the BioASQ Challenge between 2012 and 2023

Google was a proud sponsor of the BioASQ Challenge from 2018 to 2022.
BioASQ was supported by the National Library of Medicine of the National Institutes of Health under award number R13LM012214. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
The research leading to these results has received funding from the European Commission's Seventh Framework Programme (FP7/2007-2013, ICT-2011.4.4(d), Intelligent Information Management, Targeted Competition Framework) under grant agreement n° 318652.

BioASQ Participants Area

Test Results for Task 8a

+ Dry Run 1

Annotated articles:9676/9823

Flat Measures

Hierarchical Measures

+ Test batch 1, week 1

Annotated articles:6487/6510

Flat Measures

Hierarchical Measures

+ Test batch 1, week 2

Annotated articles:7074/7126

Flat Measures

Hierarchical Measures

+ Test batch 1, week 3

Annotated articles:10789/10891

Flat Measures

Hierarchical Measures

+ Test batch 1, week 4

Annotated articles:6182/6225

Flat Measures

Hierarchical Measures

+ Test batch 1, week 5

Annotated articles:6887/6953

Flat Measures

Hierarchical Measures

+ Test batch 2, week 1

Annotated articles:6787/6815

Flat Measures

Hierarchical Measures

+ Test batch 2, week 2

Annotated articles:6414/6485

Flat Measures

Hierarchical Measures

+ Test batch 2, week 3

Annotated articles:6975/7014

Flat Measures

Hierarchical Measures

+ Test batch 2, week 4

Annotated articles:6647/6726

Flat Measures

Hierarchical Measures

+ Test batch 2, week 5

Annotated articles:6246/6379

Flat Measures

Hierarchical Measures

+ Test batch 3, week 1

Annotated articles:6601/6842

Flat Measures

Hierarchical Measures

+ Test batch 3, week 2

Annotated articles:6456/7212

Flat Measures

Hierarchical Measures

+ Test batch 3, week 3

Annotated articles:4764/5430

Flat Measures

Hierarchical Measures

+ Test batch 3, week 4

Annotated articles:4858/6022

Flat Measures

Hierarchical Measures

+ Test batch 3, week 5

Annotated articles:3999/5936

Flat Measures

Hierarchical Measures