BioASQ Participants Area

Test Results for Task 9a

The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.

+ Test batch 1, week 1

Annotated articles:7808/7967

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6551	0.7063	0.6392	0.6524	0.6656	0.5641	0.5488	0.6946	0.6198	0.5050
Default MTI	0.6536	0.6587	0.6750	0.6486	0.6243	0.5933	0.5586	0.6518	0.6554	0.4969
deepmesh_dmiip_fdu	0.6706	0.7475	0.6273	0.6665	0.7042	0.5182	0.5271	0.7480	0.6077	0.5192
deepmesh_dmiip_fdu_	0.6620	0.7376	0.6162	0.6559	0.6951	0.5013	0.5107	0.7400	0.5989	0.5042
attention_dmiip_fdu	0.1894	0.2087	0.1933	0.1933	0.0788	0.0826	0.0703	0.1961	0.1831	0.1393
NLM CNN	0.6517	0.6991	0.6264	0.6407	0.6438	0.4589	0.4659	0.6944	0.6141	0.4883
dmiip_fdu	0.1128	0.1146	0.1099	0.1059	0.0010	0.0008	0.0008	0.1160	0.1099	0.0597
NLM System 1	0.6762	0.7128	0.6614	0.6687	0.6696	0.5029	0.5063	0.7163	0.6403	0.5182
NLM System 3	0.6864	0.7319	0.6692	0.6824	0.6836	0.5242	0.5282	0.7316	0.6464	0.5375
bert_dna_2	0.1045	0.7112	0.0629	0.1138	0.7112	0.0001	0.0001	0.7112	0.0564	0.0629
pi_dna_2	0.1045	0.7112	0.0629	0.1138	0.7112	0.0001	0.0001	0.7112	0.0564	0.0629

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5323	0.8083	0.7343	0.7516	0.5771	0.5191
Default MTI	0.5278	0.7696	0.7677	0.7515	0.5432	0.5408
deepmesh_dmiip_fdu	0.5359	0.8375	0.7156	0.7560	0.5991	0.5059
deepmesh_dmiip_fdu_	0.5215	0.8304	0.7059	0.7470	0.5857	0.4912
attention_dmiip_fdu	0.2371	0.3219	0.2996	0.2966	0.2565	0.2367
NLM CNN	0.5178	0.8118	0.7118	0.7388	0.5693	0.5007
dmiip_fdu	0.1718	0.2351	0.2223	0.2143	0.1866	0.1751
NLM System 1	0.5403	0.8123	0.7484	0.7620	0.5800	0.5305
NLM System 3	0.5569	0.8261	0.7546	0.7725	0.5999	0.5436
bert_dna_2	0.0770	0.7321	0.0698	0.1258	0.2861	0.0455
pi_dna_2	0.0770	0.7321	0.0698	0.1258	0.2861	0.0455

+ Test batch 1, week 2

Annotated articles:9987/10053

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6343	0.6640	0.6393	0.6310	0.6386	0.5797	0.5534	0.6533	0.6165	0.4798
Default MTI	0.6342	0.6246	0.6750	0.6294	0.6037	0.6114	0.5654	0.6179	0.6514	0.4759
deepmesh_dmiip_fdu	0.6854	0.7094	0.6898	0.6821	0.7003	0.5947	0.5870	0.7034	0.6682	0.5341
deepmesh_dmiip_fdu_	0.6827	0.7027	0.6892	0.6778	0.6951	0.5939	0.5852	0.6979	0.6681	0.5283
attention_dmiip_fdu	0.6834	0.7056	0.6881	0.6784	0.6977	0.5928	0.5855	0.7010	0.6667	0.5293
NLM CNN	0.6231	0.6596	0.6143	0.6137	0.6215	0.4371	0.4433	0.6510	0.5975	0.4588
dmiip_fdu	0.6858	0.7120	0.6880	0.6823	0.7027	0.5931	0.5868	0.7064	0.6664	0.5346
NLM System 1	0.6543	0.6795	0.6555	0.6481	0.6616	0.5001	0.5017	0.6794	0.6310	0.4947
NLM System 3	0.6589	0.6846	0.6600	0.6529	0.6650	0.5109	0.5119	0.6839	0.6356	0.5008
DistilBert	0.0001	0.0006	0.0001	0.0001	0.0006	0.0001	0.0000	0.0006	0.0000	0.0001

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5100	0.7832	0.7360	0.7397	0.5437	0.5085
Default MTI	0.5096	0.7491	0.7686	0.7406	0.5166	0.5326
deepmesh_dmiip_fdu	0.5505	0.8119	0.7739	0.7764	0.5798	0.5499
deepmesh_dmiip_fdu_	0.5453	0.8070	0.7735	0.7734	0.5731	0.5467
attention_dmiip_fdu	0.5457	0.8099	0.7715	0.7736	0.5750	0.5457
NLM CNN	0.4962	0.7895	0.7053	0.7241	0.5429	0.4861
dmiip_fdu	0.5506	0.8146	0.7715	0.7764	0.5815	0.5484
NLM System 1	0.5230	0.7916	0.7493	0.7519	0.5570	0.5200
NLM System 3	0.5271	0.7949	0.7524	0.7553	0.5611	0.5241
DistilBert	0.1188	0.7351	0.0478	0.0889	0.5421	0.0677

+ Test batch 1, week 3

Annotated articles:4854/4870

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6463	0.6902	0.6394	0.6452	0.6406	0.5686	0.5469	0.6758	0.6194	0.4966
Default MTI	0.6461	0.6388	0.6824	0.6424	0.5964	0.6003	0.5581	0.6320	0.6609	0.4905
NLM System 2	0.6726	0.6750	0.6870	0.6640	0.6734	0.5136	0.5092	0.6756	0.6697	0.5121
deepmesh_dmiip_fdu	0.7032	0.7447	0.6907	0.7016	0.7103	0.5898	0.5860	0.7390	0.6707	0.5576
deepmesh_dmiip_fdu_	0.7008	0.7254	0.6976	0.6961	0.6971	0.5895	0.5824	0.7245	0.6786	0.5497
attention_dmiip_fdu	0.7039	0.7302	0.6995	0.6991	0.7037	0.5904	0.5848	0.7294	0.6800	0.5535
NLM CNN	0.6449	0.6836	0.6272	0.6346	0.6375	0.4544	0.4575	0.6793	0.6138	0.4814
dmiip_fdu	0.7049	0.7487	0.6902	0.7030	0.7156	0.5890	0.5868	0.7431	0.6704	0.5595
NLM System 1	0.6690	0.6942	0.6666	0.6626	0.6591	0.5080	0.5054	0.6962	0.6438	0.5111
NLM System 3	0.6763	0.6811	0.6905	0.6683	0.6729	0.5271	0.5206	0.6804	0.6723	0.5180
DistilBert	0.0004	0.0027	0.0002	0.0004	0.0024	0.0000	0.0001	0.0027	0.0002	0.0002
bert_dna	0.3321	0.3695	0.3079	0.3138	0.2100	0.0166	0.0161	0.3496	0.3163	0.1970
roberta	0.0940	0.2255	0.0565	0.0885	0.1268	0.0007	0.0004	0.2402	0.0584	0.0483

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R