BioASQ Participants Area

Test Results for Task 5a

The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.

+ Dry Run 1

Annotated articles:11372/13000

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6329	0.6857	0.6080	0.6233	0.6516	0.5606	0.5422	0.6764	0.5946	0.4721
Default MTI	0.6361	0.6407	0.6587	0.6281	0.6104	0.5978	0.5582	0.6294	0.6429	0.4749

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5035	0.7954	0.7055	0.7268	0.5551	0.4882
Default MTI	0.5102	0.7594	0.7538	0.7366	0.5281	0.5238

+ Dry Run 2

Annotated articles:6179/6326

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6108	0.6727	0.5861	0.6043	0.6242	0.5350	0.5144	0.6620	0.5669	0.4506
Default MTI	0.6190	0.6317	0.6376	0.6132	0.5812	0.5785	0.5354	0.6211	0.6169	0.4591
MZ1	0.4745	0.6931	0.3881	0.4632	0.4074	0.1988	0.2048	0.6559	0.3717	0.3179

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.4916	0.7870	0.6916	0.7142	0.5471	0.4740
Default MTI	0.5039	0.7508	0.7413	0.7259	0.5254	0.5139
MZ1	0.3600	0.7892	0.4669	0.5468	0.5194	0.3056

+ Test batch 1, week 1

Annotated articles:6661/6880

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6121	0.6687	0.5834	0.6011	0.6248	0.5377	0.5164	0.6618	0.5692	0.4473
Default MTI	0.6226	0.6347	0.6343	0.6134	0.5889	0.5775	0.5379	0.6275	0.6177	0.4592
auth1	0.6254	0.6160	0.6440	0.6125	0.6057	0.5136	0.4970	0.6219	0.6291	0.4572
auth2	0.6324	0.6226	0.6502	0.6188	0.6133	0.5004	0.4832	0.6288	0.6361	0.4643
auth3	0.6335	0.6485	0.6236	0.6182	0.6317	0.4817	0.4733	0.6562	0.6123	0.4643
Search system 1	0.2707	0.2650	0.3065	0.2683	0.4201	0.4192	0.3510	0.2505	0.2944	0.1616
Search system 2	0.2570	0.2409	0.3111	0.2555	0.4039	0.4235	0.3465	0.2255	0.2988	0.1524
Search system 3	0.2103	0.2941	0.1801	0.2064	0.4012	0.2578	0.2572	0.2851	0.1665	0.1219
Search system 4	0.2430	0.2094	0.3595	0.2468	0.2936	0.4872	0.3661	0.1865	0.3485	0.1457
Search system 5	0.2559	0.2297	0.3349	0.2562	0.3136	0.4583	0.3629	0.2112	0.3244	0.1523
MZ1	0.4649	0.4888	0.4863	0.4595	0.5764	0.1913	0.2019	0.4678	0.4621	0.3101
DeepMeSH1	0.6607	0.7099	0.6346	0.6504	0.6687	0.5312	0.5283	0.7099	0.6178	0.4994
DeepMeSH2	0.6578	0.7240	0.6211	0.6494	0.6827	0.5166	0.5182	0.7280	0.5999	0.4984
DeepMeSH3	0.6498	0.6978	0.6238	0.6386	0.6465	0.5081	0.5033	0.6991	0.6070	0.4865
DeepMeSH4	0.6428	0.7083	0.6052	0.6335	0.6557	0.4791	0.4821	0.7139	0.5846	0.4809
DeepMeSH5	0.6607	0.7099	0.6346	0.6504	0.6687	0.5312	0.5283	0.7099	0.6178	0.4994

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R