BioASQ Participants Area

Test Results for Task 10a

The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.

+ Test batch 1, week 1

Annotated articles:9628/9659

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6264	0.6799	0.6146	0.6264	0.6489	0.5773	0.5546	0.6676	0.5900	0.4729
Default MTI	0.6306	0.6435	0.6516	0.6289	0.6228	0.6111	0.5689	0.6351	0.6263	0.4740
Dexstr system	0.1673	0.3644	0.1206	0.1689	0.5833	0.1609	0.1768	0.3430	0.1107	0.0981
NLM System 2	0.6961	0.7275	0.6890	0.6906	0.7101	0.5678	0.5673	0.7241	0.6701	0.5426
deepmesh_dmiip_fdu	0.6621	0.8166	0.5840	0.6630	0.7919	0.4937	0.5227	0.8182	0.5560	0.5120
attention_dmiip_fdu	0.6896	0.7281	0.6763	0.6841	0.6899	0.5830	0.5747	0.7229	0.6593	0.5350
NLM CNN	0.6481	0.6726	0.6431	0.6395	0.6202	0.4783	0.4764	0.6680	0.6292	0.4850
NLM System 1	0.6960	0.7281	0.6869	0.6901	0.7132	0.5658	0.5663	0.7251	0.6690	0.5421
Plain dict match	0.2333	0.3171	0.2042	0.2311	0.3714	0.3498	0.2861	0.2926	0.1940	0.1369
XLinear model	0.5496	0.7419	0.4526	0.5412	0.6963	0.3142	0.3478	0.7379	0.4378	0.3874

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5082	0.7902	0.7272	0.7392	0.5533	0.4966
Default MTI	0.5106	0.7598	0.7609	0.7429	0.5286	0.5223
Dexstr system	0.2228	0.6757	0.2540	0.3433	0.4164	0.1651
NLM System 2	0.5539	0.8211	0.7688	0.7777	0.5886	0.5486
deepmesh_dmiip_fdu	0.5140	0.8874	0.6667	0.7433	0.6246	0.4585
attention_dmiip_fdu	0.5473	0.8203	0.7595	0.7722	0.5857	0.5389
NLM CNN	0.5150	0.7894	0.7291	0.7402	0.5519	0.5085
NLM System 1	0.5536	0.8221	0.7672	0.7775	0.5891	0.5474
Plain dict match	0.2528	0.5484	0.4092	0.4427	0.3065	0.2395
XLinear model	0.4227	0.8544	0.5395	0.6388	0.5718	0.3553

+ Test batch 1, week 2

Annotated articles:4528/4531

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6350	0.6778	0.6412	0.6388	0.6235	0.5905	0.5603	0.6614	0.6106	0.4890
Default MTI	0.6323	0.6264	0.6779	0.6303	0.5860	0.6268	0.5746	0.6170	0.6483	0.4761
Dexstr system	0.3343	0.3166	0.4231	0.3294	0.3503	0.4367	0.3510	0.2858	0.4025	0.2043
NLM System 2	0.6971	0.7103	0.7131	0.6911	0.6856	0.5854	0.5740	0.7058	0.6887	0.5446
deepmesh_dmiip_fdu	0.6756	0.7988	0.6168	0.6759	0.7720	0.5097	0.5284	0.8000	0.5847	0.5280
attention_dmiip_fdu	0.6951	0.7155	0.7017	0.6885	0.6769	0.5984	0.5836	0.7095	0.6813	0.5413
NLM CNN	0.6498	0.6616	0.6622	0.6406	0.6032	0.4967	0.4862	0.6553	0.6444	0.4871
NLM System 1	0.6986	0.7136	0.7090	0.6917	0.6946	0.5786	0.5701	0.7102	0.6873	0.5453
Plain dict match	0.2294	0.3030	0.2060	0.2276	0.3490	0.3545	0.2913	0.2789	0.1948	0.1349
XLinear model	0.5577	0.7324	0.4683	0.5474	0.6826	0.3261	0.3514	0.7270	0.4523	0.3950
combination system	0.2466	0.2931	0.2408	0.2464	0.3159	0.3975	0.3143	0.2679	0.2285	0.1465
consensus system	0.1161	0.5388	0.0702	0.1184	0.4564	0.0997	0.0876	0.4693	0.0663	0.0666

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5178	0.7807	0.7468	0.7445	0.5503	0.5169
Default MTI	0.5088	0.7404	0.7813	0.7415	0.5117	0.5374
Dexstr system	0.3108	0.4814	0.6398	0.5137	0.2855	0.3917
NLM System 2	0.5533	0.8027	0.7895	0.7777	0.5751	0.5627
deepmesh_dmiip_fdu	0.5257	0.8727	0.6977	0.7566	0.6154	0.4826
attention_dmiip_fdu	0.5496	0.8076	0.7819	0.7765	0.5752	0.5543
NLM CNN	0.5133	0.7764	0.7454	0.7404	0.5406	0.5187
NLM System 1	0.5540	0.8067	0.7865	0.7788	0.5775	0.5611
Plain dict match	0.2511	0.5314	0.4116	0.4384	0.2960	0.2433
XLinear model	0.4267	0.8447	0.5549	0.6459	0.5632	0.3650
combination system	0.2542	0.5092	0.4493	0.4532	0.2800	0.2608
consensus system	0.1176	0.6858	0.1226	0.1965	0.3408	0.0743

+ Test batch 1, week 3

Annotated articles:4291/4291

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6324	0.6953	0.6186	0.6360	0.6637	0.5850	0.5636	0.6807	0.5905	0.4848
Default MTI	0.6293	0.6431	0.6540	0.6283	0.6172	0.6125	0.5717	0.6324	0.6262	0.4728
Dexstr system	0.3492	0.3251	0.4083	0.3477	0.3826	0.4293	0.3578	0.3168	0.3890	0.2169
NLM System 2	0.6840	0.7133	0.6844	0.6795	0.6943	0.5770	0.5693	0.7051	0.6642	0.5291
deepmesh_dmiip_fdu	0.6797	0.7172	0.6714	0.6744	0.6846	0.5823	0.5735	0.7075	0.6540	0.5240
attention_dmiip_fdu	0.6556	0.8088	0.5790	0.6567	0.7878	0.4944	0.5180	0.8067	0.5522	0.5043
NLM CNN	0.6364	0.6686	0.6326	0.6298	0.6209	0.4831	0.4792	0.6572	0.6168	0.4744
NLM System 1	0.6814	0.7121	0.6792	0.6765	0.7007	0.5706	0.5657	0.7032	0.6609	0.5260
XLinear model	0.5424	0.7341	0.4495	0.5357	0.6901	0.3237	0.3504	0.7275	0.4324	0.3818

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R