BioASQ Participants Area

Test Results for Task 10a

The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.

+ Test batch 1, week 1

Annotated articles:9450/9659

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6240	0.6791	0.6081	0.6229	0.6473	0.5732	0.5512	0.6673	0.5861	0.4686
Default MTI	0.6290	0.6438	0.6457	0.6266	0.6216	0.6073	0.5660	0.6356	0.6226	0.4714
Dexstr system	0.1664	0.3644	0.1193	0.1677	0.5829	0.1597	0.1756	0.3428	0.1099	0.0972
NLM System 2	0.6968	0.7309	0.6870	0.6916	0.7122	0.5658	0.5663	0.7274	0.6687	0.5437
deepmesh_dmiip_fdu	0.6612	0.8191	0.5804	0.6621	0.7935	0.4910	0.5207	0.8207	0.5535	0.5108
attention_dmiip_fdu	0.6901	0.7313	0.6737	0.6847	0.6928	0.5810	0.5740	0.7260	0.6575	0.5357
NLM CNN	0.6488	0.6758	0.6415	0.6406	0.6212	0.4770	0.4757	0.6710	0.6280	0.4860
NLM System 1	0.6967	0.7315	0.6849	0.6911	0.7154	0.5638	0.5652	0.7285	0.6676	0.5432
Plain dict match	0.2325	0.3170	0.2019	0.2298	0.3714	0.3485	0.2854	0.2930	0.1928	0.1359
XLinear model	0.5490	0.7443	0.4503	0.5405	0.6972	0.3129	0.3467	0.7403	0.4363	0.3867

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5057	0.7904	0.7225	0.7370	0.5528	0.4922
Default MTI	0.5096	0.7609	0.7568	0.7416	0.5293	0.5191
Dexstr system	0.2222	0.6767	0.2527	0.3424	0.4169	0.1643
NLM System 2	0.5554	0.8244	0.7669	0.7787	0.5919	0.5481
deepmesh_dmiip_fdu	0.5137	0.8899	0.6633	0.7425	0.6267	0.4563
attention_dmiip_fdu	0.5484	0.8236	0.7570	0.7729	0.5887	0.5379
NLM CNN	0.5165	0.7925	0.7274	0.7411	0.5549	0.5082
NLM System 1	0.5552	0.8254	0.7652	0.7785	0.5924	0.5469
Plain dict match	0.2524	0.5493	0.4070	0.4419	0.3070	0.2383
XLinear model	0.4226	0.8565	0.5371	0.6380	0.5737	0.3542

+ Test batch 1, week 2

Annotated articles:4512/4531

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6347	0.6778	0.6401	0.6384	0.6235	0.5898	0.5599	0.6617	0.6099	0.4885
Default MTI	0.6321	0.6265	0.6770	0.6299	0.5859	0.6262	0.5742	0.6172	0.6476	0.4758
Dexstr system	0.3341	0.3163	0.4224	0.3291	0.3507	0.4365	0.3511	0.2858	0.4019	0.2041
NLM System 2	0.6972	0.7108	0.7127	0.6912	0.6858	0.5851	0.5739	0.7063	0.6884	0.5447
deepmesh_dmiip_fdu	0.6755	0.7989	0.6165	0.6757	0.7718	0.5093	0.5280	0.8002	0.5844	0.5278
attention_dmiip_fdu	0.6951	0.7157	0.7014	0.6885	0.6769	0.5982	0.5834	0.7098	0.6811	0.5413
NLM CNN	0.6499	0.6620	0.6618	0.6406	0.6035	0.4963	0.4860	0.6558	0.6441	0.4871
NLM System 1	0.6987	0.7140	0.7087	0.6918	0.6948	0.5783	0.5701	0.7107	0.6871	0.5454
Plain dict match	0.2292	0.3026	0.2055	0.2273	0.3493	0.3541	0.2912	0.2788	0.1945	0.1347
XLinear model	0.5576	0.7329	0.4680	0.5474	0.6827	0.3257	0.3511	0.7275	0.4521	0.3950
combination system	0.2464	0.2927	0.2403	0.2462	0.3161	0.3971	0.3142	0.2679	0.2282	0.1464
consensus system	0.1160	0.5395	0.0701	0.1183	0.4573	0.0994	0.0874	0.4698	0.0662	0.0665

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5175	0.7808	0.7460	0.7442	0.5503	0.5162
Default MTI	0.5085	0.7405	0.7806	0.7412	0.5118	0.5367
Dexstr system	0.3107	0.4813	0.6395	0.5137	0.2855	0.3915
NLM System 2	0.5534	0.8030	0.7892	0.7777	0.5754	0.5626
deepmesh_dmiip_fdu	0.5257	0.8729	0.6973	0.7565	0.6157	0.4826
attention_dmiip_fdu	0.5498	0.8078	0.7817	0.7765	0.5756	0.5544
NLM CNN	0.5134	0.7767	0.7450	0.7404	0.5409	0.5186
NLM System 1	0.5541	0.8069	0.7862	0.7788	0.5778	0.5612
Plain dict match	0.2511	0.5314	0.4114	0.4383	0.2961	0.2431
XLinear model	0.4268	0.8451	0.5548	0.6459	0.5636	0.3650
combination system	0.2542	0.5093	0.4491	0.4532	0.2801	0.2607
consensus system	0.1175	0.6864	0.1224	0.1965	0.3409	0.0742

+ Test batch 1, week 3

Annotated articles:4269/4291

Flat Measures

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6317	0.6950	0.6168	0.6349	0.6637	0.5837	0.5624	0.6805	0.5894	0.4835
Default MTI	0.6288	0.6430	0.6526	0.6276	0.6175	0.6115	0.5709	0.6324	0.6252	0.4719
Dexstr system	0.3490	0.3252	0.4074	0.3475	0.3830	0.4291	0.3577	0.3168	0.3885	0.2167
NLM System 2	0.6841	0.7136	0.6842	0.6796	0.6944	0.5770	0.5694	0.7054	0.6640	0.5291
deepmesh_dmiip_fdu	0.6796	0.7173	0.6710	0.6743	0.6852	0.5822	0.5733	0.7075	0.6538	0.5238
attention_dmiip_fdu	0.6553	0.8089	0.5783	0.6563	0.7880	0.4940	0.5176	0.8068	0.5517	0.5038
NLM CNN	0.6364	0.6690	0.6324	0.6298	0.6216	0.4831	0.4792	0.6575	0.6166	0.4744
NLM System 1	0.6814	0.7124	0.6789	0.6765	0.7008	0.5705	0.5657	0.7035	0.6606	0.5260
XLinear model	0.5422	0.7344	0.4490	0.5355	0.6909	0.3238	0.3503	0.7278	0.4320	0.3815

Hierarchical Measures

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R