BioASQ Participants Area

Test Results for Task 3a

The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.

+ Test batch 1, week 1

Annotated articles:11756/21014

Flat Measures

System Name	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
BioASQ_Baseline	0.2484	0.2738	0.2846	0.2416	0.3717	0.3854	0.3195	0.2297	0.2706	0.1433
MTI First Line Index	0.5668	0.6415	0.5316	0.5598	0.6069	0.4837	0.4720	0.6367	0.5108	0.4056
Default MTI	0.5748	0.5921	0.5833	0.5673	0.5659	0.5265	0.4925	0.5879	0.5623	0.4114
it is a test submit	0.3091	0.2749	0.3769	0.3051	0.2942	0.3093	0.2578	0.2749	0.3530	0.1870
USI 10 neighbours	0.5147	0.5508	0.5082	0.5043	0.5073	0.3734	0.3708	0.5379	0.4935	0.3514
USI 20 neighbours	0.4944	0.5305	0.4990	0.4902	0.4878	0.3640	0.3572	0.5135	0.4766	0.3374
Abstract framework	0.5187	0.5600	0.5152	0.5080	0.5048	0.3763	0.3722	0.5368	0.5019	0.3555
auth1	0.5896	0.5744	0.6184	0.5788	0.5930	0.4788	0.4672	0.5793	0.6004	0.4220
auth2	0.5979	0.6373	0.5747	0.5860	0.6124	0.4657	0.4621	0.6436	0.5582	0.4304
auth4	0.5849	0.6554	0.5395	0.5710	0.7061	0.3677	0.3980	0.6585	0.5262	0.4168
MeSHLabeler	0.6266	0.6882	0.5978	0.6194	0.6356	0.5064	0.5081	0.6889	0.5746	0.4648
MeSH Now BF	0.5516	0.5671	0.5622	0.5439	0.5446	0.4767	0.4494	0.5643	0.5394	0.3885
MeSH Now HR	0.1879	0.1081	0.7510	0.1856	0.0790	0.6198	0.1992	0.1077	0.7393	0.1041
MeSHLabeler-1	0.6268	0.6879	0.5986	0.6197	0.6563	0.4934	0.4991	0.6887	0.5750	0.4650
MeSHLabeler-2	0.6164	0.7354	0.5522	0.6089	0.6826	0.4648	0.4845	0.7337	0.5314	0.4552
MeSHLabeler-3	0.6173	0.7327	0.5548	0.6094	0.6964	0.4535	0.4743	0.7308	0.5344	0.4558
MeSHLabeler-4	0.6268	0.6904	0.5966	0.6194	0.6587	0.4916	0.4978	0.6906	0.5737	0.4648

Hierarchical Measures

System Name	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
BioASQ_Baseline	0.2867	0.5323	0.5339	0.4839	0.3131	0.3037
MTI First Line Index	0.4756	0.7885	0.6552	0.6899	0.5478	0.4486
Default MTI	0.4884	0.7485	0.7138	0.7069	0.5198	0.4916
it is a test submit	0.3310	0.5234	0.6201	0.5466	0.3215	0.3661
USI 10 neighbours	0.4449	0.7178	0.6529	0.6549	0.4889	0.4403
USI 20 neighbours	0.4480	0.6999	0.6721	0.6583	0.4889	0.4451
Abstract framework	0.4437	0.7251	0.6486	0.6520	0.4902	0.4408
auth1	0.4958	0.7269	0.7418	0.7138	0.5032	0.5177
auth2	0.4936	0.7722	0.6906	0.7067	0.5370	0.4841
auth4	0.4774	0.8008	0.6415	0.6864	0.5532	0.4472
MeSHLabeler	0.5147	0.8098	0.7013	0.7288	0.5745	0.4943
MeSH Now BF	0.4597	0.7023	0.6998	0.6762	0.4775	0.4770
MeSH Now HR	0.2200	0.2255	0.9130	0.3525	0.1372	0.6139
MeSHLabeler-1	0.5145	0.8099	0.7024	0.7294	0.5736	0.4947
MeSHLabeler-2	0.4988	0.8434	0.6527	0.7106	0.5952	0.4558
MeSHLabeler-3	0.4998	0.8422	0.6552	0.7117	0.5942	0.4580
MeSHLabeler-4	0.5139	0.8119	0.7001	0.7285	0.5748	0.4929

+ Test batch 1, week 2

Annotated articles:3215/4435

Flat Measures

System Name	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
BioASQ_Baseline	0.1083	0.4188	0.0951	0.1125	0.1403	0.0771	0.0754	0.1451	0.0865	0.0637
MTI First Line Index	0.5552	0.6252	0.5174	0.5460	0.5867	0.4690	0.4545	0.6230	0.5008	0.3909
Default MTI	0.5618	0.5826	0.5620	0.5532	0.5421	0.5152	0.4805	0.5786	0.5460	0.3967
TextCategorisation1	0.5604	0.6500	0.5099	0.5484	0.6202	0.4373	0.4343	0.6461	0.4948	0.3943
TextCategorisation2	0.5608	0.6464	0.5126	0.5489	0.6160	0.4406	0.4364	0.6427	0.4974	0.3946
TextCategorisation3	0.5608	0.6476	0.5124	0.5488	0.6162	0.4405	0.4363	0.6429	0.4973	0.3946
TextCategorisation4	0.5608	0.6427	0.5134	0.5491	0.6148	0.4410	0.4366	0.6418	0.4979	0.3948
TextCategorisation5	0.5604	0.6511	0.5096	0.5481	0.6204	0.4372	0.4341	0.6463	0.4946	0.3942
spoon-spoon	0.4628	0.4871	0.4613	0.4541	0.4912	0.2962	0.2920	0.4825	0.4446	0.3071
fork-fork	0.4183	0.6442	0.3168	0.4006	0.6759	0.1602	0.1792	0.6499	0.3085	0.2664
it is a test submit	0.3021	0.2697	0.3650	0.2975	0.2771	0.3325	0.2768	0.2697	0.3434	0.1813
qaiiit system 1	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
USI 10 neighbours	0.5114	0.5471	0.5041	0.5020	0.5065	0.3877	0.3797	0.5368	0.4883	0.3483
USI 20 neighbours	0.4846	0.5139	0.4963	0.4819	0.4642	0.3747	0.3589	0.4973	0.4725	0.3289
Abstract framework	0.5159	0.5577	0.5090	0.5054	0.5051	0.3900	0.3819	0.5384	0.4951	0.3522
auth1	0.5866	0.5709	0.6154	0.5761	0.5793	0.4993	0.4786	0.5770	0.5966	0.4185
auth2	0.5931	0.6189	0.5803	0.5811	0.6026	0.4805	0.4696	0.6260	0.5635	0.4244
auth3	0.5954	0.6121	0.5906	0.5832	0.6559	0.4536	0.4550	0.6181	0.5743	0.4266
auth4	0.5806	0.6538	0.5336	0.5674	0.6864	0.3832	0.3993	0.6581	0.5195	0.4128
auth5	0.5476	0.4620	0.6852	0.5356	0.5004	0.5659	0.4919	0.4639	0.6682	0.3777
MeSHLabeler	0.6190	0.6849	0.5866	0.6118	0.6555	0.4837	0.4843	0.6868	0.5633	0.4561
MeSH Now BF	0.4361	0.4520	0.4541	0.4329	0.4302	0.3602	0.3248	0.4391	0.4332	0.2915
MeSH Now HR	0.1800	0.1032	0.7212	0.1776	0.0851	0.5921	0.2272	0.1031	0.7106	0.0991
MeSHLabeler-1	0.6193	0.6851	0.5888	0.6131	0.6445	0.4967	0.4974	0.6856	0.5646	0.4574
MeSHLabeler-2	0.6189	0.6860	0.5856	0.6117	0.6568	0.4846	0.4860	0.6875	0.5627	0.4563
MeSHLabeler-3	0.6190	0.6853	0.5876	0.6125	0.6414	0.4964	0.4971	0.6864	0.5637	0.4569
MeSHLabeler-4	0.6194	0.6855	0.5887	0.6133	0.6452	0.4967	0.4976	0.6860	0.5645	0.4575
USI baseline	0.5624	0.5804	0.5640	0.5542	0.5701	0.4640	0.4407	0.5847	0.5417	0.3984
iria-1	0.4628	0.5122	0.4487	0.4587	0.4327	0.2903	0.2912	0.5122	0.4221	0.3113

Hierarchical Measures

System Name	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
BioASQ_Baseline	0.1541	0.5787	0.2174	0.2262	0.2850	0.1398
MTI First Line Index	0.4672	0.7726	0.6445	0.6777	0.5377	0.4391
Default MTI	0.4788	0.7369	0.6963	0.6928	0.5146	0.4754
TextCategorisation1	0.4625	0.7911	0.6366	0.6778	0.5439	0.4314
TextCategorisation2	0.4630	0.7880	0.6400	0.6787	0.5414	0.4337
TextCategorisation3	0.4626	0.7885	0.6396	0.6782	0.5409	0.4334
TextCategorisation4	0.4641	0.7859	0.6424	0.6806	0.5407	0.4351
TextCategorisation5	0.4621	0.7913	0.6359	0.6770	0.5434	0.4311
spoon-spoon	0.3963	0.6653	0.5758	0.5896	0.4337	0.3952
fork-fork	0.3283	0.7944	0.3563	0.4576	0.5427	0.2563
it is a test submit	0.3287	0.5126	0.6081	0.5341	0.3218	0.3613
qaiiit system 1	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
USI 10 neighbours	0.4430	0.7112	0.6456	0.6493	0.4864	0.4375
USI 20 neighbours	0.4450	0.6826	0.6741	0.6519	0.4820	0.4457
Abstract framework	0.4416	0.7187	0.6387	0.6455	0.4882	0.4364
auth1	0.4925	0.7183	0.7394	0.7083	0.4993	0.5138
auth2	0.4912	0.7569	0.6933	0.7019	0.5270	0.4867
auth3	0.4930	0.7652	0.6974	0.7077	0.5275	0.4903
auth4	0.4754	0.7967	0.6338	0.6801	0.5534	0.4433
auth5	0.4626	0.6181	0.7986	0.6765	0.4114	0.5638
MeSHLabeler	0.5058	0.8024	0.6882	0.7173	0.5697	0.4825
MeSH Now BF	0.3817	0.5990	0.5991	0.5717	0.3992	0.3987
MeSH Now HR	0.2175	0.2160	0.8964	0.3394	0.1352	0.6112
MeSHLabeler-1	0.5069	0.8023	0.6896	0.7180	0.5691	0.4843
MeSHLabeler-2	0.5058	0.8040	0.6875	0.7175	0.5698	0.4823
MeSHLabeler-3	0.5060	0.8025	0.6890	0.7179	0.5682	0.4835
MeSHLabeler-4	0.5070	0.8025	0.6894	0.7180	0.5694	0.4842
USI baseline	0.4677	0.7095	0.6917	0.6773	0.4895	0.4765
iria-1	0.4090	0.6911	0.5798	0.6027	0.4709	0.3891

+ Test batch 1, week 3

Annotated articles:2832/3638

Flat Measures

System Name	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
BioASQ_Baseline	0.1547	0.3777	0.1431	0.1495	0.2418	0.1588	0.1464	0.1920	0.1295	0.0860
MTI First Line Index	0.5688	0.6368	0.5312	0.5573	0.5934	0.4707	0.4583	0.6368	0.5140	0.4024
Default MTI	0.5758	0.5961	0.5764	0.5643	0.5570	0.5142	0.4836	0.5936	0.5590	0.4078
TextCategorisation1	0.5691	0.6534	0.5133	0.5513	0.6169	0.4224	0.4212	0.6568	0.5020	0.3976
TextCategorisation2	0.5689	0.6497	0.5152	0.5511	0.6121	0.4245	0.4224	0.6530	0.5040	0.3973
TextCategorisation3	0.5690	0.6521	0.5149	0.5510	0.6126	0.4243	0.4223	0.6533	0.5039	0.3972
TextCategorisation4	0.5690	0.6471	0.5161	0.5518	0.6110	0.4252	0.4229	0.6522	0.5046	0.3978
TextCategorisation5	0.5691	0.6559	0.5130	0.5511	0.6173	0.4222	0.4210	0.6571	0.5018	0.3975
spoon-spoon	0.4681	0.4907	0.4673	0.4566	0.4903	0.2952	0.2917	0.4863	0.4512	0.3100
fork-fork	0.4233	0.6470	0.3205	0.4048	0.6666	0.1536	0.1714	0.6517	0.3134	0.2698
it is a test submit	0.3069	0.2731	0.3756	0.3016	0.2661	0.3337	0.2754	0.2731	0.3503	0.1848
USI 10 neighbours	0.5239	0.5618	0.5126	0.5115	0.5089	0.3810	0.3735	0.5526	0.4979	0.3591
USI 20 neighbours	0.5731	0.6304	0.5471	0.5643	0.5910	0.4688	0.4585	0.6338	0.5229	0.4082
Abstract framework	0.5279	0.5688	0.5194	0.5150	0.5041	0.3821	0.3739	0.5513	0.5065	0.3628
auth1	0.5944	0.5774	0.6200	0.5805	0.5736	0.4870	0.4695	0.5858	0.6034	0.4234
auth2	0.5936	0.5944	0.5991	0.5787	0.5907	0.4735	0.4625	0.6044	0.5832	0.4218
auth3	0.6046	0.5538	0.6714	0.5897	0.6030	0.5274	0.5023	0.5606	0.6562	0.4316
auth4	0.5856	0.5915	0.5834	0.5695	0.6571	0.4109	0.4158	0.6006	0.5712	0.4131
auth5	0.5969	0.5733	0.6277	0.5817	0.6332	0.4694	0.4600	0.5811	0.6137	0.4243
MeSHLabeler	0.6204	0.6992	0.5780	0.6094	0.6673	0.4746	0.4804	0.7012	0.5562	0.4543
MeSH Now BF	0.5998	0.6203	0.5964	0.5868	0.5962	0.4612	0.4547	0.6197	0.5812	0.4305
MeSH Now HR	0.2154	0.1237	0.8556	0.2119	0.0923	0.7567	0.3089	0.1233	0.8516	0.1209
MeSHLabeler-1	0.6217	0.6963	0.5810	0.6110	0.6634	0.4791	0.4836	0.6996	0.5595	0.4557
MeSHLabeler-2	0.6227	0.6156	0.6445	0.6107	0.5877	0.5383	0.5139	0.6218	0.6236	0.4538
MeSHLabeler-3	0.6259	0.6869	0.5939	0.6145	0.6514	0.4941	0.4936	0.6843	0.5767	0.4590
MeSHLabeler-4	0.6276	0.6846	0.5989	0.6169	0.6469	0.4998	0.4976	0.6823	0.5811	0.4617
USI baseline	0.5946	0.6596	0.5612	0.5853	0.6317	0.4189	0.4255	0.6671	0.5363	0.4292
iria-1	0.4672	0.5151	0.4550	0.4613	0.4264	0.2860	0.2850	0.5151	0.4275	0.3143

Hierarchical Measures

System Name	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R