BioASQ Participants Area

Test Results for Task 10a

The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.

+ Test batch 1, week 1

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6240	0.6791	0.6081	0.6229	0.6473	0.5732	0.5512	0.6673	0.5861	0.4686
Default MTI	0.6290	0.6438	0.6457	0.6266	0.6216	0.6073	0.5660	0.6356	0.6226	0.4714
Dexstr system	0.1664	0.3644	0.1193	0.1677	0.5829	0.1597	0.1756	0.3428	0.1099	0.0972
NLM System 2	0.6968	0.7309	0.6870	0.6916	0.7122	0.5658	0.5663	0.7274	0.6687	0.5437
deepmesh_dmiip_fdu	0.6612	0.8191	0.5804	0.6621	0.7935	0.4910	0.5207	0.8207	0.5535	0.5108
attention_dmiip_fdu	0.6901	0.7313	0.6737	0.6847	0.6928	0.5810	0.5740	0.7260	0.6575	0.5357
NLM CNN	0.6488	0.6758	0.6415	0.6406	0.6212	0.4770	0.4757	0.6710	0.6280	0.4860
NLM System 1	0.6967	0.7315	0.6849	0.6911	0.7154	0.5638	0.5652	0.7285	0.6676	0.5432
Plain dict match	0.2325	0.3170	0.2019	0.2298	0.3714	0.3485	0.2854	0.2930	0.1928	0.1359
XLinear model	0.5490	0.7443	0.4503	0.5405	0.6972	0.3129	0.3467	0.7403	0.4363	0.3867

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5057	0.7904	0.7225	0.7370	0.5528	0.4922
Default MTI	0.5096	0.7609	0.7568	0.7416	0.5293	0.5191
Dexstr system	0.2222	0.6767	0.2527	0.3424	0.4169	0.1643
NLM System 2	0.5554	0.8244	0.7669	0.7787	0.5919	0.5481
deepmesh_dmiip_fdu	0.5137	0.8899	0.6633	0.7425	0.6267	0.4563
attention_dmiip_fdu	0.5484	0.8236	0.7570	0.7729	0.5887	0.5379
NLM CNN	0.5165	0.7925	0.7274	0.7411	0.5549	0.5082
NLM System 1	0.5552	0.8254	0.7652	0.7785	0.5924	0.5469
Plain dict match	0.2524	0.5493	0.4070	0.4419	0.3070	0.2383
XLinear model	0.4226	0.8565	0.5371	0.6380	0.5737	0.3542

+ Test batch 1, week 2

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6347	0.6778	0.6401	0.6384	0.6235	0.5898	0.5599	0.6617	0.6099	0.4885
Default MTI	0.6321	0.6265	0.6770	0.6299	0.5859	0.6262	0.5742	0.6172	0.6476	0.4758
Dexstr system	0.3341	0.3163	0.4224	0.3291	0.3507	0.4365	0.3511	0.2858	0.4019	0.2041
NLM System 2	0.6972	0.7108	0.7127	0.6912	0.6858	0.5851	0.5739	0.7063	0.6884	0.5447
deepmesh_dmiip_fdu	0.6755	0.7989	0.6165	0.6757	0.7718	0.5093	0.5280	0.8002	0.5844	0.5278
attention_dmiip_fdu	0.6951	0.7157	0.7014	0.6885	0.6769	0.5982	0.5834	0.7098	0.6811	0.5413
NLM CNN	0.6499	0.6620	0.6618	0.6406	0.6035	0.4963	0.4860	0.6558	0.6441	0.4871
NLM System 1	0.6987	0.7140	0.7087	0.6918	0.6948	0.5783	0.5701	0.7107	0.6871	0.5454
Plain dict match	0.2292	0.3026	0.2055	0.2273	0.3493	0.3541	0.2912	0.2788	0.1945	0.1347
XLinear model	0.5576	0.7329	0.4680	0.5474	0.6827	0.3257	0.3511	0.7275	0.4521	0.3950
combination system	0.2464	0.2927	0.2403	0.2462	0.3161	0.3971	0.3142	0.2679	0.2282	0.1464
consensus system	0.1160	0.5395	0.0701	0.1183	0.4573	0.0994	0.0874	0.4698	0.0662	0.0665

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5175	0.7808	0.7460	0.7442	0.5503	0.5162
Default MTI	0.5085	0.7405	0.7806	0.7412	0.5118	0.5367
Dexstr system	0.3107	0.4813	0.6395	0.5137	0.2855	0.3915
NLM System 2	0.5534	0.8030	0.7892	0.7777	0.5754	0.5626
deepmesh_dmiip_fdu	0.5257	0.8729	0.6973	0.7565	0.6157	0.4826
attention_dmiip_fdu	0.5498	0.8078	0.7817	0.7765	0.5756	0.5544
NLM CNN	0.5134	0.7767	0.7450	0.7404	0.5409	0.5186
NLM System 1	0.5541	0.8069	0.7862	0.7788	0.5778	0.5612
Plain dict match	0.2511	0.5314	0.4114	0.4383	0.2961	0.2431
XLinear model	0.4268	0.8451	0.5548	0.6459	0.5636	0.3650
combination system	0.2542	0.5093	0.4491	0.4532	0.2801	0.2607
consensus system	0.1175	0.6864	0.1224	0.1965	0.3409	0.0742

+ Test batch 1, week 3

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6317	0.6950	0.6168	0.6349	0.6637	0.5837	0.5624	0.6805	0.5894	0.4835
Default MTI	0.6288	0.6430	0.6526	0.6276	0.6175	0.6115	0.5709	0.6324	0.6252	0.4719
Dexstr system	0.3490	0.3252	0.4074	0.3475	0.3830	0.4291	0.3577	0.3168	0.3885	0.2167
NLM System 2	0.6841	0.7136	0.6842	0.6796	0.6944	0.5770	0.5694	0.7054	0.6640	0.5291
deepmesh_dmiip_fdu	0.6796	0.7173	0.6710	0.6743	0.6852	0.5822	0.5733	0.7075	0.6538	0.5238
attention_dmiip_fdu	0.6553	0.8089	0.5783	0.6563	0.7880	0.4940	0.5176	0.8068	0.5517	0.5038
NLM CNN	0.6364	0.6690	0.6324	0.6298	0.6216	0.4831	0.4792	0.6575	0.6166	0.4744
NLM System 1	0.6814	0.7124	0.6789	0.6765	0.7008	0.5705	0.5657	0.7035	0.6606	0.5260
XLinear model	0.5422	0.7344	0.4490	0.5355	0.6909	0.3238	0.3503	0.7278	0.4320	0.3815

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5136	0.8018	0.7232	0.7432	0.5626	0.4979
Default MTI	0.5065	0.7604	0.7577	0.7420	0.5242	0.5185
Dexstr system	0.3273	0.5063	0.6301	0.5440	0.3044	0.3827
NLM System 2	0.5436	0.8107	0.7682	0.7727	0.5760	0.5407
deepmesh_dmiip_fdu	0.5367	0.8153	0.7566	0.7681	0.5770	0.5278
attention_dmiip_fdu	0.5046	0.8879	0.6646	0.7433	0.6163	0.4481
NLM CNN	0.5035	0.7876	0.7215	0.7349	0.5442	0.4944
NLM System 1	0.5409	0.8110	0.7648	0.7710	0.5746	0.5369
XLinear model	0.4156	0.8533	0.5361	0.6367	0.5670	0.3478

+ Test batch 1, week 4

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6209	0.6794	0.6115	0.6252	0.6421	0.5707	0.5502	0.6652	0.5822	0.4731
Default MTI	0.6173	0.6229	0.6502	0.6163	0.5940	0.6016	0.5585	0.6142	0.6204	0.4598
Dexstr system	0.3929	0.3837	0.4411	0.3916	0.3783	0.4216	0.3526	0.3689	0.4202	0.2520
Dexstr 2	0.3760	0.3748	0.4168	0.3759	0.3783	0.4215	0.3525	0.3588	0.3951	0.2398
BioASQ Filtering	0.5630	0.7530	0.4883	0.5733	0.7215	0.4576	0.4736	0.7509	0.4503	0.4196
NLM System 2	0.6751	0.6974	0.6806	0.6696	0.6842	0.5745	0.5656	0.6915	0.6594	0.5178
deepmesh_dmiip_fdu	0.6459	0.7919	0.5757	0.6467	0.7732	0.4872	0.5085	0.7906	0.5459	0.4936
deepmesh_dmiip_fdu_	0.6529	0.8055	0.5815	0.6571	0.7766	0.5011	0.5215	0.8057	0.5488	0.5077
attention_dmiip_fdu	0.6694	0.6963	0.6722	0.6638	0.6661	0.5837	0.5695	0.6862	0.6534	0.5118
NLM CNN	0.6244	0.6410	0.6335	0.6161	0.6058	0.4799	0.4717	0.6321	0.6169	0.4599
dmiip_fdu	0.6812	0.7251	0.6694	0.6799	0.6868	0.5891	0.5789	0.7170	0.6489	0.5319
NLM System 1	0.6725	0.6956	0.6777	0.6667	0.6932	0.5679	0.5619	0.6870	0.6587	0.5151
Plain dict match	0.2331	0.3081	0.2102	0.2320	0.3538	0.3527	0.2939	0.2845	0.1974	0.1377
XLinear model	0.5390	0.7190	0.4506	0.5318	0.6833	0.3172	0.3431	0.7130	0.4332	0.3780

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5014	0.7897	0.7182	0.7342	0.5469	0.4895
Default MTI	0.4932	0.7445	0.7530	0.7305	0.5066	0.5111
Dexstr system	0.3513	0.5540	0.6256	0.5658	0.3455	0.3887
Dexstr 2	0.3415	0.5526	0.6156	0.5604	0.3375	0.3768
BioASQ Filtering	0.4376	0.8494	0.6088	0.6911	0.5490	0.3843
NLM System 2	0.5302	0.8014	0.7617	0.7640	0.5604	0.5311
deepmesh_dmiip_fdu	0.4941	0.8739	0.6611	0.7343	0.5988	0.4437
deepmesh_dmiip_fdu_	0.5071	0.8811	0.6669	0.7416	0.6124	0.4550
attention_dmiip_fdu	0.5249	0.7991	0.7564	0.7590	0.5563	0.5256
NLM CNN	0.4906	0.7686	0.7222	0.7253	0.5218	0.4924
dmiip_fdu	0.5410	0.8198	0.7534	0.7696	0.5804	0.5311
NLM System 1	0.5282	0.8001	0.7614	0.7626	0.5583	0.5302
Plain dict match	0.2525	0.5384	0.4137	0.4417	0.3011	0.2427
XLinear model	0.4104	0.8401	0.5371	0.6331	0.5512	0.3471

+ Test batch 1, week 5

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6322	0.6879	0.6252	0.6362	0.6416	0.5882	0.5619	0.6722	0.5967	0.4870
Default MTI	0.6284	0.6314	0.6625	0.6272	0.5960	0.6230	0.5753	0.6221	0.6349	0.4731
Dexstr system	0.4041	0.3798	0.4598	0.3985	0.3787	0.4233	0.3515	0.3705	0.4443	0.2582
BioASQ Filtering	0.5732	0.7584	0.4973	0.5814	0.7236	0.4740	0.4889	0.7579	0.4609	0.4293
NLM System 4	0.6882	0.7064	0.6986	0.6837	0.6905	0.5826	0.5711	0.7000	0.6768	0.5346
NLM System 2	0.6889	0.7065	0.7005	0.6845	0.6826	0.5866	0.5734	0.7002	0.6780	0.5354
deepmesh_dmiip_fdu	0.6610	0.8026	0.5925	0.6620	0.7775	0.5016	0.5226	0.8024	0.5620	0.5126
deepmesh_dmiip_fdu_	0.6695	0.8146	0.6011	0.6727	0.7822	0.5174	0.5378	0.8151	0.5680	0.5278
attention_dmiip_fdu	0.6792	0.7138	0.6783	0.6751	0.6731	0.5923	0.5784	0.7042	0.6559	0.5254
NLM CNN	0.6377	0.6536	0.6499	0.6309	0.6025	0.4913	0.4804	0.6454	0.6302	0.4758
dmiip_fdu	0.6887	0.7370	0.6804	0.6887	0.6886	0.6022	0.5910	0.7245	0.6562	0.5441
NLM System 1	0.6861	0.7120	0.6911	0.6819	0.6923	0.5746	0.5655	0.7043	0.6688	0.5330
NLM System 3	0.6850	0.6943	0.7035	0.6802	0.6771	0.5890	0.5727	0.6878	0.6823	0.5306
coomat inference	0.6910	0.7323	0.6876	0.6912	0.6887	0.6018	0.5903	0.7198	0.6644	0.5465
Plain dict match	0.2263	0.3030	0.2018	0.2253	0.3486	0.3495	0.2869	0.2787	0.1905	0.1333
XLinear model	0.5797	0.6211	0.5673	0.5730	0.5974	0.4133	0.4191	0.6169	0.5468	0.4159

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5123	0.7943	0.7325	0.7441	0.5528	0.5041
Default MTI	0.5020	0.7489	0.7654	0.7390	0.5110	0.5235
Dexstr system	0.3584	0.5494	0.6393	0.5704	0.3459	0.4030
BioASQ Filtering	0.4466	0.8524	0.6210	0.7000	0.5525	0.3956
NLM System 4	0.5436	0.8033	0.7767	0.7721	0.5674	0.5510
NLM System 2	0.5439	0.8026	0.7778	0.7722	0.5674	0.5518
deepmesh_dmiip_fdu	0.5129	0.8802	0.6764	0.7463	0.6127	0.4641
deepmesh_dmiip_fdu_	0.5264	0.8866	0.6843	0.7539	0.6253	0.4779
attention_dmiip_fdu	0.5372	0.8088	0.7605	0.7652	0.5710	0.5366
NLM CNN	0.5020	0.7732	0.7335	0.7328	0.5301	0.5067
dmiip_fdu	0.5520	0.8248	0.7616	0.7742	0.5913	0.5455
NLM System 1	0.5411	0.8083	0.7693	0.7703	0.5692	0.5452
NLM System 3	0.5422	0.7945	0.7833	0.7712	0.5593	0.5555
coomat inference	0.5556	0.8228	0.7693	0.7781	0.5904	0.5518
Plain dict match	0.2508	0.5382	0.4072	0.4379	0.3002	0.2405
XLinear model	0.4619	0.7637	0.6630	0.6899	0.5150	0.4447

+ Test batch 2, week 1

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6194	0.6680	0.6096	0.6184	0.6313	0.5820	0.5537	0.6556	0.5869	0.4648
Default MTI	0.6263	0.6345	0.6516	0.6246	0.6059	0.6178	0.5699	0.6249	0.6276	0.4706
Dexstr system	0.4161	0.3626	0.5112	0.4094	0.3920	0.4505	0.3669	0.3571	0.4985	0.2663
BioASQ Filtering	0.5630	0.7485	0.4838	0.5696	0.7175	0.4763	0.4939	0.7472	0.4517	0.4151
NLM System 4	0.6853	0.7162	0.6786	0.6804	0.7036	0.5695	0.5682	0.7119	0.6607	0.5320
NLM System 2	0.6863	0.7170	0.6796	0.6814	0.7010	0.5713	0.5692	0.7132	0.6614	0.5330
deepmesh_dmiip_fdu	0.6832	0.7386	0.6583	0.6814	0.7058	0.5781	0.5778	0.7385	0.6356	0.5336
deepmesh_dmiip_fdu_	0.6848	0.7368	0.6620	0.6825	0.7109	0.5774	0.5774	0.7365	0.6399	0.5346
attention_dmiip_fdu	0.6811	0.7232	0.6680	0.6777	0.6826	0.5862	0.5778	0.7155	0.6498	0.5294
NLM CNN	0.6349	0.6574	0.6347	0.6271	0.6056	0.4747	0.4700	0.6503	0.6202	0.4723
dmiip_fdu	0.6819	0.7186	0.6734	0.6782	0.6875	0.5849	0.5761	0.7094	0.6565	0.5300
NLM System 1	0.6830	0.7115	0.6787	0.6782	0.6950	0.5652	0.5624	0.7068	0.6608	0.5293
NLM System 3	0.6835	0.7168	0.6748	0.6785	0.7038	0.5615	0.5619	0.7120	0.6571	0.5299
coomat inference	0.1294	0.1388	0.1326	0.1300	0.0234	0.0282	0.0233	0.1312	0.1275	0.0826
XLinear model	0.5711	0.6153	0.5497	0.5625	0.5856	0.3985	0.4069	0.6136	0.5341	0.4062
similar to BertMesh	0.6119	0.6764	0.5750	0.6018	0.5807	0.3790	0.3725	0.6723	0.5615	0.4472

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5001	0.7827	0.7252	0.7346	0.5414	0.4913
Default MTI	0.5058	0.7530	0.7613	0.7402	0.5185	0.5220
Dexstr system	0.3674	0.5252	0.6710	0.5715	0.3390	0.4332
BioASQ Filtering	0.4410	0.8473	0.6175	0.6966	0.5489	0.3883
NLM System 4	0.5460	0.8142	0.7634	0.7720	0.5790	0.5416
NLM System 2	0.5464	0.8145	0.7638	0.7724	0.5793	0.5421
deepmesh_dmiip_fdu	0.5437	0.8319	0.7440	0.7707	0.5933	0.5240
deepmesh_dmiip_fdu_	0.5453	0.8304	0.7477	0.7721	0.5930	0.5271
attention_dmiip_fdu	0.5429	0.8181	0.7558	0.7695	0.5813	0.5342
NLM CNN	0.5053	0.7796	0.7261	0.7337	0.5395	0.5020
dmiip_fdu	0.5445	0.8148	0.7601	0.7701	0.5808	0.5378
NLM System 1	0.5449	0.8115	0.7635	0.7706	0.5770	0.5416
NLM System 3	0.5449	0.8156	0.7605	0.7709	0.5802	0.5389
coomat inference	0.1928	0.2544	0.2440	0.2367	0.2055	0.1965
XLinear model	0.4589	0.7619	0.6546	0.6854	0.5147	0.4382
similar to BertMesh	0.4796	0.7883	0.6774	0.7086	0.5373	0.4586

+ Test batch 2, week 2

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6362	0.6783	0.6376	0.6383	0.6244	0.5813	0.5547	0.6641	0.6105	0.4892
Default MTI	0.6320	0.6284	0.6733	0.6302	0.5841	0.6167	0.5688	0.6191	0.6454	0.4765
BioASQ Filtering	0.5843	0.7477	0.5181	0.5922	0.7089	0.4740	0.4852	0.7484	0.4793	0.4405
NLM System 4	0.6957	0.7089	0.7071	0.6891	0.6889	0.5877	0.5770	0.7033	0.6882	0.5428
NLM System 2	0.6978	0.7101	0.7108	0.6919	0.6857	0.5933	0.5812	0.7050	0.6908	0.5457
deepmesh_dmiip_fdu	0.7017	0.7522	0.6849	0.7019	0.7120	0.5825	0.5810	0.7500	0.6592	0.5597
deepmesh_dmiip_fdu_	0.7148	0.7621	0.6976	0.7138	0.7304	0.5940	0.5953	0.7610	0.6739	0.5775
attention_dmiip_fdu	0.7134	0.7402	0.7144	0.7107	0.7067	0.6144	0.6056	0.7293	0.6981	0.5734
NLM CNN	0.6438	0.6512	0.6624	0.6360	0.5990	0.4989	0.4885	0.6431	0.6444	0.4823
dmiip_fdu	0.7140	0.7131	0.7388	0.7078	0.6880	0.6511	0.6302	0.7058	0.7225	0.5685
NLM System 1	0.6927	0.7038	0.7065	0.6863	0.6772	0.5812	0.5703	0.6982	0.6872	0.5394
NLM System 3	0.6948	0.7096	0.7042	0.6880	0.6894	0.5814	0.5721	0.7039	0.6859	0.5414
coomat inference	0.7088	0.7226	0.7250	0.7073	0.6834	0.6336	0.6157	0.7108	0.7068	0.5657
XLinear model	0.5827	0.6199	0.5733	0.5759	0.5792	0.4103	0.4130	0.6162	0.5527	0.4189
similar to BertMesh	0.6248	0.6740	0.6004	0.6137	0.5839	0.4062	0.3975	0.6708	0.5847	0.4603

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5206	0.7854	0.7465	0.7473	0.5540	0.5183
Default MTI	0.5116	0.7453	0.7787	0.7434	0.5176	0.5373
BioASQ Filtering	0.4617	0.8436	0.6441	0.7116	0.5572	0.4167
NLM System 4	0.5554	0.8037	0.7866	0.7778	0.5749	0.5661
NLM System 2	0.5569	0.8039	0.7888	0.7790	0.5757	0.5681
deepmesh_dmiip_fdu	0.5623	0.8361	0.7632	0.7827	0.6048	0.5494
deepmesh_dmiip_fdu_	0.5820	0.8433	0.7729	0.7916	0.6239	0.5686
attention_dmiip_fdu	0.5816	0.8248	0.7907	0.7914	0.6077	0.5833
NLM CNN	0.5116	0.7699	0.7440	0.7370	0.5336	0.5214
dmiip_fdu	0.5791	0.8033	0.8121	0.7910	0.5890	0.5986
NLM System 1	0.5542	0.8015	0.7871	0.7770	0.5731	0.5658
NLM System 3	0.5530	0.8032	0.7836	0.7757	0.5735	0.5625
coomat inference	0.5725	0.8096	0.8013	0.7893	0.5889	0.5846
XLinear model	0.4702	0.7629	0.6709	0.6940	0.5199	0.4556
similar to BertMesh	0.4904	0.7828	0.6981	0.7174	0.5369	0.4794

+ Test batch 2, week 3

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6177	0.6614	0.6141	0.6172	0.6188	0.5830	0.5517	0.6482	0.5900	0.4625
Default MTI	0.6181	0.6217	0.6481	0.6160	0.5871	0.6113	0.5622	0.6126	0.6237	0.4599
BioASQ Filtering	0.5680	0.7432	0.4917	0.5734	0.7113	0.4683	0.4804	0.7436	0.4594	0.4189
NLM System 4	0.6936	0.7082	0.7014	0.6884	0.6979	0.5949	0.5841	0.7036	0.6839	0.5407
NLM System 2	0.6945	0.7082	0.7032	0.6894	0.6949	0.6004	0.5874	0.7040	0.6853	0.5416
deepmesh_dmiip_fdu	0.7084	0.7512	0.6887	0.7039	0.7244	0.6041	0.6006	0.7525	0.6691	0.5639
deepmesh_dmiip_fdu_	0.6913	0.7388	0.6733	0.6898	0.7113	0.5819	0.5785	0.7366	0.6513	0.5427
attention_dmiip_fdu	0.7053	0.7306	0.7033	0.7004	0.7069	0.6251	0.6119	0.7223	0.6891	0.5599
NLM CNN	0.6406	0.6542	0.6504	0.6345	0.6052	0.4929	0.4824	0.6479	0.6333	0.4802
dmiip_fdu	0.5478	0.5532	0.5367	0.5322	0.5368	0.4541	0.4498	0.5662	0.5306	0.4171
NLM System 1	0.6886	0.7090	0.6921	0.6839	0.6952	0.5770	0.5685	0.7039	0.6741	0.5352
NLM System 3	0.6916	0.7069	0.6989	0.6865	0.6955	0.5883	0.5781	0.7024	0.6810	0.5388
coomat inference	0.5396	0.5559	0.5290	0.5285	0.5263	0.4431	0.4371	0.5628	0.5182	0.4062
XLinear model	0.5740	0.6183	0.5571	0.5676	0.5847	0.4129	0.4157	0.6139	0.5390	0.4106
similar to BertMesh	0.6163	0.6753	0.5843	0.6072	0.5878	0.4067	0.3965	0.6708	0.5699	0.4524
svm_baseline	0.4780	0.7530	0.3617	0.4622	0.7155	0.1672	0.1926	0.7485	0.3512	0.3199

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.4955	0.7713	0.7270	0.7296	0.5311	0.4926
Default MTI	0.4973	0.7409	0.7582	0.7315	0.5060	0.5186
BioASQ Filtering	0.4436	0.8419	0.6225	0.6974	0.5443	0.3951
NLM System 4	0.5544	0.8043	0.7813	0.7768	0.5760	0.5603
NLM System 2	0.5547	0.8036	0.7833	0.7774	0.5755	0.5615
deepmesh_dmiip_fdu	0.5700	0.8349	0.7664	0.7846	0.6094	0.5585
deepmesh_dmiip_fdu_	0.5488	0.8281	0.7544	0.7745	0.5914	0.5353
attention_dmiip_fdu	0.5711	0.8183	0.7830	0.7844	0.5978	0.5720
NLM CNN	0.5096	0.7747	0.7358	0.7372	0.5368	0.5126
dmiip_fdu	0.4657	0.6533	0.6261	0.6248	0.4884	0.4666
NLM System 1	0.5495	0.8055	0.7730	0.7730	0.5756	0.5513
NLM System 3	0.5532	0.8041	0.7788	0.7755	0.5754	0.5584
coomat inference	0.4518	0.6570	0.6225	0.6240	0.4785	0.4504
XLinear model	0.4608	0.7611	0.6584	0.6867	0.5143	0.4430
similar to BertMesh	0.4833	0.7819	0.6843	0.7104	0.5352	0.4667
svm_baseline	0.3521	0.8603	0.4273	0.5423	0.5571	0.2768

+ Test batch 2, week 4

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6521	0.6944	0.6513	0.6552	0.6407	0.6286	0.5947	0.6776	0.6285	0.5090
Default MTI	0.6407	0.6293	0.6899	0.6391	0.5851	0.6647	0.6040	0.6167	0.6665	0.4870
BioASQ Filtering	0.5995	0.7582	0.5289	0.6048	0.7178	0.5113	0.5182	0.7571	0.4961	0.4534
NLM System 4	0.6913	0.7018	0.6972	0.6833	0.6799	0.6056	0.5891	0.6968	0.6859	0.5349
NLM System 2	0.6972	0.7171	0.7001	0.6930	0.6813	0.6118	0.5948	0.7095	0.6853	0.5474
deepmesh_dmiip_fdu	0.7200	0.7525	0.7105	0.7178	0.7243	0.6418	0.6305	0.7501	0.6922	0.5828
deepmesh_dmiip_fdu_	0.7012	0.7406	0.6900	0.7004	0.7069	0.6197	0.6078	0.7357	0.6697	0.5579
attention_dmiip_fdu	0.7130	0.7366	0.7132	0.7093	0.6945	0.6466	0.6274	0.7268	0.6997	0.5733
NLM CNN	0.6325	0.6359	0.6532	0.6222	0.5826	0.5091	0.4891	0.6250	0.6403	0.4672
dmiip_fdu	0.7083	0.7113	0.7280	0.7015	0.6972	0.6594	0.6376	0.7029	0.7137	0.5605
NLM System 1	0.6827	0.6852	0.7018	0.6740	0.6685	0.6046	0.5831	0.6777	0.6877	0.5241
NLM System 3	0.6854	0.6850	0.7066	0.6771	0.6667	0.6114	0.5900	0.6788	0.6921	0.5275
coomat inference	0.7038	0.7275	0.7088	0.7024	0.6913	0.6443	0.6246	0.7159	0.6922	0.5604
XLinear model	0.5700	0.5772	0.5796	0.5583	0.5401	0.4194	0.4088	0.5706	0.5695	0.4024
similar to BertMesh	0.6197	0.6603	0.5992	0.6072	0.5727	0.4332	0.4140	0.6557	0.5875	0.4530
svm_baseline	0.4857	0.7404	0.3689	0.4626	0.7122	0.1763	0.1988	0.7357	0.3625	0.3208

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5282	0.7908	0.7535	0.7555	0.5558	0.5273
Default MTI	0.5102	0.7383	0.7882	0.7452	0.5060	0.5443
BioASQ Filtering	0.4634	0.8479	0.6499	0.7181	0.5532	0.4194
NLM System 4	0.5421	0.8011	0.7794	0.7751	0.5604	0.5499
NLM System 2	0.5518	0.8095	0.7814	0.7806	0.5735	0.5555
deepmesh_dmiip_fdu	0.5847	0.8352	0.7870	0.7976	0.6146	0.5783
deepmesh_dmiip_fdu_	0.5609	0.8274	0.7714	0.7848	0.5944	0.5524
attention_dmiip_fdu	0.5800	0.8205	0.7906	0.7907	0.6024	0.5825
NLM CNN	0.4961	0.7551	0.7406	0.7271	0.5142	0.5104
dmiip_fdu	0.5659	0.8007	0.8048	0.7866	0.5773	0.5814
NLM System 1	0.5354	0.7854	0.7845	0.7677	0.5492	0.5507
NLM System 3	0.5387	0.7858	0.7901	0.7713	0.5504	0.5550
coomat inference	0.5649	0.8133	0.7901	0.7866	0.5861	0.5688
XLinear model	0.4532	0.7198	0.6821	0.6799	0.4812	0.4568
similar to BertMesh	0.4807	0.7690	0.6987	0.7123	0.5206	0.4736
svm_baseline	0.3530	0.8510	0.4353	0.5465	0.5448	0.2813

+ Test batch 2, week 5

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6403	0.6709	0.6389	0.6370	0.6213	0.5975	0.5651	0.6595	0.6222	0.4849
Default MTI	0.6390	0.6267	0.6791	0.6340	0.5932	0.6374	0.5854	0.6174	0.6621	0.4824
BioASQ Filtering	0.5875	0.7447	0.5156	0.5904	0.7050	0.4782	0.4862	0.7427	0.4859	0.4355
NLM System 4	0.6914	0.6982	0.7009	0.6839	0.6788	0.5920	0.5780	0.6920	0.6909	0.5346
NLM System 2	0.6937	0.7030	0.7021	0.6871	0.6765	0.5955	0.5806	0.6968	0.6906	0.5385
deepmesh_dmiip_fdu	0.7023	0.7389	0.6873	0.6971	0.7032	0.6000	0.5939	0.7372	0.6706	0.5550
deepmesh_dmiip_fdu_	0.7073	0.7361	0.6970	0.7019	0.7104	0.6080	0.6003	0.7348	0.6818	0.5605
attention_dmiip_fdu	0.7008	0.7147	0.7083	0.6951	0.6778	0.6154	0.5977	0.7043	0.6973	0.5533
NLM CNN	0.6377	0.6391	0.6577	0.6287	0.5864	0.5043	0.4871	0.6287	0.6470	0.4728
dmiip_fdu	0.6977	0.7262	0.6902	0.6929	0.6974	0.6022	0.5923	0.7237	0.6736	0.5447
NLM System 1	0.6845	0.6874	0.7026	0.6772	0.6697	0.5897	0.5733	0.6792	0.6899	0.5270
NLM System 3	0.6877	0.6894	0.7051	0.6799	0.6693	0.5912	0.5747	0.6822	0.6933	0.5301
coomat inference	0.7087	0.7167	0.7213	0.7032	0.6909	0.6415	0.6220	0.7076	0.7098	0.5618
XLinear model	0.5797	0.6031	0.5734	0.5701	0.5637	0.4167	0.4136	0.5996	0.5611	0.4124
similar to BertMesh	0.6200	0.6656	0.5953	0.6092	0.5729	0.4136	0.3985	0.6583	0.5859	0.4531
svm_baseline	0.4889	0.7556	0.3717	0.4699	0.7168	0.1655	0.1892	0.7437	0.3641	0.3252

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5060	0.7749	0.7444	0.7432	0.5333	0.5061
Default MTI	0.5093	0.7398	0.7775	0.7419	0.5094	0.5367
BioASQ Filtering	0.4481	0.8379	0.6359	0.7056	0.5415	0.4029
NLM System 4	0.5412	0.7940	0.7793	0.7711	0.5597	0.5485
NLM System 2	0.5435	0.7946	0.7807	0.7721	0.5620	0.5507
deepmesh_dmiip_fdu	0.5560	0.8237	0.7645	0.7783	0.5933	0.5460
deepmesh_dmiip_fdu_	0.5616	0.8208	0.7720	0.7814	0.5950	0.5540
attention_dmiip_fdu	0.5603	0.8035	0.7846	0.7780	0.5813	0.5655
NLM CNN	0.4978	0.7545	0.7416	0.7290	0.5169	0.5079
dmiip_fdu	0.5434	0.8114	0.7683	0.7745	0.5753	0.5375
NLM System 1	0.5368	0.7853	0.7799	0.7659	0.5524	0.5488
NLM System 3	0.5380	0.7859	0.7821	0.7675	0.5532	0.5504
coomat inference	0.5667	0.8007	0.7961	0.7831	0.5833	0.5755
XLinear model	0.4550	0.7376	0.6699	0.6836	0.4957	0.4450
similar to BertMesh	0.4767	0.7737	0.6921	0.7124	0.5200	0.4649
svm_baseline	0.3511	0.8567	0.4342	0.5469	0.5505	0.2773

+ Test batch 3, week 1

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	0.6474	0.6917	0.6398	0.6459	0.6411	0.6389	0.6074	0.6771	0.6202	0.4949
Default MTI	0.6484	0.6439	0.6810	0.6448	0.6100	0.6712	0.6197	0.6356	0.6617	0.4927
Dexstr system	0.4218	0.3847	0.4964	0.4179	0.3534	0.4698	0.3927	0.3749	0.4822	0.2725
BioASQ Filtering	0.5936	0.7682	0.5157	0.5988	0.7340	0.5157	0.5242	0.7673	0.4840	0.4447
NLM System 4	0.6949	0.6996	0.7140	0.6909	0.6725	0.6255	0.6031	0.6905	0.6995	0.5428
NLM System 2	0.6932	0.7088	0.7067	0.6902	0.6777	0.6274	0.6067	0.6961	0.6904	0.5431
deepmesh_dmiip_fdu	0.7306	0.7600	0.7215	0.7252	0.7362	0.6816	0.6690	0.7559	0.7070	0.5898
deepmesh_dmiip_fdu_	0.7251	0.7492	0.7282	0.7226	0.7202	0.6899	0.6739	0.7400	0.7107	0.5876
attention_dmiip_fdu	0.7258	0.7518	0.7223	0.7207	0.7272	0.6712	0.6601	0.7454	0.7072	0.5845
NLM CNN	0.6411	0.6391	0.6656	0.6348	0.5803	0.5335	0.5090	0.6289	0.6538	0.4798
dmiip_fdu	0.7250	0.7579	0.7121	0.7192	0.7307	0.6618	0.6528	0.7549	0.6975	0.5831
NLM System 1	0.6915	0.7117	0.7010	0.6880	0.6797	0.6248	0.6060	0.6988	0.6843	0.5407
NLM System 3	0.6911	0.6895	0.7153	0.6862	0.6619	0.6239	0.6002	0.6816	0.7008	0.5376
coomat inference	0.7203	0.7396	0.7314	0.7183	0.7030	0.6828	0.6636	0.7244	0.7162	0.5817
XLinear model	0.5869	0.6125	0.5807	0.5792	0.5622	0.4446	0.4376	0.6074	0.5677	0.4220
similar to BertMesh	0.6257	0.6667	0.6073	0.6180	0.5643	0.4392	0.4202	0.6602	0.5947	0.4635
svm_baseline	0.4980	0.7572	0.3816	0.4821	0.7074	0.1909	0.2114	0.7474	0.3733	0.3358
ediranknn	0.6191	0.7392	0.5490	0.6101	0.6631	0.3563	0.3608	0.7314	0.5367	0.4571

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	0.5118	0.7891	0.7497	0.7524	0.5447	0.5101
Default MTI	0.5149	0.7496	0.7859	0.7517	0.5164	0.5433
Dexstr system	0.3647	0.5447	0.6652	0.5821	0.3425	0.4200
BioASQ Filtering	0.4518	0.8517	0.6406	0.7150	0.5505	0.4026
NLM System 4	0.5515	0.7922	0.7922	0.7781	0.5625	0.5667
NLM System 2	0.5513	0.7998	0.7865	0.7783	0.5669	0.5625
deepmesh_dmiip_fdu	0.5856	0.8341	0.7944	0.8010	0.6128	0.5836
deepmesh_dmiip_fdu_	0.5862	0.8263	0.8000	0.7995	0.6090	0.5890
attention_dmiip_fdu	0.5840	0.8289	0.7962	0.7988	0.6090	0.5847
NLM CNN	0.5049	0.7566	0.7500	0.7369	0.5177	0.5207
dmiip_fdu	0.5809	0.8340	0.7868	0.7967	0.6111	0.5759
NLM System 1	0.5489	0.8027	0.7816	0.7767	0.5675	0.5577
NLM System 3	0.5455	0.7836	0.7943	0.7741	0.5519	0.5656
coomat inference	0.5826	0.8164	0.8077	0.7978	0.5987	0.5928
XLinear model	0.4646	0.7474	0.6782	0.6939	0.5043	0.4556
similar to BertMesh	0.4854	0.7662	0.7066	0.7184	0.5227	0.4785
svm_baseline	0.3583	0.8587	0.4441	0.5590	0.5535	0.2834
ediranknn	0.4716	0.8326	0.6358	0.7016	0.5635	0.4283

+ Test batch 3, week 2

System	MiF	EBP	EBR	EBF	MaP	MaR	MaF	MiP	MiR	Acc.
MTI First Line Index	-	-	-	-	-	-	-	-	-	-
Default MTI	-	-	-	-	-	-	-	-	-	-
Dexstr system	-	-	-	-	-	-	-	-	-	-
BioASQ Filtering	-	-	-	-	-	-	-	-	-	-
deepmesh_dmiip_fdu	-	-	-	-	-	-	-	-	-	-
deepmesh_dmiip_fdu_	-	-	-	-	-	-	-	-	-	-
attention_dmiip_fdu	-	-	-	-	-	-	-	-	-	-
NLM CNN	-	-	-	-	-	-	-	-	-	-
dmiip_fdu	-	-	-	-	-	-	-	-	-	-
NLM System 3	-	-	-	-	-	-	-	-	-	-
coomat inference	-	-	-	-	-	-	-	-	-	-
XLinear model	-	-	-	-	-	-	-	-	-	-
similar to BertMesh	-	-	-	-	-	-	-	-	-	-
ediranknn	-	-	-	-	-	-	-	-	-	-
IRIS_3	-	-	-	-	-	-	-	-	-	-
100k articles, bert	-	-	-	-	-	-	-	-	-	-

System	LCA-F	HiP	HiR	HiF	LCA-P	LCA-R
MTI First Line Index	-	-	-	-	-	-
Default MTI	-	-	-	-	-	-
Dexstr system	-	-	-	-	-	-
BioASQ Filtering	-	-	-	-	-	-
deepmesh_dmiip_fdu	-	-	-	-	-	-
deepmesh_dmiip_fdu_	-	-	-	-	-	-
attention_dmiip_fdu	-	-	-	-	-	-
NLM CNN	-	-	-	-	-	-
dmiip_fdu	-	-	-	-	-	-
NLM System 3	-	-	-	-	-	-
coomat inference	-	-	-	-	-	-
XLinear model	-	-	-	-	-	-
similar to BertMesh	-	-	-	-	-	-
ediranknn	-	-	-	-	-	-
IRIS_3	-	-	-	-	-	-
100k articles, bert	-	-	-	-	-	-

Platinum sponsors of the BioASQ Challenge between 2012 and 2023

Google was a proud sponsor of the BioASQ Challenge from 2018 to 2022.
BioASQ was supported by the National Library of Medicine of the National Institutes of Health under award number R13LM012214. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
The research leading to these results has received funding from the European Commission's Seventh Framework Programme (FP7/2007-2013, ICT-2011.4.4(d), Intelligent Information Management, Targeted Competition Framework) under grant agreement n° 318652.

BioASQ Participants Area

Test Results for Task 10a

+ Test batch 1, week 1

Annotated articles:9450/9659

Flat Measures

Hierarchical Measures

+ Test batch 1, week 2

Annotated articles:4512/4531

Flat Measures

Hierarchical Measures

+ Test batch 1, week 3

Annotated articles:4269/4291

Flat Measures

Hierarchical Measures

+ Test batch 1, week 4

Annotated articles:4192/4256

Flat Measures

Hierarchical Measures

+ Test batch 1, week 5

Annotated articles:4802/4862

Flat Measures

Hierarchical Measures

+ Test batch 2, week 1

Annotated articles:8818/8874

Flat Measures

Hierarchical Measures

+ Test batch 2, week 2

Annotated articles:3858/4071

Flat Measures

Hierarchical Measures

+ Test batch 2, week 3

Annotated articles:4049/4108

Flat Measures

Hierarchical Measures

+ Test batch 2, week 4

Annotated articles:3045/3193

Flat Measures

Hierarchical Measures

+ Test batch 2, week 5

Annotated articles:2916/3078

Flat Measures

Hierarchical Measures

+ Test batch 3, week 1

Annotated articles:1870/2376

Flat Measures

Hierarchical Measures

+ Test batch 3, week 2

Annotated articles:0/28

Flat Measures

Hierarchical Measures