BioASQ Participants Area

Task Synergy - version 2023: Test Results

Test round 1

Documents

System	Mean precision	Recall	F-Measure	MAP	GMAP
bio-answerfinder	0.3113	0.2998	0.2505	0.2720	0.0097
bio-answerfinder-2	0.3787	0.4317	0.3602	0.3825	0.0484
Fleming-3	0.2922	0.3145	0.2685	0.2611	0.0113
Fleming-4	0.2643	0.3151	0.2563	0.2379	0.0121
Just for test	0.4003	0.6804	0.4347	0.5773	0.4851
dmiip1	0.3959	0.7008	0.4384	0.5870	0.4955

Snippets

System	Mean precision	Recall	F-Measure	MAP	GMAP
bio-answerfinder	0.1709	0.2763	0.1844	0.3123	0.0043
bio-answerfinder-2	0.2239	0.3103	0.2299	0.2790	0.0066
Fleming-3	-	-	-	-	-
Fleming-4	-	-	-	-	-
Just for test	0.3571	0.7758	0.4382	0.8498	0.5477
dmiip1	0.3571	0.7758	0.4382	0.8498	0.5477

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
bio-answerfinder	-	-	-	-	-	-	-	-	-	-
bio-answerfinder-2	-	-	-	-	-	-	-	-	-	-
Fleming-3	-	-	-	-	-	-	-	-	-	-
Fleming-4	-	-	-	-	-	-	-	-	-	-
Just for test	-	-	-	-	-	-	-	-	-	-
dmiip1	-	-	-	-	-	-	-	-	-	-

Ideal Answers

	Automatic scores (Rouge - R)				Manual scores
System	R-2 (Rec)	R-2 (F1)	R-SU4 (Rec)	R-SU4 (F1)	Readability	Recall	Precision	Repetition
bio-answerfinder	-	-	-	-	-	-	-	-
bio-answerfinder-2	-	-	-	-	-	-	-	-
Fleming-3	-	-	-	-	-	-	-	-
Fleming-4	-	-	-	-	-	-	-	-
Just for test	-	-	-	-	-	-	-	-
dmiip1	-	-	-	-	-	-	-	-

Test round 2

Documents

System	Mean precision	Recall	F-Measure	MAP	GMAP
bio-answerfinder	0.2846	0.1209	0.1461	0.1674	0.0052
bio-answerfinder-2	0.2965	0.1688	0.1865	0.2168	0.0102
dmiip1	0.4161	0.3806	0.3276	0.3724	0.2050
dmiip3	0.4258	0.3966	0.3369	0.4040	0.1240
dmiip4	0.3548	0.3280	0.2703	0.3237	0.1410
dmiip5	0.4161	0.3684	0.3221	0.3942	0.1471
Fleming-3	-	-	-	-	-
dmiip2	0.3516	0.2322	0.2443	0.3063	0.0289

Snippets

System	Mean precision	Recall	F-Measure	MAP	GMAP
bio-answerfinder	0.1388	0.1066	0.0981	0.1302	0.0023
bio-answerfinder-2	0.2012	0.1572	0.1422	0.1752	0.0065
dmiip1	0.3648	0.3561	0.3026	0.4601	0.0807
dmiip3	0.3471	0.3690	0.2944	0.4678	0.0656
dmiip4	0.3009	0.2995	0.2477	0.3698	0.0481
dmiip5	0.3121	0.2827	0.2517	0.4146	0.0473
Fleming-3	-	-	-	-	-
dmiip2	0.3111	0.2213	0.2297	0.3450	0.0157

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
bio-answerfinder	0.2500	0.4000	-	0.2000	0.3333	0.3333	0.3333	-	-	-
bio-answerfinder-2	0.5000	0.6667	-	0.3333	0.3333	0.3333	0.3333	-	-	-
dmiip1	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	0.2143	0.5000	0.3000
dmiip3	1.0000	1.0000	1.0000	1.0000	0.6667	1.0000	0.7778	0.1667	0.1667	0.1667
dmiip4	1.0000	1.0000	1.0000	1.0000	0.3333	1.0000	0.6111	0.2500	0.1667	0.2000
dmiip5	1.0000	1.0000	1.0000	1.0000	0.3333	0.3333	0.3333	0.0500	0.1667	0.0769
Fleming-3	0.7500	0.8000	0.6667	0.7333	-	-	-	-	-	-
dmiip2	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	0.2143	0.5000	0.3000

Ideal Answers

	Automatic scores (Rouge - R)				Manual scores
System	R-2 (Rec)	R-2 (F1)	R-SU4 (Rec)	R-SU4 (F1)	Readability	Recall	Precision	Repetition
bio-answerfinder	0.0435	0.0455	0.0649	0.0702	2.57	1.36	2.21	3.71
bio-answerfinder-2	0.0433	0.0430	0.0699	0.0715	3.07	1.64	2.07	4.00
dmiip1	0.4993	0.4223	0.5139	0.4337	3.14	3.36	2.86	2.71
dmiip3	0.4586	0.3779	0.4732	0.3892	3.43	3.71	3.00	3.00
dmiip4	0.5256	0.4402	0.5359	0.4462	3.07	3.36	2.86	2.57
dmiip5	0.4779	0.4171	0.4838	0.4206	3.79	3.71	3.50	3.29
Fleming-3	-	-	-	-	1.21	0.57	0.86	1.29
dmiip2	0.4993	0.4223	0.5139	0.4337	3.14	3.36	2.86	2.71

Test round 3

Documents

System	Mean precision	Recall	F-Measure	MAP	GMAP
bio-answerfinder	0.1485	0.1464	0.1236	0.1168	0.0016
bio-answerfinder-2	0.1366	0.1345	0.1123	0.1103	0.0016
Fleming-4	-	-	-	-	-
dmiip4	0.3639	0.4084	0.3319	0.3637	0.1008
dmiip3	0.2583	0.2988	0.2397	0.2387	0.0235
dmiip2	0.3306	0.4195	0.3090	0.3474	0.1984
dmiip1	0.2611	0.3279	0.2462	0.2421	0.0330
dmiip5	0.2139	0.2584	0.1986	0.1608	0.0140
CA-1	-	-	-	-	-

Snippets

System	Mean precision	Recall	F-Measure	MAP	GMAP
bio-answerfinder	0.0830	0.0734	0.0679	0.0603	0.0003
bio-answerfinder-2	0.0744	0.0636	0.0575	0.0498	0.0004
Fleming-4	-	-	-	-	-
dmiip4	0.2298	0.2454	0.2039	0.3360	0.0159
dmiip3	0.1921	0.1825	0.1637	0.2024	0.0048
dmiip2	0.2170	0.2473	0.2004	0.3388	0.0249
dmiip1	0.1845	0.1897	0.1563	0.2362	0.0056
dmiip5	0.1381	0.1695	0.1234	0.1616	0.0024
CA-1	0.0459	0.0231	0.0282	0.0187	0.0000

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
bio-answerfinder	0.5714	0.7273	-	0.3636	0.4286	0.5714	0.5000	0.1486	0.0803	0.1019
bio-answerfinder-2	0.5714	0.7273	-	0.3636	0.2857	0.4286	0.3214	0.1000	0.0532	0.0644
Fleming-4	0.7143	0.7500	0.6667	0.7083	-	-	-	-	-	-
dmiip4	0.4286	0.3333	0.5000	0.4167	0.4286	1.0000	0.6548	0.3641	0.2834	0.2559
dmiip3	0.8571	0.8571	0.8571	0.8571	0.5714	1.0000	0.7143	0.3555	0.2929	0.2467
dmiip2	0.8571	0.8571	0.8571	0.8571	0.7143	0.8571	0.7500	0.3554	0.3308	0.2871
dmiip1	0.8571	0.8571	0.8571	0.8571	0.7143	0.8571	0.7619	0.3571	0.3490	0.2781
dmiip5	0.7143	0.7500	0.6667	0.7083	0.5714	0.7143	0.6190	0.0667	0.2318	0.0796
CA-1	0.5714	0.7273	-	0.3636	0.1429	0.2857	0.1905	0.0083	0.0044	0.0057

Ideal Answers

	Automatic scores (Rouge - R)				Manual scores
System	R-2 (Rec)	R-2 (F1)	R-SU4 (Rec)	R-SU4 (F1)	Readability	Recall	Precision	Repetition
bio-answerfinder	0.1552	0.1587	0.1627	0.1697	2.61	2.03	2.23	2.68
bio-answerfinder-2	0.1682	0.1661	0.1772	0.1788	2.39	1.84	1.77	2.39
Fleming-4	-	-	-	-	0.23	0.13	0.13	0.13
dmiip4	0.4954	0.4168	0.5068	0.4216	2.77	2.81	2.58	2.65
dmiip3	0.4643	0.4028	0.4743	0.4085	2.74	2.77	2.58	2.71
dmiip2	0.4829	0.4095	0.4960	0.4163	2.74	2.81	2.55	2.58
dmiip1	0.4829	0.4095	0.4960	0.4163	2.74	2.81	2.55	2.58
dmiip5	0.2773	0.2485	0.2937	0.2597	2.87	2.61	2.48	2.77
CA-1	-	-	-	-	-	-	-	-

Test round 4

Documents

System	Mean precision	Recall	F-Measure	MAP	GMAP
bio-answerfinder	0.1575	0.1390	0.1244	0.1468	0.0021
bio-answerfinder-2	0.1575	0.1247	0.1232	0.1236	0.0011
Fleming-4	0.0893	0.0180	0.0285	0.0212	0.0000
dmiip1	0.2256	0.2668	0.2034	0.2001	0.0151
dmiip2	0.3026	0.3772	0.2803	0.2791	0.0572
dmiip3	0.1974	0.2116	0.1741	0.1708	0.0080
dmiip4	0.3000	0.3714	0.2760	0.2788	0.0512
dmiip5	0.2667	0.3230	0.2466	0.2525	0.0578

Snippets

System	Mean precision	Recall	F-Measure	MAP	GMAP
bio-answerfinder	0.0994	0.0823	0.0794	0.0911	0.0008
bio-answerfinder-2	0.0937	0.0882	0.0735	0.1177	0.0003
Fleming-4	-	-	-	-	-
dmiip1	0.1664	0.1698	0.1365	0.1990	0.0031
dmiip2	0.1800	0.2273	0.1588	0.2290	0.0071
dmiip3	0.1312	0.1394	0.1071	0.1692	0.0015
dmiip4	0.1775	0.2281	0.1539	0.2296	0.0052
dmiip5	0.2019	0.2029	0.1677	0.3015	0.0051

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
bio-answerfinder	0.6667	0.8000	-	0.4000	0.2500	0.2500	0.2500	0.1250	0.0617	0.0787
bio-answerfinder-2	0.6667	0.8000	-	0.4000	0.2500	0.3750	0.2917	0.1972	0.0894	0.1100
Fleming-4	0.7778	0.8333	0.6667	0.7500	0.2500	0.2500	0.2500	0.5000	0.1423	0.1852
dmiip1	0.5556	0.5000	0.6000	0.5500	0.8750	1.0000	0.9167	0.3687	0.3628	0.2917
dmiip2	0.5556	0.5000	0.6000	0.5500	0.8750	1.0000	0.9000	0.3988	0.3947	0.3247
dmiip3	0.6667	0.7273	0.5714	0.6494	0.6250	1.0000	0.7813	0.4206	0.3703	0.3032
dmiip4	0.5556	0.5000	0.6000	0.5500	0.6250	1.0000	0.7917	0.4076	0.3248	0.2879
dmiip5	0.6667	0.7273	0.5714	0.6494	0.2500	0.3750	0.3125	0.5083	0.2556	0.3288

Ideal Answers

	Automatic scores (Rouge - R)				Manual scores
System	R-2 (Rec)	R-2 (F1)	R-SU4 (Rec)	R-SU4 (F1)	Readability	Recall	Precision	Repetition
bio-answerfinder	0.6202	0.4575	0.6267	0.4623	3.59	2.95	2.78	3.59
bio-answerfinder-2	0.5969	0.4370	0.6019	0.4400	3.32	2.78	2.62	3.51
Fleming-4	-	-	-	-	0.19	0.27	0.19	0.19
dmiip1	0.4969	0.4258	0.5098	0.4328	0.54	0.54	0.57	0.54
dmiip2	0.4969	0.4258	0.5098	0.4328	0.54	0.54	0.57	0.54
dmiip3	0.4751	0.4153	0.4855	0.4216	0.65	0.68	0.65	0.65
dmiip4	0.5077	0.4320	0.5192	0.4375	0.54	0.54	0.57	0.54
dmiip5	0.3714	0.4103	0.3710	0.4104	4.43	4.49	4.38	4.51

Copyright © 2023, the BioASQ Challenge (http://bioasq.org)

Platinum sponsors of the BioASQ Challenge between 2012 and 2023

Google was a proud sponsor of the BioASQ Challenge from 2018 to 2022.
BioASQ was supported by the National Library of Medicine of the National Institutes of Health under award number R13LM012214. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
The research leading to these results has received funding from the European Commission's Seventh Framework Programme (FP7/2007-2013, ICT-2011.4.4(d), Intelligent Information Management, Targeted Competition Framework) under grant agreement n° 318652.