BioASQ Participants Area

Task 6b: Test Results of Phase B

The test results are presented in separate tables for each type of annotation. The "System Description" of each system is used.
The evaluation measures that are used in Task B are presented here .
Warning: For ideal answers, good ROUGE results do not always imply good manual scores.

Test batch 1

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
auth-qa-2	0.6875	0.8148	-	0.4074	0.1935	0.3548	0.2484	0.1545	0.5644	0.2320
auth-qa-1	0.6875	0.8148	-	0.4074	0.1935	0.3226	0.2376	0.1727	0.6023	0.2563
auth-qa-3	0.6875	0.8148	-	0.4074	0.1935	0.3226	0.2376	0.1636	0.5871	0.2450
Oaqa5b-tfidf	-	-	-	-	-	-	-	-	-	-
Oaqa5b	-	-	-	-	-	-	-	-	-	-
Oaqa 5b	-	-	-	-	-	-	-	-	-	-
Lab Zhu,Fudan Univer	0.6875	0.8148	-	0.4074	0.1935	0.2258	0.2097	0.0904	0.4091	0.1413
LabZhu,FDU	0.6875	0.8148	-	0.4074	0.1935	0.2258	0.2097	0.0934	0.4091	0.1459
MQ-1	0.6875	0.8148	-	0.4074	-	-	-	-	-	-
MQ-2	0.6875	0.8148	-	0.4074	-	-	-	-	-	-
MQ-3	0.6875	0.8148	-	0.4074	-	-	-	-	-	-
MQ-4	0.6875	0.8148	-	0.4074	-	-	-	-	-	-
MQ-5	0.6875	0.8148	-	0.4074	-	-	-	-	-	-
Oaqa-5b	-	-	-	-	-	-	-	-	-	-
OAQA based system	0.5938	0.6286	0.5517	0.5901	0.1613	0.3226	0.2366	0.3231	0.6023	0.4105
YODAQA based system	0.6875	0.8148	-	0.4074	0.0000	0.0968	0.0484	0.0909	0.0909	0.0909
Lab Zhu ,Fdan Univer	0.6875	0.8148	-	0.4074	0.2258	0.2581	0.2419	0.1015	0.4091	0.1580
LabZhu_FDU	0.6875	0.8148	-	0.4074	0.0645	0.1290	0.0968	0.0707	0.4091	0.1165
LabZhu-FDU	0.6875	0.8148	-	0.4074	0.0645	0.1290	0.0968	0.0707	0.4091	0.1165
SpanBaseline	0.6875	0.8148	-	0.4074	0.0968	0.0968	0.0968	0.1818	0.0455	0.0714
oaqa5b5	-	-	-	-	-	-	-	-	-	-
BioASQ_Baseline	0.3125	0.1538	0.4211	0.2874	0.2258	0.2903	0.2403	0.2463	0.4318	0.2807

Ideal Answers

	Automatic scores		Manual scores
System	Rouge-2	Rouge-SU4	Readability	Recall	Precision	Repetition
auth-qa-2	-	-	-	-	-	-
auth-qa-1	-	-	-	-	-	-
auth-qa-3	-	-	-	-	-	-
Oaqa5b-tfidf	0.6621	0.6651	3.30	4.45	3.22	3.25
Oaqa5b	0.5300	0.5301	3.45	4.41	3.49	3.69
Oaqa 5b	0.6002	0.5932	3.02	4.24	3.07	3.05
Lab Zhu,Fudan Univer	0.2742	0.2776	4.31	3.70	3.96	4.82
LabZhu,FDU	0.2742	0.2776	4.31	3.70	3.96	4.82
MQ-1	0.4559	0.4625	3.77	4.24	3.66	3.96
MQ-2	0.5229	0.5236	3.73	4.46	3.79	3.86
MQ-3	0.4867	0.4955	3.74	4.33	3.63	3.78
MQ-4	0.5392	0.5462	3.69	4.35	3.57	3.79
MQ-5	0.3737	0.3835	3.79	4.07	3.71	4.14
Oaqa-5b	0.6853	0.6860	3.29	4.37	3.26	3.11
OAQA based system	0.0715	0.0701	1.23	1.06	1.12	1.28
YODAQA based system	0.0118	0.0087	2.87	1.96	2.31	4.09
Lab Zhu ,Fdan Univer	0.2742	0.2776	4.31	3.70	3.96	4.82
LabZhu_FDU	0.2742	0.2776	4.31	3.70	3.96	4.82
LabZhu-FDU	0.2742	0.2776	4.31	3.70	3.96	4.82
SpanBaseline	0.0960	0.0903	2.56	1.99	2.09	3.83
oaqa5b5	0.6456	0.6444	3.08	4.39	3.19	2.87
BioASQ_Baseline	-	-	-	-	-	-

Test batch 2

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
auth-qa-1	0.7692	0.8696	-	0.4348	0.2381	0.5238	0.3468	0.1222	0.3271	0.1654
UNCC System 1	-	-	-	-	-	-	-	-	-	-
auth-qa-2	0.7692	0.8696	-	0.4348	0.2381	0.6190	0.3746	0.1753	0.2993	0.1947
auth-qa-4	0.7692	0.8696	-	0.4348	0.2381	0.5714	0.3429	0.1966	0.3289	0.2252
auth-qa-5	0.7692	0.8696	-	0.4348	0.1905	0.5238	0.3032	0.1644	0.2623	0.1781
auth-qa-3	0.7692	0.8696	-	0.4348	0.2381	0.5238	0.3270	0.1389	0.3474	0.1834
limsi-reader-UMLS-r1	0.7692	0.8696	-	0.4348	0.2381	0.2857	0.2619	0.1111	0.1667	0.1270
MQ-1	0.7692	0.8696	-	0.4348	-	-	-	-	-	-
MQ-2	0.7692	0.8696	-	0.4348	-	-	-	-	-	-
MQ-3	0.7692	0.8696	-	0.4348	-	-	-	-	-	-
MQ-5	0.7692	0.8696	-	0.4348	-	-	-	-	-	-
MQ-4	0.7692	0.8696	-	0.4348	-	-	-	-	-	-
Oaqa5b-tfidf	-	-	-	-	-	-	-	-	-	-
Oaqa5b	-	-	-	-	-	-	-	-	-	-
Oaqa 5b	-	-	-	-	-	-	-	-	-	-
Oaqa-5b	-	-	-	-	-	-	-	-	-	-
Lab Zhu ,Fdan Univer	0.7692	0.8696	-	0.4348	0.1905	0.3810	0.2619	0.0637	0.2789	0.0967
LabZhu-FDU	0.7692	0.8696	-	0.4348	-	-	-	-	-	-
Lab Zhu,Fudan Univer	0.7692	0.8696	-	0.4348	0.1905	0.3810	0.2619	0.1107	0.2743	0.1498
OAQA based system	0.6923	0.7778	0.5000	0.6389	0.2857	0.2857	0.2857	0.2086	0.2843	0.2109
LabZhu,FDU	0.7692	0.8696	-	0.4348	0.3333	0.5714	0.4325	0.2044	0.2761	0.2279
YODAQA based system	0.7692	0.8696	-	0.4348	0.0952	0.1905	0.1429	0.1944	0.0917	0.1168
BioASQ_Baseline	0.3846	0.3333	0.4286	0.3810	0.1429	0.2857	0.1841	0.1810	0.2689	0.2107

Ideal Answers

	Automatic scores		Manual scores
System	Rouge-2	Rouge-SU4	Readability	Recall	Precision	Repetition
auth-qa-1	-	-	-	-	-	-
UNCC System 1	0.6090	0.6176	3.33	4.71	3.41	3.39
auth-qa-2	-	-	-	-	-	-
auth-qa-4	-	-	-	-	-	-
auth-qa-5	-	-	-	-	-	-
auth-qa-3	-	-	-	-	-	-
limsi-reader-UMLS-r1	-	-	-	-	-	-
MQ-1	0.4955	0.5033	3.73	4.59	3.74	3.97
MQ-2	0.4882	0.4971	3.65	4.57	3.74	3.93
MQ-3	0.4957	0.5107	3.68	4.57	3.70	3.91
MQ-5	0.4206	0.4310	3.77	4.43	3.78	4.06
MQ-4	0.5289	0.5385	3.69	4.50	3.64	3.91
Oaqa5b-tfidf	0.4842	0.4943	3.36	4.42	3.45	3.69
Oaqa5b	0.5327	0.5311	2.89	4.28	3.10	3.15
Oaqa 5b	0.5704	0.5792	3.21	4.49	3.32	3.41
Oaqa-5b	0.5526	0.5581	3.12	4.45	3.22	3.25
Lab Zhu ,Fdan Univer	0.3103	0.3202	4.15	3.95	4.20	4.72
LabZhu-FDU	0.3103	0.3202	4.15	3.95	4.20	4.72
Lab Zhu,Fudan Univer	0.3103	0.3202	4.15	3.95	4.20	4.72
OAQA based system	0.1257	0.1272	1.44	1.41	1.55	1.66
LabZhu,FDU	0.3103	0.3202	4.15	3.95	4.20	4.72
YODAQA based system	0.0273	0.0245	2.61	1.83	2.08	3.62
BioASQ_Baseline	-	-	-	-	-	-

Test batch 3

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
auth-qa-1	0.6800	0.8095	-	0.4048	0.1563	0.4063	0.2604	0.1462	0.4013	0.2068
auth-qa-2	0.6800	0.8095	-	0.4048	0.1563	0.3125	0.2135	0.2120	0.3244	0.2333
auth-qa-3	0.6800	0.8095	-	0.4048	0.2500	0.4375	0.3083	0.1538	0.4577	0.2238
auth-qa-4	0.6800	0.8095	-	0.4048	0.2188	0.3750	0.2656	0.1982	0.3628	0.2311
UNCC System 1	-	-	-	-	-	-	-	-	-	-
UNCC System 2	-	-	-	-	-	-	-	-	-	-
MQ-1	0.6800	0.8095	-	0.4048	-	-	-	-	-	-
MQ-2	0.6800	0.8095	-	0.4048	-	-	-	-	-	-
MQ-3	0.6800	0.8095	-	0.4048	-	-	-	-	-	-
MQ-4	0.6800	0.8095	-	0.4048	-	-	-	-	-	-
MQ-5	0.6800	0.8095	-	0.4048	-	-	-	-	-	-
YODAQA based system	0.6800	0.8095	-	0.4048	0.0625	0.1563	0.1094	0.1154	0.0833	0.0949
Oaqa5b-tfidf	-	-	-	-	-	-	-	-	-	-
Oaqa5b	-	-	-	-	-	-	-	-	-	-
OAQA based system	0.6400	0.7097	0.5263	0.6180	0.1250	0.3438	0.2094	0.1973	0.3538	0.2432
Oaqa-5b	-	-	-	-	-	-	-	-	-	-
oaqa5b5	-	-	-	-	-	-	-	-	-	-
Lab Zhu ,Fdan Univer	0.6800	0.8095	-	0.4048	0.0938	0.2500	0.1589	0.1094	0.3654	0.1502
Lab Zhu,Fudan Univer	0.6800	0.8095	-	0.4048	0.1875	0.3125	0.2370	0.2675	0.4103	0.3051
LabZhu,FDU	0.6800	0.8095	-	0.4048	0.1875	0.3125	0.2370	0.2803	0.4103	0.3216
BioASQ_Baseline	0.5200	0.5385	0.5000	0.5192	0.2188	0.2813	0.2396	0.1406	0.3923	0.1859

Ideal Answers

	Automatic scores		Manual scores
System	Rouge-2	Rouge-SU4	Readability	Recall	Precision	Repetition
auth-qa-1	-	-	-	-	-	-
auth-qa-2	-	-	-	-	-	-
auth-qa-3	-	-	-	-	-	-
auth-qa-4	-	-	-	-	-	-
UNCC System 1	0.6445	0.6522	3.28	4.67	3.84	3.57
UNCC System 2	0.2324	0.2300	4.10	3.78	4.14	4.88
MQ-1	0.4467	0.4468	3.71	4.33	4.06	4.13
MQ-2	0.4461	0.4508	3.49	4.36	4.03	4.02
MQ-3	0.4856	0.4915	3.74	4.46	4.04	4.02
MQ-4	0.5041	0.5112	3.81	4.29	4.02	4.10
MQ-5	0.3782	0.3820	3.72	4.20	4.06	4.18
YODAQA based system	0.0143	0.0113	2.84	1.66	1.95	3.57
Oaqa5b-tfidf	0.6016	0.6129	3.40	4.61	3.79	3.55
Oaqa5b	0.4746	0.4872	3.35	4.62	4.00	3.89
OAQA based system	0.1431	0.1430	1.21	1.30	1.33	1.44
Oaqa-5b	0.5739	0.5741	3.22	4.57	3.75	3.68
oaqa5b5	0.5454	0.5479	3.10	4.53	3.74	3.74
Lab Zhu ,Fdan Univer	0.2907	0.2908	3.90	3.97	4.38	4.81
Lab Zhu,Fudan Univer	0.2941	0.2948	3.87	3.97	4.38	4.81
LabZhu,FDU	0.2941	0.2948	3.87	3.97	4.38	4.81
BioASQ_Baseline	-	-	-	-	-	-

Test batch 4

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
auth-qa-2	0.6296	0.7727	-	0.3864	0.2121	0.3030	0.2475	0.2511	0.3822	0.2925
auth-qa-4	0.6296	0.7727	-	0.3864	0.2121	0.3030	0.2434	0.2800	0.3822	0.3094
auth-qa-1	0.6296	0.7727	-	0.3864	0.2121	0.2727	0.2374	0.1600	0.4267	0.2277
auth-qa-3	0.6296	0.7727	-	0.3864	0.2121	0.2727	0.2283	0.1800	0.4711	0.2551
Lab Zhu ,Fdan Univer	0.6296	0.7727	-	0.3864	0.0909	0.1212	0.1061	0.1657	0.2833	0.1663
Oaqa5b-tfidf	-	-	-	-	-	-	-	-	-	-
Oaqa5b	-	-	-	-	-	-	-	-	-	-
Oaqa 5b	-	-	-	-	-	-	-	-	-	-
Oaqa-5b	0.6667	0.7097	0.6087	0.6592	0.0606	0.2121	0.1313	0.0867	0.2722	0.1299
MQ-1	0.6296	0.7727	-	0.3864	-	-	-	-	-	-
MQ-2	0.6296	0.7727	-	0.3864	-	-	-	-	-	-
MQ-3	0.6296	0.7727	-	0.3864	-	-	-	-	-	-
MQ-4	0.6296	0.7727	-	0.3864	-	-	-	-	-	-
MQ-5	0.6296	0.7727	-	0.3864	-	-	-	-	-	-
auth-qa-5	0.6296	0.7368	0.3750	0.5559	0.2121	0.3030	0.2434	0.2800	0.3822	0.3094
Lab Zhu,Fudan Univer	0.6296	0.7727	-	0.3864	0.2121	0.2424	0.2273	0.2944	0.3411	0.2902
LabZhu,FDU	0.6296	0.7727	-	0.3864	0.2424	0.2424	0.2424	0.4130	0.3356	0.3280
BioASQ_Baseline	0.4815	0.4167	0.5333	0.4750	0.0606	0.1212	0.0859	0.1774	0.3811	0.2227

Ideal Answers

	Automatic scores		Manual scores
System	Rouge-2	Rouge-SU4	Readability	Recall	Precision	Repetition
auth-qa-2	-	-	-	-	-	-
auth-qa-4	-	-	-	-	-	-
auth-qa-1	-	-	-	-	-	-
auth-qa-3	-	-	-	-	-	-
Lab Zhu ,Fdan Univer	0.3004	0.3095	4.14	4.14	4.20	4.86
Oaqa5b-tfidf	0.5390	0.5425	3.56	4.61	3.76	3.96
Oaqa5b	0.5477	0.5513	3.49	4.60	3.74	3.95
Oaqa 5b	0.4311	0.4421	3.47	4.25	3.72	4.17
Oaqa-5b	0.6292	0.6398	3.56	4.53	3.61	3.67
MQ-1	0.4967	0.5101	3.94	4.52	3.97	4.23
MQ-2	0.5291	0.5307	3.83	4.62	3.98	4.18
MQ-3	0.5103	0.5160	3.84	4.65	4.01	4.18
MQ-4	0.5578	0.5653	3.95	4.69	3.94	4.18
MQ-5	0.4473	0.4624	3.89	4.57	3.96	4.32
auth-qa-5	-	-	-	-	-	-
Lab Zhu,Fudan Univer	0.3004	0.3095	4.15	4.14	4.19	4.87
LabZhu,FDU	0.3004	0.3095	4.15	4.14	4.19	4.87
BioASQ_Baseline	-	-	-	-	-	-

Test batch 5

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
Lab Zhu ,Fdan Univer	0.7000	0.8235	-	0.4118	0.0909	0.1818	0.1250	0.0960	0.1808	0.1141
auth-qa-2	0.7000	0.8235	-	0.4118	0.1136	0.2727	0.1758	0.2143	0.2580	0.2187
auth-qa-4	0.7000	0.8235	-	0.4118	0.1136	0.2727	0.1758	0.1952	0.2419	0.1949
auth-qa-1	0.7000	0.8235	-	0.4118	0.1136	0.2955	0.1758	0.1500	0.3858	0.2070
auth-qa-3	0.7000	0.8235	-	0.4118	0.1136	0.2955	0.1777	0.1429	0.3654	0.1980
UNCC System 1	-	-	-	-	-	-	-	-	-	-
UNCC System 2	-	-	-	-	-	-	-	-	-	-
Oaqa5b-tfidf	-	-	-	-	-	-	-	-	-	-
Oaqa5b	-	-	-	-	-	-	-	-	-	-
Oaqa 5b	-	-	-	-	-	-	-	-	-	-
MQ-1	0.7000	0.8235	-	0.4118	-	-	-	-	-	-
MQ-2	0.7000	0.8235	-	0.4118	-	-	-	-	-	-
MQ-3	0.7000	0.8235	-	0.4118	-	-	-	-	-	-
MQ-4	0.7000	0.8235	-	0.4118	-	-	-	-	-	-
MQ-5	0.7000	0.8235	-	0.4118	-	-	-	-	-	-
Lab Zhu,Fudan Univer	0.7000	0.8235	-	0.4118	0.1136	0.2045	0.1477	0.0960	0.1808	0.1141
Olelo system at HPI	0.7000	0.8235	-	0.4118	0.0000	0.0227	0.0114	0.0204	0.0204	0.0204
LabZhu,FDU	0.7000	0.8235	-	0.4118	0.1136	0.2045	0.1477	0.0960	0.1808	0.1141
LabZhu_FDU	-	-	-	-	0.0455	0.0682	0.0511	0.0560	0.0316	0.0397
auth-qa-5	0.4500	0.6207	-	0.3103	0.1136	0.2727	0.1758	0.1952	0.2419	0.1949
Oaqa-5b	-	-	-	-	-	-	-	-	-	-
oaqa5b5	0.6000	0.7143	0.3333	0.5238	0.1818	0.2273	0.1951	0.0929	0.1757	0.1135
LabZhu-FDU	0.7000	0.8235	-	0.4118	0.2045	0.2727	0.2273	0.1736	0.2344	0.1832
BioASQ_Baseline	0.6000	0.6364	0.5556	0.5960	0.0000	0.1818	0.0795	0.2862	0.2999	0.2544

Ideal Answers

	Automatic scores		Manual scores
System	Rouge-2	Rouge-SU4	Readability	Recall	Precision	Repetition
Lab Zhu ,Fdan Univer	0.4334	0.4273	4.48	4.08	4.07	4.82
auth-qa-2	-	-	-	-	-	-
auth-qa-4	-	-	-	-	-	-
auth-qa-1	-	-	-	-	-	-
auth-qa-3	-	-	-	-	-	-
UNCC System 1	0.7607	0.7483	4.50	4.19	4.18	4.75
UNCC System 2	0.4329	0.4261	3.60	4.71	3.29	3.41
Oaqa5b-tfidf	0.6427	0.6386	3.78	4.66	3.54	3.67
Oaqa5b	0.6782	0.6657	3.69	4.58	3.30	3.54
Oaqa 5b	0.6762	0.6663	3.67	4.64	3.43	3.60
MQ-1	0.5936	0.5890	4.00	4.49	3.69	4.02
MQ-2	0.5942	0.5869	3.96	4.54	3.76	3.98
MQ-3	0.6207	0.6167	3.93	4.49	3.59	3.92
MQ-4	0.6439	0.6414	3.97	4.58	3.67	3.87
MQ-5	0.5220	0.5196	4.10	4.36	3.89	4.23
Lab Zhu,Fudan Univer	0.4334	0.4273	4.48	4.08	4.07	4.82
Olelo system at HPI	0.1880	0.2352	3.11	3.18	2.14	3.17
LabZhu,FDU	0.4334	0.4273	4.48	4.08	4.07	4.82
LabZhu_FDU	0.4334	0.4273	4.48	4.08	4.07	4.82
auth-qa-5	-	-	-	-	-	-
Oaqa-5b	0.4854	0.4901	3.57	4.04	3.40	3.69
oaqa5b5	0.2588	0.2538	3.80	3.57	3.71	4.51
LabZhu-FDU	0.4334	0.4273	4.48	4.08	4.07	4.82
BioASQ_Baseline	-	-	-	-	-	-