BioASQ Participants Area

Task 11b: Test Results of Phase B

The test results are presented in separate tables for each type of annotation. The "System Description" of each system is used.
The evaluation measures that are used in Task B are presented here .
Warning: For ideal answers, good ROUGE results do not always imply good manual scores.

Test batch 1

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
OWLMan-phaseB-TaskV1	0.7083	0.8000	0.4615	0.6308	0.3158	0.5263	0.4123	0.2167	0.3973	0.2618
IISR-1	0.9167	0.9375	0.8750	0.9063	0.4211	0.4211	0.4211	0.7185	0.6666	0.6841
IISR-3	0.7083	0.8293	-	0.4146	-	-	-	-	-	-
ELECTROLBERT-3	0.8333	0.8947	0.6000	0.7474	-	-	-	-	-	-
AsqAway_1	0.7083	0.8293	-	0.4146	0.3158	0.3158	0.3158	0.4754	0.4905	0.4595
AsqAway_2	0.7083	0.8293	-	0.4146	0.3158	0.4737	0.3947	0.4368	0.6140	0.4804
AsqAway_3	0.7083	0.8293	-	0.4146	0.3158	0.4737	0.3947	0.4388	0.6953	0.5106
AsqAway_4	0.7083	0.8293	-	0.4146	0.3684	0.4737	0.4211	0.4388	0.6953	0.5106
MQ-1	0.7083	0.8293	-	0.4146	-	-	-	-	-	-
MQ-2	0.7083	0.8293	-	0.4146	-	-	-	-	-	-
MQ-3	0.7083	0.8293	-	0.4146	-	-	-	-	-	-
Deep ML methods for	0.7083	0.8000	0.4615	0.6308	0.1579	0.2105	0.1754	0.4444	0.2695	0.2903
MQ-4	0.7083	0.8293	-	0.4146	-	-	-	-	-	-
MQ-5	0.7083	0.8293	-	0.4146	-	-	-	-	-	-
simple truncation	0.9167	0.9375	0.8750	0.9063	0.2632	0.2632	0.2632	0.2389	0.2369	0.2374
dmiip1	0.8333	0.8824	0.7143	0.7983	0.3684	0.5263	0.4474	0.4183	0.4417	0.4099
dmiip2	0.8333	0.8824	0.7143	0.7983	0.5263	0.6316	0.5570	0.4698	0.4271	0.4199
dmiip4	0.7917	0.8485	0.6667	0.7576	0.3684	0.6316	0.4737	0.4715	0.5307	0.4686
dmiip5	0.9167	0.9412	0.8571	0.8992	0.3684	0.5789	0.4474	0.4815	0.5092	0.4609
dmiip3	0.7917	0.8485	0.6667	0.7576	0.3684	0.5789	0.4474	0.4590	0.5326	0.4696
IISR-2	0.9583	0.9697	0.9333	0.9515	0.5263	0.5263	0.5263	0.6417	0.5697	0.5972
UR-gpt4-zero-ret	0.9167	0.9412	0.8571	0.8992	0.5789	0.5789	0.5789	0.6472	0.6530	0.6495
UR-gpt3.5-turbo-zero	0.8750	0.9091	0.8000	0.8545	0.5263	0.6316	0.5789	0.6742	0.7249	0.6917
UR-gpt4-simple	0.9167	0.9412	0.8571	0.8992	0.2105	0.2632	0.2368	0.4000	0.4014	0.3939
extractive	0.9167	0.9375	0.8750	0.9063	0.2632	0.2632	0.2632	0.1667	0.1667	0.1667
abstractive	0.9167	0.9375	0.8750	0.9063	0.2105	0.2105	0.2105	0.1875	0.1833	0.1852
bioinfo-0	0.7083	0.8293	-	0.4146	-	-	-	-	-	-
BART	0.2500	0.3077	0.1818	0.2448	-	-	-	-	-	-
DMIS-KU-2	0.9167	0.9412	0.8571	0.8992	0.4737	0.6316	0.5526	0.7365	0.6910	0.6965
DMIS-KU-3	0.9167	0.9412	0.8571	0.8992	0.4211	0.6316	0.5088	0.7861	0.6668	0.7027
DMIS-KU-5	0.9583	0.9697	0.9333	0.9515	0.4737	0.6316	0.5526	0.7861	0.6668	0.7027
DMIS-KU-1	0.9167	0.9412	0.8571	0.8992	0.4737	0.6316	0.5526	0.7778	0.6592	0.6937
DMIS-KU-4	0.8750	0.9091	0.8000	0.8545	0.4737	0.6316	0.5439	0.7986	0.6592	0.6992

Ideal Answers

	Automatic scores (Rouge - R)				Manual scores
System	R-2 (Rec)	R-2 (F1)	R-SU4 (Rec)	R-SU4 (F1)	Readability	Recall	Precision	Repetition
OWLMan-phaseB-TaskV1	-	-	-	-	-	-	-	-
IISR-1	0.3780	0.3143	0.3609	0.2902	-	-	-	-
IISR-3	0.4477	0.4063	0.4389	0.3960	-	-	-	-
ELECTROLBERT-3	-	-	-	-	-	-	-	-
AsqAway_1	-	-	-	-	-	-	-	-
AsqAway_2	-	-	-	-	-	-	-	-
AsqAway_3	-	-	-	-	-	-	-	-
AsqAway_4	-	-	-	-	-	-	-	-
MQ-1	0.2370	0.1458	0.2414	0.1389	-	-	-	-
MQ-2	0.2608	0.2112	0.2651	0.2048	-	-	-	-
MQ-3	0.5268	0.3555	0.5118	0.3334	-	-	-	-
Deep ML methods for	0.0982	0.1452	0.0739	0.1103	-	-	-	-
MQ-4	0.6046	0.3367	0.5891	0.3168	-	-	-	-
MQ-5	0.6167	0.3436	0.5997	0.3222	-	-	-	-
simple truncation	0.1515	0.0665	0.1474	0.0623	-	-	-	-
dmiip1	0.5780	0.3103	0.5674	0.2927	-	-	-	-
dmiip2	0.5711	0.3206	0.5629	0.3059	-	-	-	-
dmiip4	0.5905	0.3281	0.5809	0.3117	-	-	-	-
dmiip5	0.6052	0.3479	0.5931	0.3308	-	-	-	-
dmiip3	0.5711	0.3206	0.5629	0.3059	-	-	-	-
IISR-2	0.4149	0.3289	0.4026	0.3093	-	-	-	-
UR-gpt4-zero-ret	0.5610	0.1855	0.5547	0.1700	-	-	-	-
UR-gpt3.5-turbo-zero	0.5365	0.1500	0.5273	0.1377	-	-	-	-
UR-gpt4-simple	0.3346	0.0729	0.3666	0.0744	-	-	-	-
extractive	0.1662	0.0725	0.1625	0.0684	-	-	-	-
abstractive	0.1562	0.0691	0.1533	0.0656	-	-	-	-
bioinfo-0	0.3256	0.2907	0.3182	0.2741	-	-	-	-
BART	0.1920	0.2125	0.1813	0.2009	-	-	-	-
DMIS-KU-2	-	-	-	-	-	-	-	-
DMIS-KU-3	-	-	-	-	-	-	-	-
DMIS-KU-5	-	-	-	-	-	-	-	-
DMIS-KU-1	-	-	-	-	-	-	-	-
DMIS-KU-4	-	-	-	-	-	-	-	-

Test batch 2

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
IISR-2	1.0000	1.0000	1.0000	1.0000	0.5455	0.6364	0.5909	0.5099	0.3577	0.3980
IISR-3	0.5833	0.7368	-	0.3684	0.5455	0.6364	0.5909	0.5099	0.3577	0.3980
IISR-1	1.0000	1.0000	1.0000	1.0000	0.5000	0.5455	0.5227	0.4861	0.3310	0.3678
OWLMan-phaseB-TaskV1	0.4583	0.5806	0.2353	0.4080	0.4545	0.5000	0.4697	0.1667	0.1395	0.1374
UR-gpt3.5-turbo-zero	0.9167	0.9333	0.8889	0.9111	0.5455	0.5909	0.5682	0.4598	0.4671	0.4316
UR-gpt4-zero-ret	0.9583	0.9655	0.9474	0.9564	0.5455	0.5909	0.5682	0.3742	0.4369	0.3828
ELECTROLBERT-3	0.7083	0.8000	0.4615	0.6308	-	-	-	-	-	-
capstone-1	0.9167	0.9231	0.9091	0.9161	0.4091	0.5455	0.4561	0.2085	0.3810	0.2617
capstone-3	0.8333	0.8571	0.8000	0.8286	0.2273	0.3636	0.2727	0.2485	0.3810	0.2923
capstone-2	0.8333	0.8571	0.8000	0.8286	0.2273	0.3636	0.2727	0.3250	0.3750	0.3173
bioinfo-0	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
bioinfo-1	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
bioinfo-2	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
bioinfo-3	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
bioinfo-4	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
AsqAway_1	0.8750	0.8966	0.8421	0.8693	0.4545	0.4545	0.4545	0.1780	0.1968	0.1756
AsqAway_2	0.8750	0.8966	0.8421	0.8693	0.4545	0.4545	0.4545	0.2023	0.3226	0.2327
AsqAway_3	0.8750	0.8966	0.8421	0.8693	0.4545	0.5000	0.4773	0.1951	0.3504	0.2329
AsqAway_4	0.8750	0.8966	0.8421	0.8693	0.4545	0.5000	0.4773	0.1951	0.3504	0.2329
Deep ML methods for	0.8333	0.8667	0.7778	0.8222	0.2727	0.2727	0.2727	0.3472	0.2008	0.2125
MindLab QA Reloaded	0.7917	0.8276	0.7368	0.7822	0.2727	0.2727	0.2727	0.3264	0.1341	0.1755
MindLab QA System	0.8333	0.8667	0.7778	0.8222	0.2727	0.4091	0.3000	0.1333	0.1841	0.1402
MindLab QA System ++	0.5417	0.6857	0.1538	0.4198	0.1818	0.2273	0.2045	0.3889	0.1063	0.1597
MindLab Red Lions++	0.7917	0.8276	0.7368	0.7822	0.2727	0.2727	0.2727	0.3264	0.1341	0.1755
MQ-1	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
MQ-2	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
MQ-3	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
MQ-4	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
MQ-5	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
DMIS-KU-1	0.9583	0.9630	0.9524	0.9577	0.3182	0.6818	0.4773	0.3349	0.3623	0.3080
DMIS-KU-2	0.9583	0.9630	0.9524	0.9577	0.3182	0.6818	0.4561	0.3486	0.3456	0.3087
DMIS-KU-3	0.9583	0.9630	0.9524	0.9577	0.3636	0.5909	0.4621	0.2818	0.4058	0.3178
DMIS-KU-4	1.0000	1.0000	1.0000	1.0000	0.3636	0.5909	0.4697	0.2983	0.3683	0.2871
DMIS-KU-5	0.9167	0.9286	0.9000	0.9143	0.3636	0.5909	0.4621	0.2534	0.4593	0.3022
dmiip2	0.5833	0.7368	-	0.3684	0.4091	0.5909	0.4811	0.3488	0.3256	0.3022
dmiip3	0.8750	0.8889	0.8571	0.8730	0.3182	0.5455	0.3992	0.2851	0.2464	0.2232
dmiip5	0.9583	0.9655	0.9474	0.9564	0.4091	0.4545	0.4242	0.1413	0.2200	0.1676
dmiip4	0.7917	0.8276	0.7368	0.7822	0.4545	0.5909	0.5227	0.2573	0.3200	0.2531
dmiip1	0.9167	0.9286	0.9000	0.9143	0.3182	0.5909	0.4318	0.2271	0.3760	0.2501
simple truncation	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
extractive	0.5833	0.7368	-	0.3684	-	-	-	-	-	-
abstractive	0.5833	0.7368	-	0.3684	-	-	-	-	-	-

Ideal Answers

	Automatic scores (Rouge - R)				Manual scores
System	R-2 (Rec)	R-2 (F1)	R-SU4 (Rec)	R-SU4 (F1)	Readability	Recall	Precision	Repetition
IISR-2	0.3552	0.3007	0.3520	0.2904	-	-	-	-
IISR-3	0.3390	0.3056	0.3359	0.2954	-	-	-	-
IISR-1	0.3394	0.2870	0.3403	0.2821	-	-	-	-
OWLMan-phaseB-TaskV1	-	-	-	-	-	-	-	-
UR-gpt3.5-turbo-zero	0.4983	0.1720	0.5163	0.1638	-	-	-	-
UR-gpt4-zero-ret	0.4802	0.1785	0.4911	0.1684	-	-	-	-
ELECTROLBERT-3	-	-	-	-	-	-	-	-
capstone-1	0.4953	0.3278	0.4919	0.3185	-	-	-	-
capstone-3	0.4953	0.3278	0.4919	0.3185	-	-	-	-
capstone-2	0.4953	0.3278	0.4919	0.3185	-	-	-	-
bioinfo-0	0.2192	0.1784	0.2206	0.1733	-	-	-	-
bioinfo-1	0.2608	0.2257	0.2627	0.2205	-	-	-	-
bioinfo-2	0.2912	0.2853	0.2937	0.2825	-	-	-	-
bioinfo-3	0.3323	0.3242	0.3300	0.3172	-	-	-	-
bioinfo-4	0.3263	0.3169	0.3253	0.3111	-	-	-	-
AsqAway_1	-	-	-	-	-	-	-	-
AsqAway_2	-	-	-	-	-	-	-	-
AsqAway_3	-	-	-	-	-	-	-	-
AsqAway_4	-	-	-	-	-	-	-	-
Deep ML methods for	0.0553	0.0822	0.0486	0.0706	-	-	-	-
MindLab QA Reloaded	0.0583	0.0851	0.0514	0.0736	-	-	-	-
MindLab QA System	0.0218	0.0308	0.0284	0.0367	-	-	-	-
MindLab QA System ++	0.0785	0.1169	0.0607	0.0940	-	-	-	-
MindLab Red Lions++	0.0583	0.0851	0.0514	0.0736	-	-	-	-
MQ-1	0.1950	0.1404	0.2224	0.1530	-	-	-	-
MQ-2	0.2436	0.1971	0.2463	0.1961	-	-	-	-
MQ-3	0.4678	0.3290	0.4690	0.3203	-	-	-	-
MQ-4	0.4953	0.3278	0.4919	0.3185	-	-	-	-
MQ-5	0.4953	0.3259	0.4924	0.3169	-	-	-	-
DMIS-KU-1	-	-	-	-	-	-	-	-
DMIS-KU-2	-	-	-	-	-	-	-	-
DMIS-KU-3	-	-	-	-	-	-	-	-
DMIS-KU-4	-	-	-	-	-	-	-	-
DMIS-KU-5	-	-	-	-	-	-	-	-
dmiip2	0.4737	0.3153	0.4750	0.3078	-	-	-	-
dmiip3	0.4737	0.3153	0.4750	0.3078	-	-	-	-
dmiip5	0.3998	0.3255	0.3831	0.3041	-	-	-	-
dmiip4	0.4973	0.3252	0.4948	0.3159	-	-	-	-
dmiip1	0.4128	0.2739	0.4260	0.2748	-	-	-	-
simple truncation	0.1466	0.0847	0.1431	0.0812	-	-	-	-
extractive	0.1420	0.0842	0.1367	0.0799	-	-	-	-
abstractive	0.1449	0.0844	0.1424	0.0806	-	-	-	-

Test batch 3

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
UR-gpt3.5-t-simple	0.7917	0.8485	0.6667	0.7576	0.3077	0.3077	0.3077	0.2690	0.2385	0.2333
UR-gpt4-simple	0.8750	0.9032	0.8235	0.8634	0.2692	0.4615	0.3654	0.2519	0.2343	0.2305
UR-gpt3.5-turbo-zero	0.8750	0.9091	0.8000	0.8545	0.5000	0.5000	0.5000	0.5600	0.5140	0.5101
UR-gpt4-zero-ret	0.9167	0.9375	0.8750	0.9063	0.4615	0.5000	0.4808	0.5518	0.6597	0.5736
bioinfo-0	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
bioinfo-2	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
bioinfo-3	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
MQ-1	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
MQ-2	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
MQ-3	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
MQ-4	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
MQ-5	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
Deep ML methods for	0.7917	0.8485	0.6667	0.7576	0.2692	0.2692	0.2692	0.2778	0.0930	0.1287
capstone-1	0.7083	0.7742	0.5882	0.6812	0.3846	0.6154	0.4936	0.5104	0.4063	0.3982
capstone-2	0.7500	0.8000	0.6667	0.7333	0.4231	0.6154	0.5128	0.4467	0.4896	0.4161
capstone-3	0.9167	0.9333	0.8889	0.9111	0.3462	0.3846	0.3654	0.2444	0.2319	0.2182
capstone-4	0.9167	0.9333	0.8889	0.9111	0.2692	0.4231	0.3237	0.2222	0.2504	0.2140
capstone-5	0.7083	0.7742	0.5882	0.6812	-	-	-	0.5000	0.1409	0.2095
MindLab QA Reloaded	0.5417	0.5600	0.5217	0.5409	0.1923	0.1923	0.1923	0.0833	0.0278	0.0417
IISR-1	0.9167	0.9375	0.8750	0.9063	0.4231	0.4231	0.4231	0.5153	0.4581	0.4592
IISR-2	0.9167	0.9333	0.8889	0.9111	0.3846	0.4231	0.4038	0.6519	0.6058	0.6049
IISR-3	0.6250	0.7692	-	0.3846	0.3846	0.4231	0.4038	0.6519	0.6058	0.6049
AsqAway_1	0.8750	0.9091	0.8000	0.8545	0.3077	0.3462	0.3154	0.4101	0.4905	0.4290
MindLab QA System	0.5833	0.5833	0.5833	0.5833	0.0769	0.1154	0.0962	0.2306	0.1547	0.1786
AsqAway_2	0.8750	0.9091	0.8000	0.8545	0.4231	0.4615	0.4423	0.4565	0.5650	0.4813
MindLab Red Lions++	0.5417	0.5600	0.5217	0.5409	0.1923	0.1923	0.1923	0.0833	0.0278	0.0417
AsqAway_3	0.8750	0.9091	0.8000	0.8545	0.4231	0.5000	0.4615	0.3764	0.6113	0.4431
AsqAway_4	0.8750	0.9091	0.8000	0.8545	0.4231	0.5000	0.4615	0.3764	0.6113	0.4431
kmeans	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
extractive	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
similarity measures	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
abstractive	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
simple truncation	0.6250	0.7692	-	0.3846	-	-	-	-	-	-
dmiip1	0.8333	0.8824	0.7143	0.7983	0.4615	0.6538	0.5205	0.2742	0.5094	0.3279
dmiip2	0.8333	0.8667	0.7778	0.8222	0.3462	0.5000	0.4006	0.3019	0.4422	0.3204
dmiip3	0.7083	0.8000	0.4615	0.6308	0.3462	0.6538	0.4821	0.3534	0.6104	0.4134
dmiip4	0.7500	0.8235	0.5714	0.6975	0.3846	0.6923	0.4865	0.3516	0.5039	0.3678
dmiip5	1.0000	1.0000	1.0000	1.0000	0.4231	0.5769	0.4712	0.2731	0.4757	0.3144
ELECTROLBERT-3	0.7500	0.8333	0.5000	0.6667	-	-	-	-	-	-
ELErank	0.9583	0.9677	0.9412	0.9545	0.4615	0.5385	0.4904	-	-	-
DMIS-KU-1	0.9583	0.9677	0.9412	0.9545	0.4615	0.6154	0.5154	0.5458	0.5689	0.5341
DMIS-KU-2	0.8750	0.8966	0.8421	0.8693	0.4231	0.6154	0.5077	0.5258	0.6041	0.5437
DMIS-KU-3	0.8750	0.9032	0.8235	0.8634	0.4231	0.6154	0.4981	0.5529	0.5809	0.5454
DMIS-KU-4	1.0000	1.0000	1.0000	1.0000	0.4231	0.5000	0.4500	0.5680	0.5689	0.5466
DMIS-KU-5	0.9167	0.9333	0.8889	0.9111	0.3462	0.6154	0.4647	0.5287	0.6272	0.5477
DICE1	0.6667	0.6923	0.6364	0.6643	0.3462	0.3462	0.3462	0.3593	0.3241	0.3096
DICE2	0.6250	0.6897	0.5263	0.6080	0.3077	0.3077	0.3077	0.2894	0.2980	0.2797

Ideal Answers

	Automatic scores (Rouge - R)				Manual scores
System	R-2 (Rec)	R-2 (F1)	R-SU4 (Rec)	R-SU4 (F1)	Readability	Recall	Precision	Repetition
UR-gpt3.5-t-simple	0.2839	0.0701	0.3182	0.0740	-	-	-	-
UR-gpt4-simple	0.3123	0.0735	0.3455	0.0773	-	-	-	-
UR-gpt3.5-turbo-zero	0.5203	0.1631	0.5217	0.1522	-	-	-	-
UR-gpt4-zero-ret	0.5541	0.1886	0.5436	0.1719	-	-	-	-
bioinfo-0	0.1802	0.1208	0.1957	0.1253	-	-	-	-
bioinfo-2	0.1815	0.1214	0.1992	0.1244	-	-	-	-
bioinfo-3	0.1487	0.0891	0.1723	0.0968	-	-	-	-
MQ-1	0.2461	0.2054	0.2488	0.2026	-	-	-	-
MQ-2	0.2437	0.2106	0.2503	0.2112	-	-	-	-
MQ-3	0.5057	0.3741	0.5068	0.3618	-	-	-	-
MQ-4	0.5456	0.3186	0.5425	0.3025	-	-	-	-
MQ-5	0.5265	0.3062	0.5264	0.2921	-	-	-	-
Deep ML methods for	0.0696	0.1104	0.0495	0.0807	-	-	-	-
capstone-1	0.5456	0.3186	0.5425	0.3025	-	-	-	-
capstone-2	0.5456	0.3186	0.5425	0.3025	-	-	-	-
capstone-3	0.5456	0.3186	0.5425	0.3025	-	-	-	-
capstone-4	0.5456	0.3186	0.5425	0.3025	-	-	-	-
capstone-5	0.5456	0.3186	0.5425	0.3025	-	-	-	-
MindLab QA Reloaded	0.0590	0.0889	0.0460	0.0719	-	-	-	-
IISR-1	0.3805	0.3449	0.3834	0.3420	-	-	-	-
IISR-2	0.3777	0.3233	0.3780	0.3169	-	-	-	-
IISR-3	0.3758	0.3307	0.3640	0.3139	-	-	-	-
AsqAway_1	-	-	-	-	-	-	-	-
MindLab QA System	0.0465	0.0657	0.0398	0.0581	-	-	-	-
AsqAway_2	-	-	-	-	-	-	-	-
MindLab Red Lions++	0.0590	0.0889	0.0460	0.0719	-	-	-	-
AsqAway_3	-	-	-	-	-	-	-	-
AsqAway_4	-	-	-	-	-	-	-	-
kmeans	0.1340	0.1175	0.1307	0.1126	-	-	-	-
extractive	0.1407	0.1255	0.1356	0.1197	-	-	-	-
similarity measures	0.1296	0.1174	0.1268	0.1132	-	-	-	-
abstractive	0.1416	0.1283	0.1388	0.1236	-	-	-	-
simple truncation	0.1475	0.1376	0.1417	0.1308	-	-	-	-
dmiip1	0.3519	0.1235	0.3760	0.1245	-	-	-	-
dmiip2	0.5173	0.2299	0.5238	0.2212	-	-	-	-
dmiip3	0.5256	0.2328	0.5323	0.2239	-	-	-	-
dmiip4	0.5132	0.2282	0.5156	0.2164	-	-	-	-
dmiip5	0.4406	0.3565	0.4451	0.3509	-	-	-	-
ELECTROLBERT-3	-	-	-	-	-	-	-	-
ELErank	0.2939	0.2935	0.2841	0.2786	-	-	-	-
DMIS-KU-1	-	-	-	-	-	-	-	-
DMIS-KU-2	-	-	-	-	-	-	-	-
DMIS-KU-3	-	-	-	-	-	-	-	-
DMIS-KU-4	-	-	-	-	-	-	-	-
DMIS-KU-5	-	-	-	-	-	-	-	-
DICE1	-	-	-	-	-	-	-	-
DICE2	-	-	-	-	-	-	-	-

Test batch 4

Exact Answers

	Yes/No				Factoid			List
System	Accuracy	F1 Yes	F1 No	Macro F1	Strict Acc.	Lenient Acc.	MRR	Mean Prec.	Recall	F-Measure
UR-gpt3.5-t-simple	0.4286	0.5000	0.3333	0.4167	0.2581	0.2903	0.2742	0.3944	0.3362	0.3470
UR-gpt4-simple	0.7857	0.7273	0.8235	0.7754	0.2258	0.2581	0.2366	0.4440	0.4214	0.4127
UR-gpt3.5-turbo-zero	0.9286	0.8571	0.9524	0.9048	0.6452	0.6452	0.6452	0.6090	0.6710	0.6196
UR-gpt4-zero-ret	0.9286	0.8889	0.9474	0.9181	0.5161	0.6129	0.5645	0.6902	0.7818	0.7191
IISR-2	0.9286	0.8889	0.9474	0.9181	0.4194	0.4194	0.4194	0.6404	0.6620	0.6364
IISR-1	1.0000	1.0000	1.0000	1.0000	0.3871	0.4194	0.4032	0.7171	0.6479	0.6706
bioinfo-0	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
bioinfo-1	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
bioinfo-2	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
bioinfo-3	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
bioinfo-4	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
dmiip1	0.2857	0.4444	-	0.2222	0.4839	0.7742	0.5914	0.4778	0.4448	0.4460
dmiip2	0.7857	0.6667	0.8421	0.7544	0.4516	0.7419	0.5710	0.5575	0.5199	0.5073
dmiip3	0.7857	0.6667	0.8421	0.7544	0.5484	0.7742	0.6371	0.5528	0.4584	0.4784
dmiip4	0.7143	0.6667	0.7500	0.7083	0.4194	0.6774	0.5226	0.4871	0.4536	0.4424
dmiip5	1.0000	1.0000	1.0000	1.0000	0.4839	0.7419	0.5968	0.4689	0.4486	0.4380
MQ-5	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
MQ-3	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
MQ-1	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
ELErank	0.8571	0.7500	0.9000	0.8250	0.5161	0.7097	0.5978	-	-	-
ELErank+	0.8571	0.7500	0.9000	0.8250	0.5161	0.7097	0.5978	-	-	-
capstone-1	0.7857	0.6667	0.8421	0.7544	0.4839	0.6774	0.5522	0.4840	0.3068	0.3510
MQ-2	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
MQ-4	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
capstone-2	0.7857	0.6667	0.8421	0.7544	0.4839	0.6774	0.5522	0.4924	0.4429	0.4226
capstone-3	0.7857	0.6667	0.8421	0.7544	0.4839	0.6774	0.5522	0.4840	0.3068	0.3510
capstone-4	0.7857	0.6667	0.8421	0.7544	0.2581	0.4839	0.3548	0.4924	0.4429	0.4226
capstone-5	0.7857	0.6667	0.8421	0.7544	0.4839	0.6774	0.5522	0.4394	0.4498	0.4061
AUEB-System1	0.2857	0.4444	-	0.2222	0.2581	0.3226	0.2849	0.0125	0.0197	0.0153
Deep ML methods for	0.3571	0.4000	0.3077	0.3538	0.2258	0.2258	0.2258	0.2847	0.1144	0.1463
MindLab QA System	0.3571	0.4000	0.3077	0.3538	0.2258	0.2258	0.2258	0.2847	0.1144	0.1463
DICE1	1.0000	1.0000	1.0000	1.0000	0.5484	0.5484	0.5484	0.5489	0.4997	0.5130
MindLab QA System ++	0.3571	0.4000	0.3077	0.3538	0.2258	0.2258	0.2258	0.2847	0.1144	0.1463
ELECTROLBERT-3	0.5714	0.4000	0.6667	0.5333	-	-	-	-	-	-
OWLMan-phaseB-TaskV1	0.4286	0.5000	0.3333	0.4167	0.4194	0.4839	0.4435	0.2417	0.3152	0.2572
DMIS-KU-1	1.0000	1.0000	1.0000	1.0000	0.6452	0.8710	0.7323	0.7139	0.8061	0.7440
DMIS-KU-2	0.9286	0.8889	0.9474	0.9181	0.6452	0.8387	0.7108	0.7271	0.6606	0.6806
DMIS-KU-3	0.9286	0.8571	0.9524	0.9048	0.5806	0.8387	0.6882	0.7331	0.6467	0.6752
DMIS-KU-4	0.8571	0.7500	0.9000	0.8250	0.5484	0.8387	0.6570	0.7734	0.6282	0.6747
DMIS-KU-5	0.9286	0.8889	0.9474	0.9181	0.5484	0.8065	0.6473	0.7234	0.6652	0.6787
IISR-3	0.2857	0.4444	-	0.2222	0.4194	0.4194	0.4194	0.2417	0.1836	0.1969
AUEB-System2	0.2857	0.4444	-	0.2222	0.1935	0.3226	0.2419	0.0396	0.0424	0.0395
IISR-4	1.0000	1.0000	1.0000	1.0000	0.4194	0.4194	0.4194	0.6667	0.5747	0.5956
DICE2	0.9286	0.8889	0.9474	0.9181	0.2581	0.2581	0.2581	-	-	-
abstractive	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
DICE_Lab	1.0000	1.0000	1.0000	1.0000	0.4839	0.4839	0.4839	0.4978	0.4989	0.4868
extractive	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
kmeans	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
similarity measures	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
simple truncation	0.2857	0.4444	-	0.2222	-	-	-	-	-	-
DICE_Lab2	0.9286	0.8889	0.9474	0.9181	0.5484	0.5484	0.5484	0.3743	0.3701	0.3650
IISR-5	0.9286	0.8889	0.9474	0.9181	0.4516	0.4516	0.4516	0.7134	0.6840	0.6807

Ideal Answers

	Automatic scores (Rouge - R)				Manual scores
System	R-2 (Rec)	R-2 (F1)	R-SU4 (Rec)	R-SU4 (F1)	Readability	Recall	Precision	Repetition
UR-gpt3.5-t-simple	0.2678	0.0757	0.2938	0.0852	-	-	-	-
UR-gpt4-simple	0.2869	0.0804	0.3138	0.0870	-	-	-	-
UR-gpt3.5-turbo-zero	0.5381	0.2159	0.5235	0.2006	-	-	-	-
UR-gpt4-zero-ret	0.5570	0.2500	0.5309	0.2320	-	-	-	-
IISR-2	0.3306	0.2999	0.3322	0.2932	-	-	-	-
IISR-1	0.3495	0.3419	0.3448	0.3311	-	-	-	-
bioinfo-0	0.1916	0.1586	0.2023	0.1581	-	-	-	-
bioinfo-1	0.2811	0.2433	0.2715	0.2292	-	-	-	-
bioinfo-2	0.1325	0.1076	0.1471	0.1110	-	-	-	-
bioinfo-3	0.2071	0.1582	0.2118	0.1570	-	-	-	-
bioinfo-4	0.1200	0.0892	0.1356	0.0962	-	-	-	-
dmiip1	0.5458	0.3864	0.5286	0.3658	-	-	-	-
dmiip2	0.5458	0.3864	0.5286	0.3658	-	-	-	-
dmiip3	0.5424	0.3825	0.5252	0.3618	-	-	-	-
dmiip4	0.5645	0.3905	0.5449	0.3682	-	-	-	-
dmiip5	0.3746	0.3488	0.3664	0.3361	-	-	-	-
MQ-5	0.5582	0.3889	0.5382	0.3671	-	-	-	-
MQ-3	0.5383	0.4084	0.5163	0.3840	-	-	-	-
MQ-1	0.4557	0.4019	0.4352	0.3768	-	-	-	-
ELErank	0.3035	0.3021	0.2859	0.2838	-	-	-	-
ELErank+	0.3035	0.3021	0.2859	0.2838	-	-	-	-
capstone-1	0.5604	0.3919	0.5411	0.3702	-	-	-	-
MQ-2	0.4184	0.3954	0.4056	0.3778	-	-	-	-
MQ-4	0.5604	0.3919	0.5411	0.3702	-	-	-	-
capstone-2	0.5604	0.3919	0.5411	0.3702	-	-	-	-
capstone-3	0.6335	0.3761	0.6244	0.3569	-	-	-	-
capstone-4	0.6335	0.3761	0.6244	0.3569	-	-	-	-
capstone-5	0.5604	0.3919	0.5411	0.3702	-	-	-	-
AUEB-System1	-	-	-	-	-	-	-	-
Deep ML methods for	0.0433	0.0619	0.0384	0.0578	-	-	-	-
MindLab QA System	0.0776	0.1253	0.0537	0.0905	-	-	-	-
DICE1	-	-	-	-	-	-	-	-
MindLab QA System ++	0.0524	0.0823	0.0401	0.0661	-	-	-	-
ELECTROLBERT-3	-	-	-	-	-	-	-	-
OWLMan-phaseB-TaskV1	-	-	-	-	-	-	-	-
DMIS-KU-1	-	-	-	-	-	-	-	-
DMIS-KU-2	-	-	-	-	-	-	-	-
DMIS-KU-3	-	-	-	-	-	-	-	-
DMIS-KU-4	-	-	-	-	-	-	-	-
DMIS-KU-5	-	-	-	-	-	-	-	-
IISR-3	0.3330	0.3400	0.3140	0.3173	-	-	-	-
AUEB-System2	-	-	-	-	-	-	-	-
IISR-4	0.3447	0.3358	0.3388	0.3255	-	-	-	-
DICE2	-	-	-	-	-	-	-	-
abstractive	0.1111	0.1173	0.1083	0.1135	-	-	-	-
DICE_Lab	-	-	-	-	-	-	-	-
extractive	0.1176	0.1216	0.1141	0.1171	-	-	-	-
kmeans	0.1328	0.1320	0.1299	0.1282	-	-	-	-
similarity measures	0.1002	0.1066	0.0976	0.1031	-	-	-	-
simple truncation	0.1195	0.1269	0.1162	0.1225	-	-	-	-
DICE_Lab2	-	-	-	-	-	-	-	-
IISR-5	0.3451	0.3212	0.3405	0.3086	-	-	-	-