BioASQ Participants Area
Task 11b: Test Results of Phase B
The test results are presented in separate tables for each type of annotation. The "System Description" of each system is used.
The evaluation measures that are used in Task B are presented
here .
Warning: For ideal answers, good ROUGE results do not always imply good manual scores.
Test batch 1
Exact Answers
|
Yes/No |
Factoid |
List |
System |
Accuracy |
F1 Yes |
F1 No |
Macro F1 |
Strict Acc. |
Lenient Acc. |
MRR |
Mean Prec. |
Recall |
F-Measure |
OWLMan-phaseB-TaskV1 |
0.7083 |
0.8000 |
0.4615 |
0.6308 |
0.3158 |
0.5263 |
0.4123 |
0.2167 |
0.3973 |
0.2618 |
IISR-1 |
0.9167 |
0.9375 |
0.8750 |
0.9063 |
0.4211 |
0.4211 |
0.4211 |
0.7185 |
0.6666 |
0.6841 |
IISR-3 |
0.7083 |
0.8293 |
- |
0.4146 |
- | - | - |
- | - | - |
ELECTROLBERT-3 |
0.8333 |
0.8947 |
0.6000 |
0.7474 |
- | - | - |
- | - | - |
AsqAway_1 |
0.7083 |
0.8293 |
- |
0.4146 |
0.3158 |
0.3158 |
0.3158 |
0.4754 |
0.4905 |
0.4595 |
AsqAway_2 |
0.7083 |
0.8293 |
- |
0.4146 |
0.3158 |
0.4737 |
0.3947 |
0.4368 |
0.6140 |
0.4804 |
AsqAway_3 |
0.7083 |
0.8293 |
- |
0.4146 |
0.3158 |
0.4737 |
0.3947 |
0.4388 |
0.6953 |
0.5106 |
AsqAway_4 |
0.7083 |
0.8293 |
- |
0.4146 |
0.3684 |
0.4737 |
0.4211 |
0.4388 |
0.6953 |
0.5106 |
MQ-1 |
0.7083 |
0.8293 |
- |
0.4146 |
- | - | - |
- | - | - |
MQ-2 |
0.7083 |
0.8293 |
- |
0.4146 |
- | - | - |
- | - | - |
MQ-3 |
0.7083 |
0.8293 |
- |
0.4146 |
- | - | - |
- | - | - |
Deep ML methods for |
0.7083 |
0.8000 |
0.4615 |
0.6308 |
0.1579 |
0.2105 |
0.1754 |
0.4444 |
0.2695 |
0.2903 |
MQ-4 |
0.7083 |
0.8293 |
- |
0.4146 |
- | - | - |
- | - | - |
MQ-5 |
0.7083 |
0.8293 |
- |
0.4146 |
- | - | - |
- | - | - |
simple truncation |
0.9167 |
0.9375 |
0.8750 |
0.9063 |
0.2632 |
0.2632 |
0.2632 |
0.2389 |
0.2369 |
0.2374 |
dmiip1 |
0.8333 |
0.8824 |
0.7143 |
0.7983 |
0.3684 |
0.5263 |
0.4474 |
0.4183 |
0.4417 |
0.4099 |
dmiip2 |
0.8333 |
0.8824 |
0.7143 |
0.7983 |
0.5263 |
0.6316 |
0.5570 |
0.4698 |
0.4271 |
0.4199 |
dmiip4 |
0.7917 |
0.8485 |
0.6667 |
0.7576 |
0.3684 |
0.6316 |
0.4737 |
0.4715 |
0.5307 |
0.4686 |
dmiip5 |
0.9167 |
0.9412 |
0.8571 |
0.8992 |
0.3684 |
0.5789 |
0.4474 |
0.4815 |
0.5092 |
0.4609 |
dmiip3 |
0.7917 |
0.8485 |
0.6667 |
0.7576 |
0.3684 |
0.5789 |
0.4474 |
0.4590 |
0.5326 |
0.4696 |
IISR-2 |
0.9583 |
0.9697 |
0.9333 |
0.9515 |
0.5263 |
0.5263 |
0.5263 |
0.6417 |
0.5697 |
0.5972 |
UR-gpt4-zero-ret |
0.9167 |
0.9412 |
0.8571 |
0.8992 |
0.5789 |
0.5789 |
0.5789 |
0.6472 |
0.6530 |
0.6495 |
UR-gpt3.5-turbo-zero |
0.8750 |
0.9091 |
0.8000 |
0.8545 |
0.5263 |
0.6316 |
0.5789 |
0.6742 |
0.7249 |
0.6917 |
UR-gpt4-simple |
0.9167 |
0.9412 |
0.8571 |
0.8992 |
0.2105 |
0.2632 |
0.2368 |
0.4000 |
0.4014 |
0.3939 |
extractive |
0.9167 |
0.9375 |
0.8750 |
0.9063 |
0.2632 |
0.2632 |
0.2632 |
0.1667 |
0.1667 |
0.1667 |
abstractive |
0.9167 |
0.9375 |
0.8750 |
0.9063 |
0.2105 |
0.2105 |
0.2105 |
0.1875 |
0.1833 |
0.1852 |
bioinfo-0 |
0.7083 |
0.8293 |
- |
0.4146 |
- | - | - |
- | - | - |
BART |
0.2500 |
0.3077 |
0.1818 |
0.2448 |
- | - | - |
- | - | - |
DMIS-KU-2 |
0.9167 |
0.9412 |
0.8571 |
0.8992 |
0.4737 |
0.6316 |
0.5526 |
0.7365 |
0.6910 |
0.6965 |
DMIS-KU-3 |
0.9167 |
0.9412 |
0.8571 |
0.8992 |
0.4211 |
0.6316 |
0.5088 |
0.7861 |
0.6668 |
0.7027 |
DMIS-KU-5 |
0.9583 |
0.9697 |
0.9333 |
0.9515 |
0.4737 |
0.6316 |
0.5526 |
0.7861 |
0.6668 |
0.7027 |
DMIS-KU-1 |
0.9167 |
0.9412 |
0.8571 |
0.8992 |
0.4737 |
0.6316 |
0.5526 |
0.7778 |
0.6592 |
0.6937 |
DMIS-KU-4 |
0.8750 |
0.9091 |
0.8000 |
0.8545 |
0.4737 |
0.6316 |
0.5439 |
0.7986 |
0.6592 |
0.6992 |
Ideal Answers
|
Automatic scores (Rouge - R) |
Manual scores |
System |
R-2 (Rec) |
R-2 (F1) |
R-SU4 (Rec) |
R-SU4 (F1) |
Readability |
Recall |
Precision |
Repetition |
OWLMan-phaseB-TaskV1 |
- |
- |
- |
- |
- |
- |
- |
- |
IISR-1 |
0.3780 |
0.3143 |
0.3609 |
0.2902 |
- |
- |
- |
- |
IISR-3 |
0.4477 |
0.4063 |
0.4389 |
0.3960 |
- |
- |
- |
- |
ELECTROLBERT-3 |
- |
- |
- |
- |
- |
- |
- |
- |
AsqAway_1 |
- |
- |
- |
- |
- |
- |
- |
- |
AsqAway_2 |
- |
- |
- |
- |
- |
- |
- |
- |
AsqAway_3 |
- |
- |
- |
- |
- |
- |
- |
- |
AsqAway_4 |
- |
- |
- |
- |
- |
- |
- |
- |
MQ-1 |
0.2370 |
0.1458 |
0.2414 |
0.1389 |
- |
- |
- |
- |
MQ-2 |
0.2608 |
0.2112 |
0.2651 |
0.2048 |
- |
- |
- |
- |
MQ-3 |
0.5268 |
0.3555 |
0.5118 |
0.3334 |
- |
- |
- |
- |
Deep ML methods for |
0.0982 |
0.1452 |
0.0739 |
0.1103 |
- |
- |
- |
- |
MQ-4 |
0.6046 |
0.3367 |
0.5891 |
0.3168 |
- |
- |
- |
- |
MQ-5 |
0.6167 |
0.3436 |
0.5997 |
0.3222 |
- |
- |
- |
- |
simple truncation |
0.1515 |
0.0665 |
0.1474 |
0.0623 |
- |
- |
- |
- |
dmiip1 |
0.5780 |
0.3103 |
0.5674 |
0.2927 |
- |
- |
- |
- |
dmiip2 |
0.5711 |
0.3206 |
0.5629 |
0.3059 |
- |
- |
- |
- |
dmiip4 |
0.5905 |
0.3281 |
0.5809 |
0.3117 |
- |
- |
- |
- |
dmiip5 |
0.6052 |
0.3479 |
0.5931 |
0.3308 |
- |
- |
- |
- |
dmiip3 |
0.5711 |
0.3206 |
0.5629 |
0.3059 |
- |
- |
- |
- |
IISR-2 |
0.4149 |
0.3289 |
0.4026 |
0.3093 |
- |
- |
- |
- |
UR-gpt4-zero-ret |
0.5610 |
0.1855 |
0.5547 |
0.1700 |
- |
- |
- |
- |
UR-gpt3.5-turbo-zero |
0.5365 |
0.1500 |
0.5273 |
0.1377 |
- |
- |
- |
- |
UR-gpt4-simple |
0.3346 |
0.0729 |
0.3666 |
0.0744 |
- |
- |
- |
- |
extractive |
0.1662 |
0.0725 |
0.1625 |
0.0684 |
- |
- |
- |
- |
abstractive |
0.1562 |
0.0691 |
0.1533 |
0.0656 |
- |
- |
- |
- |
bioinfo-0 |
0.3256 |
0.2907 |
0.3182 |
0.2741 |
- |
- |
- |
- |
BART |
0.1920 |
0.2125 |
0.1813 |
0.2009 |
- |
- |
- |
- |
DMIS-KU-2 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-3 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-5 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-1 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-4 |
- |
- |
- |
- |
- |
- |
- |
- |
Test batch 2
Exact Answers
|
Yes/No |
Factoid |
List |
System |
Accuracy |
F1 Yes |
F1 No |
Macro F1 |
Strict Acc. |
Lenient Acc. |
MRR |
Mean Prec. |
Recall |
F-Measure |
IISR-2 |
1.0000 |
1.0000 |
1.0000 |
1.0000 |
0.5455 |
0.6364 |
0.5909 |
0.5099 |
0.3577 |
0.3980 |
IISR-3 |
0.5833 |
0.7368 |
- |
0.3684 |
0.5455 |
0.6364 |
0.5909 |
0.5099 |
0.3577 |
0.3980 |
IISR-1 |
1.0000 |
1.0000 |
1.0000 |
1.0000 |
0.5000 |
0.5455 |
0.5227 |
0.4861 |
0.3310 |
0.3678 |
OWLMan-phaseB-TaskV1 |
0.4583 |
0.5806 |
0.2353 |
0.4080 |
0.4545 |
0.5000 |
0.4697 |
0.1667 |
0.1395 |
0.1374 |
UR-gpt3.5-turbo-zero |
0.9167 |
0.9333 |
0.8889 |
0.9111 |
0.5455 |
0.5909 |
0.5682 |
0.4598 |
0.4671 |
0.4316 |
UR-gpt4-zero-ret |
0.9583 |
0.9655 |
0.9474 |
0.9564 |
0.5455 |
0.5909 |
0.5682 |
0.3742 |
0.4369 |
0.3828 |
ELECTROLBERT-3 |
0.7083 |
0.8000 |
0.4615 |
0.6308 |
- | - | - |
- | - | - |
capstone-1 |
0.9167 |
0.9231 |
0.9091 |
0.9161 |
0.4091 |
0.5455 |
0.4561 |
0.2085 |
0.3810 |
0.2617 |
capstone-3 |
0.8333 |
0.8571 |
0.8000 |
0.8286 |
0.2273 |
0.3636 |
0.2727 |
0.2485 |
0.3810 |
0.2923 |
capstone-2 |
0.8333 |
0.8571 |
0.8000 |
0.8286 |
0.2273 |
0.3636 |
0.2727 |
0.3250 |
0.3750 |
0.3173 |
bioinfo-0 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
bioinfo-1 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
bioinfo-2 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
bioinfo-3 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
bioinfo-4 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
AsqAway_1 |
0.8750 |
0.8966 |
0.8421 |
0.8693 |
0.4545 |
0.4545 |
0.4545 |
0.1780 |
0.1968 |
0.1756 |
AsqAway_2 |
0.8750 |
0.8966 |
0.8421 |
0.8693 |
0.4545 |
0.4545 |
0.4545 |
0.2023 |
0.3226 |
0.2327 |
AsqAway_3 |
0.8750 |
0.8966 |
0.8421 |
0.8693 |
0.4545 |
0.5000 |
0.4773 |
0.1951 |
0.3504 |
0.2329 |
AsqAway_4 |
0.8750 |
0.8966 |
0.8421 |
0.8693 |
0.4545 |
0.5000 |
0.4773 |
0.1951 |
0.3504 |
0.2329 |
Deep ML methods for |
0.8333 |
0.8667 |
0.7778 |
0.8222 |
0.2727 |
0.2727 |
0.2727 |
0.3472 |
0.2008 |
0.2125 |
MindLab QA Reloaded |
0.7917 |
0.8276 |
0.7368 |
0.7822 |
0.2727 |
0.2727 |
0.2727 |
0.3264 |
0.1341 |
0.1755 |
MindLab QA System |
0.8333 |
0.8667 |
0.7778 |
0.8222 |
0.2727 |
0.4091 |
0.3000 |
0.1333 |
0.1841 |
0.1402 |
MindLab QA System ++ |
0.5417 |
0.6857 |
0.1538 |
0.4198 |
0.1818 |
0.2273 |
0.2045 |
0.3889 |
0.1063 |
0.1597 |
MindLab Red Lions++ |
0.7917 |
0.8276 |
0.7368 |
0.7822 |
0.2727 |
0.2727 |
0.2727 |
0.3264 |
0.1341 |
0.1755 |
MQ-1 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
MQ-2 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
MQ-3 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
MQ-4 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
MQ-5 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
DMIS-KU-1 |
0.9583 |
0.9630 |
0.9524 |
0.9577 |
0.3182 |
0.6818 |
0.4773 |
0.3349 |
0.3623 |
0.3080 |
DMIS-KU-2 |
0.9583 |
0.9630 |
0.9524 |
0.9577 |
0.3182 |
0.6818 |
0.4561 |
0.3486 |
0.3456 |
0.3087 |
DMIS-KU-3 |
0.9583 |
0.9630 |
0.9524 |
0.9577 |
0.3636 |
0.5909 |
0.4621 |
0.2818 |
0.4058 |
0.3178 |
DMIS-KU-4 |
1.0000 |
1.0000 |
1.0000 |
1.0000 |
0.3636 |
0.5909 |
0.4697 |
0.2983 |
0.3683 |
0.2871 |
DMIS-KU-5 |
0.9167 |
0.9286 |
0.9000 |
0.9143 |
0.3636 |
0.5909 |
0.4621 |
0.2534 |
0.4593 |
0.3022 |
dmiip2 |
0.5833 |
0.7368 |
- |
0.3684 |
0.4091 |
0.5909 |
0.4811 |
0.3488 |
0.3256 |
0.3022 |
dmiip3 |
0.8750 |
0.8889 |
0.8571 |
0.8730 |
0.3182 |
0.5455 |
0.3992 |
0.2851 |
0.2464 |
0.2232 |
dmiip5 |
0.9583 |
0.9655 |
0.9474 |
0.9564 |
0.4091 |
0.4545 |
0.4242 |
0.1413 |
0.2200 |
0.1676 |
dmiip4 |
0.7917 |
0.8276 |
0.7368 |
0.7822 |
0.4545 |
0.5909 |
0.5227 |
0.2573 |
0.3200 |
0.2531 |
dmiip1 |
0.9167 |
0.9286 |
0.9000 |
0.9143 |
0.3182 |
0.5909 |
0.4318 |
0.2271 |
0.3760 |
0.2501 |
simple truncation |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
extractive |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
abstractive |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
Ideal Answers
|
Automatic scores (Rouge - R) |
Manual scores |
System |
R-2 (Rec) |
R-2 (F1) |
R-SU4 (Rec) |
R-SU4 (F1) |
Readability |
Recall |
Precision |
Repetition |
IISR-2 |
0.3552 |
0.3007 |
0.3520 |
0.2904 |
- |
- |
- |
- |
IISR-3 |
0.3390 |
0.3056 |
0.3359 |
0.2954 |
- |
- |
- |
- |
IISR-1 |
0.3394 |
0.2870 |
0.3403 |
0.2821 |
- |
- |
- |
- |
OWLMan-phaseB-TaskV1 |
- |
- |
- |
- |
- |
- |
- |
- |
UR-gpt3.5-turbo-zero |
0.4983 |
0.1720 |
0.5163 |
0.1638 |
- |
- |
- |
- |
UR-gpt4-zero-ret |
0.4802 |
0.1785 |
0.4911 |
0.1684 |
- |
- |
- |
- |
ELECTROLBERT-3 |
- |
- |
- |
- |
- |
- |
- |
- |
capstone-1 |
0.4953 |
0.3278 |
0.4919 |
0.3185 |
- |
- |
- |
- |
capstone-3 |
0.4953 |
0.3278 |
0.4919 |
0.3185 |
- |
- |
- |
- |
capstone-2 |
0.4953 |
0.3278 |
0.4919 |
0.3185 |
- |
- |
- |
- |
bioinfo-0 |
0.2192 |
0.1784 |
0.2206 |
0.1733 |
- |
- |
- |
- |
bioinfo-1 |
0.2608 |
0.2257 |
0.2627 |
0.2205 |
- |
- |
- |
- |
bioinfo-2 |
0.2912 |
0.2853 |
0.2937 |
0.2825 |
- |
- |
- |
- |
bioinfo-3 |
0.3323 |
0.3242 |
0.3300 |
0.3172 |
- |
- |
- |
- |
bioinfo-4 |
0.3263 |
0.3169 |
0.3253 |
0.3111 |
- |
- |
- |
- |
AsqAway_1 |
- |
- |
- |
- |
- |
- |
- |
- |
AsqAway_2 |
- |
- |
- |
- |
- |
- |
- |
- |
AsqAway_3 |
- |
- |
- |
- |
- |
- |
- |
- |
AsqAway_4 |
- |
- |
- |
- |
- |
- |
- |
- |
Deep ML methods for |
0.0553 |
0.0822 |
0.0486 |
0.0706 |
- |
- |
- |
- |
MindLab QA Reloaded |
0.0583 |
0.0851 |
0.0514 |
0.0736 |
- |
- |
- |
- |
MindLab QA System |
0.0218 |
0.0308 |
0.0284 |
0.0367 |
- |
- |
- |
- |
MindLab QA System ++ |
0.0785 |
0.1169 |
0.0607 |
0.0940 |
- |
- |
- |
- |
MindLab Red Lions++ |
0.0583 |
0.0851 |
0.0514 |
0.0736 |
- |
- |
- |
- |
MQ-1 |
0.1950 |
0.1404 |
0.2224 |
0.1530 |
- |
- |
- |
- |
MQ-2 |
0.2436 |
0.1971 |
0.2463 |
0.1961 |
- |
- |
- |
- |
MQ-3 |
0.4678 |
0.3290 |
0.4690 |
0.3203 |
- |
- |
- |
- |
MQ-4 |
0.4953 |
0.3278 |
0.4919 |
0.3185 |
- |
- |
- |
- |
MQ-5 |
0.4953 |
0.3259 |
0.4924 |
0.3169 |
- |
- |
- |
- |
DMIS-KU-1 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-2 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-3 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-4 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-5 |
- |
- |
- |
- |
- |
- |
- |
- |
dmiip2 |
0.4737 |
0.3153 |
0.4750 |
0.3078 |
- |
- |
- |
- |
dmiip3 |
0.4737 |
0.3153 |
0.4750 |
0.3078 |
- |
- |
- |
- |
dmiip5 |
0.3998 |
0.3255 |
0.3831 |
0.3041 |
- |
- |
- |
- |
dmiip4 |
0.4973 |
0.3252 |
0.4948 |
0.3159 |
- |
- |
- |
- |
dmiip1 |
0.4128 |
0.2739 |
0.4260 |
0.2748 |
- |
- |
- |
- |
simple truncation |
0.1466 |
0.0847 |
0.1431 |
0.0812 |
- |
- |
- |
- |
extractive |
0.1420 |
0.0842 |
0.1367 |
0.0799 |
- |
- |
- |
- |
abstractive |
0.1449 |
0.0844 |
0.1424 |
0.0806 |
- |
- |
- |
- |
Test batch 3
Exact Answers
|
Yes/No |
Factoid |
List |
System |
Accuracy |
F1 Yes |
F1 No |
Macro F1 |
Strict Acc. |
Lenient Acc. |
MRR |
Mean Prec. |
Recall |
F-Measure |
UR-gpt3.5-t-simple |
0.7917 |
0.8485 |
0.6667 |
0.7576 |
0.3077 |
0.3077 |
0.3077 |
0.2690 |
0.2385 |
0.2333 |
UR-gpt4-simple |
0.8750 |
0.9032 |
0.8235 |
0.8634 |
0.2692 |
0.4615 |
0.3654 |
0.2519 |
0.2343 |
0.2305 |
UR-gpt3.5-turbo-zero |
0.8750 |
0.9091 |
0.8000 |
0.8545 |
0.5000 |
0.5000 |
0.5000 |
0.5600 |
0.5140 |
0.5101 |
UR-gpt4-zero-ret |
0.9167 |
0.9375 |
0.8750 |
0.9063 |
0.4615 |
0.5000 |
0.4808 |
0.5518 |
0.6597 |
0.5736 |
bioinfo-0 |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
bioinfo-2 |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
bioinfo-3 |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
MQ-1 |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
MQ-2 |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
MQ-3 |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
MQ-4 |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
MQ-5 |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
Deep ML methods for |
0.7917 |
0.8485 |
0.6667 |
0.7576 |
0.2692 |
0.2692 |
0.2692 |
0.2778 |
0.0930 |
0.1287 |
capstone-1 |
0.7083 |
0.7742 |
0.5882 |
0.6812 |
0.3846 |
0.6154 |
0.4936 |
0.5104 |
0.4063 |
0.3982 |
capstone-2 |
0.7500 |
0.8000 |
0.6667 |
0.7333 |
0.4231 |
0.6154 |
0.5128 |
0.4467 |
0.4896 |
0.4161 |
capstone-3 |
0.9167 |
0.9333 |
0.8889 |
0.9111 |
0.3462 |
0.3846 |
0.3654 |
0.2444 |
0.2319 |
0.2182 |
capstone-4 |
0.9167 |
0.9333 |
0.8889 |
0.9111 |
0.2692 |
0.4231 |
0.3237 |
0.2222 |
0.2504 |
0.2140 |
capstone-5 |
0.7083 |
0.7742 |
0.5882 |
0.6812 |
- | - | - |
0.5000 |
0.1409 |
0.2095 |
MindLab QA Reloaded |
0.5417 |
0.5600 |
0.5217 |
0.5409 |
0.1923 |
0.1923 |
0.1923 |
0.0833 |
0.0278 |
0.0417 |
IISR-1 |
0.9167 |
0.9375 |
0.8750 |
0.9063 |
0.4231 |
0.4231 |
0.4231 |
0.5153 |
0.4581 |
0.4592 |
IISR-2 |
0.9167 |
0.9333 |
0.8889 |
0.9111 |
0.3846 |
0.4231 |
0.4038 |
0.6519 |
0.6058 |
0.6049 |
IISR-3 |
0.6250 |
0.7692 |
- |
0.3846 |
0.3846 |
0.4231 |
0.4038 |
0.6519 |
0.6058 |
0.6049 |
AsqAway_1 |
0.8750 |
0.9091 |
0.8000 |
0.8545 |
0.3077 |
0.3462 |
0.3154 |
0.4101 |
0.4905 |
0.4290 |
MindLab QA System |
0.5833 |
0.5833 |
0.5833 |
0.5833 |
0.0769 |
0.1154 |
0.0962 |
0.2306 |
0.1547 |
0.1786 |
AsqAway_2 |
0.8750 |
0.9091 |
0.8000 |
0.8545 |
0.4231 |
0.4615 |
0.4423 |
0.4565 |
0.5650 |
0.4813 |
MindLab Red Lions++ |
0.5417 |
0.5600 |
0.5217 |
0.5409 |
0.1923 |
0.1923 |
0.1923 |
0.0833 |
0.0278 |
0.0417 |
AsqAway_3 |
0.8750 |
0.9091 |
0.8000 |
0.8545 |
0.4231 |
0.5000 |
0.4615 |
0.3764 |
0.6113 |
0.4431 |
AsqAway_4 |
0.8750 |
0.9091 |
0.8000 |
0.8545 |
0.4231 |
0.5000 |
0.4615 |
0.3764 |
0.6113 |
0.4431 |
kmeans |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
extractive |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
similarity measures |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
abstractive |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
simple truncation |
0.6250 |
0.7692 |
- |
0.3846 |
- | - | - |
- | - | - |
dmiip1 |
0.8333 |
0.8824 |
0.7143 |
0.7983 |
0.4615 |
0.6538 |
0.5205 |
0.2742 |
0.5094 |
0.3279 |
dmiip2 |
0.8333 |
0.8667 |
0.7778 |
0.8222 |
0.3462 |
0.5000 |
0.4006 |
0.3019 |
0.4422 |
0.3204 |
dmiip3 |
0.7083 |
0.8000 |
0.4615 |
0.6308 |
0.3462 |
0.6538 |
0.4821 |
0.3534 |
0.6104 |
0.4134 |
dmiip4 |
0.7500 |
0.8235 |
0.5714 |
0.6975 |
0.3846 |
0.6923 |
0.4865 |
0.3516 |
0.5039 |
0.3678 |
dmiip5 |
1.0000 |
1.0000 |
1.0000 |
1.0000 |
0.4231 |
0.5769 |
0.4712 |
0.2731 |
0.4757 |
0.3144 |
ELECTROLBERT-3 |
0.7500 |
0.8333 |
0.5000 |
0.6667 |
- | - | - |
- | - | - |
ELErank |
0.9583 |
0.9677 |
0.9412 |
0.9545 |
0.4615 |
0.5385 |
0.4904 |
- | - | - |
DMIS-KU-1 |
0.9583 |
0.9677 |
0.9412 |
0.9545 |
0.4615 |
0.6154 |
0.5154 |
0.5458 |
0.5689 |
0.5341 |
DMIS-KU-2 |
0.8750 |
0.8966 |
0.8421 |
0.8693 |
0.4231 |
0.6154 |
0.5077 |
0.5258 |
0.6041 |
0.5437 |
DMIS-KU-3 |
0.8750 |
0.9032 |
0.8235 |
0.8634 |
0.4231 |
0.6154 |
0.4981 |
0.5529 |
0.5809 |
0.5454 |
DMIS-KU-4 |
1.0000 |
1.0000 |
1.0000 |
1.0000 |
0.4231 |
0.5000 |
0.4500 |
0.5680 |
0.5689 |
0.5466 |
DMIS-KU-5 |
0.9167 |
0.9333 |
0.8889 |
0.9111 |
0.3462 |
0.6154 |
0.4647 |
0.5287 |
0.6272 |
0.5477 |
DICE1 |
0.6667 |
0.6923 |
0.6364 |
0.6643 |
0.3462 |
0.3462 |
0.3462 |
0.3593 |
0.3241 |
0.3096 |
DICE2 |
0.6250 |
0.6897 |
0.5263 |
0.6080 |
0.3077 |
0.3077 |
0.3077 |
0.2894 |
0.2980 |
0.2797 |
Ideal Answers
|
Automatic scores (Rouge - R) |
Manual scores |
System |
R-2 (Rec) |
R-2 (F1) |
R-SU4 (Rec) |
R-SU4 (F1) |
Readability |
Recall |
Precision |
Repetition |
UR-gpt3.5-t-simple |
0.2839 |
0.0701 |
0.3182 |
0.0740 |
- |
- |
- |
- |
UR-gpt4-simple |
0.3123 |
0.0735 |
0.3455 |
0.0773 |
- |
- |
- |
- |
UR-gpt3.5-turbo-zero |
0.5203 |
0.1631 |
0.5217 |
0.1522 |
- |
- |
- |
- |
UR-gpt4-zero-ret |
0.5541 |
0.1886 |
0.5436 |
0.1719 |
- |
- |
- |
- |
bioinfo-0 |
0.1802 |
0.1208 |
0.1957 |
0.1253 |
- |
- |
- |
- |
bioinfo-2 |
0.1815 |
0.1214 |
0.1992 |
0.1244 |
- |
- |
- |
- |
bioinfo-3 |
0.1487 |
0.0891 |
0.1723 |
0.0968 |
- |
- |
- |
- |
MQ-1 |
0.2461 |
0.2054 |
0.2488 |
0.2026 |
- |
- |
- |
- |
MQ-2 |
0.2437 |
0.2106 |
0.2503 |
0.2112 |
- |
- |
- |
- |
MQ-3 |
0.5057 |
0.3741 |
0.5068 |
0.3618 |
- |
- |
- |
- |
MQ-4 |
0.5456 |
0.3186 |
0.5425 |
0.3025 |
- |
- |
- |
- |
MQ-5 |
0.5265 |
0.3062 |
0.5264 |
0.2921 |
- |
- |
- |
- |
Deep ML methods for |
0.0696 |
0.1104 |
0.0495 |
0.0807 |
- |
- |
- |
- |
capstone-1 |
0.5456 |
0.3186 |
0.5425 |
0.3025 |
- |
- |
- |
- |
capstone-2 |
0.5456 |
0.3186 |
0.5425 |
0.3025 |
- |
- |
- |
- |
capstone-3 |
0.5456 |
0.3186 |
0.5425 |
0.3025 |
- |
- |
- |
- |
capstone-4 |
0.5456 |
0.3186 |
0.5425 |
0.3025 |
- |
- |
- |
- |
capstone-5 |
0.5456 |
0.3186 |
0.5425 |
0.3025 |
- |
- |
- |
- |
MindLab QA Reloaded |
0.0590 |
0.0889 |
0.0460 |
0.0719 |
- |
- |
- |
- |
IISR-1 |
0.3805 |
0.3449 |
0.3834 |
0.3420 |
- |
- |
- |
- |
IISR-2 |
0.3777 |
0.3233 |
0.3780 |
0.3169 |
- |
- |
- |
- |
IISR-3 |
0.3758 |
0.3307 |
0.3640 |
0.3139 |
- |
- |
- |
- |
AsqAway_1 |
- |
- |
- |
- |
- |
- |
- |
- |
MindLab QA System |
0.0465 |
0.0657 |
0.0398 |
0.0581 |
- |
- |
- |
- |
AsqAway_2 |
- |
- |
- |
- |
- |
- |
- |
- |
MindLab Red Lions++ |
0.0590 |
0.0889 |
0.0460 |
0.0719 |
- |
- |
- |
- |
AsqAway_3 |
- |
- |
- |
- |
- |
- |
- |
- |
AsqAway_4 |
- |
- |
- |
- |
- |
- |
- |
- |
kmeans |
0.1340 |
0.1175 |
0.1307 |
0.1126 |
- |
- |
- |
- |
extractive |
0.1407 |
0.1255 |
0.1356 |
0.1197 |
- |
- |
- |
- |
similarity measures |
0.1296 |
0.1174 |
0.1268 |
0.1132 |
- |
- |
- |
- |
abstractive |
0.1416 |
0.1283 |
0.1388 |
0.1236 |
- |
- |
- |
- |
simple truncation |
0.1475 |
0.1376 |
0.1417 |
0.1308 |
- |
- |
- |
- |
dmiip1 |
0.3519 |
0.1235 |
0.3760 |
0.1245 |
- |
- |
- |
- |
dmiip2 |
0.5173 |
0.2299 |
0.5238 |
0.2212 |
- |
- |
- |
- |
dmiip3 |
0.5256 |
0.2328 |
0.5323 |
0.2239 |
- |
- |
- |
- |
dmiip4 |
0.5132 |
0.2282 |
0.5156 |
0.2164 |
- |
- |
- |
- |
dmiip5 |
0.4406 |
0.3565 |
0.4451 |
0.3509 |
- |
- |
- |
- |
ELECTROLBERT-3 |
- |
- |
- |
- |
- |
- |
- |
- |
ELErank |
0.2939 |
0.2935 |
0.2841 |
0.2786 |
- |
- |
- |
- |
DMIS-KU-1 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-2 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-3 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-4 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-5 |
- |
- |
- |
- |
- |
- |
- |
- |
DICE1 |
- |
- |
- |
- |
- |
- |
- |
- |
DICE2 |
- |
- |
- |
- |
- |
- |
- |
- |
Test batch 4
Exact Answers
|
Yes/No |
Factoid |
List |
System |
Accuracy |
F1 Yes |
F1 No |
Macro F1 |
Strict Acc. |
Lenient Acc. |
MRR |
Mean Prec. |
Recall |
F-Measure |
UR-gpt3.5-t-simple |
0.4286 |
0.5000 |
0.3333 |
0.4167 |
0.2581 |
0.2903 |
0.2742 |
0.3944 |
0.3362 |
0.3470 |
UR-gpt4-simple |
0.7857 |
0.7273 |
0.8235 |
0.7754 |
0.2258 |
0.2581 |
0.2366 |
0.4440 |
0.4214 |
0.4127 |
UR-gpt3.5-turbo-zero |
0.9286 |
0.8571 |
0.9524 |
0.9048 |
0.6452 |
0.6452 |
0.6452 |
0.6090 |
0.6710 |
0.6196 |
UR-gpt4-zero-ret |
0.9286 |
0.8889 |
0.9474 |
0.9181 |
0.5161 |
0.6129 |
0.5645 |
0.6902 |
0.7818 |
0.7191 |
IISR-2 |
0.9286 |
0.8889 |
0.9474 |
0.9181 |
0.4194 |
0.4194 |
0.4194 |
0.6404 |
0.6620 |
0.6364 |
IISR-1 |
1.0000 |
1.0000 |
1.0000 |
1.0000 |
0.3871 |
0.4194 |
0.4032 |
0.7171 |
0.6479 |
0.6706 |
bioinfo-0 |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
bioinfo-1 |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
bioinfo-2 |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
bioinfo-3 |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
bioinfo-4 |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
dmiip1 |
0.2857 |
0.4444 |
- |
0.2222 |
0.4839 |
0.7742 |
0.5914 |
0.4778 |
0.4448 |
0.4460 |
dmiip2 |
0.7857 |
0.6667 |
0.8421 |
0.7544 |
0.4516 |
0.7419 |
0.5710 |
0.5575 |
0.5199 |
0.5073 |
dmiip3 |
0.7857 |
0.6667 |
0.8421 |
0.7544 |
0.5484 |
0.7742 |
0.6371 |
0.5528 |
0.4584 |
0.4784 |
dmiip4 |
0.7143 |
0.6667 |
0.7500 |
0.7083 |
0.4194 |
0.6774 |
0.5226 |
0.4871 |
0.4536 |
0.4424 |
dmiip5 |
1.0000 |
1.0000 |
1.0000 |
1.0000 |
0.4839 |
0.7419 |
0.5968 |
0.4689 |
0.4486 |
0.4380 |
MQ-5 |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
MQ-3 |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
MQ-1 |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
ELErank |
0.8571 |
0.7500 |
0.9000 |
0.8250 |
0.5161 |
0.7097 |
0.5978 |
- | - | - |
ELErank+ |
0.8571 |
0.7500 |
0.9000 |
0.8250 |
0.5161 |
0.7097 |
0.5978 |
- | - | - |
capstone-1 |
0.7857 |
0.6667 |
0.8421 |
0.7544 |
0.4839 |
0.6774 |
0.5522 |
0.4840 |
0.3068 |
0.3510 |
MQ-2 |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
MQ-4 |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
capstone-2 |
0.7857 |
0.6667 |
0.8421 |
0.7544 |
0.4839 |
0.6774 |
0.5522 |
0.4924 |
0.4429 |
0.4226 |
capstone-3 |
0.7857 |
0.6667 |
0.8421 |
0.7544 |
0.4839 |
0.6774 |
0.5522 |
0.4840 |
0.3068 |
0.3510 |
capstone-4 |
0.7857 |
0.6667 |
0.8421 |
0.7544 |
0.2581 |
0.4839 |
0.3548 |
0.4924 |
0.4429 |
0.4226 |
capstone-5 |
0.7857 |
0.6667 |
0.8421 |
0.7544 |
0.4839 |
0.6774 |
0.5522 |
0.4394 |
0.4498 |
0.4061 |
AUEB-System1 |
0.2857 |
0.4444 |
- |
0.2222 |
0.2581 |
0.3226 |
0.2849 |
0.0125 |
0.0197 |
0.0153 |
Deep ML methods for |
0.3571 |
0.4000 |
0.3077 |
0.3538 |
0.2258 |
0.2258 |
0.2258 |
0.2847 |
0.1144 |
0.1463 |
MindLab QA System |
0.3571 |
0.4000 |
0.3077 |
0.3538 |
0.2258 |
0.2258 |
0.2258 |
0.2847 |
0.1144 |
0.1463 |
DICE1 |
1.0000 |
1.0000 |
1.0000 |
1.0000 |
0.5484 |
0.5484 |
0.5484 |
0.5489 |
0.4997 |
0.5130 |
MindLab QA System ++ |
0.3571 |
0.4000 |
0.3077 |
0.3538 |
0.2258 |
0.2258 |
0.2258 |
0.2847 |
0.1144 |
0.1463 |
ELECTROLBERT-3 |
0.5714 |
0.4000 |
0.6667 |
0.5333 |
- | - | - |
- | - | - |
OWLMan-phaseB-TaskV1 |
0.4286 |
0.5000 |
0.3333 |
0.4167 |
0.4194 |
0.4839 |
0.4435 |
0.2417 |
0.3152 |
0.2572 |
DMIS-KU-1 |
1.0000 |
1.0000 |
1.0000 |
1.0000 |
0.6452 |
0.8710 |
0.7323 |
0.7139 |
0.8061 |
0.7440 |
DMIS-KU-2 |
0.9286 |
0.8889 |
0.9474 |
0.9181 |
0.6452 |
0.8387 |
0.7108 |
0.7271 |
0.6606 |
0.6806 |
DMIS-KU-3 |
0.9286 |
0.8571 |
0.9524 |
0.9048 |
0.5806 |
0.8387 |
0.6882 |
0.7331 |
0.6467 |
0.6752 |
DMIS-KU-4 |
0.8571 |
0.7500 |
0.9000 |
0.8250 |
0.5484 |
0.8387 |
0.6570 |
0.7734 |
0.6282 |
0.6747 |
DMIS-KU-5 |
0.9286 |
0.8889 |
0.9474 |
0.9181 |
0.5484 |
0.8065 |
0.6473 |
0.7234 |
0.6652 |
0.6787 |
IISR-3 |
0.2857 |
0.4444 |
- |
0.2222 |
0.4194 |
0.4194 |
0.4194 |
0.2417 |
0.1836 |
0.1969 |
AUEB-System2 |
0.2857 |
0.4444 |
- |
0.2222 |
0.1935 |
0.3226 |
0.2419 |
0.0396 |
0.0424 |
0.0395 |
IISR-4 |
1.0000 |
1.0000 |
1.0000 |
1.0000 |
0.4194 |
0.4194 |
0.4194 |
0.6667 |
0.5747 |
0.5956 |
DICE2 |
0.9286 |
0.8889 |
0.9474 |
0.9181 |
0.2581 |
0.2581 |
0.2581 |
- | - | - |
abstractive |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
DICE_Lab |
1.0000 |
1.0000 |
1.0000 |
1.0000 |
0.4839 |
0.4839 |
0.4839 |
0.4978 |
0.4989 |
0.4868 |
extractive |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
kmeans |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
similarity measures |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
simple truncation |
0.2857 |
0.4444 |
- |
0.2222 |
- | - | - |
- | - | - |
DICE_Lab2 |
0.9286 |
0.8889 |
0.9474 |
0.9181 |
0.5484 |
0.5484 |
0.5484 |
0.3743 |
0.3701 |
0.3650 |
IISR-5 |
0.9286 |
0.8889 |
0.9474 |
0.9181 |
0.4516 |
0.4516 |
0.4516 |
0.7134 |
0.6840 |
0.6807 |
Ideal Answers
|
Automatic scores (Rouge - R) |
Manual scores |
System |
R-2 (Rec) |
R-2 (F1) |
R-SU4 (Rec) |
R-SU4 (F1) |
Readability |
Recall |
Precision |
Repetition |
UR-gpt3.5-t-simple |
0.2678 |
0.0757 |
0.2938 |
0.0852 |
- |
- |
- |
- |
UR-gpt4-simple |
0.2869 |
0.0804 |
0.3138 |
0.0870 |
- |
- |
- |
- |
UR-gpt3.5-turbo-zero |
0.5381 |
0.2159 |
0.5235 |
0.2006 |
- |
- |
- |
- |
UR-gpt4-zero-ret |
0.5570 |
0.2500 |
0.5309 |
0.2320 |
- |
- |
- |
- |
IISR-2 |
0.3306 |
0.2999 |
0.3322 |
0.2932 |
- |
- |
- |
- |
IISR-1 |
0.3495 |
0.3419 |
0.3448 |
0.3311 |
- |
- |
- |
- |
bioinfo-0 |
0.1916 |
0.1586 |
0.2023 |
0.1581 |
- |
- |
- |
- |
bioinfo-1 |
0.2811 |
0.2433 |
0.2715 |
0.2292 |
- |
- |
- |
- |
bioinfo-2 |
0.1325 |
0.1076 |
0.1471 |
0.1110 |
- |
- |
- |
- |
bioinfo-3 |
0.2071 |
0.1582 |
0.2118 |
0.1570 |
- |
- |
- |
- |
bioinfo-4 |
0.1200 |
0.0892 |
0.1356 |
0.0962 |
- |
- |
- |
- |
dmiip1 |
0.5458 |
0.3864 |
0.5286 |
0.3658 |
- |
- |
- |
- |
dmiip2 |
0.5458 |
0.3864 |
0.5286 |
0.3658 |
- |
- |
- |
- |
dmiip3 |
0.5424 |
0.3825 |
0.5252 |
0.3618 |
- |
- |
- |
- |
dmiip4 |
0.5645 |
0.3905 |
0.5449 |
0.3682 |
- |
- |
- |
- |
dmiip5 |
0.3746 |
0.3488 |
0.3664 |
0.3361 |
- |
- |
- |
- |
MQ-5 |
0.5582 |
0.3889 |
0.5382 |
0.3671 |
- |
- |
- |
- |
MQ-3 |
0.5383 |
0.4084 |
0.5163 |
0.3840 |
- |
- |
- |
- |
MQ-1 |
0.4557 |
0.4019 |
0.4352 |
0.3768 |
- |
- |
- |
- |
ELErank |
0.3035 |
0.3021 |
0.2859 |
0.2838 |
- |
- |
- |
- |
ELErank+ |
0.3035 |
0.3021 |
0.2859 |
0.2838 |
- |
- |
- |
- |
capstone-1 |
0.5604 |
0.3919 |
0.5411 |
0.3702 |
- |
- |
- |
- |
MQ-2 |
0.4184 |
0.3954 |
0.4056 |
0.3778 |
- |
- |
- |
- |
MQ-4 |
0.5604 |
0.3919 |
0.5411 |
0.3702 |
- |
- |
- |
- |
capstone-2 |
0.5604 |
0.3919 |
0.5411 |
0.3702 |
- |
- |
- |
- |
capstone-3 |
0.6335 |
0.3761 |
0.6244 |
0.3569 |
- |
- |
- |
- |
capstone-4 |
0.6335 |
0.3761 |
0.6244 |
0.3569 |
- |
- |
- |
- |
capstone-5 |
0.5604 |
0.3919 |
0.5411 |
0.3702 |
- |
- |
- |
- |
AUEB-System1 |
- |
- |
- |
- |
- |
- |
- |
- |
Deep ML methods for |
0.0433 |
0.0619 |
0.0384 |
0.0578 |
- |
- |
- |
- |
MindLab QA System |
0.0776 |
0.1253 |
0.0537 |
0.0905 |
- |
- |
- |
- |
DICE1 |
- |
- |
- |
- |
- |
- |
- |
- |
MindLab QA System ++ |
0.0524 |
0.0823 |
0.0401 |
0.0661 |
- |
- |
- |
- |
ELECTROLBERT-3 |
- |
- |
- |
- |
- |
- |
- |
- |
OWLMan-phaseB-TaskV1 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-1 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-2 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-3 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-4 |
- |
- |
- |
- |
- |
- |
- |
- |
DMIS-KU-5 |
- |
- |
- |
- |
- |
- |
- |
- |
IISR-3 |
0.3330 |
0.3400 |
0.3140 |
0.3173 |
- |
- |
- |
- |
AUEB-System2 |
- |
- |
- |
- |
- |
- |
- |
- |
IISR-4 |
0.3447 |
0.3358 |
0.3388 |
0.3255 |
- |
- |
- |
- |
DICE2 |
- |
- |
- |
- |
- |
- |
- |
- |
abstractive |
0.1111 |
0.1173 |
0.1083 |
0.1135 |
- |
- |
- |
- |
DICE_Lab |
- |
- |
- |
- |
- |
- |
- |
- |
extractive |
0.1176 |
0.1216 |
0.1141 |
0.1171 |
- |
- |
- |
- |
kmeans |
0.1328 |
0.1320 |
0.1299 |
0.1282 |
- |
- |
- |
- |
similarity measures |
0.1002 |
0.1066 |
0.0976 |
0.1031 |
- |
- |
- |
- |
simple truncation |
0.1195 |
0.1269 |
0.1162 |
0.1225 |
- |
- |
- |
- |
DICE_Lab2 |
- |
- |
- |
- |
- |
- |
- |
- |
IISR-5 |
0.3451 |
0.3212 |
0.3405 |
0.3086 |
- |
- |
- |
- |