BioASQ Participants Area
Task 12b: Test Results of Phase A+
The test results are presented in separate tables for each type of annotation. The "System Description" of each system is used.
The evaluation measures that are used in Task A+ are presented
here .
Warning: For ideal answers, good ROUGE results do not always imply good manual scores.
Test batch 1
Exact Answers
|
Yes/No |
Factoid |
List |
System |
Accuracy |
F1 Yes |
F1 No |
Macro F1 |
Strict Acc. |
Lenient Acc. |
MRR |
Mean Prec. |
Recall |
F-Measure |
mibi_rag_snippet |
0.7600 |
0.8000 |
0.7000 |
0.7500 |
0.0476 |
0.0476 |
0.0476 |
0.2635 |
0.2106 |
0.2180 |
mibi_rag_abstract |
0.7200 |
0.7742 |
0.6316 |
0.7029 |
0.0476 |
0.0476 |
0.0476 |
0.3032 |
0.2673 |
0.2706 |
UR-IW-5 |
0.8000 |
0.8148 |
0.7826 |
0.7987 |
0.0952 |
0.0952 |
0.0952 |
0.4119 |
0.4182 |
0.3976 |
Fleming-1 |
0.8000 |
0.8387 |
0.7368 |
0.7878 |
- | - | - |
0.2186 |
0.2103 |
0.2079 |
GTBioASQsys2 |
0.8000 |
0.8148 |
0.7826 |
0.7987 |
0.0952 |
0.0952 |
0.0952 |
0.2722 |
0.2356 |
0.2350 |
Gatech competition |
0.8400 |
0.8333 |
0.8462 |
0.8397 |
0.1429 |
0.1429 |
0.1429 |
0.4452 |
0.3415 |
0.3661 |
GTBioASQsys3 |
0.8400 |
0.8462 |
0.8333 |
0.8397 |
0.1429 |
0.1429 |
0.1429 |
0.2421 |
0.1765 |
0.1866 |
UR-IW-4 |
0.8400 |
0.8462 |
0.8333 |
0.8397 |
0.0476 |
0.0952 |
0.0714 |
0.3948 |
0.4063 |
0.3798 |
UR-IW-2 |
0.8400 |
0.8462 |
0.8333 |
0.8397 |
0.0952 |
0.0952 |
0.0952 |
0.5250 |
0.4914 |
0.4808 |
bioinfo-0 |
0.5600 |
0.7179 |
- |
0.3590 |
- | - | - |
- | - | - |
UR-IW-3 |
0.9200 |
0.9333 |
0.9000 |
0.9167 |
0.0952 |
0.0952 |
0.0952 |
0.4016 |
0.4778 |
0.4089 |
UR-IW-1 |
0.8000 |
0.8276 |
0.7619 |
0.7947 |
0.1905 |
0.2381 |
0.2143 |
0.3224 |
0.4273 |
0.3418 |
bioinfo-1 |
0.5600 |
0.7179 |
- |
0.3590 |
- | - | - |
- | - | - |
bioinfo-2 |
0.5600 |
0.7179 |
- |
0.3590 |
- | - | - |
- | - | - |
bioinfo-3 |
0.5600 |
0.7179 |
- |
0.3590 |
- | - | - |
- | - | - |
bioinfo-4 |
0.5600 |
0.7179 |
- |
0.3590 |
- | - | - |
- | - | - |
dmiip2024 |
0.7600 |
0.8235 |
0.6250 |
0.7243 |
0.1905 |
0.1905 |
0.1905 |
0.4960 |
0.4269 |
0.4471 |
dmiip2024_1 |
0.7600 |
0.8235 |
0.6250 |
0.7243 |
0.2381 |
0.5238 |
0.3349 |
0.3317 |
0.3591 |
0.3109 |
dmiip2024_3 |
0.7600 |
0.8235 |
0.6250 |
0.7243 |
0.2381 |
0.5238 |
0.3611 |
0.3341 |
0.3746 |
0.3315 |
dmiip2024_2 |
0.7600 |
0.8235 |
0.6250 |
0.7243 |
0.2381 |
0.3810 |
0.2730 |
0.3270 |
0.3591 |
0.2993 |
dmiip2024_4 |
0.7600 |
0.8235 |
0.6250 |
0.7243 |
0.0952 |
0.2857 |
0.1683 |
0.2651 |
0.2810 |
0.2540 |
simple truncation |
0.8000 |
0.8485 |
0.7059 |
0.7772 |
0.0952 |
0.1429 |
0.1190 |
0.1733 |
0.2046 |
0.1760 |
Ideal Answers
|
Automatic scores (Rouge - R) |
Manual scores |
System |
R-2 (Rec) |
R-2 (F1) |
R-SU4 (Rec) |
R-SU4 (F1) |
Readability |
Recall |
Precision |
Repetition |
mibi_rag_snippet |
0.2539 |
0.1447 |
0.2784 |
0.1429 |
- |
- |
- |
- |
mibi_rag_abstract |
0.2584 |
0.1586 |
0.2807 |
0.1572 |
- |
- |
- |
- |
UR-IW-5 |
0.2280 |
0.1065 |
0.2557 |
0.1073 |
- |
- |
- |
- |
Fleming-1 |
0.2158 |
0.0704 |
0.2552 |
0.0747 |
- |
- |
- |
- |
GTBioASQsys2 |
0.1655 |
0.1078 |
0.1861 |
0.1091 |
- |
- |
- |
- |
Gatech competition |
0.1474 |
0.1080 |
0.1624 |
0.1112 |
- |
- |
- |
- |
GTBioASQsys3 |
0.1464 |
0.1020 |
0.1778 |
0.1042 |
- |
- |
- |
- |
UR-IW-4 |
0.2516 |
0.1201 |
0.2752 |
0.1193 |
- |
- |
- |
- |
UR-IW-2 |
0.2356 |
0.2345 |
0.2428 |
0.2308 |
- |
- |
- |
- |
bioinfo-0 |
0.2962 |
0.0671 |
0.3321 |
0.0679 |
- |
- |
- |
- |
UR-IW-3 |
0.2338 |
0.2098 |
0.2505 |
0.2044 |
- |
- |
- |
- |
UR-IW-1 |
0.2393 |
0.1254 |
0.2642 |
0.1228 |
- |
- |
- |
- |
bioinfo-1 |
0.2939 |
0.0735 |
0.3343 |
0.0746 |
- |
- |
- |
- |
bioinfo-2 |
0.2756 |
0.0736 |
0.3139 |
0.0774 |
- |
- |
- |
- |
bioinfo-3 |
0.3007 |
0.0721 |
0.3336 |
0.0730 |
- |
- |
- |
- |
bioinfo-4 |
0.2861 |
0.0657 |
0.3324 |
0.0693 |
- |
- |
- |
- |
dmiip2024 |
0.1674 |
0.1485 |
0.1818 |
0.1498 |
- |
- |
- |
- |
dmiip2024_1 |
0.1674 |
0.1485 |
0.1818 |
0.1498 |
- |
- |
- |
- |
dmiip2024_3 |
0.1674 |
0.1485 |
0.1818 |
0.1498 |
- |
- |
- |
- |
dmiip2024_2 |
0.1674 |
0.1485 |
0.1818 |
0.1498 |
- |
- |
- |
- |
dmiip2024_4 |
0.1674 |
0.1485 |
0.1818 |
0.1498 |
- |
- |
- |
- |
simple truncation |
0.0938 |
0.0723 |
0.1064 |
0.0726 |
- |
- |
- |
- |
Test batch 2
Exact Answers
|
Yes/No |
Factoid |
List |
System |
Accuracy |
F1 Yes |
F1 No |
Macro F1 |
Strict Acc. |
Lenient Acc. |
MRR |
Mean Prec. |
Recall |
F-Measure |
mibi_rag_snippet |
0.7692 |
0.8125 |
0.7000 |
0.7563 |
0.1579 |
0.1579 |
0.1579 |
0.3306 |
0.2830 |
0.3005 |
mibi_rag_abstract |
0.6923 |
0.7500 |
0.6000 |
0.6750 |
0.1053 |
0.1053 |
0.1053 |
0.2769 |
0.1967 |
0.2140 |
Gatech competition |
0.8077 |
0.8276 |
0.7826 |
0.8051 |
0.2105 |
0.2105 |
0.2105 |
0.2290 |
0.2310 |
0.2133 |
GTBioASQsys2 |
0.6923 |
0.7143 |
0.6667 |
0.6905 |
0.2105 |
0.2105 |
0.2105 |
0.1549 |
0.1260 |
0.1268 |
GTBioASQsys3 |
0.8077 |
0.8387 |
0.7619 |
0.8003 |
0.2105 |
0.2105 |
0.2105 |
0.1376 |
0.1481 |
0.1364 |
UR-IW-5 |
0.8846 |
0.8966 |
0.8696 |
0.8831 |
0.3158 |
0.3158 |
0.3158 |
0.1589 |
0.1725 |
0.1497 |
UR-IW-4 |
0.8462 |
0.8571 |
0.8333 |
0.8452 |
0.1579 |
0.2105 |
0.1842 |
0.2628 |
0.2299 |
0.2179 |
UR-IW-3 |
0.8846 |
0.8966 |
0.8696 |
0.8831 |
0.3158 |
0.3158 |
0.3158 |
0.2625 |
0.2400 |
0.2411 |
UR-IW-2 |
0.8462 |
0.8571 |
0.8333 |
0.8452 |
0.2632 |
0.3158 |
0.2895 |
0.2045 |
0.2569 |
0.2182 |
simple truncation |
0.7692 |
0.8235 |
0.6667 |
0.7451 |
0.1579 |
0.1579 |
0.1579 |
0.0773 |
0.0662 |
0.0675 |
kmeans |
0.7692 |
0.8125 |
0.7000 |
0.7563 |
0.1579 |
0.1579 |
0.1579 |
0.0930 |
0.0935 |
0.0894 |
similarity measures |
0.7692 |
0.8235 |
0.6667 |
0.7451 |
0.1579 |
0.1579 |
0.1579 |
0.0773 |
0.0662 |
0.0675 |
UR-IW-1 |
0.7692 |
0.8000 |
0.7273 |
0.7636 |
0.2632 |
0.2632 |
0.2632 |
0.1953 |
0.1906 |
0.1766 |
Fleming-3 |
0.8077 |
0.8485 |
0.7368 |
0.7927 |
0.2632 |
0.3684 |
0.3070 |
0.2335 |
0.1478 |
0.1708 |
dmiip2024 |
0.9615 |
0.9677 |
0.9524 |
0.9601 |
0.2632 |
0.4737 |
0.3596 |
0.4299 |
0.4543 |
0.4074 |
dmiip2024_1 |
0.8077 |
0.8387 |
0.7619 |
0.8003 |
0.2632 |
0.3684 |
0.3158 |
0.4470 |
0.4451 |
0.4088 |
dmiip2024_2 |
0.8846 |
0.9143 |
0.8235 |
0.8689 |
0.1579 |
0.3158 |
0.2237 |
0.3520 |
0.3935 |
0.3230 |
dmiip2024_4 |
0.3846 |
- |
0.5556 |
0.2778 |
0.1579 |
0.4211 |
0.2807 |
0.2685 |
0.3199 |
0.2606 |
dmiip2024_3 |
0.8846 |
0.9143 |
0.8235 |
0.8689 |
0.3684 |
0.4211 |
0.3947 |
0.3769 |
0.3793 |
0.3542 |
bioinfo-0 |
0.6154 |
0.7619 |
- |
0.3810 |
- | - | - |
- | - | - |
bioinfo-1 |
0.6154 |
0.7619 |
- |
0.3810 |
- | - | - |
- | - | - |
bioinfo-2 |
0.6154 |
0.7619 |
- |
0.3810 |
- | - | - |
- | - | - |
bioinfo-3 |
0.6154 |
0.7619 |
- |
0.3810 |
- | - | - |
- | - | - |
bioinfo-4 |
0.6154 |
0.7619 |
- |
0.3810 |
- | - | - |
- | - | - |
CPS |
0.6923 |
0.7895 |
0.4286 |
0.6090 |
- | - | - |
- | - | - |
CPS2 |
0.6923 |
0.7778 |
0.5000 |
0.6389 |
- | - | - |
- | - | - |
Ideal Answers
|
Automatic scores (Rouge - R) |
Manual scores |
System |
R-2 (Rec) |
R-2 (F1) |
R-SU4 (Rec) |
R-SU4 (F1) |
Readability |
Recall |
Precision |
Repetition |
mibi_rag_snippet |
0.2181 |
0.1199 |
0.2345 |
0.1200 |
- |
- |
- |
- |
mibi_rag_abstract |
0.2098 |
0.1095 |
0.2394 |
0.1127 |
- |
- |
- |
- |
Gatech competition |
0.1225 |
0.0879 |
0.1339 |
0.0925 |
- |
- |
- |
- |
GTBioASQsys2 |
0.1459 |
0.1018 |
0.1703 |
0.1101 |
- |
- |
- |
- |
GTBioASQsys3 |
0.1148 |
0.0946 |
0.1342 |
0.1002 |
- |
- |
- |
- |
UR-IW-5 |
0.2234 |
0.1002 |
0.2450 |
0.1024 |
- |
- |
- |
- |
UR-IW-4 |
0.2181 |
0.1066 |
0.2514 |
0.1108 |
- |
- |
- |
- |
UR-IW-3 |
0.1980 |
0.1785 |
0.2132 |
0.1815 |
- |
- |
- |
- |
UR-IW-2 |
0.1890 |
0.1747 |
0.1989 |
0.1783 |
- |
- |
- |
- |
simple truncation |
0.0525 |
0.0166 |
0.0591 |
0.0172 |
- |
- |
- |
- |
kmeans |
0.0850 |
0.0352 |
0.0986 |
0.0382 |
- |
- |
- |
- |
similarity measures |
0.0525 |
0.0166 |
0.0591 |
0.0172 |
- |
- |
- |
- |
UR-IW-1 |
0.1892 |
0.1033 |
0.2295 |
0.1076 |
- |
- |
- |
- |
Fleming-3 |
0.2033 |
0.0682 |
0.2411 |
0.0737 |
- |
- |
- |
- |
dmiip2024 |
0.1741 |
0.1604 |
0.1831 |
0.1641 |
- |
- |
- |
- |
dmiip2024_1 |
0.1724 |
0.1548 |
0.1818 |
0.1596 |
- |
- |
- |
- |
dmiip2024_2 |
0.1809 |
0.1661 |
0.1969 |
0.1680 |
- |
- |
- |
- |
dmiip2024_4 |
0.1594 |
0.1485 |
0.1745 |
0.1531 |
- |
- |
- |
- |
dmiip2024_3 |
0.1497 |
0.1265 |
0.1710 |
0.1374 |
- |
- |
- |
- |
bioinfo-0 |
0.2459 |
0.0762 |
0.2885 |
0.0815 |
- |
- |
- |
- |
bioinfo-1 |
0.2500 |
0.0771 |
0.2898 |
0.0806 |
- |
- |
- |
- |
bioinfo-2 |
0.2498 |
0.0721 |
0.2927 |
0.0778 |
- |
- |
- |
- |
bioinfo-3 |
0.2187 |
0.1898 |
0.2292 |
0.1898 |
- |
- |
- |
- |
bioinfo-4 |
0.1608 |
0.1373 |
0.1706 |
0.1381 |
- |
- |
- |
- |
CPS |
0.1729 |
0.1235 |
0.1949 |
0.1272 |
- |
- |
- |
- |
CPS2 |
0.1569 |
0.0861 |
0.1737 |
0.0904 |
- |
- |
- |
- |
Test batch 3
Exact Answers
|
Yes/No |
Factoid |
List |
System |
Accuracy |
F1 Yes |
F1 No |
Macro F1 |
Strict Acc. |
Lenient Acc. |
MRR |
Mean Prec. |
Recall |
F-Measure |
bioinfo-0 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
bioinfo-1 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
bioinfo-2 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
bioinfo-3 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
bioinfo-4 |
0.5833 |
0.7368 |
- |
0.3684 |
- | - | - |
- | - | - |
GTBioASQsys2 |
0.6667 |
0.6923 |
0.6364 |
0.6643 |
0.0769 |
0.0769 |
0.0769 |
0.2132 |
0.2430 |
0.2098 |
GTBioASQsys4 |
0.7917 |
0.8148 |
0.7619 |
0.7884 |
0.1154 |
0.1154 |
0.1154 |
0.1018 |
0.1583 |
0.1196 |
Gatech competition |
0.7917 |
0.8276 |
0.7368 |
0.7822 |
0.2308 |
0.2308 |
0.2308 |
0.1782 |
0.2133 |
0.1774 |
GTBioASQsys3 |
0.7500 |
0.7692 |
0.7273 |
0.7483 |
0.1538 |
0.1538 |
0.1538 |
0.1877 |
0.2064 |
0.1860 |
Fleming-3 |
0.7500 |
0.7857 |
0.7000 |
0.7429 |
0.0769 |
0.2308 |
0.1250 |
0.1775 |
0.1702 |
0.1643 |
mibi_rag_abstract |
0.4583 |
0.3158 |
0.5517 |
0.4338 |
0.3077 |
0.3077 |
0.3077 |
0.2673 |
0.2540 |
0.2554 |
mibi_rag_snippet |
0.8750 |
0.8966 |
0.8421 |
0.8693 |
0.1538 |
0.1538 |
0.1538 |
0.2561 |
0.2434 |
0.2410 |
mibi_rag_3 |
0.8750 |
0.8966 |
0.8421 |
0.8693 |
0.1538 |
0.1538 |
0.1538 |
0.2561 |
0.2434 |
0.2410 |
mibi_rag_4 |
0.5417 |
0.4762 |
0.5926 |
0.5344 |
0.2692 |
0.2692 |
0.2692 |
0.3006 |
0.2803 |
0.2851 |
mibi_rag_5 |
0.8333 |
0.8571 |
0.8000 |
0.8286 |
0.1538 |
0.1538 |
0.1538 |
0.2491 |
0.2401 |
0.2364 |
CPS |
0.6250 |
0.7273 |
0.4000 |
0.5636 |
0.1154 |
0.1154 |
0.1154 |
0.1140 |
0.0965 |
0.1009 |
CPS2 |
0.5833 |
0.7059 |
0.2857 |
0.4958 |
- | - | - |
- | - | - |
CPS3 |
0.6667 |
0.7500 |
0.5000 |
0.6250 |
- | - | - |
- | - | - |
UR-IW-1 |
0.8333 |
0.8667 |
0.7778 |
0.8222 |
0.1923 |
0.3077 |
0.2340 |
0.2657 |
0.4232 |
0.3000 |
UR-IW-3 |
0.7917 |
0.8000 |
0.7826 |
0.7913 |
0.1538 |
0.1538 |
0.1538 |
0.1373 |
0.2326 |
0.1627 |
UR-IW-4 |
0.7917 |
0.8148 |
0.7619 |
0.7884 |
0.1923 |
0.2308 |
0.2019 |
0.2014 |
0.2655 |
0.2186 |
UR-IW-5 |
0.9167 |
0.9286 |
0.9000 |
0.9143 |
0.1538 |
0.1538 |
0.1538 |
0.2208 |
0.2881 |
0.2392 |
UR-IW-2 |
0.8333 |
0.8462 |
0.8182 |
0.8322 |
0.1538 |
0.1923 |
0.1731 |
0.2125 |
0.2892 |
0.2303 |
dmiip2024_1 |
0.8750 |
0.8889 |
0.8571 |
0.8730 |
0.3077 |
0.3462 |
0.3269 |
0.3706 |
0.3988 |
0.3571 |
dmiip2024 |
0.8750 |
0.8889 |
0.8571 |
0.8730 |
0.2692 |
0.3462 |
0.3077 |
0.3487 |
0.3502 |
0.3252 |
dmiip2024_2 |
0.8750 |
0.8966 |
0.8421 |
0.8693 |
0.2692 |
0.4231 |
0.3301 |
0.2639 |
0.4094 |
0.2684 |
dmiip2024_3 |
0.8750 |
0.8966 |
0.8421 |
0.8693 |
0.2308 |
0.3077 |
0.2628 |
0.3750 |
0.4069 |
0.3708 |
dmiip2024_4 |
0.4167 |
- |
0.5882 |
0.2941 |
0.3077 |
0.3462 |
0.3269 |
0.1945 |
0.3490 |
0.2118 |
Ideal Answers
|
Automatic scores (Rouge - R) |
Manual scores |
System |
R-2 (Rec) |
R-2 (F1) |
R-SU4 (Rec) |
R-SU4 (F1) |
Readability |
Recall |
Precision |
Repetition |
bioinfo-0 |
0.3314 |
0.1135 |
0.3473 |
0.1121 |
- |
- |
- |
- |
bioinfo-1 |
0.3077 |
0.1047 |
0.3283 |
0.1044 |
- |
- |
- |
- |
bioinfo-2 |
0.3230 |
0.1085 |
0.3404 |
0.1075 |
- |
- |
- |
- |
bioinfo-3 |
0.2286 |
0.1197 |
0.2445 |
0.1226 |
- |
- |
- |
- |
bioinfo-4 |
0.2544 |
0.1320 |
0.2711 |
0.1355 |
- |
- |
- |
- |
GTBioASQsys2 |
0.2033 |
0.1404 |
0.1982 |
0.1338 |
- |
- |
- |
- |
GTBioASQsys4 |
0.1943 |
0.1488 |
0.1902 |
0.1444 |
- |
- |
- |
- |
Gatech competition |
0.2105 |
0.1603 |
0.2110 |
0.1579 |
- |
- |
- |
- |
GTBioASQsys3 |
0.1795 |
0.1586 |
0.1840 |
0.1589 |
- |
- |
- |
- |
Fleming-3 |
0.2866 |
0.0752 |
0.3196 |
0.0812 |
- |
- |
- |
- |
mibi_rag_abstract |
0.2267 |
0.2006 |
0.2268 |
0.2012 |
- |
- |
- |
- |
mibi_rag_snippet |
0.2279 |
0.1899 |
0.2306 |
0.1878 |
- |
- |
- |
- |
mibi_rag_3 |
0.2197 |
0.1860 |
0.2216 |
0.1833 |
- |
- |
- |
- |
mibi_rag_4 |
0.2416 |
0.1958 |
0.2501 |
0.1978 |
- |
- |
- |
- |
mibi_rag_5 |
0.2199 |
0.1888 |
0.2204 |
0.1847 |
- |
- |
- |
- |
CPS |
0.2335 |
0.1892 |
0.2339 |
0.1870 |
- |
- |
- |
- |
CPS2 |
0.2414 |
0.1955 |
0.2452 |
0.1946 |
- |
- |
- |
- |
CPS3 |
0.2196 |
0.1900 |
0.2238 |
0.1904 |
- |
- |
- |
- |
UR-IW-1 |
0.2513 |
0.1352 |
0.2619 |
0.1352 |
- |
- |
- |
- |
UR-IW-3 |
0.2708 |
0.1093 |
0.2937 |
0.1125 |
- |
- |
- |
- |
UR-IW-4 |
0.3027 |
0.1438 |
0.3216 |
0.1447 |
- |
- |
- |
- |
UR-IW-5 |
0.2836 |
0.1078 |
0.3054 |
0.1131 |
- |
- |
- |
- |
UR-IW-2 |
0.3138 |
0.1454 |
0.3334 |
0.1470 |
- |
- |
- |
- |
dmiip2024_1 |
0.2268 |
0.2112 |
0.2411 |
0.2160 |
- |
- |
- |
- |
dmiip2024 |
0.2418 |
0.2266 |
0.2421 |
0.2249 |
- |
- |
- |
- |
dmiip2024_2 |
0.2589 |
0.2433 |
0.2524 |
0.2370 |
- |
- |
- |
- |
dmiip2024_3 |
0.1898 |
0.1847 |
0.1981 |
0.1899 |
- |
- |
- |
- |
dmiip2024_4 |
0.2322 |
0.2192 |
0.2288 |
0.2160 |
- |
- |
- |
- |
Test batch 4
Exact Answers
|
Yes/No |
Factoid |
List |
System |
Accuracy |
F1 Yes |
F1 No |
Macro F1 |
Strict Acc. |
Lenient Acc. |
MRR |
Mean Prec. |
Recall |
F-Measure |
UR-IW-1 |
0.8519 |
0.8947 |
0.7500 |
0.8224 |
0.3158 |
0.4737 |
0.3816 |
0.1529 |
0.2641 |
0.1774 |
UR-IW-3 |
0.7037 |
0.7647 |
0.6000 |
0.6824 |
0.2105 |
0.3158 |
0.2412 |
0.1191 |
0.1239 |
0.1155 |
UR-IW-4 |
0.7778 |
0.8333 |
0.6667 |
0.7500 |
0.1579 |
0.2632 |
0.2018 |
0.1364 |
0.1845 |
0.1418 |
UR-IW-5 |
0.7407 |
0.8000 |
0.6316 |
0.7158 |
0.3684 |
0.3684 |
0.3684 |
0.1125 |
0.1610 |
0.1269 |
Fleming-1 |
0.8148 |
0.8571 |
0.7368 |
0.7970 |
0.1053 |
0.1579 |
0.1158 |
0.1424 |
0.1850 |
0.1494 |
mibi_rag_snippet |
0.4444 |
0.5161 |
0.3478 |
0.4320 |
0.1053 |
0.1053 |
0.1053 |
0.1484 |
0.1486 |
0.1473 |
mibi_rag_abstract |
0.3704 |
0.1905 |
0.4848 |
0.3377 |
0.2105 |
0.2105 |
0.2105 |
0.2318 |
0.1910 |
0.2062 |
mibi_rag_3 |
0.4444 |
0.5161 |
0.3478 |
0.4320 |
0.1053 |
0.1053 |
0.1053 |
0.1484 |
0.1486 |
0.1473 |
mibi_rag_4 |
0.3704 |
0.1905 |
0.4848 |
0.3377 |
0.1579 |
0.1579 |
0.1579 |
0.2386 |
0.2202 |
0.2196 |
mibi_rag_5 |
0.4815 |
0.5333 |
0.4167 |
0.4750 |
0.1053 |
0.1053 |
0.1053 |
0.1484 |
0.1486 |
0.1473 |
bioinfo-0 |
0.7037 |
0.8261 |
- |
0.4130 |
- | - | - |
- | - | - |
bioinfo-1 |
0.7037 |
0.8261 |
- |
0.4130 |
- | - | - |
- | - | - |
bioinfo-2 |
0.7037 |
0.8261 |
- |
0.4130 |
- | - | - |
- | - | - |
bioinfo-3 |
0.7037 |
0.8261 |
- |
0.4130 |
- | - | - |
- | - | - |
bioinfo-4 |
0.7037 |
0.8261 |
- |
0.4130 |
- | - | - |
- | - | - |
GTBioASQsys2 |
0.6296 |
0.6875 |
0.5455 |
0.6165 |
0.1053 |
0.1053 |
0.1053 |
0.0840 |
0.1098 |
0.0928 |
GTBioASQsys3 |
0.7037 |
0.7647 |
0.6000 |
0.6824 |
0.0526 |
0.0526 |
0.0526 |
0.0811 |
0.1019 |
0.0816 |
GTBioASQsys4 |
0.6667 |
0.7097 |
0.6087 |
0.6592 |
0.1579 |
0.1579 |
0.1579 |
0.1338 |
0.1531 |
0.1353 |
UR-IW-2 |
0.8519 |
0.8947 |
0.7500 |
0.8224 |
0.3158 |
0.3684 |
0.3421 |
0.1161 |
0.2069 |
0.1366 |
CPS |
0.7778 |
0.8571 |
0.5000 |
0.6786 |
0.0526 |
0.0526 |
0.0526 |
0.1076 |
0.1038 |
0.1037 |
CPS2 |
0.8148 |
0.8837 |
0.5455 |
0.7146 |
- | - | - |
- | - | - |
CPS3 |
0.8148 |
0.8837 |
0.5455 |
0.7146 |
- | - | - |
- | - | - |
dmiip2024_3 |
0.8148 |
0.8649 |
0.7059 |
0.7854 |
0.3158 |
0.3158 |
0.3158 |
0.2368 |
0.2747 |
0.2461 |
Fleming-2 |
0.7778 |
0.8421 |
0.6250 |
0.7336 |
0.1053 |
0.1579 |
0.1158 |
0.1424 |
0.1850 |
0.1494 |
dmiip2024 |
0.8148 |
0.8571 |
0.7368 |
0.7970 |
0.2632 |
0.4211 |
0.3333 |
0.2704 |
0.3497 |
0.2970 |
dmiip2024_1 |
0.8889 |
0.9231 |
0.8000 |
0.8615 |
0.3684 |
0.4211 |
0.3947 |
0.3139 |
0.3433 |
0.3219 |
dmiip2024_2 |
0.8889 |
0.9189 |
0.8235 |
0.8712 |
0.3158 |
0.3158 |
0.3158 |
0.1966 |
0.3815 |
0.2460 |
dmiip2024_4 |
0.2963 |
- |
0.4571 |
0.2286 |
0.1579 |
0.2632 |
0.2018 |
0.1599 |
0.3027 |
0.2002 |
extractive |
0.8148 |
0.8649 |
0.7059 |
0.7854 |
- | - | - |
- | - | - |
Ideal Answers
|
Automatic scores (Rouge - R) |
Manual scores |
System |
R-2 (Rec) |
R-2 (F1) |
R-SU4 (Rec) |
R-SU4 (F1) |
Readability |
Recall |
Precision |
Repetition |
UR-IW-1 |
0.1979 |
0.1058 |
0.2184 |
0.1087 |
- |
- |
- |
- |
UR-IW-3 |
0.2114 |
0.0919 |
0.2359 |
0.0970 |
- |
- |
- |
- |
UR-IW-4 |
0.2203 |
0.1046 |
0.2435 |
0.1108 |
- |
- |
- |
- |
UR-IW-5 |
0.2423 |
0.1112 |
0.2539 |
0.1110 |
- |
- |
- |
- |
Fleming-1 |
0.2549 |
0.0650 |
0.2769 |
0.0696 |
- |
- |
- |
- |
mibi_rag_snippet |
0.2279 |
0.1815 |
0.2395 |
0.1861 |
- |
- |
- |
- |
mibi_rag_abstract |
0.1882 |
0.1716 |
0.1972 |
0.1774 |
- |
- |
- |
- |
mibi_rag_3 |
0.2285 |
0.1814 |
0.2396 |
0.1858 |
- |
- |
- |
- |
mibi_rag_4 |
0.1954 |
0.1629 |
0.2061 |
0.1683 |
- |
- |
- |
- |
mibi_rag_5 |
0.2061 |
0.1668 |
0.2202 |
0.1733 |
- |
- |
- |
- |
bioinfo-0 |
0.2961 |
0.0937 |
0.3219 |
0.0983 |
- |
- |
- |
- |
bioinfo-1 |
0.2757 |
0.0866 |
0.2984 |
0.0913 |
- |
- |
- |
- |
bioinfo-2 |
0.3037 |
0.1055 |
0.3261 |
0.1096 |
- |
- |
- |
- |
bioinfo-3 |
0.2118 |
0.1097 |
0.2311 |
0.1177 |
- |
- |
- |
- |
bioinfo-4 |
0.2029 |
0.1037 |
0.2219 |
0.1103 |
- |
- |
- |
- |
GTBioASQsys2 |
0.1530 |
0.1080 |
0.1676 |
0.1163 |
- |
- |
- |
- |
GTBioASQsys3 |
0.1256 |
0.0945 |
0.1342 |
0.0999 |
- |
- |
- |
- |
GTBioASQsys4 |
0.1253 |
0.0893 |
0.1358 |
0.0967 |
- |
- |
- |
- |
UR-IW-2 |
0.2267 |
0.1441 |
0.2386 |
0.1465 |
- |
- |
- |
- |
CPS |
0.2158 |
0.1340 |
0.2435 |
0.1439 |
- |
- |
- |
- |
CPS2 |
0.2224 |
0.1376 |
0.2517 |
0.1495 |
- |
- |
- |
- |
CPS3 |
0.1954 |
0.1600 |
0.2084 |
0.1642 |
- |
- |
- |
- |
dmiip2024_3 |
0.2783 |
0.2461 |
0.2795 |
0.2431 |
- |
- |
- |
- |
Fleming-2 |
0.2549 |
0.0650 |
0.2769 |
0.0696 |
- |
- |
- |
- |
dmiip2024 |
0.2269 |
0.1911 |
0.2343 |
0.1944 |
- |
- |
- |
- |
dmiip2024_1 |
0.2386 |
0.2101 |
0.2527 |
0.2174 |
- |
- |
- |
- |
dmiip2024_2 |
0.2386 |
0.2101 |
0.2527 |
0.2174 |
- |
- |
- |
- |
dmiip2024_4 |
0.2068 |
0.1915 |
0.2131 |
0.1957 |
- |
- |
- |
- |
extractive |
- |
- |
- |
- |
- |
- |
- |
- |