BioASQ Participants Area
Oracle: Results for task B-Phase B
The test results are presented in seperate tables for each type of annotation. The "System Description" of each system is used.
The evaluation measures that are used in Task B are presented
here .
+ Task 1b, Test batch 1
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
Wishart-S1 |
0.9200 |
0.2222 |
0.2778 |
0.2778 |
0.3186 |
0.2147 |
0.2290 |
Wishart-S2 |
0.9200 |
0.2222 |
0.2778 |
0.2778 |
0.3186 |
0.2147 |
0.2290 |
Wishart-S3 |
0.9200 |
0.2222 |
0.3333 |
0.3056 |
0.3067 |
0.2082 |
0.2207 |
main system |
0.3200 |
- | - | - |
0.0037 |
0.0394 |
0.0066 |
BioASQ_Baseline |
0.4400 |
- | - | - |
0.0153 |
0.0402 |
0.0204 |
BioASQ Baseline 2 |
0.4800 |
- | - | - |
0.0153 |
0.0402 |
0.0204 |
HPI-S2 |
- |
- | - | - |
- | - | - |
UNCC System 1 |
0.8400 |
- | - | - |
0.0115 |
0.1499 |
0.0209 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
Wishart-S1 |
0.2059 |
0.1463 |
0.2202 |
0.1522 |
Wishart-S2 |
0.2059 |
0.1463 |
0.2202 |
0.1522 |
Wishart-S3 |
0.2059 |
0.1463 |
0.2202 |
0.1522 |
main system |
0.2165 |
0.1231 |
0.2396 |
0.1309 |
BioASQ_Baseline |
0.2266 |
0.0735 |
0.2636 |
0.0843 |
BioASQ Baseline 2 |
0.1935 |
0.0628 |
0.2305 |
0.0745 |
HPI-S2 |
0.0872 |
0.1093 |
0.0915 |
0.1155 |
UNCC System 1 |
0.3032 |
0.1266 |
0.3276 |
0.1329 |
+ Task 1b, Test batch 2
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
main system |
0.4231 |
- | - | - |
0.0603 |
0.1040 |
0.0680 |
Wishart-S1 |
0.9615 |
0.2500 |
0.3000 |
0.3000 |
0.4060 |
0.3127 |
0.3336 |
system 2 |
0.4231 |
- | - | - |
0.0437 |
0.0445 |
0.0414 |
system 3 |
0.4231 |
- | - | - |
0.0644 |
0.0440 |
0.0488 |
system 4 |
0.4231 |
- | - | - |
0.0644 |
0.0440 |
0.0488 |
BioASQ_Baseline |
0.2692 |
- | - | - |
0.0612 |
0.2062 |
0.0789 |
BioASQ Baseline 2 |
0.5000 |
- | - | - |
0.0612 |
0.2062 |
0.0789 |
UNCC System 1 |
0.8077 |
- | - | - |
0.0065 |
0.1243 |
0.0120 |
lalala |
- |
- | - | - |
- | - | - |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
main system |
0.2122 |
0.0795 |
0.2596 |
0.0890 |
Wishart-S1 |
0.2106 |
0.1197 |
0.2387 |
0.1292 |
system 2 |
0.2204 |
0.0826 |
0.2659 |
0.0899 |
system 3 |
0.2152 |
0.0809 |
0.2592 |
0.0886 |
system 4 |
0.2152 |
0.0809 |
0.2592 |
0.0886 |
BioASQ_Baseline |
0.2074 |
0.0573 |
0.2533 |
0.0675 |
BioASQ Baseline 2 |
0.2052 |
0.0563 |
0.2577 |
0.0663 |
UNCC System 1 |
0.3319 |
0.1079 |
0.3596 |
0.1114 |
lalala |
- | - |
+ Task 1b, Test batch 3
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
main system |
0.5769 |
- | - | - |
0.0671 |
0.1013 |
0.0727 |
system 2 |
0.5769 |
- | - | - |
0.0642 |
0.0580 |
0.0575 |
system 3 |
0.5769 |
- | - | - |
0.0594 |
0.0399 |
0.0450 |
BioASQ_Baseline |
0.6538 |
- | - | - |
0.0209 |
0.0635 |
0.0278 |
BioASQ Baseline 2 |
0.6154 |
- | - | - |
0.0209 |
0.0635 |
0.0278 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
main system |
0.2563 |
0.0885 |
0.2828 |
0.0912 |
system 2 |
0.2505 |
0.0876 |
0.2806 |
0.0910 |
system 3 |
0.2555 |
0.0884 |
0.2835 |
0.0916 |
BioASQ_Baseline |
0.2670 |
0.0558 |
0.2982 |
0.0596 |
BioASQ Baseline 2 |
0.2547 |
0.0534 |
0.2941 |
0.0587 |
+ Task 2b, Test batch 1
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
main system |
0.5938 |
0.0400 |
0.1600 |
0.1000 |
- | - | - |
Biomedical Text Ming |
0.9375 |
0.1600 |
0.1600 |
0.1600 |
0.0572 |
0.0702 |
0.0614 |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
system 2 |
0.9375 |
0.0400 |
0.1600 |
0.1000 |
- | - | - |
Wishart-S3 |
0.8438 |
0.4400 |
0.4800 |
0.4600 |
0.4478 |
0.3335 |
0.3456 |
Wishart-S2 |
0.8438 |
0.4400 |
0.4800 |
0.4600 |
0.4774 |
0.3335 |
0.3621 |
system 3 |
0.9375 |
0.0400 |
0.1600 |
0.1000 |
- | - | - |
BioASQ_Baseline |
0.5313 |
- | - | - |
0.0418 |
0.0766 |
0.0501 |
BioASQ Baseline 2 |
0.5000 |
- | - | - |
0.0418 |
0.0766 |
0.0501 |
HPI Master Project |
0.8750 |
- | - | - |
0.0080 |
0.0136 |
0.0084 |
UNCC System 1 |
0.9375 |
- | - | - |
0.0073 |
0.0681 |
0.0130 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
main system |
0.4387 |
0.1381 |
0.4538 |
0.1281 |
Biomedical Text Ming |
0.1724 |
0.1301 |
0.1823 |
0.1278 |
SNUMedinfo1 |
0.4065 |
0.1907 |
0.4100 |
0.1808 |
SNUMedinfo2 |
0.3778 |
0.1855 |
0.3793 |
0.1736 |
SNUMedinfo3 |
0.4971 |
0.1698 |
0.4971 |
0.1562 |
SNUMedinfo4 |
0.3529 |
0.1686 |
0.3542 |
0.1583 |
SNUMedinfo5 |
0.4602 |
0.1588 |
0.4679 |
0.1477 |
system 2 |
0.4400 |
0.1384 |
0.4544 |
0.1282 |
Wishart-S3 |
0.4802 |
0.1724 |
0.4814 |
0.1602 |
Wishart-S2 |
0.4802 |
0.1724 |
0.4814 |
0.1602 |
system 3 |
0.4395 |
0.1383 |
0.4535 |
0.1281 |
BioASQ_Baseline |
0.3741 |
0.0744 |
0.3943 |
0.0727 |
BioASQ Baseline 2 |
0.3622 |
0.0758 |
0.3970 |
0.0765 |
HPI Master Project |
0.0222 |
0.0344 |
0.0294 |
0.0444 |
UNCC System 1 |
0.5313 |
0.1454 |
0.5326 |
0.1334 |
+ Task 2b, Test batch 2
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
Biomedical Text Ming |
0.8214 |
- | - | - |
0.1596 |
0.2057 |
0.1618 |
main system |
0.8214 |
0.0435 |
0.1739 |
0.0942 |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
system 2 |
0.8214 |
0.0435 |
0.1739 |
0.0942 |
- | - | - |
system 3 |
0.8214 |
0.0435 |
0.1739 |
0.0942 |
- | - | - |
system 4 |
0.7500 |
0.0435 |
0.1739 |
0.0942 |
- | - | - |
Wishart-S1 |
0.9286 |
0.1304 |
0.1304 |
0.1304 |
0.4396 |
0.4739 |
0.4122 |
Wishart-S2 |
0.9286 |
0.1304 |
0.1304 |
0.1304 |
0.5120 |
0.4399 |
0.4261 |
system 5 |
0.7143 |
0.0435 |
0.1739 |
0.0942 |
- | - | - |
BioASQ_Baseline |
0.5000 |
- | - | - |
0.1083 |
0.0962 |
0.0951 |
BioASQ Baseline 2 |
0.3929 |
- | - | - |
0.1083 |
0.0962 |
0.0951 |
UNCC System 1 |
0.8214 |
- | - | - |
0.0118 |
0.1974 |
0.0219 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
Biomedical Text Ming |
0.1846 |
0.1638 |
0.1948 |
0.1669 |
main system |
0.3280 |
0.1224 |
0.3411 |
0.1225 |
SNUMedinfo2 |
0.3157 |
0.1794 |
0.3198 |
0.1751 |
SNUMedinfo1 |
0.3169 |
0.1829 |
0.3225 |
0.1792 |
SNUMedinfo3 |
0.3116 |
0.1829 |
0.3174 |
0.1794 |
SNUMedinfo4 |
0.2910 |
0.1792 |
0.2973 |
0.1761 |
SNUMedinfo5 |
0.2894 |
0.1677 |
0.2982 |
0.1663 |
system 2 |
0.3247 |
0.1184 |
0.3392 |
0.1186 |
system 3 |
0.3352 |
0.1252 |
0.3493 |
0.1255 |
system 4 |
0.3282 |
0.1219 |
0.3419 |
0.1223 |
Wishart-S1 |
0.3914 |
0.1533 |
0.4089 |
0.1541 |
Wishart-S2 |
0.3914 |
0.1533 |
0.4089 |
0.1541 |
system 5 |
0.3231 |
0.1185 |
0.3385 |
0.1191 |
BioASQ_Baseline |
0.3183 |
0.0813 |
0.3577 |
0.0884 |
BioASQ Baseline 2 |
0.3121 |
0.0840 |
0.3359 |
0.0877 |
UNCC System 1 |
0.4075 |
0.1404 |
0.4258 |
0.1399 |
+ Task 2b, Test batch 3
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
Biomedical Text Ming |
0.8333 |
0.0417 |
0.1250 |
0.0833 |
0.1195 |
0.1780 |
0.1373 |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
Wishart-S1 |
0.8889 |
0.0417 |
0.0833 |
0.0556 |
0.4584 |
0.3763 |
0.3909 |
main system |
0.8333 |
- | - | - |
- | - | - |
Asclepius |
0.8333 |
0.0417 |
0.0417 |
0.0417 |
- | - | - |
BioASQ_Baseline |
0.7222 |
- | - | - |
0.1010 |
0.2340 |
0.1114 |
BioASQ Baseline 2 |
0.6111 |
- | - | - |
0.1010 |
0.2340 |
0.1114 |
Lab Zhu ,Fdan Univer |
0.8333 |
0.0417 |
0.2083 |
0.0931 |
0.0947 |
0.3322 |
0.1403 |
UNCC System 1 |
0.8333 |
- | - | - |
0.0154 |
0.1522 |
0.0269 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
Biomedical Text Ming |
0.2134 |
0.1564 |
0.2190 |
0.1524 |
SNUMedinfo1 |
0.3382 |
0.1769 |
0.3446 |
0.1723 |
SNUMedinfo2 |
0.3560 |
0.1860 |
0.3605 |
0.1789 |
SNUMedinfo3 |
0.3321 |
0.1793 |
0.3385 |
0.1746 |
SNUMedinfo4 |
0.3307 |
0.1857 |
0.3369 |
0.1808 |
SNUMedinfo5 |
0.3343 |
0.1654 |
0.3407 |
0.1597 |
Wishart-S1 |
0.4331 |
0.1582 |
0.4427 |
0.1525 |
main system |
0.4282 |
0.1552 |
0.4386 |
0.1501 |
Asclepius |
- | - |
BioASQ_Baseline |
0.3319 |
0.0698 |
0.3564 |
0.0711 |
BioASQ Baseline 2 |
0.3219 |
0.0630 |
0.3589 |
0.0679 |
Lab Zhu ,Fdan Univer |
0.2046 |
0.2032 |
0.2092 |
0.1963 |
UNCC System 1 |
0.4731 |
0.1662 |
0.4754 |
0.1577 |
+ Task 2b, Test batch 4
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
Hippocrates |
0.8750 |
0.0625 |
0.1875 |
0.1120 |
- | - | - |
Asclepius |
0.8750 |
0.0625 |
0.2188 |
0.1354 |
- | - | - |
Biomedical Text Ming |
0.8750 |
0.0938 |
0.1250 |
0.1042 |
- | - | - |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
Wishart-S1 |
0.9375 |
0.2500 |
0.2813 |
0.2813 |
0.2659 |
0.4029 |
0.2963 |
BioASQ_Baseline |
0.5000 |
- | - | - |
0.1233 |
0.1365 |
0.1062 |
BioASQ Baseline 2 |
0.3438 |
- | - | - |
0.1233 |
0.1365 |
0.1062 |
UNCC System 1 |
0.8750 |
- | - | - |
0.0180 |
0.0863 |
0.0286 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
Hippocrates |
- | - |
Asclepius |
- | - |
Biomedical Text Ming |
0.1796 |
0.1518 |
0.1938 |
0.1560 |
SNUMedinfo1 |
0.2963 |
0.1746 |
0.3158 |
0.1788 |
SNUMedinfo2 |
0.2858 |
0.1687 |
0.3087 |
0.1735 |
SNUMedinfo3 |
0.2927 |
0.1743 |
0.3130 |
0.1790 |
SNUMedinfo4 |
0.2756 |
0.1737 |
0.2968 |
0.1794 |
SNUMedinfo5 |
0.3164 |
0.1934 |
0.3384 |
0.1977 |
Wishart-S1 |
0.4072 |
0.1606 |
0.4295 |
0.1596 |
BioASQ_Baseline |
0.3273 |
0.0833 |
0.3677 |
0.0888 |
BioASQ Baseline 2 |
0.3075 |
0.0814 |
0.3569 |
0.0874 |
UNCC System 1 |
0.4201 |
0.1404 |
0.4458 |
0.1416 |
+ Task 2b, Test batch 5
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
Biomedical Text Ming |
1.0000 |
0.1379 |
0.1724 |
0.1466 |
- | - | - |
Asclepius |
0.8333 |
0.0345 |
0.0690 |
0.0460 |
- | - | - |
BioASQ_Baseline |
0.3750 |
- | - | - |
0.1548 |
0.1665 |
0.1319 |
BioASQ Baseline 2 |
0.4583 |
- | - | - |
0.1548 |
0.1665 |
0.1319 |
UNCC System 1 |
1.0000 |
- | - | - |
0.0098 |
0.1183 |
0.0177 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
SNUMedinfo1 |
0.2929 |
0.1558 |
0.3178 |
0.1591 |
SNUMedinfo2 |
0.2945 |
0.1575 |
0.3185 |
0.1613 |
SNUMedinfo3 |
0.2895 |
0.1581 |
0.3146 |
0.1616 |
SNUMedinfo4 |
0.2765 |
0.1633 |
0.2984 |
0.1664 |
SNUMedinfo5 |
0.2919 |
0.1509 |
0.3163 |
0.1547 |
Biomedical Text Ming |
0.1685 |
0.1386 |
0.1817 |
0.1429 |
Asclepius |
- | - |
BioASQ_Baseline |
0.3297 |
0.0773 |
0.3599 |
0.0845 |
BioASQ Baseline 2 |
0.3366 |
0.0736 |
0.3639 |
0.0827 |
UNCC System 1 |
0.3967 |
0.1429 |
0.4180 |
0.1444 |
+ Task 3b, Test batch 1
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
HPI-S2 |
0.6667 |
- | - | - |
0.0327 |
0.0830 |
0.0424 |
auth-qa-1 |
0.8485 |
0.1154 |
0.1154 |
0.1154 |
- | - | - |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
ilsp.aueb.1 |
- |
- | - | - |
- | - | - |
ilsp.aueb.2 |
- |
- | - | - |
- | - | - |
fdu |
0.8485 |
0.1154 |
0.2692 |
0.1744 |
0.0958 |
0.4312 |
0.1520 |
fdu2 |
0.8485 |
0.1154 |
0.2692 |
0.1744 |
0.0958 |
0.4312 |
0.1520 |
fdu3 |
0.8485 |
0.1154 |
0.2692 |
0.1744 |
0.0931 |
0.4085 |
0.1472 |
fdu4 |
0.8485 |
0.1538 |
0.3077 |
0.2128 |
0.0681 |
0.4998 |
0.1166 |
main system |
0.8485 |
0.1923 |
0.3846 |
0.2641 |
0.1841 |
0.2597 |
0.1987 |
BioASQ_Baseline |
0.4545 |
- | - | - |
- | - | - |
BioASQ Baseline 2 |
0.5455 |
- | - | - |
- | - | - |
Lab Zhu ,Fdan Univer |
0.8485 |
0.2308 |
0.3077 |
0.2481 |
0.1255 |
0.5791 |
0.1823 |
Question Summary |
- |
- | - | - |
- | - | - |
test json format |
0.8485 |
- | - | - |
0.0265 |
0.0149 |
0.0170 |
test oracle format |
0.8485 |
- | - | - |
0.1179 |
0.1944 |
0.1408 |
UNCC System 1 |
0.8485 |
0.0769 |
0.1154 |
0.0897 |
0.0122 |
0.2364 |
0.0228 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
HPI-S2 |
0.1897 |
0.1596 |
0.2015 |
0.1636 |
auth-qa-1 |
- | - |
SNUMedinfo1 |
0.3413 |
0.2651 |
0.3550 |
0.2699 |
SNUMedinfo2 |
0.3459 |
0.2699 |
0.3585 |
0.2731 |
SNUMedinfo3 |
0.3386 |
0.2710 |
0.3524 |
0.2747 |
SNUMedinfo4 |
0.2942 |
0.2543 |
0.3076 |
0.2585 |
SNUMedinfo5 |
0.3139 |
0.2658 |
0.3270 |
0.2700 |
ilsp.aueb.1 |
0.3894 |
0.1055 |
0.4144 |
0.1090 |
ilsp.aueb.2 |
0.4183 |
0.1142 |
0.4387 |
0.1165 |
fdu |
0.2801 |
0.2142 |
0.2811 |
0.2098 |
fdu2 |
0.2932 |
0.2142 |
0.3045 |
0.2127 |
fdu3 |
0.2881 |
0.1895 |
0.3076 |
0.1974 |
fdu4 |
0.2881 |
0.1895 |
0.3076 |
0.1974 |
main system |
0.3076 |
0.1473 |
0.3209 |
0.1483 |
BioASQ_Baseline |
0.4053 |
0.1104 |
0.4267 |
0.1133 |
BioASQ Baseline 2 |
0.3789 |
0.0994 |
0.3985 |
0.1018 |
Lab Zhu ,Fdan Univer |
0.2490 |
0.1864 |
0.2614 |
0.1864 |
Question Summary |
0.0635 |
0.0950 |
0.0663 |
0.0988 |
test json format |
0.0645 |
0.0396 |
0.0620 |
0.0359 |
test oracle format |
0.0674 |
0.0423 |
0.0653 |
0.0389 |
UNCC System 1 |
0.5240 |
0.2008 |
0.5368 |
0.1964 |
+ Task 3b, Test batch 2
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
main system |
0.8125 |
0.1250 |
0.1875 |
0.1510 |
0.1149 |
0.1798 |
0.1356 |
system 2 |
0.8125 |
0.0938 |
0.1875 |
0.1260 |
0.1149 |
0.1798 |
0.1356 |
system 3 |
0.8125 |
0.0625 |
0.2500 |
0.1391 |
0.1149 |
0.1798 |
0.1356 |
auth-qa-1 |
0.8125 |
0.0313 |
0.0313 |
0.0313 |
0.0357 |
0.0089 |
0.0143 |
ilsp.aueb.1 |
- |
- | - | - |
- | - | - |
ilsp.aueb.2 |
- |
- | - | - |
- | - | - |
HPI-S2 |
0.5625 |
- | - | - |
0.0714 |
0.0161 |
0.0262 |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
fdu4 |
0.8125 |
0.0313 |
0.1563 |
0.0859 |
0.0828 |
0.3772 |
0.1280 |
fdu2 |
0.8125 |
0.0625 |
0.1875 |
0.1172 |
0.0828 |
0.3772 |
0.1280 |
fdu |
0.8125 |
0.0625 |
0.1875 |
0.1172 |
0.0828 |
0.3772 |
0.1280 |
fdu3 |
0.8125 |
0.0625 |
0.1875 |
0.1172 |
0.0829 |
0.3772 |
0.1281 |
fdu5 |
0.7500 |
0.0313 |
0.0313 |
0.0313 |
0.0007 |
0.0083 |
0.0013 |
qaiiit system 1 |
- |
- | - | - |
- | - | - |
BioASQ_Baseline |
0.3125 |
- | - | - |
0.0357 |
0.0089 |
0.0143 |
BioASQ Baseline 2 |
0.4375 |
- | - | - |
0.0357 |
0.0089 |
0.0143 |
UNCC System 1 |
- |
- | - | - |
- | - | - |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
main system |
0.2968 |
0.1322 |
0.3055 |
0.1312 |
system 2 |
0.2984 |
0.1326 |
0.3069 |
0.1313 |
system 3 |
0.2983 |
0.1324 |
0.3080 |
0.1314 |
auth-qa-1 |
- | - |
ilsp.aueb.1 |
0.3852 |
0.1043 |
0.4160 |
0.1072 |
ilsp.aueb.2 |
0.3989 |
0.1081 |
0.4368 |
0.1113 |
HPI-S2 |
0.2063 |
0.1664 |
0.2243 |
0.1757 |
SNUMedinfo1 |
0.3953 |
0.2866 |
0.4114 |
0.2847 |
SNUMedinfo2 |
0.3897 |
0.2828 |
0.4073 |
0.2835 |
SNUMedinfo3 |
0.3918 |
0.2909 |
0.4087 |
0.2911 |
SNUMedinfo4 |
0.3509 |
0.2902 |
0.3708 |
0.2923 |
SNUMedinfo5 |
0.3641 |
0.2937 |
0.3848 |
0.2946 |
fdu4 |
0.2604 |
0.1770 |
0.2859 |
0.1850 |
fdu2 |
0.2590 |
0.1761 |
0.2853 |
0.1846 |
fdu |
0.2590 |
0.1761 |
0.2853 |
0.1846 |
fdu3 |
0.2604 |
0.1770 |
0.2859 |
0.1850 |
fdu5 |
- | - |
qaiiit system 1 |
0.3081 |
0.1657 |
0.3353 |
0.1710 |
BioASQ_Baseline |
0.4570 |
0.1251 |
0.4772 |
0.1247 |
BioASQ Baseline 2 |
0.4105 |
0.1146 |
0.4388 |
0.1153 |
UNCC System 1 |
0.5451 |
0.1894 |
0.5674 |
0.1852 |
+ Task 3b, Test batch 3
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
HPI-S2 |
0.6207 |
- | - | - |
- | - | - |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
main system |
0.9655 |
0.1923 |
0.2692 |
0.2308 |
0.1529 |
0.1765 |
0.1587 |
system 2 |
0.9655 |
0.0385 |
0.1538 |
0.0897 |
0.1529 |
0.1765 |
0.1587 |
system 3 |
0.9655 |
0.1538 |
0.2692 |
0.1987 |
0.1529 |
0.1765 |
0.1587 |
auth-qa-1 |
0.9655 |
0.0385 |
0.0769 |
0.0577 |
- | - | - |
ilsp.aueb.1 |
- |
- | - | - |
- | - | - |
ilsp.aueb.2 |
- |
- | - | - |
- | - | - |
ilsp.aueb.3 |
- |
- | - | - |
- | - | - |
fdu4 |
0.9655 |
0.0769 |
0.1154 |
0.0962 |
0.0175 |
0.2735 |
0.0324 |
oaqa-3b-3 |
0.0345 |
0.1538 |
0.3462 |
0.2321 |
0.0520 |
0.7383 |
0.0940 |
oaqa-3b-3-e |
0.0345 |
0.1538 |
0.3462 |
0.2321 |
0.0594 |
0.8130 |
0.1072 |
fdu2 |
0.6207 |
0.1154 |
0.1923 |
0.1359 |
0.0851 |
0.3964 |
0.1353 |
fdu3 |
0.6207 |
0.1154 |
0.1923 |
0.1359 |
0.0851 |
0.3964 |
0.1353 |
fdu5 |
0.9655 |
0.0385 |
0.0385 |
0.0385 |
0.0722 |
0.1775 |
0.0390 |
fdu |
0.6207 |
0.1154 |
0.1923 |
0.1359 |
0.1008 |
0.4993 |
0.1608 |
BioASQ_Baseline |
0.3448 |
- | - | - |
- | - | - |
BioASQ Baseline 2 |
0.4828 |
- | - | - |
- | - | - |