BioASQ Participants Area
Oracle: Results for task B-Phase B
The test results are presented in seperate tables for each type of annotation. The "System Description" of each system is used.
The evaluation measures that are used in Task B are presented
here .
+ Task 1b, Test batch 1
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
Wishart-S1 |
0.9200 |
0.2222 |
0.2778 |
0.2778 |
0.3186 |
0.2147 |
0.2290 |
Wishart-S2 |
0.9200 |
0.2222 |
0.2778 |
0.2778 |
0.3186 |
0.2147 |
0.2290 |
Wishart-S3 |
0.9200 |
0.2222 |
0.3333 |
0.3056 |
0.3067 |
0.2082 |
0.2207 |
main system |
0.3200 |
- | - | - |
0.0037 |
0.0394 |
0.0066 |
BioASQ_Baseline |
0.4400 |
- | - | - |
0.0153 |
0.0402 |
0.0204 |
BioASQ Baseline FS |
0.4800 |
- | - | - |
0.0153 |
0.0402 |
0.0204 |
HPI-S2 |
- |
- | - | - |
- | - | - |
UNCC System 1 |
0.8400 |
- | - | - |
0.0115 |
0.1499 |
0.0209 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
Wishart-S1 |
0.2059 |
0.1463 |
0.2202 |
0.1522 |
Wishart-S2 |
0.2059 |
0.1463 |
0.2202 |
0.1522 |
Wishart-S3 |
0.2059 |
0.1463 |
0.2202 |
0.1522 |
main system |
0.2165 |
0.1231 |
0.2396 |
0.1309 |
BioASQ_Baseline |
0.2266 |
0.0735 |
0.2636 |
0.0843 |
BioASQ Baseline FS |
0.1935 |
0.0628 |
0.2305 |
0.0745 |
HPI-S2 |
0.0872 |
0.1093 |
0.0915 |
0.1155 |
UNCC System 1 |
0.3032 |
0.1266 |
0.3276 |
0.1329 |
+ Task 1b, Test batch 2
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
main system |
0.4231 |
- | - | - |
0.0603 |
0.1040 |
0.0680 |
Wishart-S1 |
0.9615 |
0.2500 |
0.3000 |
0.3000 |
0.4060 |
0.3127 |
0.3336 |
system 2 |
0.4231 |
- | - | - |
0.0437 |
0.0445 |
0.0414 |
system 3 |
0.4231 |
- | - | - |
0.0644 |
0.0440 |
0.0488 |
system 4 |
0.4231 |
- | - | - |
0.0644 |
0.0440 |
0.0488 |
BioASQ_Baseline |
0.2692 |
- | - | - |
0.0612 |
0.2062 |
0.0789 |
BioASQ Baseline FS |
0.5000 |
- | - | - |
0.0612 |
0.2062 |
0.0789 |
UNCC System 1 |
0.8077 |
- | - | - |
0.0065 |
0.1243 |
0.0120 |
lalala |
- |
- | - | - |
- | - | - |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
main system |
0.2122 |
0.0795 |
0.2596 |
0.0890 |
Wishart-S1 |
0.2106 |
0.1197 |
0.2387 |
0.1292 |
system 2 |
0.2204 |
0.0826 |
0.2659 |
0.0899 |
system 3 |
0.2152 |
0.0809 |
0.2592 |
0.0886 |
system 4 |
0.2152 |
0.0809 |
0.2592 |
0.0886 |
BioASQ_Baseline |
0.2074 |
0.0573 |
0.2533 |
0.0675 |
BioASQ Baseline FS |
0.2052 |
0.0563 |
0.2577 |
0.0663 |
UNCC System 1 |
0.3319 |
0.1079 |
0.3596 |
0.1114 |
lalala |
- | - |
+ Task 1b, Test batch 3
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
main system |
0.5769 |
- | - | - |
0.0671 |
0.1013 |
0.0727 |
system 2 |
0.5769 |
- | - | - |
0.0642 |
0.0580 |
0.0575 |
system 3 |
0.5769 |
- | - | - |
0.0594 |
0.0399 |
0.0450 |
BioASQ_Baseline |
0.6538 |
- | - | - |
0.0209 |
0.0635 |
0.0278 |
BioASQ Baseline FS |
0.6154 |
- | - | - |
0.0209 |
0.0635 |
0.0278 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
main system |
0.2563 |
0.0885 |
0.2828 |
0.0912 |
system 2 |
0.2505 |
0.0876 |
0.2806 |
0.0910 |
system 3 |
0.2555 |
0.0884 |
0.2835 |
0.0916 |
BioASQ_Baseline |
0.2670 |
0.0558 |
0.2982 |
0.0596 |
BioASQ Baseline FS |
0.2547 |
0.0534 |
0.2941 |
0.0587 |
+ Task 2b, Test batch 1
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
main system |
0.5938 |
0.0400 |
0.1600 |
0.1000 |
- | - | - |
Biomedical Text Ming |
0.9375 |
0.1600 |
0.1600 |
0.1600 |
0.0572 |
0.0702 |
0.0614 |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
system 2 |
0.9375 |
0.0400 |
0.1600 |
0.1000 |
- | - | - |
Wishart-S3 |
0.8438 |
0.4400 |
0.4800 |
0.4600 |
0.4478 |
0.3335 |
0.3456 |
Wishart-S2 |
0.8438 |
0.4400 |
0.4800 |
0.4600 |
0.4774 |
0.3335 |
0.3621 |
system 3 |
0.9375 |
0.0400 |
0.1600 |
0.1000 |
- | - | - |
BioASQ_Baseline |
0.5313 |
- | - | - |
0.0418 |
0.0766 |
0.0501 |
BioASQ Baseline FS |
0.5000 |
- | - | - |
0.0418 |
0.0766 |
0.0501 |
HPI Master Project |
0.8750 |
- | - | - |
0.0080 |
0.0136 |
0.0084 |
UNCC System 1 |
0.9375 |
- | - | - |
0.0073 |
0.0681 |
0.0130 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
main system |
0.4387 |
0.1381 |
0.4538 |
0.1281 |
Biomedical Text Ming |
0.1724 |
0.1301 |
0.1823 |
0.1278 |
SNUMedinfo1 |
0.4065 |
0.1907 |
0.4100 |
0.1808 |
SNUMedinfo2 |
0.3778 |
0.1855 |
0.3793 |
0.1736 |
SNUMedinfo3 |
0.4971 |
0.1698 |
0.4971 |
0.1562 |
SNUMedinfo4 |
0.3529 |
0.1686 |
0.3542 |
0.1583 |
SNUMedinfo5 |
0.4602 |
0.1588 |
0.4679 |
0.1477 |
system 2 |
0.4400 |
0.1384 |
0.4544 |
0.1282 |
Wishart-S3 |
0.4802 |
0.1724 |
0.4814 |
0.1602 |
Wishart-S2 |
0.4802 |
0.1724 |
0.4814 |
0.1602 |
system 3 |
0.4395 |
0.1383 |
0.4535 |
0.1281 |
BioASQ_Baseline |
0.3741 |
0.0744 |
0.3943 |
0.0727 |
BioASQ Baseline FS |
0.3622 |
0.0758 |
0.3970 |
0.0765 |
HPI Master Project |
0.0222 |
0.0344 |
0.0294 |
0.0444 |
UNCC System 1 |
0.5313 |
0.1454 |
0.5326 |
0.1334 |
+ Task 2b, Test batch 2
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
Biomedical Text Ming |
0.8214 |
- | - | - |
0.1596 |
0.2057 |
0.1618 |
main system |
0.8214 |
0.0435 |
0.1739 |
0.0942 |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
system 2 |
0.8214 |
0.0435 |
0.1739 |
0.0942 |
- | - | - |
system 3 |
0.8214 |
0.0435 |
0.1739 |
0.0942 |
- | - | - |
system 4 |
0.7500 |
0.0435 |
0.1739 |
0.0942 |
- | - | - |
Wishart-S1 |
0.9286 |
0.1304 |
0.1304 |
0.1304 |
0.4396 |
0.4739 |
0.4122 |
Wishart-S2 |
0.9286 |
0.1304 |
0.1304 |
0.1304 |
0.5120 |
0.4399 |
0.4261 |
system 5 |
0.7143 |
0.0435 |
0.1739 |
0.0942 |
- | - | - |
BioASQ_Baseline |
0.5000 |
- | - | - |
0.1083 |
0.0962 |
0.0951 |
BioASQ Baseline FS |
0.3929 |
- | - | - |
0.1083 |
0.0962 |
0.0951 |
UNCC System 1 |
0.8214 |
- | - | - |
0.0118 |
0.1974 |
0.0219 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
Biomedical Text Ming |
0.1846 |
0.1638 |
0.1948 |
0.1669 |
main system |
0.3280 |
0.1224 |
0.3411 |
0.1225 |
SNUMedinfo2 |
0.3157 |
0.1794 |
0.3198 |
0.1751 |
SNUMedinfo1 |
0.3169 |
0.1829 |
0.3225 |
0.1792 |
SNUMedinfo3 |
0.3116 |
0.1829 |
0.3174 |
0.1794 |
SNUMedinfo4 |
0.2910 |
0.1792 |
0.2973 |
0.1761 |
SNUMedinfo5 |
0.2894 |
0.1677 |
0.2982 |
0.1663 |
system 2 |
0.3247 |
0.1184 |
0.3392 |
0.1186 |
system 3 |
0.3352 |
0.1252 |
0.3493 |
0.1255 |
system 4 |
0.3282 |
0.1219 |
0.3419 |
0.1223 |
Wishart-S1 |
0.3914 |
0.1533 |
0.4089 |
0.1541 |
Wishart-S2 |
0.3914 |
0.1533 |
0.4089 |
0.1541 |
system 5 |
0.3231 |
0.1185 |
0.3385 |
0.1191 |
BioASQ_Baseline |
0.3183 |
0.0813 |
0.3577 |
0.0884 |
BioASQ Baseline FS |
0.3121 |
0.0840 |
0.3359 |
0.0877 |
UNCC System 1 |
0.4075 |
0.1404 |
0.4258 |
0.1399 |
+ Task 2b, Test batch 3
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
Biomedical Text Ming |
0.8333 |
0.0417 |
0.1250 |
0.0833 |
0.1195 |
0.1780 |
0.1373 |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
Wishart-S1 |
0.8889 |
0.0417 |
0.0833 |
0.0556 |
0.4584 |
0.3763 |
0.3909 |
main system |
0.8333 |
- | - | - |
- | - | - |
Asclepius |
0.8333 |
0.0417 |
0.0417 |
0.0417 |
- | - | - |
BioASQ_Baseline |
0.7222 |
- | - | - |
0.1010 |
0.2340 |
0.1114 |
BioASQ Baseline FS |
0.6111 |
- | - | - |
0.1010 |
0.2340 |
0.1114 |
Lab Zhu ,Fdan Univer |
0.8333 |
0.0417 |
0.2083 |
0.0931 |
0.0947 |
0.3322 |
0.1403 |
UNCC System 1 |
0.8333 |
- | - | - |
0.0154 |
0.1522 |
0.0269 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
Biomedical Text Ming |
0.2134 |
0.1564 |
0.2190 |
0.1524 |
SNUMedinfo1 |
0.3382 |
0.1769 |
0.3446 |
0.1723 |
SNUMedinfo2 |
0.3560 |
0.1860 |
0.3605 |
0.1789 |
SNUMedinfo3 |
0.3321 |
0.1793 |
0.3385 |
0.1746 |
SNUMedinfo4 |
0.3307 |
0.1857 |
0.3369 |
0.1808 |
SNUMedinfo5 |
0.3343 |
0.1654 |
0.3407 |
0.1597 |
Wishart-S1 |
0.4331 |
0.1582 |
0.4427 |
0.1525 |
main system |
0.4282 |
0.1552 |
0.4386 |
0.1501 |
Asclepius |
- | - |
BioASQ_Baseline |
0.3319 |
0.0698 |
0.3564 |
0.0711 |
BioASQ Baseline FS |
0.3219 |
0.0630 |
0.3589 |
0.0679 |
Lab Zhu ,Fdan Univer |
0.2046 |
0.2032 |
0.2092 |
0.1963 |
UNCC System 1 |
0.4731 |
0.1662 |
0.4754 |
0.1577 |
+ Task 2b, Test batch 4
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
Hippocrates |
0.8750 |
0.0625 |
0.1875 |
0.1120 |
- | - | - |
Asclepius |
0.8750 |
0.0625 |
0.2188 |
0.1354 |
- | - | - |
Biomedical Text Ming |
0.8750 |
0.0938 |
0.1250 |
0.1042 |
- | - | - |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
Wishart-S1 |
0.9375 |
0.2500 |
0.2813 |
0.2813 |
0.2659 |
0.4029 |
0.2963 |
BioASQ_Baseline |
0.5000 |
- | - | - |
0.1233 |
0.1365 |
0.1062 |
BioASQ Baseline FS |
0.3438 |
- | - | - |
0.1233 |
0.1365 |
0.1062 |
UNCC System 1 |
0.8750 |
- | - | - |
0.0180 |
0.0863 |
0.0286 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
Hippocrates |
- | - |
Asclepius |
- | - |
Biomedical Text Ming |
0.1796 |
0.1518 |
0.1938 |
0.1560 |
SNUMedinfo1 |
0.2963 |
0.1746 |
0.3158 |
0.1788 |
SNUMedinfo2 |
0.2858 |
0.1687 |
0.3087 |
0.1735 |
SNUMedinfo3 |
0.2927 |
0.1743 |
0.3130 |
0.1790 |
SNUMedinfo4 |
0.2756 |
0.1737 |
0.2968 |
0.1794 |
SNUMedinfo5 |
0.3164 |
0.1934 |
0.3384 |
0.1977 |
Wishart-S1 |
0.4072 |
0.1606 |
0.4295 |
0.1596 |
BioASQ_Baseline |
0.3273 |
0.0833 |
0.3677 |
0.0888 |
BioASQ Baseline FS |
0.3075 |
0.0814 |
0.3569 |
0.0874 |
UNCC System 1 |
0.4201 |
0.1404 |
0.4458 |
0.1416 |
+ Task 2b, Test batch 5
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
Biomedical Text Ming |
1.0000 |
0.1379 |
0.1724 |
0.1466 |
- | - | - |
Asclepius |
0.8333 |
0.0345 |
0.0690 |
0.0460 |
- | - | - |
BioASQ_Baseline |
0.3750 |
- | - | - |
0.1548 |
0.1665 |
0.1319 |
BioASQ Baseline FS |
0.4583 |
- | - | - |
0.1548 |
0.1665 |
0.1319 |
UNCC System 1 |
1.0000 |
- | - | - |
0.0098 |
0.1183 |
0.0177 |
Ideal Answers
|
Automatic scores |
System Name |
Rouge-2 |
Rouge-2 - F1 |
Rouge-SU4 |
Rouge-SU4 - F1 |
SNUMedinfo1 |
0.2929 |
0.1558 |
0.3178 |
0.1591 |
SNUMedinfo2 |
0.2945 |
0.1575 |
0.3185 |
0.1613 |
SNUMedinfo3 |
0.2895 |
0.1581 |
0.3146 |
0.1616 |
SNUMedinfo4 |
0.2765 |
0.1633 |
0.2984 |
0.1664 |
SNUMedinfo5 |
0.2919 |
0.1509 |
0.3163 |
0.1547 |
Biomedical Text Ming |
0.1685 |
0.1386 |
0.1817 |
0.1429 |
Asclepius |
- | - |
BioASQ_Baseline |
0.3297 |
0.0773 |
0.3599 |
0.0845 |
BioASQ Baseline FS |
0.3366 |
0.0736 |
0.3639 |
0.0827 |
UNCC System 1 |
0.3967 |
0.1429 |
0.4180 |
0.1444 |
+ Task 3b, Test batch 1
Exact Answers
|
Yes/No |
Factoid |
List |
System Name |
Accuracy |
Strict Acc. |
Lenient Acc. |
MRR |
Mean precision |
Recall |
F-Measure |
HPI-S2 |
0.6667 |
- | - | - |
0.0327 |
0.0830 |
0.0424 |
auth-qa-1 |
0.8485 |
0.1154 |
0.1154 |
0.1154 |
- | - | - |
SNUMedinfo1 |
- |
- | - | - |
- | - | - |
SNUMedinfo2 |
- |
- | - | - |
- | - | - |
SNUMedinfo3 |
- |
- | - | - |
- | - | - |
SNUMedinfo4 |
- |
- | - | - |
- | - | - |
SNUMedinfo5 |
- |
- | - | - |
- | - | - |
ilsp.aueb.1 |
- |
- | - | - |
- | - | - |
ilsp.aueb.2 |
- |
- | - | - |
- | - | - |
fdu |
0.8485 |
0.1154 |
0.2692 |
0.1744 |
0.0958 |
0.4312 |
0.1520 |
fdu2 |
0.8485 |
0.1154 |
0.2692 |
0.1744 |
0.0958 |
0.4312 |
0.1520 |
fdu3 |
0.8485 |
0.1154 |
0.2692 |
0.1744 |
0.0931 |
0.4085 |
0.1472 |
fdu4 |
0.8485 |
0.1538 |
0.3077 |
0.2128 |
0.0681 |
0.4998 |
0.1166 |
main system |
0.8485 |
0.1923 |
0.3846 |
0.2641 |
0.1841 |
0.2597 |
0.1987 |
BioASQ_Baseline |
0.4545 |
- | - | - |
- | - | - |
BioASQ Baseline FS |
0.5455 |
- | - | - |
- | - | - |