BioASQ Participants Area
Task 13b: Test Results of Phase A
* The "-" replace the scores of systems that didn't submit the corresponding annotations. *
The test results are presented in separate tables for each type of annotation. They are sorted based on the scores of MAP. The "System Description" of each system is used.
The evaluation measures that are used are presented
here .
Test batch 1
Documents
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.0606 |
0.3863 |
0.1005 |
0.2716 |
0.0020 |
IR1 |
0.0718 |
0.4312 |
0.1173 |
0.3394 |
0.0036 |
UniTor_0 |
0.0565 |
0.3847 |
0.0948 |
0.3020 |
0.0026 |
UniTor_1 |
0.1042 |
0.3608 |
0.1486 |
0.3004 |
0.0016 |
Baseline top 20 |
0.0788 |
0.4720 |
0.1300 |
0.3806 |
0.0051 |
Baseline top 10 |
0.0788 |
0.4720 |
0.1300 |
0.3806 |
0.0051 |
Using LLM alone |
0.0753 |
0.4598 |
0.1249 |
0.3622 |
0.0040 |
Using KG for list q |
0.1453 |
0.4225 |
0.1940 |
0.3004 |
0.0029 |
Main pipeline |
0.1512 |
0.3971 |
0.1908 |
0.2885 |
0.0020 |
13b_phase_a |
0.0430 |
0.1396 |
0.0591 |
0.1005 |
0.0001 |
13b-1 |
- |
- |
- |
- |
- |
bioinfo-0 |
0.1047 |
0.5037 |
0.1602 |
0.4141 |
0.0101 |
bioinfo-1 |
0.1071 |
0.5106 |
0.1636 |
0.4082 |
0.0123 |
bioinfo-2 |
0.1000 |
0.4916 |
0.1537 |
0.4214 |
0.0077 |
bioinfo-3 |
0.1047 |
0.5057 |
0.1605 |
0.4175 |
0.0123 |
bioinfo-4 |
0.1047 |
0.5043 |
0.1605 |
0.4246 |
0.0104 |
UniTor_2 |
0.0553 |
0.3725 |
0.0933 |
0.2784 |
0.0012 |
UniTor_3 |
0.0852 |
0.3471 |
0.1303 |
0.2712 |
0.0011 |
DB_vector_&_LLM |
- |
- |
- |
- |
- |
UR-IW-1 |
0.1415 |
0.3194 |
0.1776 |
0.2527 |
0.0010 |
UR-IW-2 |
0.1376 |
0.2941 |
0.1699 |
0.2272 |
0.0007 |
UR-IW-3 |
0.1344 |
0.2547 |
0.1557 |
0.2064 |
0.0005 |
UR-IW-4 |
0.0979 |
0.1892 |
0.1135 |
0.1739 |
0.0001 |
google_serach_&_LLM |
- |
- |
- |
- |
- |
UR-IW-5 |
0.1677 |
0.3471 |
0.2038 |
0.2865 |
0.0015 |
no expand only rank |
0.1133 |
0.1480 |
0.1196 |
0.1268 |
0.0001 |
DS@GT BioASQ 1b-1 |
0.0584 |
0.2325 |
0.0843 |
0.1980 |
0.0003 |
IRIS_1 |
0.0682 |
0.4282 |
0.1129 |
0.3081 |
0.0033 |
IRIS_3 |
0.0035 |
0.0186 |
0.0056 |
0.0082 |
0.0000 |
IRIS_2 |
0.0035 |
0.0235 |
0.0061 |
0.0034 |
0.0000 |
deepseek32b-me |
0.1378 |
0.1690 |
0.1425 |
0.1428 |
0.0001 |
dmiip2024 |
0.0682 |
0.4353 |
0.1140 |
0.3590 |
0.0031 |
dmiip2024_1 |
0.0718 |
0.4465 |
0.1192 |
0.3660 |
0.0034 |
dmiip2024_2 |
0.0682 |
0.4249 |
0.1133 |
0.3428 |
0.0024 |
dmiip2024_3 |
0.0694 |
0.4278 |
0.1149 |
0.3469 |
0.0024 |
dmiip2024_4 |
0.0706 |
0.4327 |
0.1169 |
0.3528 |
0.0033 |
config-1 |
0.0553 |
0.3745 |
0.0934 |
0.2903 |
0.0021 |
config-2 |
0.0506 |
0.3706 |
0.0869 |
0.2114 |
0.0011 |
config-3 |
0.0494 |
0.3539 |
0.0844 |
0.2342 |
0.0010 |
config-4 |
0.0619 |
0.4059 |
0.1042 |
0.1314 |
0.0017 |
config-5 |
0.0619 |
0.4059 |
0.1042 |
0.1302 |
0.0017 |
mistral |
0.1260 |
0.2876 |
0.1542 |
0.2271 |
0.0005 |
deepseek32b-full |
0.1514 |
0.2298 |
0.1641 |
0.1712 |
0.0002 |
GPT4O |
0.0343 |
0.1363 |
0.0532 |
0.0665 |
0.0001 |
deepseek-r1:32b |
0.0381 |
0.1945 |
0.0598 |
0.0724 |
0.0001 |
deepseek-r1:14b |
0.0251 |
0.1167 |
0.0395 |
0.0764 |
0.0001 |
using free 7b LLM |
- |
- |
- |
- |
- |
llama |
0.1339 |
0.2853 |
0.1642 |
0.2466 |
0.0006 |
dense |
0.0295 |
0.1857 |
0.0492 |
0.1324 |
0.0002 |
deepseek-r1:8b |
0.0381 |
0.1876 |
0.0611 |
0.0932 |
0.0001 |
lasigeBioTM |
0.0624 |
0.3920 |
0.1029 |
0.3207 |
0.0023 |
Snippets
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.0280 |
0.0924 |
0.0398 |
0.1009 |
0.0003 |
IR1 |
- |
- |
- |
- |
- |
UniTor_0 |
0.0957 |
0.2120 |
0.1170 |
0.2770 |
0.0011 |
UniTor_1 |
0.0833 |
0.2120 |
0.1081 |
0.2770 |
0.0011 |
Baseline top 20 |
0.0671 |
0.2042 |
0.0933 |
0.2109 |
0.0014 |
Baseline top 10 |
0.0671 |
0.2042 |
0.0933 |
0.2109 |
0.0014 |
Using LLM alone |
0.0679 |
0.1959 |
0.0936 |
0.2051 |
0.0010 |
Using KG for list q |
0.0897 |
0.1797 |
0.0939 |
0.1629 |
0.0007 |
Main pipeline |
0.0989 |
0.1630 |
0.0958 |
0.1566 |
0.0005 |
13b_phase_a |
0.0209 |
0.0168 |
0.0173 |
0.0494 |
0.0000 |
13b-1 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
0.0624 |
0.1930 |
0.0874 |
0.2282 |
0.0007 |
UniTor_3 |
0.0569 |
0.1930 |
0.0817 |
0.2282 |
0.0007 |
DB_vector_&_LLM |
- |
- |
- |
- |
- |
UR-IW-1 |
0.0978 |
0.1594 |
0.1071 |
0.2762 |
0.0005 |
UR-IW-2 |
0.1136 |
0.1633 |
0.1110 |
0.2478 |
0.0005 |
UR-IW-3 |
0.0863 |
0.1393 |
0.0912 |
0.2447 |
0.0003 |
UR-IW-4 |
0.0795 |
0.1035 |
0.0778 |
0.1844 |
0.0001 |
google_serach_&_LLM |
0.0428 |
0.0334 |
0.0289 |
0.0000 |
0.0000 |
UR-IW-5 |
0.1189 |
0.1928 |
0.1202 |
0.2768 |
0.0006 |
no expand only rank |
0.0389 |
0.0176 |
0.0220 |
0.0478 |
0.0000 |
DS@GT BioASQ 1b-1 |
0.0562 |
0.1310 |
0.0761 |
0.1390 |
0.0001 |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
deepseek32b-me |
0.0955 |
0.0517 |
0.0603 |
0.1085 |
0.0001 |
dmiip2024 |
0.0783 |
0.2891 |
0.1157 |
0.4533 |
0.0012 |
dmiip2024_1 |
0.0803 |
0.3050 |
0.1186 |
0.4535 |
0.0014 |
dmiip2024_2 |
0.0713 |
0.2821 |
0.1076 |
0.4156 |
0.0009 |
dmiip2024_3 |
0.0741 |
0.2865 |
0.1105 |
0.4158 |
0.0010 |
dmiip2024_4 |
0.0716 |
0.2767 |
0.1077 |
0.4158 |
0.0009 |
config-1 |
0.0229 |
0.0630 |
0.0310 |
0.0824 |
0.0001 |
config-2 |
0.0226 |
0.0510 |
0.0287 |
0.0561 |
0.0001 |
config-3 |
0.0177 |
0.0448 |
0.0235 |
0.0687 |
0.0001 |
config-4 |
0.0284 |
0.0714 |
0.0367 |
0.0411 |
0.0001 |
config-5 |
0.0284 |
0.0714 |
0.0367 |
0.0396 |
0.0001 |
mistral |
0.1077 |
0.0717 |
0.0703 |
0.1170 |
0.0001 |
deepseek32b-full |
0.0998 |
0.0575 |
0.0651 |
0.1131 |
0.0001 |
GPT4O |
0.0094 |
0.0166 |
0.0101 |
0.0162 |
0.0000 |
deepseek-r1:32b |
0.0201 |
0.0355 |
0.0236 |
0.0304 |
0.0001 |
deepseek-r1:14b |
0.0122 |
0.0326 |
0.0173 |
0.0419 |
0.0000 |
using free 7b LLM |
- |
- |
- |
- |
- |
llama |
0.1159 |
0.1054 |
0.0870 |
0.1644 |
0.0002 |
dense |
0.0456 |
0.1117 |
0.0583 |
0.1466 |
0.0001 |
deepseek-r1:8b |
0.0176 |
0.0386 |
0.0218 |
0.0446 |
0.0000 |
lasigeBioTM |
0.0385 |
0.0560 |
0.0417 |
0.1426 |
0.0006 |
Concepts
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
13b_phase_a |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
DB_vector_&_LLM |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
google_serach_&_LLM |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
no expand only rank |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-1 |
- |
- |
- |
- |
- |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
config-1 |
- |
- |
- |
- |
- |
config-2 |
- |
- |
- |
- |
- |
config-3 |
- |
- |
- |
- |
- |
config-4 |
- |
- |
- |
- |
- |
config-5 |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
using free 7b LLM |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
RDF Triples
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
13b_phase_a |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
DB_vector_&_LLM |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
google_serach_&_LLM |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
no expand only rank |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-1 |
- |
- |
- |
- |
- |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
config-1 |
- |
- |
- |
- |
- |
config-2 |
- |
- |
- |
- |
- |
config-3 |
- |
- |
- |
- |
- |
config-4 |
- |
- |
- |
- |
- |
config-5 |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
using free 7b LLM |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
Test batch 2
Documents
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.0861 |
0.4333 |
0.1342 |
0.2957 |
0.0026 |
Fleming-2 |
0.0993 |
0.4333 |
0.1477 |
0.3066 |
0.0026 |
UniTor_0 |
0.0590 |
0.3551 |
0.0965 |
0.2952 |
0.0013 |
UniTor_1 |
0.1086 |
0.3178 |
0.1457 |
0.2921 |
0.0009 |
UniTor_2 |
0.0588 |
0.3633 |
0.0964 |
0.3065 |
0.0015 |
UniTor_3 |
0.1075 |
0.3243 |
0.1433 |
0.2986 |
0.0008 |
UR-IW-1 |
0.1643 |
0.2890 |
0.1855 |
0.2523 |
0.0008 |
UR-IW-2 |
0.1181 |
0.2399 |
0.1335 |
0.1846 |
0.0005 |
UR-IW-3 |
0.1742 |
0.3064 |
0.1996 |
0.2443 |
0.0008 |
UR-IW-4 |
0.1575 |
0.3184 |
0.1820 |
0.2601 |
0.0009 |
Baseline top 10 |
0.0976 |
0.5093 |
0.1546 |
0.4425 |
0.0096 |
Baseline top 20 |
0.0976 |
0.5093 |
0.1546 |
0.4425 |
0.0096 |
Main pipeline |
0.2165 |
0.4394 |
0.2527 |
0.3362 |
0.0054 |
Using KG for list q |
0.2142 |
0.4213 |
0.2483 |
0.3296 |
0.0044 |
Using LLM alone |
0.0953 |
0.4973 |
0.1516 |
0.3973 |
0.0097 |
dmiip2024 |
0.0976 |
0.4968 |
0.1526 |
0.4049 |
0.0090 |
dmiip2024_1 |
0.0976 |
0.4978 |
0.1529 |
0.4179 |
0.0100 |
dmiip2024_2 |
0.1012 |
0.4944 |
0.1574 |
0.3939 |
0.0093 |
dmiip2024_3 |
0.1000 |
0.4935 |
0.1563 |
0.4055 |
0.0096 |
dmiip2024_4 |
0.0988 |
0.4866 |
0.1538 |
0.3938 |
0.0077 |
bioinfo-0 |
0.1094 |
0.5137 |
0.1658 |
0.4193 |
0.0109 |
bioinfo-1 |
0.1165 |
0.5337 |
0.1766 |
0.4307 |
0.0141 |
bioinfo-2 |
0.1094 |
0.5137 |
0.1658 |
0.4193 |
0.0109 |
bioinfo-3 |
0.1071 |
0.4961 |
0.1617 |
0.4226 |
0.0098 |
bioinfo-4 |
0.1082 |
0.5020 |
0.1637 |
0.4166 |
0.0097 |
IR5 |
0.0741 |
0.3914 |
0.1171 |
0.3225 |
0.0024 |
lasigeBioTM |
- |
- |
- |
- |
- |
UR-IW-5 |
0.1930 |
0.3237 |
0.2088 |
0.2634 |
0.0011 |
deepseek32b-full |
0.1170 |
0.2131 |
0.1285 |
0.1738 |
0.0003 |
mistral |
0.0927 |
0.1345 |
0.0920 |
0.1216 |
0.0001 |
llama |
0.1785 |
0.2666 |
0.1890 |
0.2284 |
0.0006 |
dense |
0.1501 |
0.2764 |
0.1696 |
0.2393 |
0.0008 |
GPT4O |
0.0243 |
0.1100 |
0.0376 |
0.0971 |
0.0001 |
deepseek-r1:32b |
0.0611 |
0.2059 |
0.0863 |
0.1492 |
0.0002 |
deepseek-r1:14b |
0.0283 |
0.1292 |
0.0448 |
0.0562 |
0.0001 |
deepseek-r1:8b |
0.0487 |
0.1451 |
0.0681 |
0.0785 |
0.0001 |
gpt 01 mini |
0.0502 |
0.1757 |
0.0736 |
0.0735 |
0.0001 |
13b-1 |
0.0863 |
0.2396 |
0.1138 |
0.1771 |
0.0003 |
IR1 |
0.0882 |
0.4383 |
0.1378 |
0.3548 |
0.0051 |
Fleming-3 |
0.0993 |
0.4333 |
0.1477 |
0.3066 |
0.0026 |
deepseek32b-me |
0.1132 |
0.1772 |
0.1118 |
0.1392 |
0.0002 |
Snippets
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.0645 |
0.2041 |
0.0913 |
0.2012 |
0.0008 |
Fleming-2 |
0.0710 |
0.1392 |
0.0753 |
0.1562 |
0.0004 |
UniTor_0 |
0.0901 |
0.1981 |
0.1071 |
0.3315 |
0.0007 |
UniTor_1 |
0.0780 |
0.1981 |
0.0977 |
0.3315 |
0.0007 |
UniTor_2 |
0.0876 |
0.1976 |
0.1032 |
0.3396 |
0.0007 |
UniTor_3 |
0.0729 |
0.1976 |
0.0924 |
0.3396 |
0.0007 |
UR-IW-1 |
0.1233 |
0.1713 |
0.1200 |
0.3023 |
0.0005 |
UR-IW-2 |
0.0916 |
0.1287 |
0.0877 |
0.1654 |
0.0003 |
UR-IW-3 |
0.1287 |
0.1715 |
0.1212 |
0.2949 |
0.0007 |
UR-IW-4 |
0.1397 |
0.1635 |
0.1149 |
0.2543 |
0.0007 |
Baseline top 10 |
0.1111 |
0.2838 |
0.1469 |
0.3154 |
0.0051 |
Baseline top 20 |
0.1111 |
0.2838 |
0.1469 |
0.3154 |
0.0051 |
Main pipeline |
0.1926 |
0.2524 |
0.1786 |
0.2589 |
0.0032 |
Using KG for list q |
0.1912 |
0.2491 |
0.1761 |
0.2507 |
0.0028 |
Using LLM alone |
0.1117 |
0.2722 |
0.1445 |
0.2914 |
0.0051 |
dmiip2024 |
0.0840 |
0.3395 |
0.1279 |
0.4866 |
0.0025 |
dmiip2024_1 |
0.0941 |
0.3625 |
0.1421 |
0.5522 |
0.0035 |
dmiip2024_2 |
0.0828 |
0.3393 |
0.1261 |
0.4612 |
0.0022 |
dmiip2024_3 |
0.0877 |
0.3529 |
0.1337 |
0.5006 |
0.0025 |
dmiip2024_4 |
0.0833 |
0.3420 |
0.1270 |
0.4583 |
0.0022 |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
lasigeBioTM |
0.0657 |
0.0942 |
0.0629 |
0.1034 |
0.0002 |
UR-IW-5 |
0.1407 |
0.1885 |
0.1290 |
0.3080 |
0.0009 |
deepseek32b-full |
0.0812 |
0.1200 |
0.0780 |
0.1883 |
0.0002 |
mistral |
0.0624 |
0.0325 |
0.0355 |
0.0595 |
0.0000 |
llama |
0.1608 |
0.1095 |
0.1054 |
0.1998 |
0.0004 |
dense |
0.1442 |
0.1376 |
0.1031 |
0.2091 |
0.0006 |
GPT4O |
0.0173 |
0.0412 |
0.0213 |
0.0538 |
0.0000 |
deepseek-r1:32b |
0.0316 |
0.0527 |
0.0352 |
0.0751 |
0.0001 |
deepseek-r1:14b |
0.0182 |
0.0410 |
0.0244 |
0.0322 |
0.0000 |
deepseek-r1:8b |
0.0262 |
0.0465 |
0.0314 |
0.0477 |
0.0000 |
gpt 01 mini |
0.0232 |
0.0437 |
0.0272 |
0.0275 |
0.0000 |
13b-1 |
0.0826 |
0.0978 |
0.0756 |
0.1406 |
0.0002 |
IR1 |
0.0766 |
0.2285 |
0.1048 |
0.2638 |
0.0014 |
Fleming-3 |
0.0710 |
0.1392 |
0.0753 |
0.1562 |
0.0004 |
deepseek32b-me |
0.0884 |
0.0985 |
0.0690 |
0.1523 |
0.0002 |
Concepts
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
Fleming-2 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
gpt 01 mini |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
Fleming-3 |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
RDF Triples
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
Fleming-2 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
gpt 01 mini |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
Fleming-3 |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
Test batch 3
Documents
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
NLP_BioBert |
0.0047 |
0.0235 |
0.0075 |
0.0097 |
0.0000 |
NLP_Group5_TU |
0.0047 |
0.0235 |
0.0075 |
0.0097 |
0.0000 |
Fleming-1 |
0.0697 |
0.3105 |
0.1064 |
0.1794 |
0.0009 |
IR5 |
0.0588 |
0.2934 |
0.0917 |
0.2099 |
0.0007 |
IR1 |
0.0671 |
0.3417 |
0.1052 |
0.2665 |
0.0021 |
IR4 |
0.0659 |
0.3312 |
0.1025 |
0.2344 |
0.0013 |
UR-IW-1 |
0.1114 |
0.2283 |
0.1273 |
0.1615 |
0.0004 |
UR-IW-2 |
0.0703 |
0.1588 |
0.0871 |
0.1187 |
0.0002 |
UR-IW-4 |
0.0644 |
0.1490 |
0.0818 |
0.1043 |
0.0001 |
UR-IW-5 |
0.1341 |
0.2507 |
0.1560 |
0.1834 |
0.0005 |
lasigeBioTM |
0.0597 |
0.1330 |
0.0716 |
0.1089 |
0.0001 |
lasigeBioTM-onto-sm |
0.0597 |
0.1330 |
0.0716 |
0.1089 |
0.0001 |
Baseline top 10 |
0.0888 |
0.4279 |
0.1394 |
0.3049 |
0.0053 |
Using LLM alone |
0.0735 |
0.3562 |
0.1144 |
0.2646 |
0.0024 |
Using KG for list q |
0.1377 |
0.3291 |
0.1701 |
0.2208 |
0.0014 |
Main pipeline |
0.1381 |
0.3232 |
0.1700 |
0.2183 |
0.0014 |
Baseline top 20 |
0.0888 |
0.4279 |
0.1394 |
0.3049 |
0.0053 |
UniTor_0 |
0.0482 |
0.2970 |
0.0794 |
0.2072 |
0.0007 |
UniTor_1 |
0.0818 |
0.2617 |
0.1149 |
0.2011 |
0.0005 |
bioinfo-0 |
0.0888 |
0.3932 |
0.1355 |
0.3079 |
0.0044 |
bioinfo-1 |
0.0941 |
0.4228 |
0.1445 |
0.3236 |
0.0059 |
bioinfo-2 |
0.0853 |
0.3849 |
0.1306 |
0.3148 |
0.0040 |
bioinfo-3 |
0.0865 |
0.3873 |
0.1322 |
0.3079 |
0.0037 |
bioinfo-4 |
0.0859 |
0.3887 |
0.1309 |
0.3084 |
0.0037 |
UniTor_2 |
0.0506 |
0.2911 |
0.0821 |
0.2029 |
0.0007 |
UniTor_3 |
0.0820 |
0.2730 |
0.1184 |
0.2000 |
0.0005 |
dmiip2024 |
0.0824 |
0.3999 |
0.1278 |
0.3030 |
0.0032 |
dmiip2024_1 |
0.0753 |
0.3783 |
0.1180 |
0.2988 |
0.0031 |
dmiip2024_2 |
0.0835 |
0.3773 |
0.1272 |
0.2969 |
0.0029 |
dmiip2024_4 |
0.0871 |
0.3886 |
0.1324 |
0.2976 |
0.0032 |
dmiip2024_3 |
0.0800 |
0.3705 |
0.1229 |
0.2902 |
0.0028 |
UR-IW-3 |
0.0854 |
0.2086 |
0.1093 |
0.1456 |
0.0004 |
lasigeBioTM-onto-bl |
0.0597 |
0.1330 |
0.0716 |
0.1089 |
0.0001 |
IRIS_1 |
0.0635 |
0.3022 |
0.0973 |
0.2230 |
0.0012 |
IRIS_2 |
- |
- |
- |
- |
- |
IRIS_3 |
0.0024 |
0.0176 |
0.0041 |
0.0071 |
0.0000 |
13b-1 |
- |
- |
- |
- |
- |
13b_phase_a |
0.0495 |
0.0862 |
0.0521 |
0.0669 |
0.0000 |
google_serach_&_LLM |
- |
- |
- |
- |
- |
mistral |
0.1266 |
0.2269 |
0.1428 |
0.1587 |
0.0003 |
llama |
0.1509 |
0.2407 |
0.1541 |
0.1772 |
0.0006 |
dense |
0.1188 |
0.2168 |
0.1372 |
0.1533 |
0.0003 |
GPT4O |
0.0400 |
0.1249 |
0.0559 |
0.0623 |
0.0001 |
deepseek-r1:32b |
0.0429 |
0.1159 |
0.0581 |
0.0757 |
0.0001 |
deepseek-r1:14b |
0.0552 |
0.1861 |
0.0792 |
0.0813 |
0.0002 |
sp_lasigebiotm |
0.0597 |
0.1330 |
0.0716 |
0.1089 |
0.0001 |
AQAMS |
- |
- |
- |
- |
- |
Snippets
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
NLP_BioBert |
- |
- |
- |
- |
- |
NLP_Group5_TU |
- |
- |
- |
- |
- |
Fleming-1 |
0.0410 |
0.0932 |
0.0544 |
0.0970 |
0.0002 |
IR5 |
- |
- |
- |
- |
- |
IR1 |
0.0574 |
0.1583 |
0.0793 |
0.2167 |
0.0003 |
IR4 |
0.0580 |
0.1714 |
0.0815 |
0.2164 |
0.0004 |
UR-IW-1 |
0.0838 |
0.1160 |
0.0806 |
0.1534 |
0.0003 |
UR-IW-2 |
0.0680 |
0.0662 |
0.0595 |
0.0968 |
0.0001 |
UR-IW-4 |
0.0478 |
0.0623 |
0.0475 |
0.0721 |
0.0001 |
UR-IW-5 |
0.0961 |
0.1271 |
0.0938 |
0.1488 |
0.0003 |
lasigeBioTM |
0.0527 |
0.0475 |
0.0397 |
0.0518 |
0.0000 |
lasigeBioTM-onto-sm |
0.0527 |
0.0475 |
0.0397 |
0.0518 |
0.0000 |
Baseline top 10 |
0.0778 |
0.2113 |
0.1054 |
0.2197 |
0.0019 |
Using LLM alone |
0.0633 |
0.1646 |
0.0834 |
0.1960 |
0.0008 |
Using KG for list q |
0.1031 |
0.1435 |
0.1025 |
0.1526 |
0.0004 |
Main pipeline |
0.1041 |
0.1408 |
0.1017 |
0.1499 |
0.0004 |
Baseline top 20 |
0.0778 |
0.2113 |
0.1054 |
0.2197 |
0.0019 |
UniTor_0 |
0.0474 |
0.1307 |
0.0653 |
0.1957 |
0.0002 |
UniTor_1 |
0.0458 |
0.1352 |
0.0641 |
0.1974 |
0.0002 |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
0.0466 |
0.1372 |
0.0656 |
0.1849 |
0.0003 |
UniTor_3 |
0.0405 |
0.1372 |
0.0602 |
0.1849 |
0.0003 |
dmiip2024 |
0.0749 |
0.2855 |
0.1098 |
0.4322 |
0.0015 |
dmiip2024_1 |
0.0745 |
0.2866 |
0.1099 |
0.4146 |
0.0015 |
dmiip2024_2 |
0.0738 |
0.2818 |
0.1082 |
0.4113 |
0.0014 |
dmiip2024_4 |
0.0711 |
0.2764 |
0.1055 |
0.4039 |
0.0012 |
dmiip2024_3 |
0.0732 |
0.2720 |
0.1070 |
0.4026 |
0.0012 |
UR-IW-3 |
0.0602 |
0.1130 |
0.0745 |
0.1463 |
0.0002 |
lasigeBioTM-onto-bl |
0.0527 |
0.0475 |
0.0397 |
0.0518 |
0.0000 |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
13b_phase_a |
0.0056 |
0.0030 |
0.0034 |
0.0215 |
0.0000 |
google_serach_&_LLM |
0.0339 |
0.0676 |
0.0405 |
0.0000 |
0.0000 |
mistral |
0.0974 |
0.1065 |
0.0819 |
0.1197 |
0.0002 |
llama |
0.1157 |
0.1086 |
0.0843 |
0.1244 |
0.0002 |
dense |
0.0944 |
0.0954 |
0.0771 |
0.1191 |
0.0001 |
GPT4O |
0.0233 |
0.0336 |
0.0245 |
0.0286 |
0.0000 |
deepseek-r1:32b |
0.0258 |
0.0343 |
0.0266 |
0.0333 |
0.0000 |
deepseek-r1:14b |
0.0282 |
0.0425 |
0.0313 |
0.0338 |
0.0000 |
sp_lasigebiotm |
0.0527 |
0.0475 |
0.0397 |
0.0518 |
0.0000 |
AQAMS |
0.0254 |
0.1097 |
0.0392 |
0.0000 |
0.0000 |
Concepts
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
NLP_BioBert |
- |
- |
- |
- |
- |
NLP_Group5_TU |
- |
- |
- |
- |
- |
Fleming-1 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
IR4 |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
lasigeBioTM-onto-sm |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
lasigeBioTM-onto-bl |
- |
- |
- |
- |
- |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
13b_phase_a |
- |
- |
- |
- |
- |
google_serach_&_LLM |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
sp_lasigebiotm |
- |
- |
- |
- |
- |
AQAMS |
- |
- |
- |
- |
- |
RDF Triples
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
NLP_BioBert |
- |
- |
- |
- |
- |
NLP_Group5_TU |
- |
- |
- |
- |
- |
Fleming-1 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
IR4 |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
lasigeBioTM-onto-sm |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
lasigeBioTM-onto-bl |
- |
- |
- |
- |
- |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
13b_phase_a |
- |
- |
- |
- |
- |
google_serach_&_LLM |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
sp_lasigebiotm |
- |
- |
- |
- |
- |
AQAMS |
- |
- |
- |
- |
- |
Test batch 4
Documents
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.0383 |
0.1550 |
0.0595 |
0.0863 |
0.0002 |
Data Wizards BM25 |
0.0012 |
0.0039 |
0.0018 |
0.0039 |
0.0000 |
NLP_BioBert |
0.0047 |
0.0137 |
0.0070 |
0.0080 |
0.0000 |
Data Wizards Re-Rank |
0.0059 |
0.0039 |
0.0047 |
0.0039 |
0.0000 |
First abrotsch |
0.0035 |
0.0098 |
0.0052 |
0.0053 |
0.0000 |
ms-marco-MiniLML6v2 |
0.0035 |
0.0098 |
0.0052 |
0.0053 |
0.0000 |
IR5 |
0.0376 |
0.1652 |
0.0591 |
0.0751 |
0.0001 |
BM25-Okapi |
0.0024 |
0.0069 |
0.0035 |
0.0037 |
0.0000 |
IR1 |
0.0529 |
0.2578 |
0.0848 |
0.1586 |
0.0006 |
IR4 |
0.0482 |
0.2383 |
0.0770 |
0.1333 |
0.0004 |
UniTor_0 |
0.0259 |
0.1333 |
0.0420 |
0.0667 |
0.0001 |
UniTor_1 |
0.0452 |
0.0990 |
0.0579 |
0.0626 |
0.0001 |
Reranking rankings |
0.0335 |
0.0814 |
0.0404 |
0.0416 |
0.0001 |
query-engineering |
0.0276 |
0.0644 |
0.0317 |
0.0346 |
0.0000 |
Re-Ranker with BM25 |
0.0035 |
0.0108 |
0.0053 |
0.0058 |
0.0000 |
RetrievalRivals |
0.0353 |
0.1385 |
0.0536 |
0.0676 |
0.0001 |
RetrievalRivals_BM25 |
0.0353 |
0.1385 |
0.0536 |
0.0676 |
0.0001 |
RetrieverRivals |
0.0353 |
0.1385 |
0.0536 |
0.0676 |
0.0001 |
UniTor_2 |
0.0294 |
0.1301 |
0.0463 |
0.0724 |
0.0001 |
UniTor_3 |
0.0423 |
0.1134 |
0.0583 |
0.0695 |
0.0001 |
TEsting123 |
0.0201 |
0.0474 |
0.0269 |
0.0287 |
0.0000 |
Traditional IR Methd |
0.0201 |
0.0474 |
0.0269 |
0.0287 |
0.0000 |
traditional IR |
0.0217 |
0.0451 |
0.0231 |
0.0193 |
0.0000 |
Neural IR approach |
0.0141 |
0.0527 |
0.0207 |
0.0240 |
0.0000 |
Machinen Results |
0.0306 |
0.1266 |
0.0475 |
0.0740 |
0.0001 |
Tuwien Group 6 AIR |
0.0141 |
0.0527 |
0.0207 |
0.0240 |
0.0000 |
reranking after IR |
0.0182 |
0.0157 |
0.0154 |
0.0110 |
0.0000 |
bioinfo-0 |
0.0576 |
0.2491 |
0.0899 |
0.1737 |
0.0007 |
bioinfo-1 |
0.0600 |
0.2512 |
0.0927 |
0.1801 |
0.0008 |
bioinfo-2 |
0.0565 |
0.2364 |
0.0877 |
0.1625 |
0.0006 |
bioinfo-3 |
0.0576 |
0.2491 |
0.0899 |
0.1619 |
0.0006 |
bioinfo-4 |
0.0565 |
0.2364 |
0.0877 |
0.1624 |
0.0006 |
Okapi BM25 IR |
0.0026 |
0.0069 |
0.0038 |
0.0011 |
0.0000 |
NNR Model |
0.0026 |
0.0069 |
0.0038 |
0.0012 |
0.0000 |
GoldenRetrievers |
0.0260 |
0.1002 |
0.0396 |
0.0597 |
0.0001 |
Using KG for list q |
0.0634 |
0.2528 |
0.0946 |
0.1476 |
0.0007 |
Baseline top 20 |
0.0587 |
0.2929 |
0.0939 |
0.1686 |
0.0007 |
Baseline top 10 |
0.0587 |
0.2929 |
0.0939 |
0.1686 |
0.0007 |
Using LLM alone |
0.0610 |
0.3009 |
0.0968 |
0.1701 |
0.0010 |
Main pipeline |
0.0622 |
0.2512 |
0.0934 |
0.1466 |
0.0007 |
dmiip2024 |
0.0494 |
0.2298 |
0.0784 |
0.1433 |
0.0004 |
dmiip2024_1 |
0.0471 |
0.2239 |
0.0750 |
0.1423 |
0.0003 |
dmiip2024_2 |
0.0447 |
0.2253 |
0.0719 |
0.1244 |
0.0002 |
dmiip2024_3 |
0.0424 |
0.2171 |
0.0684 |
0.1263 |
0.0002 |
dmiip2024_4 |
0.0459 |
0.2270 |
0.0733 |
0.1270 |
0.0003 |
AQAMS |
- |
- |
- |
- |
- |
deepseek32b-me |
0.0741 |
0.1595 |
0.0809 |
0.1014 |
0.0001 |
deepseek32b-full |
0.0513 |
0.1047 |
0.0565 |
0.0586 |
0.0001 |
DB_vector_&_LLM |
0.0302 |
0.1318 |
0.0467 |
0.0700 |
0.0001 |
UR-IW-1 |
0.0418 |
0.1040 |
0.0522 |
0.0627 |
0.0001 |
UR-IW-2 |
0.0408 |
0.1227 |
0.0555 |
0.0655 |
0.0001 |
UR-IW-3 |
0.0396 |
0.0900 |
0.0460 |
0.0574 |
0.0001 |
UR-IW-4 |
0.0451 |
0.1371 |
0.0622 |
0.0713 |
0.0001 |
UR-IW-5 |
0.0427 |
0.1391 |
0.0632 |
0.0794 |
0.0002 |
The Relevants |
0.0212 |
0.0686 |
0.0301 |
0.0314 |
0.0000 |
Data Wizards NN |
0.0024 |
0.0069 |
0.0035 |
0.0069 |
0.0000 |
API and weirdnesss |
0.0278 |
0.0955 |
0.0406 |
0.0574 |
0.0001 |
DS@GT BioASQ 1b-1 |
0.0435 |
0.2159 |
0.0696 |
0.1331 |
0.0003 |
Neural Re-Ranking |
0.0012 |
0.0029 |
0.0017 |
0.0003 |
0.0000 |
Approach 1 |
0.0047 |
0.0137 |
0.0070 |
0.0046 |
0.0000 |
DS@GT BioASQ 1b-2 |
0.0447 |
0.2163 |
0.0708 |
0.1337 |
0.0004 |
GoldenRetrievers-2 |
0.0447 |
0.0833 |
0.0503 |
0.0393 |
0.0001 |
dl approach |
0.0025 |
0.0069 |
0.0036 |
0.0012 |
0.0000 |
GPT4O |
0.0566 |
0.1821 |
0.0822 |
0.0941 |
0.0003 |
DS@GTBioASQT13b5 |
0.0435 |
0.2065 |
0.0690 |
0.1309 |
0.0003 |
sp_lasigebiotm |
0.0248 |
0.0916 |
0.0370 |
0.0452 |
0.0001 |
mistral |
0.0406 |
0.0829 |
0.0420 |
0.0603 |
0.0001 |
lasigeBioTM |
0.0248 |
0.0916 |
0.0370 |
0.0452 |
0.0001 |
llama |
0.0341 |
0.0838 |
0.0407 |
0.0505 |
0.0001 |
lasigeBioTM-onto-bl |
0.0248 |
0.0916 |
0.0370 |
0.0452 |
0.0001 |
lasigeBioTM-onto-sm |
0.0248 |
0.0916 |
0.0370 |
0.0452 |
0.0001 |
1.PhaseA_System |
0.0154 |
0.0346 |
0.0202 |
0.0291 |
0.0000 |
deepseek-r1:32b |
0.0387 |
0.1099 |
0.0550 |
0.0351 |
0.0001 |
deepseek-r1:14b |
0.0387 |
0.1099 |
0.0550 |
0.0351 |
0.0001 |
deepseek-r1:8b |
0.0319 |
0.0795 |
0.0418 |
0.0346 |
0.0000 |
DS@GTBioASQT13b4 |
0.0495 |
0.2476 |
0.0790 |
0.1562 |
0.0005 |
DS@GTBioASQT13b3 |
0.0482 |
0.2467 |
0.0773 |
0.1581 |
0.0004 |
dense |
0.0482 |
0.1185 |
0.0616 |
0.0578 |
0.0001 |
Fleming-2 |
0.0383 |
0.1550 |
0.0595 |
0.0942 |
0.0002 |
Snippets
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.0222 |
0.0562 |
0.0288 |
0.0382 |
0.0001 |
Data Wizards BM25 |
0.0038 |
0.0052 |
0.0044 |
0.0076 |
0.0000 |
NLP_BioBert |
0.0104 |
0.0103 |
0.0100 |
0.0074 |
0.0000 |
Data Wizards Re-Rank |
- |
- |
- |
- |
- |
First abrotsch |
- |
- |
- |
- |
- |
ms-marco-MiniLML6v2 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
BM25-Okapi |
- |
- |
- |
- |
- |
IR1 |
0.0427 |
0.1298 |
0.0594 |
0.0951 |
0.0003 |
IR4 |
0.0360 |
0.1217 |
0.0513 |
0.0922 |
0.0002 |
UniTor_0 |
0.0259 |
0.0606 |
0.0346 |
0.0559 |
0.0000 |
UniTor_1 |
0.0259 |
0.0606 |
0.0346 |
0.0559 |
0.0000 |
Reranking rankings |
0.0191 |
0.0562 |
0.0259 |
0.0460 |
0.0000 |
query-engineering |
0.0092 |
0.0191 |
0.0122 |
0.0195 |
0.0000 |
Re-Ranker with BM25 |
0.0016 |
0.0056 |
0.0025 |
0.0010 |
0.0000 |
RetrievalRivals |
0.0256 |
0.0030 |
0.0053 |
0.0065 |
0.0000 |
RetrievalRivals_BM25 |
0.0256 |
0.0030 |
0.0053 |
0.0065 |
0.0000 |
RetrieverRivals |
0.0256 |
0.0030 |
0.0053 |
0.0065 |
0.0000 |
UniTor_2 |
0.0439 |
0.0974 |
0.0576 |
0.0825 |
0.0001 |
UniTor_3 |
0.0414 |
0.0974 |
0.0551 |
0.0825 |
0.0001 |
TEsting123 |
0.0060 |
0.0173 |
0.0085 |
0.0035 |
0.0000 |
Traditional IR Methd |
0.0060 |
0.0173 |
0.0085 |
0.0035 |
0.0000 |
traditional IR |
0.0135 |
0.0086 |
0.0100 |
0.0068 |
0.0000 |
Neural IR approach |
0.0093 |
0.0169 |
0.0118 |
0.0168 |
0.0000 |
Machinen Results |
- |
- |
- |
- |
- |
Tuwien Group 6 AIR |
0.0093 |
0.0169 |
0.0118 |
0.0168 |
0.0000 |
reranking after IR |
0.0135 |
0.0086 |
0.0100 |
0.0113 |
0.0000 |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
Okapi BM25 IR |
0.0010 |
0.0065 |
0.0017 |
0.0002 |
0.0000 |
NNR Model |
0.0010 |
0.0065 |
0.0017 |
0.0002 |
0.0000 |
GoldenRetrievers |
- |
- |
- |
- |
- |
Using KG for list q |
0.0560 |
0.1479 |
0.0706 |
0.1007 |
0.0004 |
Baseline top 20 |
0.0557 |
0.1688 |
0.0773 |
0.1069 |
0.0005 |
Baseline top 10 |
0.0557 |
0.1688 |
0.0773 |
0.1069 |
0.0005 |
Using LLM alone |
0.0561 |
0.1716 |
0.0789 |
0.1152 |
0.0007 |
Main pipeline |
0.0542 |
0.1472 |
0.0690 |
0.1005 |
0.0004 |
dmiip2024 |
0.0425 |
0.1282 |
0.0580 |
0.1622 |
0.0002 |
dmiip2024_1 |
0.0416 |
0.1250 |
0.0569 |
0.1618 |
0.0001 |
dmiip2024_2 |
0.0417 |
0.1180 |
0.0573 |
0.1530 |
0.0001 |
dmiip2024_3 |
0.0411 |
0.1135 |
0.0560 |
0.1634 |
0.0001 |
dmiip2024_4 |
0.0408 |
0.1166 |
0.0561 |
0.1478 |
0.0001 |
AQAMS |
0.0177 |
0.1318 |
0.0292 |
0.0000 |
0.0000 |
deepseek32b-me |
0.0488 |
0.0939 |
0.0548 |
0.0682 |
0.0001 |
deepseek32b-full |
0.0428 |
0.0520 |
0.0368 |
0.0343 |
0.0000 |
DB_vector_&_LLM |
0.0154 |
0.0439 |
0.0219 |
0.0310 |
0.0000 |
UR-IW-1 |
0.0254 |
0.0639 |
0.0323 |
0.0434 |
0.0000 |
UR-IW-2 |
0.0250 |
0.0700 |
0.0344 |
0.0411 |
0.0001 |
UR-IW-3 |
0.0247 |
0.0492 |
0.0297 |
0.0459 |
0.0000 |
UR-IW-4 |
0.0229 |
0.0564 |
0.0314 |
0.0446 |
0.0001 |
UR-IW-5 |
0.0308 |
0.0633 |
0.0399 |
0.0511 |
0.0001 |
The Relevants |
- |
- |
- |
- |
- |
Data Wizards NN |
- |
- |
- |
- |
- |
API and weirdnesss |
0.0150 |
0.0374 |
0.0207 |
0.0226 |
0.0000 |
DS@GT BioASQ 1b-1 |
0.0217 |
0.0422 |
0.0274 |
0.0239 |
0.0000 |
Neural Re-Ranking |
0.0022 |
0.0033 |
0.0026 |
0.0039 |
0.0000 |
Approach 1 |
0.0015 |
0.0016 |
0.0015 |
0.0006 |
0.0000 |
DS@GT BioASQ 1b-2 |
0.0217 |
0.0422 |
0.0274 |
0.0239 |
0.0000 |
GoldenRetrievers-2 |
- |
- |
- |
- |
- |
dl approach |
0.0010 |
0.0065 |
0.0017 |
0.0002 |
0.0000 |
GPT4O |
0.0160 |
0.0437 |
0.0218 |
0.0271 |
0.0000 |
DS@GTBioASQT13b5 |
0.0217 |
0.0422 |
0.0274 |
0.0239 |
0.0000 |
sp_lasigebiotm |
0.0119 |
0.0251 |
0.0155 |
0.0265 |
0.0000 |
mistral |
0.0457 |
0.0441 |
0.0339 |
0.0262 |
0.0000 |
lasigeBioTM |
0.0119 |
0.0251 |
0.0155 |
0.0265 |
0.0000 |
llama |
0.0348 |
0.0419 |
0.0308 |
0.0273 |
0.0000 |
lasigeBioTM-onto-bl |
0.0119 |
0.0251 |
0.0155 |
0.0265 |
0.0000 |
lasigeBioTM-onto-sm |
0.0119 |
0.0251 |
0.0155 |
0.0265 |
0.0000 |
1.PhaseA_System |
0.0068 |
0.0138 |
0.0080 |
0.0125 |
0.0000 |
deepseek-r1:32b |
0.0159 |
0.0273 |
0.0198 |
0.0197 |
0.0000 |
deepseek-r1:14b |
0.0159 |
0.0273 |
0.0198 |
0.0197 |
0.0000 |
deepseek-r1:8b |
0.0106 |
0.0203 |
0.0136 |
0.0077 |
0.0000 |
DS@GTBioASQT13b4 |
0.0217 |
0.0422 |
0.0274 |
0.0239 |
0.0000 |
DS@GTBioASQT13b3 |
0.0217 |
0.0422 |
0.0274 |
0.0239 |
0.0000 |
dense |
0.0407 |
0.0494 |
0.0366 |
0.0325 |
0.0000 |
Fleming-2 |
0.0222 |
0.0562 |
0.0288 |
0.0382 |
0.0001 |
Concepts
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
Data Wizards BM25 |
- |
- |
- |
- |
- |
NLP_BioBert |
- |
- |
- |
- |
- |
Data Wizards Re-Rank |
- |
- |
- |
- |
- |
First abrotsch |
- |
- |
- |
- |
- |
ms-marco-MiniLML6v2 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
BM25-Okapi |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
IR4 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
Reranking rankings |
- |
- |
- |
- |
- |
query-engineering |
- |
- |
- |
- |
- |
Re-Ranker with BM25 |
- |
- |
- |
- |
- |
RetrievalRivals |
- |
- |
- |
- |
- |
RetrievalRivals_BM25 |
- |
- |
- |
- |
- |
RetrieverRivals |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
TEsting123 |
- |
- |
- |
- |
- |
Traditional IR Methd |
- |
- |
- |
- |
- |
traditional IR |
- |
- |
- |
- |
- |
Neural IR approach |
- |
- |
- |
- |
- |
Machinen Results |
- |
- |
- |
- |
- |
Tuwien Group 6 AIR |
- |
- |
- |
- |
- |
reranking after IR |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
Okapi BM25 IR |
- |
- |
- |
- |
- |
NNR Model |
- |
- |
- |
- |
- |
GoldenRetrievers |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
AQAMS |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
DB_vector_&_LLM |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
The Relevants |
- |
- |
- |
- |
- |
Data Wizards NN |
- |
- |
- |
- |
- |
API and weirdnesss |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-1 |
- |
- |
- |
- |
- |
Neural Re-Ranking |
- |
- |
- |
- |
- |
Approach 1 |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-2 |
- |
- |
- |
- |
- |
GoldenRetrievers-2 |
- |
- |
- |
- |
- |
dl approach |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
DS@GTBioASQT13b5 |
- |
- |
- |
- |
- |
sp_lasigebiotm |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
lasigeBioTM-onto-bl |
- |
- |
- |
- |
- |
lasigeBioTM-onto-sm |
- |
- |
- |
- |
- |
1.PhaseA_System |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
DS@GTBioASQT13b4 |
- |
- |
- |
- |
- |
DS@GTBioASQT13b3 |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
Fleming-2 |
- |
- |
- |
- |
- |
RDF Triples
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
Data Wizards BM25 |
- |
- |
- |
- |
- |
NLP_BioBert |
- |
- |
- |
- |
- |
Data Wizards Re-Rank |
- |
- |
- |
- |
- |
First abrotsch |
- |
- |
- |
- |
- |
ms-marco-MiniLML6v2 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
BM25-Okapi |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
IR4 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
Reranking rankings |
- |
- |
- |
- |
- |
query-engineering |
- |
- |
- |
- |
- |
Re-Ranker with BM25 |
- |
- |
- |
- |
- |
RetrievalRivals |
- |
- |
- |
- |
- |
RetrievalRivals_BM25 |
- |
- |
- |
- |
- |
RetrieverRivals |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
TEsting123 |
- |
- |
- |
- |
- |
Traditional IR Methd |
- |
- |
- |
- |
- |
traditional IR |
- |
- |
- |
- |
- |
Neural IR approach |
- |
- |
- |
- |
- |
Machinen Results |
- |
- |
- |
- |
- |
Tuwien Group 6 AIR |
- |
- |
- |
- |
- |
reranking after IR |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
Okapi BM25 IR |
- |
- |
- |
- |
- |
NNR Model |
- |
- |
- |
- |
- |
GoldenRetrievers |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
AQAMS |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
DB_vector_&_LLM |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
The Relevants |
- |
- |
- |
- |
- |
Data Wizards NN |
- |
- |
- |
- |
- |
API and weirdnesss |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-1 |
- |
- |
- |
- |
- |
Neural Re-Ranking |
- |
- |
- |
- |
- |
Approach 1 |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-2 |
- |
- |
- |
- |
- |
GoldenRetrievers-2 |
- |
- |
- |
- |
- |
dl approach |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
DS@GTBioASQT13b5 |
- |
- |
- |
- |
- |
sp_lasigebiotm |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
lasigeBioTM-onto-bl |
- |
- |
- |
- |
- |
lasigeBioTM-onto-sm |
- |
- |
- |
- |
- |
1.PhaseA_System |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
DS@GTBioASQT13b4 |
- |
- |
- |
- |
- |
DS@GTBioASQT13b3 |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
Fleming-2 |
- |
- |
- |
- |
- |