BioASQ Participants Area
Task 13b: Test Results of Phase A
* The "-" replace the scores of systems that didn't submit the corresponding annotations. *
The test results are presented in separate tables for each type of annotation. They are sorted based on the scores of MAP. The "System Description" of each system is used.
The evaluation measures that are used are presented
here .
Test batch 1
Documents
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.2030 |
0.3477 |
0.1528 |
0.3349 |
0.0254 |
IR1 |
0.3024 |
0.4103 |
0.2145 |
0.4594 |
0.1415 |
UniTor_0 |
0.2376 |
0.3617 |
0.1715 |
0.3979 |
0.0766 |
UniTor_1 |
0.3629 |
0.3307 |
0.2073 |
0.3711 |
0.0552 |
Baseline top 20 |
0.3953 |
0.4688 |
0.2728 |
0.5627 |
0.3121 |
Baseline top 10 |
0.3953 |
0.4688 |
0.2728 |
0.5627 |
0.3121 |
Using LLM alone |
0.3694 |
0.4403 |
0.2514 |
0.5219 |
0.2729 |
Using KG for list q |
0.4641 |
0.4230 |
0.3264 |
0.4800 |
0.1905 |
Main pipeline |
0.4659 |
0.4256 |
0.3309 |
0.4796 |
0.1766 |
13b_phase_a |
0.0906 |
0.1465 |
0.0810 |
0.1190 |
0.0004 |
13b-1 |
- |
- |
- |
- |
- |
bioinfo-0 |
0.3259 |
0.4362 |
0.2467 |
0.4865 |
0.2568 |
bioinfo-1 |
0.3271 |
0.4360 |
0.2467 |
0.4816 |
0.2525 |
bioinfo-2 |
0.3353 |
0.4379 |
0.2501 |
0.5057 |
0.2410 |
bioinfo-3 |
0.3200 |
0.4334 |
0.2418 |
0.4933 |
0.2345 |
bioinfo-4 |
0.3318 |
0.4368 |
0.2479 |
0.5015 |
0.2627 |
UniTor_2 |
0.2906 |
0.3845 |
0.2057 |
0.4196 |
0.0799 |
UniTor_3 |
0.3953 |
0.3501 |
0.2208 |
0.3778 |
0.0722 |
DB_vector_&_LLM |
0.0200 |
0.0056 |
0.0085 |
0.0096 |
0.0000 |
UR-IW-1 |
0.3565 |
0.3045 |
0.2595 |
0.3726 |
0.0277 |
UR-IW-2 |
0.3659 |
0.2931 |
0.2589 |
0.3652 |
0.0180 |
UR-IW-3 |
0.3650 |
0.2752 |
0.2556 |
0.3534 |
0.0221 |
UR-IW-4 |
0.2575 |
0.1964 |
0.1724 |
0.2724 |
0.0019 |
google_serach_&_LLM |
0.0773 |
0.0303 |
0.0380 |
0.0430 |
0.0002 |
UR-IW-5 |
0.4079 |
0.3428 |
0.3013 |
0.4216 |
0.0577 |
no expand only rank |
0.1618 |
0.1291 |
0.1304 |
0.1359 |
0.0004 |
DS@GT BioASQ 1b-1 |
0.1396 |
0.2154 |
0.1066 |
0.2387 |
0.0021 |
IRIS_1 |
0.2153 |
0.3799 |
0.1688 |
0.3693 |
0.0851 |
IRIS_3 |
0.0082 |
0.0171 |
0.0075 |
0.0056 |
0.0000 |
IRIS_2 |
0.0282 |
0.0168 |
0.0159 |
0.0121 |
0.0001 |
deepseek32b-me |
0.1902 |
0.1532 |
0.1508 |
0.1474 |
0.0003 |
dmiip2024 |
0.3047 |
0.4236 |
0.2226 |
0.4765 |
0.1741 |
dmiip2024_1 |
0.3094 |
0.4290 |
0.2250 |
0.4844 |
0.1760 |
dmiip2024_2 |
0.3259 |
0.4229 |
0.2308 |
0.4851 |
0.1989 |
dmiip2024_3 |
0.3318 |
0.4270 |
0.2355 |
0.4889 |
0.2057 |
dmiip2024_4 |
0.3294 |
0.4217 |
0.2337 |
0.4926 |
0.2062 |
config-1 |
0.2847 |
0.3653 |
0.1973 |
0.3959 |
0.0968 |
config-2 |
0.2247 |
0.3447 |
0.1622 |
0.2831 |
0.0431 |
config-3 |
0.2165 |
0.3329 |
0.1588 |
0.3109 |
0.0270 |
config-4 |
0.2626 |
0.3651 |
0.1864 |
0.2302 |
0.0633 |
config-5 |
0.2614 |
0.3650 |
0.1861 |
0.2262 |
0.0628 |
mistral |
0.2755 |
0.2675 |
0.1999 |
0.2909 |
0.0124 |
deepseek32b-full |
0.2355 |
0.1966 |
0.1812 |
0.1759 |
0.0023 |
GPT4O |
0.0853 |
0.1075 |
0.0643 |
0.0742 |
0.0004 |
deepseek-r1:32b |
0.0946 |
0.1633 |
0.0795 |
0.0840 |
0.0006 |
deepseek-r1:14b |
0.0630 |
0.1076 |
0.0505 |
0.0875 |
0.0002 |
using free 7b LLM |
- |
- |
- |
- |
- |
llama |
0.2890 |
0.2904 |
0.2300 |
0.3271 |
0.0186 |
dense |
0.1178 |
0.1746 |
0.0833 |
0.1619 |
0.0033 |
deepseek-r1:8b |
0.0805 |
0.1682 |
0.0765 |
0.1002 |
0.0006 |
lasigeBioTM |
0.2188 |
0.3619 |
0.1650 |
0.3827 |
0.0733 |
Snippets
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.1412 |
0.1076 |
0.0876 |
0.1867 |
0.0055 |
IR1 |
- |
- |
- |
- |
- |
UniTor_0 |
0.3118 |
0.2078 |
0.1945 |
0.4130 |
0.0477 |
UniTor_1 |
0.2993 |
0.2111 |
0.1897 |
0.4199 |
0.0494 |
Baseline top 20 |
0.3648 |
0.2422 |
0.2314 |
0.4040 |
0.1150 |
Baseline top 10 |
0.3648 |
0.2422 |
0.2314 |
0.4040 |
0.1150 |
Using LLM alone |
0.3389 |
0.2241 |
0.2140 |
0.3745 |
0.1158 |
Using KG for list q |
0.3971 |
0.2100 |
0.2270 |
0.3470 |
0.0818 |
Main pipeline |
0.4081 |
0.2176 |
0.2366 |
0.3499 |
0.0832 |
13b_phase_a |
0.0619 |
0.0195 |
0.0235 |
0.0605 |
0.0001 |
13b-1 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
0.3109 |
0.2181 |
0.1925 |
0.4231 |
0.0579 |
UniTor_3 |
0.3093 |
0.2291 |
0.1919 |
0.4288 |
0.0660 |
DB_vector_&_LLM |
0.0288 |
0.0067 |
0.0105 |
0.0121 |
0.0000 |
UR-IW-1 |
0.3017 |
0.1589 |
0.1689 |
0.4009 |
0.0152 |
UR-IW-2 |
0.3346 |
0.1681 |
0.1865 |
0.3947 |
0.0153 |
UR-IW-3 |
0.3082 |
0.1590 |
0.1739 |
0.3994 |
0.0137 |
UR-IW-4 |
0.2447 |
0.1171 |
0.1311 |
0.2878 |
0.0014 |
google_serach_&_LLM |
0.1797 |
0.0536 |
0.0726 |
0.0327 |
0.0002 |
UR-IW-5 |
0.3447 |
0.1835 |
0.1972 |
0.4298 |
0.0217 |
no expand only rank |
0.0805 |
0.0148 |
0.0225 |
0.0533 |
0.0001 |
DS@GT BioASQ 1b-1 |
0.1280 |
0.1421 |
0.1011 |
0.1620 |
0.0004 |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
deepseek32b-me |
0.1301 |
0.0499 |
0.0654 |
0.1130 |
0.0002 |
dmiip2024 |
0.2663 |
0.3248 |
0.2003 |
0.6156 |
0.1195 |
dmiip2024_1 |
0.2687 |
0.3308 |
0.2042 |
0.6137 |
0.1113 |
dmiip2024_2 |
0.2763 |
0.3285 |
0.2068 |
0.6062 |
0.1003 |
dmiip2024_3 |
0.2805 |
0.3275 |
0.2085 |
0.5935 |
0.1137 |
dmiip2024_4 |
0.2721 |
0.3193 |
0.2027 |
0.6020 |
0.0978 |
config-1 |
0.1688 |
0.0797 |
0.0812 |
0.1826 |
0.0067 |
config-2 |
0.1437 |
0.0578 |
0.0625 |
0.1254 |
0.0024 |
config-3 |
0.1283 |
0.0550 |
0.0595 |
0.1281 |
0.0015 |
config-4 |
0.1665 |
0.0751 |
0.0778 |
0.1111 |
0.0045 |
config-5 |
0.1662 |
0.0751 |
0.0779 |
0.1101 |
0.0044 |
mistral |
0.2562 |
0.0749 |
0.1040 |
0.1865 |
0.0033 |
deepseek32b-full |
0.1665 |
0.0593 |
0.0785 |
0.1340 |
0.0007 |
GPT4O |
0.0625 |
0.0194 |
0.0240 |
0.0410 |
0.0001 |
deepseek-r1:32b |
0.0612 |
0.0353 |
0.0337 |
0.0504 |
0.0002 |
deepseek-r1:14b |
0.0467 |
0.0302 |
0.0277 |
0.0553 |
0.0001 |
using free 7b LLM |
- |
- |
- |
- |
- |
llama |
0.2729 |
0.1056 |
0.1341 |
0.2407 |
0.0081 |
dense |
0.1659 |
0.1106 |
0.0904 |
0.2330 |
0.0019 |
deepseek-r1:8b |
0.0525 |
0.0447 |
0.0358 |
0.0609 |
0.0002 |
lasigeBioTM |
0.1012 |
0.0615 |
0.0574 |
0.1817 |
0.0043 |
Concepts
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
13b_phase_a |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
DB_vector_&_LLM |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
google_serach_&_LLM |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
no expand only rank |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-1 |
- |
- |
- |
- |
- |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
config-1 |
- |
- |
- |
- |
- |
config-2 |
- |
- |
- |
- |
- |
config-3 |
- |
- |
- |
- |
- |
config-4 |
- |
- |
- |
- |
- |
config-5 |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
using free 7b LLM |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
RDF Triples
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
13b_phase_a |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
DB_vector_&_LLM |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
google_serach_&_LLM |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
no expand only rank |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-1 |
- |
- |
- |
- |
- |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
config-1 |
- |
- |
- |
- |
- |
config-2 |
- |
- |
- |
- |
- |
config-3 |
- |
- |
- |
- |
- |
config-4 |
- |
- |
- |
- |
- |
config-5 |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
using free 7b LLM |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
Test batch 2
Documents
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.2941 |
0.4245 |
0.2103 |
0.4449 |
0.0965 |
Fleming-2 |
0.3089 |
0.4245 |
0.2240 |
0.4317 |
0.0893 |
UniTor_0 |
0.2695 |
0.3289 |
0.1672 |
0.4210 |
0.0447 |
UniTor_1 |
0.3982 |
0.3000 |
0.2016 |
0.3841 |
0.0413 |
UniTor_2 |
0.2859 |
0.3360 |
0.1780 |
0.4468 |
0.0526 |
UniTor_3 |
0.4254 |
0.3130 |
0.2140 |
0.3996 |
0.0485 |
UR-IW-1 |
0.4596 |
0.2847 |
0.2796 |
0.4405 |
0.0476 |
UR-IW-2 |
0.4161 |
0.2586 |
0.2417 |
0.4167 |
0.0250 |
UR-IW-3 |
0.4085 |
0.2926 |
0.2791 |
0.4034 |
0.0245 |
UR-IW-4 |
0.4670 |
0.3041 |
0.2804 |
0.4778 |
0.0592 |
Baseline top 10 |
0.4635 |
0.4888 |
0.2922 |
0.6768 |
0.3837 |
Baseline top 20 |
0.4635 |
0.4888 |
0.2922 |
0.6768 |
0.3837 |
Main pipeline |
0.5646 |
0.4121 |
0.3636 |
0.5423 |
0.2186 |
Using KG for list q |
0.5650 |
0.3984 |
0.3586 |
0.5299 |
0.1906 |
Using LLM alone |
0.4082 |
0.4581 |
0.2552 |
0.5945 |
0.2823 |
dmiip2024 |
0.4059 |
0.4834 |
0.2692 |
0.6167 |
0.3445 |
dmiip2024_1 |
0.4047 |
0.4827 |
0.2682 |
0.6145 |
0.3393 |
dmiip2024_2 |
0.4271 |
0.4711 |
0.2774 |
0.6263 |
0.3217 |
dmiip2024_3 |
0.4259 |
0.4698 |
0.2759 |
0.6212 |
0.3176 |
dmiip2024_4 |
0.4224 |
0.4701 |
0.2758 |
0.6245 |
0.3202 |
bioinfo-0 |
0.4106 |
0.4717 |
0.2699 |
0.6183 |
0.2800 |
bioinfo-1 |
0.4365 |
0.4869 |
0.2846 |
0.6424 |
0.3932 |
bioinfo-2 |
0.4106 |
0.4717 |
0.2699 |
0.6183 |
0.2800 |
bioinfo-3 |
0.4071 |
0.4704 |
0.2684 |
0.6161 |
0.3009 |
bioinfo-4 |
0.4035 |
0.4700 |
0.2673 |
0.6066 |
0.2936 |
IR5 |
0.2659 |
0.3824 |
0.1861 |
0.4285 |
0.0932 |
lasigeBioTM |
- |
- |
- |
- |
- |
UR-IW-5 |
0.5113 |
0.2988 |
0.3117 |
0.4745 |
0.0561 |
deepseek32b-full |
0.3347 |
0.2031 |
0.1910 |
0.3029 |
0.0108 |
mistral |
0.2962 |
0.1643 |
0.1640 |
0.2716 |
0.0054 |
llama |
0.4120 |
0.2593 |
0.2632 |
0.3936 |
0.0344 |
dense |
0.4089 |
0.2618 |
0.2562 |
0.4055 |
0.0303 |
GPT4O |
0.1518 |
0.1053 |
0.0792 |
0.1634 |
0.0009 |
deepseek-r1:32b |
0.2598 |
0.2027 |
0.1533 |
0.2841 |
0.0103 |
deepseek-r1:14b |
0.1253 |
0.0765 |
0.0641 |
0.0802 |
0.0008 |
deepseek-r1:8b |
0.1384 |
0.1166 |
0.0880 |
0.1134 |
0.0006 |
gpt 01 mini |
0.1660 |
0.1363 |
0.1025 |
0.1166 |
0.0016 |
13b-1 |
0.2455 |
0.2224 |
0.1642 |
0.2829 |
0.0071 |
IR1 |
0.3744 |
0.4106 |
0.2318 |
0.5454 |
0.1657 |
Fleming-3 |
0.3100 |
0.4250 |
0.2247 |
0.4329 |
0.0996 |
deepseek32b-me |
0.3009 |
0.1465 |
0.1532 |
0.2096 |
0.0050 |
Snippets
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.2040 |
0.1756 |
0.1335 |
0.2974 |
0.0302 |
Fleming-2 |
0.2058 |
0.1199 |
0.1058 |
0.2411 |
0.0099 |
UniTor_0 |
0.3075 |
0.1961 |
0.1741 |
0.4331 |
0.0376 |
UniTor_1 |
0.2933 |
0.1971 |
0.1659 |
0.4366 |
0.0385 |
UniTor_2 |
0.3217 |
0.2165 |
0.1879 |
0.4564 |
0.0421 |
UniTor_3 |
0.3065 |
0.2185 |
0.1798 |
0.4604 |
0.0435 |
UR-IW-1 |
0.3939 |
0.1680 |
0.1966 |
0.4737 |
0.0333 |
UR-IW-2 |
0.3731 |
0.1437 |
0.1744 |
0.3897 |
0.0208 |
UR-IW-3 |
0.3626 |
0.1605 |
0.1852 |
0.4504 |
0.0248 |
UR-IW-4 |
0.4125 |
0.1610 |
0.1941 |
0.4654 |
0.0325 |
Baseline top 10 |
0.4450 |
0.2813 |
0.2729 |
0.5267 |
0.2319 |
Baseline top 20 |
0.4450 |
0.2813 |
0.2729 |
0.5267 |
0.2319 |
Main pipeline |
0.4913 |
0.2389 |
0.2746 |
0.4330 |
0.0912 |
Using KG for list q |
0.4909 |
0.2330 |
0.2684 |
0.4157 |
0.0858 |
Using LLM alone |
0.3987 |
0.2489 |
0.2371 |
0.4596 |
0.1562 |
dmiip2024 |
0.2712 |
0.3346 |
0.2099 |
0.6604 |
0.1676 |
dmiip2024_1 |
0.2785 |
0.3376 |
0.2149 |
0.6860 |
0.1896 |
dmiip2024_2 |
0.2649 |
0.3297 |
0.2061 |
0.6462 |
0.1411 |
dmiip2024_3 |
0.2698 |
0.3225 |
0.2040 |
0.6464 |
0.1504 |
dmiip2024_4 |
0.2671 |
0.3331 |
0.2089 |
0.6402 |
0.1533 |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
lasigeBioTM |
0.2609 |
0.0831 |
0.0931 |
0.2184 |
0.0058 |
UR-IW-5 |
0.4345 |
0.1654 |
0.2020 |
0.5208 |
0.0393 |
deepseek32b-full |
0.2735 |
0.1032 |
0.1234 |
0.3136 |
0.0065 |
mistral |
0.2587 |
0.0653 |
0.0954 |
0.1872 |
0.0022 |
llama |
0.3801 |
0.1257 |
0.1707 |
0.3612 |
0.0206 |
dense |
0.3690 |
0.1264 |
0.1674 |
0.3680 |
0.0186 |
GPT4O |
0.1227 |
0.0388 |
0.0463 |
0.1126 |
0.0004 |
deepseek-r1:32b |
0.1986 |
0.0621 |
0.0780 |
0.1999 |
0.0029 |
deepseek-r1:14b |
0.0989 |
0.0293 |
0.0390 |
0.0551 |
0.0003 |
deepseek-r1:8b |
0.0991 |
0.0407 |
0.0457 |
0.0795 |
0.0002 |
gpt 01 mini |
0.1125 |
0.0399 |
0.0489 |
0.0679 |
0.0006 |
13b-1 |
0.2348 |
0.0942 |
0.1065 |
0.2349 |
0.0033 |
IR1 |
0.3503 |
0.2150 |
0.1912 |
0.4698 |
0.0772 |
Fleming-3 |
0.2058 |
0.1199 |
0.1058 |
0.2411 |
0.0099 |
deepseek32b-me |
0.2540 |
0.0680 |
0.0906 |
0.2204 |
0.0036 |
Concepts
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
Fleming-2 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
gpt 01 mini |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
Fleming-3 |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
RDF Triples
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
Fleming-2 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
gpt 01 mini |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
Fleming-3 |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
Test batch 3
Documents
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
NLP_BioBert |
0.0576 |
0.0251 |
0.0307 |
0.0403 |
0.0001 |
NLP_Group5_TU |
0.0576 |
0.0251 |
0.0307 |
0.0403 |
0.0001 |
Fleming-1 |
0.3063 |
0.2996 |
0.2093 |
0.3288 |
0.0526 |
IR5 |
0.2718 |
0.2783 |
0.1816 |
0.3250 |
0.0592 |
IR1 |
0.4071 |
0.3471 |
0.2575 |
0.4749 |
0.2414 |
IR4 |
0.4094 |
0.3505 |
0.2613 |
0.4681 |
0.2014 |
UR-IW-1 |
0.4478 |
0.2473 |
0.2580 |
0.3975 |
0.0560 |
UR-IW-2 |
0.3654 |
0.1926 |
0.2114 |
0.3224 |
0.0235 |
UR-IW-4 |
0.3778 |
0.1824 |
0.2144 |
0.3127 |
0.0172 |
UR-IW-5 |
0.4804 |
0.2641 |
0.2969 |
0.4066 |
0.0615 |
lasigeBioTM |
0.2714 |
0.1486 |
0.1462 |
0.2265 |
0.0031 |
lasigeBioTM-onto-sm |
0.2714 |
0.1486 |
0.1462 |
0.2265 |
0.0031 |
Baseline top 10 |
0.4959 |
0.3977 |
0.3139 |
0.5757 |
0.3983 |
Using LLM alone |
0.4641 |
0.3753 |
0.2905 |
0.5193 |
0.3428 |
Using KG for list q |
0.5065 |
0.3308 |
0.3188 |
0.4571 |
0.1211 |
Main pipeline |
0.5093 |
0.3261 |
0.3143 |
0.4513 |
0.1180 |
Baseline top 20 |
0.4959 |
0.3977 |
0.3139 |
0.5757 |
0.3983 |
UniTor_0 |
0.2855 |
0.2628 |
0.1759 |
0.3554 |
0.0408 |
UniTor_1 |
0.3980 |
0.2422 |
0.1990 |
0.3067 |
0.0342 |
bioinfo-0 |
0.4347 |
0.3676 |
0.2778 |
0.5229 |
0.2641 |
bioinfo-1 |
0.4565 |
0.3784 |
0.2887 |
0.5410 |
0.2545 |
bioinfo-2 |
0.4194 |
0.3614 |
0.2693 |
0.5123 |
0.2551 |
bioinfo-3 |
0.4312 |
0.3651 |
0.2746 |
0.5168 |
0.2629 |
bioinfo-4 |
0.4341 |
0.3677 |
0.2754 |
0.5212 |
0.2648 |
UniTor_2 |
0.3271 |
0.2836 |
0.2011 |
0.3868 |
0.0515 |
UniTor_3 |
0.4243 |
0.2518 |
0.2035 |
0.3216 |
0.0392 |
dmiip2024 |
0.4518 |
0.3800 |
0.2892 |
0.5283 |
0.3132 |
dmiip2024_1 |
0.4318 |
0.3724 |
0.2783 |
0.5178 |
0.2739 |
dmiip2024_2 |
0.4659 |
0.3719 |
0.2922 |
0.5464 |
0.2869 |
dmiip2024_4 |
0.4682 |
0.3722 |
0.2931 |
0.5490 |
0.2664 |
dmiip2024_3 |
0.4494 |
0.3625 |
0.2808 |
0.5332 |
0.2543 |
UR-IW-3 |
0.3596 |
0.1936 |
0.2031 |
0.3170 |
0.0300 |
lasigeBioTM-onto-bl |
0.2714 |
0.1486 |
0.1462 |
0.2265 |
0.0031 |
IRIS_1 |
0.3235 |
0.2895 |
0.2099 |
0.3711 |
0.0581 |
IRIS_2 |
0.0365 |
0.0099 |
0.0145 |
0.0138 |
0.0001 |
IRIS_3 |
0.0494 |
0.0234 |
0.0201 |
0.0180 |
0.0001 |
13b-1 |
- |
- |
- |
- |
- |
13b_phase_a |
0.1421 |
0.0899 |
0.0790 |
0.1085 |
0.0006 |
google_serach_&_LLM |
0.0376 |
0.0140 |
0.0161 |
0.0249 |
0.0000 |
mistral |
0.4165 |
0.2132 |
0.2405 |
0.3324 |
0.0330 |
llama |
0.4139 |
0.2084 |
0.2244 |
0.3080 |
0.0318 |
dense |
0.4045 |
0.2066 |
0.2334 |
0.3184 |
0.0302 |
GPT4O |
0.1399 |
0.0827 |
0.0852 |
0.0822 |
0.0007 |
deepseek-r1:32b |
0.1304 |
0.0819 |
0.0815 |
0.0991 |
0.0005 |
deepseek-r1:14b |
0.1695 |
0.1356 |
0.1156 |
0.1165 |
0.0015 |
sp_lasigebiotm |
0.2714 |
0.1486 |
0.1462 |
0.2265 |
0.0031 |
AQAMS |
0.0951 |
0.0289 |
0.0394 |
0.0430 |
0.0001 |
Snippets
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
NLP_BioBert |
0.0907 |
0.0221 |
0.0328 |
0.0342 |
0.0001 |
NLP_Group5_TU |
0.0907 |
0.0221 |
0.0328 |
0.0342 |
0.0001 |
Fleming-1 |
0.2200 |
0.1020 |
0.1070 |
0.2028 |
0.0110 |
IR5 |
- |
- |
- |
- |
- |
IR1 |
0.4157 |
0.1962 |
0.2197 |
0.4155 |
0.0725 |
IR4 |
0.4570 |
0.2215 |
0.2469 |
0.4633 |
0.1086 |
UR-IW-1 |
0.4221 |
0.1490 |
0.1897 |
0.3558 |
0.0338 |
UR-IW-2 |
0.3772 |
0.1132 |
0.1508 |
0.2934 |
0.0107 |
UR-IW-4 |
0.3739 |
0.1117 |
0.1509 |
0.2941 |
0.0104 |
UR-IW-5 |
0.4520 |
0.1621 |
0.2115 |
0.3732 |
0.0353 |
lasigeBioTM |
0.1918 |
0.0462 |
0.0645 |
0.1181 |
0.0006 |
lasigeBioTM-onto-sm |
0.1918 |
0.0462 |
0.0645 |
0.1181 |
0.0006 |
Baseline top 10 |
0.4600 |
0.2236 |
0.2501 |
0.4597 |
0.2228 |
Using LLM alone |
0.4166 |
0.2022 |
0.2218 |
0.4111 |
0.1829 |
Using KG for list q |
0.4565 |
0.1958 |
0.2355 |
0.3554 |
0.0660 |
Main pipeline |
0.4531 |
0.1937 |
0.2319 |
0.3467 |
0.0684 |
Baseline top 20 |
0.4600 |
0.2236 |
0.2501 |
0.4597 |
0.2228 |
UniTor_0 |
0.3354 |
0.1485 |
0.1601 |
0.3495 |
0.0238 |
UniTor_1 |
0.3327 |
0.1540 |
0.1616 |
0.3524 |
0.0246 |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
0.3357 |
0.1675 |
0.1735 |
0.3637 |
0.0277 |
UniTor_3 |
0.3268 |
0.1677 |
0.1686 |
0.3638 |
0.0299 |
dmiip2024 |
0.3465 |
0.2917 |
0.2391 |
0.6352 |
0.1546 |
dmiip2024_1 |
0.3520 |
0.3001 |
0.2436 |
0.6425 |
0.1773 |
dmiip2024_2 |
0.3519 |
0.3059 |
0.2439 |
0.6296 |
0.1365 |
dmiip2024_4 |
0.3493 |
0.3035 |
0.2420 |
0.6235 |
0.1387 |
dmiip2024_3 |
0.3630 |
0.3023 |
0.2467 |
0.6446 |
0.1439 |
UR-IW-3 |
0.3207 |
0.1246 |
0.1532 |
0.3096 |
0.0141 |
lasigeBioTM-onto-bl |
0.1918 |
0.0462 |
0.0645 |
0.1181 |
0.0006 |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
13b_phase_a |
0.0604 |
0.0063 |
0.0108 |
0.0356 |
0.0001 |
google_serach_&_LLM |
0.1837 |
0.0763 |
0.0938 |
0.0533 |
0.0002 |
mistral |
0.3965 |
0.1173 |
0.1574 |
0.2743 |
0.0193 |
llama |
0.4109 |
0.1150 |
0.1534 |
0.2445 |
0.0161 |
dense |
0.3843 |
0.1066 |
0.1463 |
0.2647 |
0.0176 |
GPT4O |
0.1165 |
0.0350 |
0.0459 |
0.0470 |
0.0003 |
deepseek-r1:32b |
0.1041 |
0.0310 |
0.0429 |
0.0503 |
0.0002 |
deepseek-r1:14b |
0.1328 |
0.0382 |
0.0538 |
0.0642 |
0.0006 |
sp_lasigebiotm |
0.1918 |
0.0462 |
0.0645 |
0.1181 |
0.0006 |
AQAMS |
0.1094 |
0.1348 |
0.0985 |
0.0398 |
0.0001 |
Concepts
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
NLP_BioBert |
- |
- |
- |
- |
- |
NLP_Group5_TU |
- |
- |
- |
- |
- |
Fleming-1 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
IR4 |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
lasigeBioTM-onto-sm |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
lasigeBioTM-onto-bl |
- |
- |
- |
- |
- |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
13b_phase_a |
- |
- |
- |
- |
- |
google_serach_&_LLM |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
sp_lasigebiotm |
- |
- |
- |
- |
- |
AQAMS |
- |
- |
- |
- |
- |
RDF Triples
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
NLP_BioBert |
- |
- |
- |
- |
- |
NLP_Group5_TU |
- |
- |
- |
- |
- |
Fleming-1 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
IR4 |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
lasigeBioTM-onto-sm |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
lasigeBioTM-onto-bl |
- |
- |
- |
- |
- |
IRIS_1 |
- |
- |
- |
- |
- |
IRIS_2 |
- |
- |
- |
- |
- |
IRIS_3 |
- |
- |
- |
- |
- |
13b-1 |
- |
- |
- |
- |
- |
13b_phase_a |
- |
- |
- |
- |
- |
google_serach_&_LLM |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
sp_lasigebiotm |
- |
- |
- |
- |
- |
AQAMS |
- |
- |
- |
- |
- |
Test batch 4
Documents
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.3172 |
0.1289 |
0.1504 |
0.2524 |
0.0223 |
Data Wizards BM25 |
0.0376 |
0.0109 |
0.0158 |
0.0234 |
0.0001 |
NLP_BioBert |
0.0718 |
0.0220 |
0.0310 |
0.0467 |
0.0002 |
Data Wizards Re-Rank |
0.1078 |
0.0062 |
0.0113 |
0.0180 |
0.0001 |
First abrotsch |
0.0706 |
0.0232 |
0.0321 |
0.0441 |
0.0002 |
ms-marco-MiniLML6v2 |
0.0706 |
0.0232 |
0.0321 |
0.0441 |
0.0002 |
IR5 |
0.2918 |
0.1180 |
0.1374 |
0.2123 |
0.0249 |
BM25-Okapi |
0.0600 |
0.0195 |
0.0276 |
0.0366 |
0.0001 |
IR1 |
0.4471 |
0.1641 |
0.2029 |
0.3763 |
0.1165 |
IR4 |
0.4353 |
0.1674 |
0.2037 |
0.3592 |
0.1195 |
UniTor_0 |
0.2706 |
0.0899 |
0.1157 |
0.2195 |
0.0183 |
UniTor_1 |
0.3880 |
0.0784 |
0.1177 |
0.1917 |
0.0119 |
Reranking rankings |
0.1489 |
0.0641 |
0.0710 |
0.0943 |
0.0010 |
query-engineering |
0.1242 |
0.0549 |
0.0582 |
0.0771 |
0.0005 |
Re-Ranker with BM25 |
0.0988 |
0.0291 |
0.0415 |
0.0698 |
0.0004 |
RetrievalRivals |
0.2247 |
0.0951 |
0.1089 |
0.1532 |
0.0074 |
RetrievalRivals_BM25 |
0.2247 |
0.0951 |
0.1089 |
0.1532 |
0.0074 |
RetrieverRivals |
0.2247 |
0.0951 |
0.1089 |
0.1532 |
0.0074 |
UniTor_2 |
0.3635 |
0.1289 |
0.1606 |
0.3002 |
0.0353 |
UniTor_3 |
0.4303 |
0.0967 |
0.1329 |
0.2216 |
0.0203 |
TEsting123 |
0.0676 |
0.0303 |
0.0317 |
0.0300 |
0.0001 |
Traditional IR Methd |
0.0676 |
0.0303 |
0.0317 |
0.0300 |
0.0001 |
traditional IR |
0.1569 |
0.0315 |
0.0422 |
0.0641 |
0.0006 |
Neural IR approach |
0.1878 |
0.0346 |
0.0534 |
0.0883 |
0.0008 |
Machinen Results |
0.2953 |
0.1079 |
0.1360 |
0.2302 |
0.0232 |
Tuwien Group 6 AIR |
0.1878 |
0.0346 |
0.0534 |
0.0883 |
0.0008 |
reranking after IR |
0.1539 |
0.0111 |
0.0195 |
0.0333 |
0.0001 |
bioinfo-0 |
0.4776 |
0.1829 |
0.2244 |
0.4139 |
0.1311 |
bioinfo-1 |
0.5129 |
0.1983 |
0.2442 |
0.4342 |
0.2073 |
bioinfo-2 |
0.4635 |
0.1785 |
0.2181 |
0.3998 |
0.1391 |
bioinfo-3 |
0.4741 |
0.1817 |
0.2227 |
0.4107 |
0.1387 |
bioinfo-4 |
0.4624 |
0.1783 |
0.2178 |
0.3992 |
0.1385 |
Okapi BM25 IR |
0.0442 |
0.0142 |
0.0201 |
0.0229 |
0.0001 |
NNR Model |
0.0443 |
0.0143 |
0.0203 |
0.0229 |
0.0001 |
GoldenRetrievers |
0.1802 |
0.0715 |
0.0819 |
0.1349 |
0.0047 |
Using KG for list q |
0.5434 |
0.1818 |
0.2336 |
0.3990 |
0.1491 |
Baseline top 20 |
0.5622 |
0.2107 |
0.2621 |
0.4847 |
0.2452 |
Baseline top 10 |
0.5622 |
0.2107 |
0.2621 |
0.4847 |
0.2452 |
Using LLM alone |
0.5022 |
0.1905 |
0.2334 |
0.3930 |
0.1934 |
Main pipeline |
0.5520 |
0.1860 |
0.2391 |
0.4051 |
0.1865 |
dmiip2024 |
0.4635 |
0.1756 |
0.2171 |
0.3932 |
0.1655 |
dmiip2024_1 |
0.4624 |
0.1742 |
0.2153 |
0.3917 |
0.1661 |
dmiip2024_2 |
0.4976 |
0.1840 |
0.2340 |
0.4190 |
0.1462 |
dmiip2024_3 |
0.4965 |
0.1819 |
0.2317 |
0.4142 |
0.1438 |
dmiip2024_4 |
0.5012 |
0.1833 |
0.2336 |
0.4211 |
0.1332 |
AQAMS |
0.1680 |
0.0581 |
0.0727 |
0.0903 |
0.0026 |
deepseek32b-me |
0.3215 |
0.1005 |
0.1346 |
0.2198 |
0.0198 |
deepseek32b-full |
0.2542 |
0.0804 |
0.1040 |
0.1488 |
0.0081 |
DB_vector_&_LLM |
0.2298 |
0.0869 |
0.1045 |
0.1396 |
0.0086 |
UR-IW-1 |
0.3617 |
0.1283 |
0.1605 |
0.2998 |
0.0174 |
UR-IW-2 |
0.4047 |
0.1260 |
0.1735 |
0.3373 |
0.0376 |
UR-IW-3 |
0.4052 |
0.1169 |
0.1640 |
0.3379 |
0.0340 |
UR-IW-4 |
0.4419 |
0.1487 |
0.1942 |
0.3785 |
0.0569 |
UR-IW-5 |
0.4350 |
0.1448 |
0.1918 |
0.3589 |
0.0866 |
The Relevants |
0.1867 |
0.0350 |
0.0538 |
0.0875 |
0.0008 |
Data Wizards NN |
0.0753 |
0.0218 |
0.0311 |
0.0499 |
0.0002 |
API and weirdnesss |
0.1773 |
0.0704 |
0.0829 |
0.1016 |
0.0024 |
DS@GT BioASQ 1b-1 |
0.4200 |
0.1636 |
0.1948 |
0.3483 |
0.1391 |
Neural Re-Ranking |
0.0400 |
0.0118 |
0.0167 |
0.0164 |
0.0001 |
Approach 1 |
0.0918 |
0.0269 |
0.0383 |
0.0638 |
0.0002 |
DS@GT BioASQ 1b-2 |
0.4306 |
0.1661 |
0.1980 |
0.3577 |
0.1694 |
GoldenRetrievers-2 |
0.1910 |
0.0617 |
0.0802 |
0.1043 |
0.0020 |
dl approach |
0.0419 |
0.0138 |
0.0192 |
0.0238 |
0.0001 |
GPT4O |
0.3242 |
0.1271 |
0.1536 |
0.2789 |
0.0319 |
DS@GTBioASQT13b5 |
0.4318 |
0.1643 |
0.1970 |
0.3600 |
0.1550 |
sp_lasigebiotm |
0.2573 |
0.0711 |
0.0952 |
0.1313 |
0.0043 |
mistral |
0.3605 |
0.0999 |
0.1400 |
0.2626 |
0.0341 |
lasigeBioTM |
0.2573 |
0.0711 |
0.0952 |
0.1313 |
0.0043 |
llama |
0.3150 |
0.0833 |
0.1163 |
0.2065 |
0.0234 |
lasigeBioTM-onto-bl |
0.2573 |
0.0711 |
0.0952 |
0.1313 |
0.0043 |
lasigeBioTM-onto-sm |
0.2573 |
0.0711 |
0.0952 |
0.1313 |
0.0043 |
1.PhaseA_System |
0.0688 |
0.0267 |
0.0298 |
0.0368 |
0.0001 |
deepseek-r1:32b |
0.1906 |
0.0779 |
0.0944 |
0.0942 |
0.0038 |
deepseek-r1:14b |
0.1906 |
0.0779 |
0.0944 |
0.0942 |
0.0038 |
deepseek-r1:8b |
0.1561 |
0.0538 |
0.0672 |
0.0747 |
0.0011 |
DS@GTBioASQT13b4 |
0.4759 |
0.1874 |
0.2260 |
0.4096 |
0.1968 |
DS@GTBioASQT13b3 |
0.4800 |
0.1875 |
0.2264 |
0.4137 |
0.1993 |
dense |
0.3677 |
0.1186 |
0.1602 |
0.2694 |
0.0418 |
Fleming-2 |
0.3172 |
0.1289 |
0.1504 |
0.2579 |
0.0224 |
Snippets
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
0.2216 |
0.0569 |
0.0775 |
0.1788 |
0.0070 |
Data Wizards BM25 |
0.0392 |
0.0074 |
0.0117 |
0.0398 |
0.0000 |
NLP_BioBert |
0.0981 |
0.0147 |
0.0242 |
0.0420 |
0.0001 |
Data Wizards Re-Rank |
- |
- |
- |
- |
- |
First abrotsch |
- |
- |
- |
- |
- |
ms-marco-MiniLML6v2 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
BM25-Okapi |
- |
- |
- |
- |
- |
IR1 |
0.4392 |
0.1158 |
0.1568 |
0.3702 |
0.0935 |
IR4 |
0.4621 |
0.1302 |
0.1763 |
0.4061 |
0.1265 |
UniTor_0 |
0.3216 |
0.0796 |
0.1101 |
0.2327 |
0.0105 |
UniTor_1 |
0.3156 |
0.0826 |
0.1144 |
0.2419 |
0.0108 |
Reranking rankings |
0.1544 |
0.0452 |
0.0567 |
0.1272 |
0.0011 |
query-engineering |
0.0935 |
0.0220 |
0.0329 |
0.0751 |
0.0002 |
Re-Ranker with BM25 |
0.0369 |
0.0200 |
0.0238 |
0.0329 |
0.0001 |
RetrievalRivals |
0.2082 |
0.0033 |
0.0064 |
0.0185 |
0.0001 |
RetrievalRivals_BM25 |
0.2082 |
0.0033 |
0.0064 |
0.0185 |
0.0001 |
RetrieverRivals |
0.2082 |
0.0033 |
0.0064 |
0.0185 |
0.0001 |
UniTor_2 |
0.3456 |
0.0981 |
0.1306 |
0.2797 |
0.0217 |
UniTor_3 |
0.3393 |
0.0981 |
0.1283 |
0.2801 |
0.0218 |
TEsting123 |
0.0512 |
0.0128 |
0.0174 |
0.0123 |
0.0001 |
Traditional IR Methd |
0.0512 |
0.0128 |
0.0174 |
0.0123 |
0.0001 |
traditional IR |
0.0749 |
0.0068 |
0.0118 |
0.0351 |
0.0001 |
Neural IR approach |
0.0958 |
0.0172 |
0.0278 |
0.0760 |
0.0004 |
Machinen Results |
- |
- |
- |
- |
- |
Tuwien Group 6 AIR |
0.0958 |
0.0172 |
0.0278 |
0.0760 |
0.0004 |
reranking after IR |
0.0749 |
0.0068 |
0.0118 |
0.0448 |
0.0001 |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
Okapi BM25 IR |
0.0163 |
0.0234 |
0.0175 |
0.0082 |
0.0001 |
NNR Model |
0.0171 |
0.0243 |
0.0185 |
0.0082 |
0.0001 |
GoldenRetrievers |
- |
- |
- |
- |
- |
Using KG for list q |
0.5055 |
0.1364 |
0.1916 |
0.3557 |
0.1237 |
Baseline top 20 |
0.5210 |
0.1606 |
0.2164 |
0.4178 |
0.2039 |
Baseline top 10 |
0.5210 |
0.1606 |
0.2164 |
0.4178 |
0.2039 |
Using LLM alone |
0.4695 |
0.1451 |
0.1942 |
0.3481 |
0.1551 |
Main pipeline |
0.5040 |
0.1381 |
0.1926 |
0.3599 |
0.1402 |
dmiip2024 |
0.3161 |
0.1185 |
0.1354 |
0.2930 |
0.0524 |
dmiip2024_1 |
0.3152 |
0.1187 |
0.1358 |
0.3011 |
0.0547 |
dmiip2024_2 |
0.3249 |
0.1159 |
0.1417 |
0.3003 |
0.0496 |
dmiip2024_3 |
0.3420 |
0.1197 |
0.1472 |
0.3169 |
0.0568 |
dmiip2024_4 |
0.3237 |
0.1142 |
0.1397 |
0.3017 |
0.0441 |
AQAMS |
0.2091 |
0.1769 |
0.1541 |
0.0750 |
0.0006 |
deepseek32b-me |
0.3039 |
0.0700 |
0.1013 |
0.2198 |
0.0121 |
deepseek32b-full |
0.2371 |
0.0467 |
0.0665 |
0.1310 |
0.0045 |
DB_vector_&_LLM |
0.1726 |
0.0371 |
0.0577 |
0.0860 |
0.0020 |
UR-IW-1 |
0.3390 |
0.0941 |
0.1219 |
0.2982 |
0.0127 |
UR-IW-2 |
0.4131 |
0.0845 |
0.1309 |
0.3405 |
0.0187 |
UR-IW-3 |
0.3980 |
0.0913 |
0.1303 |
0.3558 |
0.0211 |
UR-IW-4 |
0.4440 |
0.1013 |
0.1473 |
0.3616 |
0.0435 |
UR-IW-5 |
0.4034 |
0.0983 |
0.1389 |
0.3413 |
0.0542 |
The Relevants |
- |
- |
- |
- |
- |
Data Wizards NN |
0.0158 |
0.0016 |
0.0029 |
0.0147 |
0.0000 |
API and weirdnesss |
0.1203 |
0.0329 |
0.0457 |
0.0544 |
0.0005 |
DS@GT BioASQ 1b-1 |
0.2921 |
0.0665 |
0.0900 |
0.1776 |
0.0040 |
Neural Re-Ranking |
0.0585 |
0.0099 |
0.0157 |
0.0442 |
0.0001 |
Approach 1 |
0.0546 |
0.0098 |
0.0158 |
0.0379 |
0.0001 |
DS@GT BioASQ 1b-2 |
0.2921 |
0.0665 |
0.0900 |
0.1776 |
0.0040 |
GoldenRetrievers-2 |
- |
- |
- |
- |
- |
dl approach |
0.0202 |
0.0244 |
0.0202 |
0.0097 |
0.0000 |
GPT4O |
0.2423 |
0.0545 |
0.0761 |
0.2108 |
0.0110 |
DS@GTBioASQT13b5 |
0.2921 |
0.0665 |
0.0900 |
0.1776 |
0.0040 |
sp_lasigebiotm |
0.1561 |
0.0363 |
0.0531 |
0.0852 |
0.0007 |
mistral |
0.3680 |
0.0732 |
0.1106 |
0.2413 |
0.0260 |
lasigeBioTM |
0.1561 |
0.0363 |
0.0531 |
0.0852 |
0.0007 |
llama |
0.3081 |
0.0529 |
0.0822 |
0.1686 |
0.0116 |
lasigeBioTM-onto-bl |
0.1561 |
0.0363 |
0.0531 |
0.0852 |
0.0007 |
lasigeBioTM-onto-sm |
0.1561 |
0.0363 |
0.0531 |
0.0852 |
0.0007 |
1.PhaseA_System |
0.0419 |
0.0111 |
0.0142 |
0.0223 |
0.0001 |
deepseek-r1:32b |
0.1433 |
0.0366 |
0.0503 |
0.0770 |
0.0014 |
deepseek-r1:14b |
0.1433 |
0.0366 |
0.0503 |
0.0770 |
0.0014 |
deepseek-r1:8b |
0.1146 |
0.0202 |
0.0317 |
0.0481 |
0.0003 |
DS@GTBioASQT13b4 |
0.2921 |
0.0665 |
0.0900 |
0.1776 |
0.0040 |
DS@GTBioASQT13b3 |
0.2921 |
0.0665 |
0.0900 |
0.1776 |
0.0040 |
dense |
0.3550 |
0.0854 |
0.1257 |
0.2465 |
0.0201 |
Fleming-2 |
0.2216 |
0.0569 |
0.0775 |
0.1788 |
0.0070 |
Concepts
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
Data Wizards BM25 |
- |
- |
- |
- |
- |
NLP_BioBert |
- |
- |
- |
- |
- |
Data Wizards Re-Rank |
- |
- |
- |
- |
- |
First abrotsch |
- |
- |
- |
- |
- |
ms-marco-MiniLML6v2 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
BM25-Okapi |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
IR4 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
Reranking rankings |
- |
- |
- |
- |
- |
query-engineering |
- |
- |
- |
- |
- |
Re-Ranker with BM25 |
- |
- |
- |
- |
- |
RetrievalRivals |
- |
- |
- |
- |
- |
RetrievalRivals_BM25 |
- |
- |
- |
- |
- |
RetrieverRivals |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
TEsting123 |
- |
- |
- |
- |
- |
Traditional IR Methd |
- |
- |
- |
- |
- |
traditional IR |
- |
- |
- |
- |
- |
Neural IR approach |
- |
- |
- |
- |
- |
Machinen Results |
- |
- |
- |
- |
- |
Tuwien Group 6 AIR |
- |
- |
- |
- |
- |
reranking after IR |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
Okapi BM25 IR |
- |
- |
- |
- |
- |
NNR Model |
- |
- |
- |
- |
- |
GoldenRetrievers |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
AQAMS |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
DB_vector_&_LLM |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
The Relevants |
- |
- |
- |
- |
- |
Data Wizards NN |
- |
- |
- |
- |
- |
API and weirdnesss |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-1 |
- |
- |
- |
- |
- |
Neural Re-Ranking |
- |
- |
- |
- |
- |
Approach 1 |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-2 |
- |
- |
- |
- |
- |
GoldenRetrievers-2 |
- |
- |
- |
- |
- |
dl approach |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
DS@GTBioASQT13b5 |
- |
- |
- |
- |
- |
sp_lasigebiotm |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
lasigeBioTM-onto-bl |
- |
- |
- |
- |
- |
lasigeBioTM-onto-sm |
- |
- |
- |
- |
- |
1.PhaseA_System |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
DS@GTBioASQT13b4 |
- |
- |
- |
- |
- |
DS@GTBioASQT13b3 |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
Fleming-2 |
- |
- |
- |
- |
- |
RDF Triples
System |
Mean precision |
Recall |
F-Measure |
MAP |
GMAP |
Fleming-1 |
- |
- |
- |
- |
- |
Data Wizards BM25 |
- |
- |
- |
- |
- |
NLP_BioBert |
- |
- |
- |
- |
- |
Data Wizards Re-Rank |
- |
- |
- |
- |
- |
First abrotsch |
- |
- |
- |
- |
- |
ms-marco-MiniLML6v2 |
- |
- |
- |
- |
- |
IR5 |
- |
- |
- |
- |
- |
BM25-Okapi |
- |
- |
- |
- |
- |
IR1 |
- |
- |
- |
- |
- |
IR4 |
- |
- |
- |
- |
- |
UniTor_0 |
- |
- |
- |
- |
- |
UniTor_1 |
- |
- |
- |
- |
- |
Reranking rankings |
- |
- |
- |
- |
- |
query-engineering |
- |
- |
- |
- |
- |
Re-Ranker with BM25 |
- |
- |
- |
- |
- |
RetrievalRivals |
- |
- |
- |
- |
- |
RetrievalRivals_BM25 |
- |
- |
- |
- |
- |
RetrieverRivals |
- |
- |
- |
- |
- |
UniTor_2 |
- |
- |
- |
- |
- |
UniTor_3 |
- |
- |
- |
- |
- |
TEsting123 |
- |
- |
- |
- |
- |
Traditional IR Methd |
- |
- |
- |
- |
- |
traditional IR |
- |
- |
- |
- |
- |
Neural IR approach |
- |
- |
- |
- |
- |
Machinen Results |
- |
- |
- |
- |
- |
Tuwien Group 6 AIR |
- |
- |
- |
- |
- |
reranking after IR |
- |
- |
- |
- |
- |
bioinfo-0 |
- |
- |
- |
- |
- |
bioinfo-1 |
- |
- |
- |
- |
- |
bioinfo-2 |
- |
- |
- |
- |
- |
bioinfo-3 |
- |
- |
- |
- |
- |
bioinfo-4 |
- |
- |
- |
- |
- |
Okapi BM25 IR |
- |
- |
- |
- |
- |
NNR Model |
- |
- |
- |
- |
- |
GoldenRetrievers |
- |
- |
- |
- |
- |
Using KG for list q |
- |
- |
- |
- |
- |
Baseline top 20 |
- |
- |
- |
- |
- |
Baseline top 10 |
- |
- |
- |
- |
- |
Using LLM alone |
- |
- |
- |
- |
- |
Main pipeline |
- |
- |
- |
- |
- |
dmiip2024 |
- |
- |
- |
- |
- |
dmiip2024_1 |
- |
- |
- |
- |
- |
dmiip2024_2 |
- |
- |
- |
- |
- |
dmiip2024_3 |
- |
- |
- |
- |
- |
dmiip2024_4 |
- |
- |
- |
- |
- |
AQAMS |
- |
- |
- |
- |
- |
deepseek32b-me |
- |
- |
- |
- |
- |
deepseek32b-full |
- |
- |
- |
- |
- |
DB_vector_&_LLM |
- |
- |
- |
- |
- |
UR-IW-1 |
- |
- |
- |
- |
- |
UR-IW-2 |
- |
- |
- |
- |
- |
UR-IW-3 |
- |
- |
- |
- |
- |
UR-IW-4 |
- |
- |
- |
- |
- |
UR-IW-5 |
- |
- |
- |
- |
- |
The Relevants |
- |
- |
- |
- |
- |
Data Wizards NN |
- |
- |
- |
- |
- |
API and weirdnesss |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-1 |
- |
- |
- |
- |
- |
Neural Re-Ranking |
- |
- |
- |
- |
- |
Approach 1 |
- |
- |
- |
- |
- |
DS@GT BioASQ 1b-2 |
- |
- |
- |
- |
- |
GoldenRetrievers-2 |
- |
- |
- |
- |
- |
dl approach |
- |
- |
- |
- |
- |
GPT4O |
- |
- |
- |
- |
- |
DS@GTBioASQT13b5 |
- |
- |
- |
- |
- |
sp_lasigebiotm |
- |
- |
- |
- |
- |
mistral |
- |
- |
- |
- |
- |
lasigeBioTM |
- |
- |
- |
- |
- |
llama |
- |
- |
- |
- |
- |
lasigeBioTM-onto-bl |
- |
- |
- |
- |
- |
lasigeBioTM-onto-sm |
- |
- |
- |
- |
- |
1.PhaseA_System |
- |
- |
- |
- |
- |
deepseek-r1:32b |
- |
- |
- |
- |
- |
deepseek-r1:14b |
- |
- |
- |
- |
- |
deepseek-r1:8b |
- |
- |
- |
- |
- |
DS@GTBioASQT13b4 |
- |
- |
- |
- |
- |
DS@GTBioASQT13b3 |
- |
- |
- |
- |
- |
dense |
- |
- |
- |
- |
- |
Fleming-2 |
- |
- |
- |
- |
- |