BioASQ Participants Area
Test Results for Task 7a
The evaluation measures indicating the performance of the systems that submitted results are presented below. The evaluation is incremental; as new MeSH become available the tables are updated.Annotated articles:16780/17711
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| BioASQ_Baseline | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| BioASQ_Baseline | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
+ Dry Run 2
Annotated articles:5408/5577
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| NLM System 4 | 0.6561 | 0.6900 | 0.6371 | 0.6449 | 0.6237 | 0.5146 | 0.5090 | 0.6838 | 0.6306 | 0.4920 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| NLM System 4 | 0.5252 | 0.8082 | 0.7248 | 0.7475 | 0.5668 | 0.5121 |
+ Test batch 1, week 1
Annotated articles:7194/7358
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| MTI First Line Index | 0.6268 | 0.6730 | 0.6050 | 0.6170 | 0.6284 | 0.5552 | 0.5320 | 0.6610 | 0.5960 | 0.4640 |
| Default MTI | 0.6397 | 0.6419 | 0.6586 | 0.6322 | 0.5937 | 0.5953 | 0.5524 | 0.6328 | 0.6468 | 0.4797 |
| DeepMeSH1 | 0.6628 | 0.7286 | 0.6226 | 0.6564 | 0.6886 | 0.5137 | 0.5139 | 0.7319 | 0.6056 | 0.5053 |
| DeepMeSH2 | 0.6382 | 0.7160 | 0.5883 | 0.6305 | 0.6705 | 0.4526 | 0.4599 | 0.7213 | 0.5724 | 0.4767 |
| DeepMeSH3 | 0.6607 | 0.7260 | 0.6209 | 0.6544 | 0.6870 | 0.5108 | 0.5116 | 0.7293 | 0.6039 | 0.5034 |
| MeSHProbeNet-P | 0.6849 | 0.7141 | 0.6714 | 0.6753 | 0.6728 | 0.5670 | 0.5556 | 0.7090 | 0.6624 | 0.5264 |
| NLM System 4 | 0.6515 | 0.6840 | 0.6343 | 0.6404 | 0.6125 | 0.5045 | 0.4967 | 0.6768 | 0.6280 | 0.4873 |
| MeSHProbeNet-P0 | 0.6848 | 0.7024 | 0.6817 | 0.6753 | 0.6604 | 0.5781 | 0.5609 | 0.6970 | 0.6730 | 0.5262 |
| MeSHProbeNet-P1 | 0.6839 | 0.6903 | 0.6917 | 0.6746 | 0.6486 | 0.5893 | 0.5657 | 0.6846 | 0.6832 | 0.5252 |
| MeSHProbeNet-P2 | 0.6849 | 0.7140 | 0.6714 | 0.6753 | 0.6728 | 0.5670 | 0.5556 | 0.7089 | 0.6624 | 0.5264 |
| MeSHProbeNet-P3 | 0.6843 | 0.7128 | 0.6711 | 0.6746 | 0.6700 | 0.5651 | 0.5529 | 0.7077 | 0.6624 | 0.5256 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| MTI First Line Index | 0.5035 | 0.7889 | 0.7105 | 0.7277 | 0.5508 | 0.4909 |
| Default MTI | 0.5207 | 0.7601 | 0.7567 | 0.7408 | 0.5358 | 0.5333 |
| DeepMeSH1 | 0.5298 | 0.8304 | 0.7120 | 0.7507 | 0.5905 | 0.5015 |
| DeepMeSH2 | 0.5068 | 0.8239 | 0.6815 | 0.7291 | 0.5772 | 0.4722 |
| DeepMeSH3 | 0.5288 | 0.8293 | 0.7099 | 0.7491 | 0.5895 | 0.5006 |
| MeSHProbeNet-P | 0.5510 | 0.8186 | 0.7572 | 0.7704 | 0.5874 | 0.5429 |
| NLM System 4 | 0.5228 | 0.8013 | 0.7231 | 0.7420 | 0.5624 | 0.5130 |
| MeSHProbeNet-P0 | 0.5523 | 0.8099 | 0.7667 | 0.7715 | 0.5812 | 0.5508 |
| MeSHProbeNet-P1 | 0.5535 | 0.8011 | 0.7761 | 0.7725 | 0.5746 | 0.5587 |
| MeSHProbeNet-P2 | 0.5509 | 0.8186 | 0.7572 | 0.7703 | 0.5874 | 0.5429 |
| MeSHProbeNet-P3 | 0.5509 | 0.8178 | 0.7576 | 0.7701 | 0.5870 | 0.5432 |
+ Test batch 1, week 2
Annotated articles:7021/7166
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| MTI First Line Index | 0.6272 | 0.6688 | 0.6083 | 0.6160 | 0.6383 | 0.5640 | 0.5421 | 0.6613 | 0.5964 | 0.4644 |
| Default MTI | 0.6405 | 0.6391 | 0.6621 | 0.6313 | 0.6068 | 0.6072 | 0.5650 | 0.6343 | 0.6469 | 0.4792 |
| DeepMeSH1 | 0.6330 | 0.7105 | 0.5857 | 0.6246 | 0.6636 | 0.4596 | 0.4664 | 0.7177 | 0.5662 | 0.4714 |
| DeepMeSH2 | 0.6382 | 0.7252 | 0.5866 | 0.6313 | 0.6801 | 0.4886 | 0.4915 | 0.7324 | 0.5655 | 0.4786 |
| DeepMeSH3 | 0.6134 | 0.7367 | 0.5462 | 0.6084 | 0.6781 | 0.4732 | 0.4782 | 0.7428 | 0.5223 | 0.4547 |
| MetaEmb | 0.0001 | 0.0001 | 0.0001 | 0.0001 | 0.0001 | 0.0004 | 0.0000 | 0.0001 | 0.0001 | 0.0001 |
| MeSHProbeNet-P | 0.6853 | 0.7109 | 0.6751 | 0.6748 | 0.6828 | 0.5806 | 0.5699 | 0.7090 | 0.6632 | 0.5266 |
| NLM System 4 | 0.6499 | 0.6898 | 0.6254 | 0.6373 | 0.6281 | 0.5000 | 0.4978 | 0.6855 | 0.6177 | 0.4848 |
| MeSHProbeNet-P0 | 0.6868 | 0.7108 | 0.6768 | 0.6758 | 0.6855 | 0.5754 | 0.5668 | 0.7090 | 0.6659 | 0.5273 |
| MeSHProbeNet-P1 | 0.6819 | 0.7027 | 0.6731 | 0.6703 | 0.6801 | 0.5588 | 0.5533 | 0.7005 | 0.6642 | 0.5207 |
| MeSHProbeNet-P2 | 0.6815 | 0.6837 | 0.6861 | 0.6705 | 0.6557 | 0.5903 | 0.5737 | 0.6891 | 0.6741 | 0.5205 |
| MeSHProbeNet-P3 | 0.6799 | 0.6888 | 0.6772 | 0.6687 | 0.6701 | 0.5677 | 0.5588 | 0.6939 | 0.6665 | 0.5185 |
| baseline | 0.0001 | 0.0001 | 0.0001 | 0.0001 | 0.0001 | 0.0004 | 0.0000 | 0.0001 | 0.0001 | 0.0001 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| MTI First Line Index | 0.5015 | 0.7858 | 0.7097 | 0.7252 | 0.5460 | 0.4913 |
| Default MTI | 0.5169 | 0.7579 | 0.7567 | 0.7387 | 0.5315 | 0.5311 |
| DeepMeSH1 | 0.5008 | 0.8195 | 0.6812 | 0.7262 | 0.5695 | 0.4687 |
| DeepMeSH2 | 0.5079 | 0.8295 | 0.6937 | 0.7386 | 0.5747 | 0.4763 |
| DeepMeSH3 | 0.4868 | 0.8358 | 0.6636 | 0.7217 | 0.5708 | 0.4457 |
| MetaEmb | 0.1224 | 0.1192 | 0.1926 | 0.1429 | 0.1059 | 0.1529 |
| MeSHProbeNet-P | 0.5501 | 0.8177 | 0.7583 | 0.7701 | 0.5849 | 0.5438 |
| NLM System 4 | 0.5193 | 0.8089 | 0.7146 | 0.7403 | 0.5648 | 0.5045 |
| MeSHProbeNet-P0 | 0.5509 | 0.8194 | 0.7596 | 0.7717 | 0.5848 | 0.5449 |
| MeSHProbeNet-P1 | 0.5474 | 0.8167 | 0.7577 | 0.7694 | 0.5811 | 0.5416 |
| MeSHProbeNet-P2 | 0.5481 | 0.8005 | 0.7701 | 0.7703 | 0.5703 | 0.5502 |
| MeSHProbeNet-P3 | 0.5477 | 0.8066 | 0.7637 | 0.7698 | 0.5741 | 0.5455 |
| baseline | 0.1211 | 0.1223 | 0.1828 | 0.1421 | 0.1052 | 0.1506 |
+ Test batch 1, week 3
Annotated articles:10831/11019
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| MTI First Line Index | 0.6191 | 0.6682 | 0.5952 | 0.6083 | 0.6416 | 0.5624 | 0.5423 | 0.6576 | 0.5849 | 0.4556 |
| Default MTI | 0.6329 | 0.6352 | 0.6521 | 0.6245 | 0.6072 | 0.6058 | 0.5634 | 0.6279 | 0.6379 | 0.4716 |
| DeepMeSH1 | 0.6649 | 0.7277 | 0.6300 | 0.6591 | 0.7049 | 0.5469 | 0.5527 | 0.7309 | 0.6098 | 0.5084 |
| DeepMeSH2 | 0.6647 | 0.7250 | 0.6316 | 0.6587 | 0.7032 | 0.5470 | 0.5533 | 0.7280 | 0.6115 | 0.5079 |
| DeepMeSH3 | 0.6494 | 0.7181 | 0.6093 | 0.6430 | 0.6937 | 0.4991 | 0.5075 | 0.7224 | 0.5898 | 0.4899 |
| DeepMeSH4 | 0.6450 | 0.7170 | 0.6021 | 0.6382 | 0.6912 | 0.4910 | 0.5006 | 0.7215 | 0.5832 | 0.4849 |
| DeepMeSH5 | 0.6453 | 0.7133 | 0.6044 | 0.6381 | 0.6884 | 0.4890 | 0.4981 | 0.7182 | 0.5858 | 0.4848 |
| MeSHProbeNet-P | 0.6738 | 0.7033 | 0.6634 | 0.6639 | 0.6780 | 0.5674 | 0.5590 | 0.6977 | 0.6515 | 0.5134 |
| NLM System 4 | 0.6384 | 0.6839 | 0.6145 | 0.6277 | 0.6254 | 0.4911 | 0.4912 | 0.6762 | 0.6045 | 0.4737 |
| MeSHProbeNet-P0 | 0.6747 | 0.6983 | 0.6694 | 0.6648 | 0.6735 | 0.5673 | 0.5581 | 0.6926 | 0.6577 | 0.5142 |
| MeSHProbeNet-P1 | 0.6737 | 0.6996 | 0.6661 | 0.6636 | 0.6760 | 0.5589 | 0.5527 | 0.6938 | 0.6547 | 0.5129 |
| MeSHProbeNet-P2 | 0.6709 | 0.6799 | 0.6727 | 0.6610 | 0.6631 | 0.5699 | 0.5593 | 0.6817 | 0.6605 | 0.5092 |
| MeSHProbeNet-P3 | 0.6693 | 0.6747 | 0.6750 | 0.6595 | 0.6489 | 0.5849 | 0.5663 | 0.6764 | 0.6622 | 0.5076 |
| UNCC_IR_01 | 0.5101 | 0.5572 | 0.4855 | 0.4939 | 0.5511 | 0.3250 | 0.3388 | 0.5468 | 0.4781 | 0.3434 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| MTI First Line Index | 0.4924 | 0.7864 | 0.7044 | 0.7229 | 0.5407 | 0.4793 |
| Default MTI | 0.5077 | 0.7564 | 0.7529 | 0.7366 | 0.5237 | 0.5208 |
| DeepMeSH1 | 0.5252 | 0.8292 | 0.7199 | 0.7542 | 0.5831 | 0.5000 |
| DeepMeSH2 | 0.5253 | 0.8272 | 0.7216 | 0.7541 | 0.5820 | 0.5011 |
| DeepMeSH3 | 0.5122 | 0.8252 | 0.7035 | 0.7427 | 0.5759 | 0.4831 |
| DeepMeSH4 | 0.5083 | 0.8236 | 0.6975 | 0.7384 | 0.5739 | 0.4778 |
| DeepMeSH5 | 0.5084 | 0.8211 | 0.6994 | 0.7384 | 0.5718 | 0.4793 |
| MeSHProbeNet-P | 0.5361 | 0.8111 | 0.7529 | 0.7630 | 0.5725 | 0.5299 |
| NLM System 4 | 0.5062 | 0.8038 | 0.7081 | 0.7335 | 0.5543 | 0.4909 |
| MeSHProbeNet-P0 | 0.5379 | 0.8083 | 0.7580 | 0.7646 | 0.5713 | 0.5342 |
| MeSHProbeNet-P1 | 0.5375 | 0.8101 | 0.7555 | 0.7641 | 0.5729 | 0.5319 |
| MeSHProbeNet-P2 | 0.5375 | 0.7965 | 0.7647 | 0.7649 | 0.5632 | 0.5375 |
| MeSHProbeNet-P3 | 0.5343 | 0.7908 | 0.7663 | 0.7629 | 0.5573 | 0.5373 |
| UNCC_IR_01 | 0.4107 | 0.7163 | 0.5929 | 0.6212 | 0.4736 | 0.3923 |
+ Test batch 1, week 4
Annotated articles:5482/5566
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| MTI First Line Index | 0.6202 | 0.6693 | 0.6017 | 0.6137 | 0.6277 | 0.5596 | 0.5364 | 0.6590 | 0.5856 | 0.4623 |
| Default MTI | 0.6387 | 0.6420 | 0.6622 | 0.6341 | 0.5965 | 0.6000 | 0.5598 | 0.6357 | 0.6419 | 0.4817 |
| DeepMeSH1 | 0.6702 | 0.7368 | 0.6357 | 0.6670 | 0.6986 | 0.5427 | 0.5460 | 0.7403 | 0.6122 | 0.5164 |
| DeepMeSH2 | 0.6687 | 0.7350 | 0.6346 | 0.6654 | 0.6968 | 0.5400 | 0.5429 | 0.7382 | 0.6110 | 0.5147 |
| DeepMeSH3 | 0.6518 | 0.7250 | 0.6088 | 0.6463 | 0.6873 | 0.4903 | 0.4969 | 0.7300 | 0.5887 | 0.4936 |
| DeepMeSH4 | 0.6522 | 0.7246 | 0.6096 | 0.6467 | 0.6863 | 0.4931 | 0.4997 | 0.7292 | 0.5899 | 0.4938 |
| MeSHProbeNet-P | 0.6816 | 0.7164 | 0.6689 | 0.6747 | 0.6800 | 0.5662 | 0.5583 | 0.7127 | 0.6532 | 0.5253 |
| NLM System 4 | 0.6476 | 0.6930 | 0.6238 | 0.6380 | 0.6246 | 0.4936 | 0.4914 | 0.6862 | 0.6130 | 0.4843 |
| MeSHProbeNet-P0 | 0.6831 | 0.7074 | 0.6790 | 0.6761 | 0.6715 | 0.5724 | 0.5614 | 0.7035 | 0.6639 | 0.5265 |
| MeSHProbeNet-P1 | 0.6829 | 0.7100 | 0.6750 | 0.6754 | 0.6754 | 0.5629 | 0.5551 | 0.7064 | 0.6608 | 0.5257 |
| MeSHProbeNet-P2 | 0.6819 | 0.7045 | 0.6796 | 0.6751 | 0.6689 | 0.5783 | 0.5652 | 0.7008 | 0.6640 | 0.5255 |
| MeSHProbeNet-P3 | 0.6693 | 0.7128 | 0.6494 | 0.6620 | 0.6773 | 0.5347 | 0.5288 | 0.7097 | 0.6333 | 0.5113 |
| UNCC_IR_01 | 0.5249 | 0.5744 | 0.5008 | 0.5120 | 0.5433 | 0.3289 | 0.3377 | 0.5640 | 0.4909 | 0.3595 |
| MeSHedPotato | 0.2839 | 0.7562 | 0.1770 | 0.2591 | 0.6752 | 0.0523 | 0.0679 | 0.7873 | 0.1732 | 0.1668 |
| txt_sys0 | 0.4508 | 0.4401 | 0.4851 | 0.4420 | 0.2706 | 0.3207 | 0.2847 | 0.4280 | 0.4760 | 0.2964 |
| txt_sys1 | 0.4672 | 0.5123 | 0.4444 | 0.4550 | 0.3536 | 0.2818 | 0.2759 | 0.5027 | 0.4363 | 0.3085 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| MTI First Line Index | 0.5038 | 0.7870 | 0.7097 | 0.7273 | 0.5489 | 0.4911 |
| Default MTI | 0.5212 | 0.7613 | 0.7582 | 0.7424 | 0.5353 | 0.5343 |
| DeepMeSH1 | 0.5370 | 0.8359 | 0.7214 | 0.7584 | 0.5968 | 0.5093 |
| DeepMeSH2 | 0.5356 | 0.8349 | 0.7208 | 0.7576 | 0.5957 | 0.5078 |
| DeepMeSH3 | 0.5207 | 0.8298 | 0.6996 | 0.7426 | 0.5882 | 0.4880 |
| DeepMeSH4 | 0.5209 | 0.8310 | 0.6993 | 0.7431 | 0.5879 | 0.4882 |
| MeSHProbeNet-P | 0.5514 | 0.8209 | 0.7536 | 0.7694 | 0.5902 | 0.5408 |
| NLM System 4 | 0.5192 | 0.8082 | 0.7127 | 0.7394 | 0.5654 | 0.5029 |
| MeSHProbeNet-P0 | 0.5541 | 0.8139 | 0.7619 | 0.7708 | 0.5862 | 0.5488 |
| MeSHProbeNet-P1 | 0.5532 | 0.8167 | 0.7579 | 0.7699 | 0.5881 | 0.5450 |
| MeSHProbeNet-P2 | 0.5526 | 0.8117 | 0.7633 | 0.7706 | 0.5836 | 0.5485 |
| MeSHProbeNet-P3 | 0.5420 | 0.8216 | 0.7366 | 0.7594 | 0.5891 | 0.5255 |
| UNCC_IR_01 | 0.4281 | 0.7295 | 0.6014 | 0.6340 | 0.4919 | 0.4063 |
| MeSHedPotato | 0.1959 | 0.8160 | 0.2045 | 0.2963 | 0.4259 | 0.1386 |
| txt_sys0 | 0.3866 | 0.5834 | 0.6241 | 0.5799 | 0.3904 | 0.4142 |
| txt_sys1 | 0.3883 | 0.6539 | 0.5656 | 0.5816 | 0.4379 | 0.3757 |
+ Test batch 1, week 5
Annotated articles:6353/6729
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| MTI First Line Index | 0.6288 | 0.6719 | 0.6132 | 0.6207 | 0.6319 | 0.5629 | 0.5393 | 0.6612 | 0.5995 | 0.4692 |
| Default MTI | 0.6424 | 0.6431 | 0.6662 | 0.6358 | 0.6003 | 0.6035 | 0.5621 | 0.6361 | 0.6489 | 0.4842 |
| DeepMeSH1 | 0.6857 | 0.7491 | 0.6502 | 0.6805 | 0.7249 | 0.5518 | 0.5579 | 0.7537 | 0.6289 | 0.5329 |
| DeepMeSH2 | 0.6771 | 0.7408 | 0.6409 | 0.6718 | 0.7138 | 0.5290 | 0.5353 | 0.7455 | 0.6202 | 0.5227 |
| DeepMeSH3 | 0.6729 | 0.7431 | 0.6312 | 0.6671 | 0.7206 | 0.5140 | 0.5239 | 0.7488 | 0.6110 | 0.5176 |
| DeepMeSH4 | 0.6728 | 0.7463 | 0.6288 | 0.6669 | 0.7207 | 0.5098 | 0.5202 | 0.7516 | 0.6090 | 0.5175 |
| DeepMeSH5 | 0.6993 | 0.7628 | 0.6638 | 0.6954 | 0.7419 | 0.5677 | 0.5754 | 0.7674 | 0.6423 | 0.5546 |
| MeSHProbeNet-P | 0.6854 | 0.7146 | 0.6750 | 0.6763 | 0.6787 | 0.5709 | 0.5616 | 0.7109 | 0.6617 | 0.5281 |
| NLM System 4 | 0.6522 | 0.6926 | 0.6315 | 0.6420 | 0.6260 | 0.4985 | 0.4957 | 0.6860 | 0.6216 | 0.4892 |
| MeSHProbeNet-P0 | 0.6854 | 0.7025 | 0.6854 | 0.6763 | 0.6670 | 0.5830 | 0.5685 | 0.6991 | 0.6723 | 0.5278 |
| MeSHProbeNet-P1 | 0.6847 | 0.6905 | 0.6958 | 0.6759 | 0.6542 | 0.5941 | 0.5735 | 0.6869 | 0.6825 | 0.5268 |
| MeSHProbeNet-P2 | 0.6862 | 0.6997 | 0.6894 | 0.6771 | 0.6667 | 0.5813 | 0.5670 | 0.6960 | 0.6767 | 0.5284 |
| MeSHProbeNet-P3 | 0.6736 | 0.6999 | 0.6659 | 0.6644 | 0.6649 | 0.5476 | 0.5370 | 0.6963 | 0.6523 | 0.5143 |
| UNCC_IR_01 | 0.5203 | 0.5323 | 0.5243 | 0.5051 | 0.5083 | 0.3476 | 0.3492 | 0.5244 | 0.5163 | 0.3532 |
| MeSHedPotato1 | 0.3344 | 0.7613 | 0.2253 | 0.3040 | 0.6267 | 0.1267 | 0.1517 | 0.7445 | 0.2156 | 0.2021 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| MTI First Line Index | 0.5075 | 0.7883 | 0.7163 | 0.7312 | 0.5509 | 0.4976 |
| Default MTI | 0.5210 | 0.7621 | 0.7621 | 0.7446 | 0.5349 | 0.5359 |
| DeepMeSH1 | 0.5472 | 0.8430 | 0.7344 | 0.7697 | 0.6043 | 0.5218 |
| DeepMeSH2 | 0.5401 | 0.8369 | 0.7279 | 0.7633 | 0.5968 | 0.5148 |
| DeepMeSH3 | 0.5359 | 0.8398 | 0.7186 | 0.7589 | 0.5973 | 0.5073 |
| DeepMeSH4 | 0.5353 | 0.8424 | 0.7165 | 0.7588 | 0.5991 | 0.5051 |
| DeepMeSH5 | 0.5682 | 0.8521 | 0.7448 | 0.7805 | 0.6243 | 0.5424 |
| MeSHProbeNet-P | 0.5508 | 0.8216 | 0.7593 | 0.7726 | 0.5875 | 0.5441 |
| NLM System 4 | 0.5199 | 0.8097 | 0.7210 | 0.7449 | 0.5633 | 0.5077 |
| MeSHProbeNet-P0 | 0.5520 | 0.8122 | 0.7690 | 0.7737 | 0.5807 | 0.5519 |
| MeSHProbeNet-P1 | 0.5523 | 0.8027 | 0.7784 | 0.7742 | 0.5734 | 0.5587 |
| MeSHProbeNet-P2 | 0.5527 | 0.8103 | 0.7724 | 0.7746 | 0.5794 | 0.5544 |
| MeSHProbeNet-P3 | 0.5435 | 0.8142 | 0.7539 | 0.7659 | 0.5791 | 0.5374 |
| UNCC_IR_01 | 0.4244 | 0.6958 | 0.6323 | 0.6371 | 0.4632 | 0.4224 |
| MeSHedPotato1 | 0.2323 | 0.8265 | 0.2771 | 0.3638 | 0.4243 | 0.1802 |
+ Test batch 2, week 1
Annotated articles:6098/6380
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| MTI First Line Index | 0.6265 | 0.6690 | 0.6039 | 0.6161 | 0.6296 | 0.5627 | 0.5394 | 0.6598 | 0.5965 | 0.4635 |
| Default MTI | 0.6403 | 0.6394 | 0.6590 | 0.6322 | 0.5985 | 0.6051 | 0.5614 | 0.6332 | 0.6476 | 0.4793 |
| iria-mix | 0.5202 | 0.4616 | 0.5971 | 0.5091 | 0.4434 | 0.4807 | 0.4318 | 0.4658 | 0.5890 | 0.3545 |
| DeepMeSH1 | 0.6917 | 0.7214 | 0.6745 | 0.6841 | 0.6845 | 0.5961 | 0.5863 | 0.7232 | 0.6629 | 0.5364 |
| DeepMeSH2 | 0.6900 | 0.7215 | 0.6711 | 0.6824 | 0.6838 | 0.5907 | 0.5828 | 0.7237 | 0.6593 | 0.5346 |
| DeepMeSH3 | 0.6895 | 0.7214 | 0.6701 | 0.6820 | 0.6834 | 0.5888 | 0.5810 | 0.7239 | 0.6582 | 0.5341 |
| DeepMeSH4 | 0.6786 | 0.7078 | 0.6585 | 0.6690 | 0.6681 | 0.5637 | 0.5561 | 0.7111 | 0.6490 | 0.5188 |
| DeepMeSH5 | 0.7128 | 0.7391 | 0.6959 | 0.7047 | 0.7070 | 0.6228 | 0.6141 | 0.7424 | 0.6855 | 0.5668 |
| MeSHProbeNet-P | 0.6848 | 0.6897 | 0.6899 | 0.6742 | 0.6563 | 0.6020 | 0.5780 | 0.6865 | 0.6832 | 0.5243 |
| NLM System 4 | 0.6509 | 0.6901 | 0.6254 | 0.6386 | 0.6198 | 0.5029 | 0.4978 | 0.6840 | 0.6210 | 0.4855 |
| MeSHProbeNet-P0 | 0.6855 | 0.7017 | 0.6796 | 0.6746 | 0.6679 | 0.5902 | 0.5719 | 0.6986 | 0.6729 | 0.5251 |
| MeSHProbeNet-P1 | 0.6871 | 0.7104 | 0.6746 | 0.6759 | 0.6776 | 0.5791 | 0.5657 | 0.7072 | 0.6681 | 0.5267 |
| MeSHProbeNet-P2 | 0.6824 | 0.7002 | 0.6747 | 0.6711 | 0.6655 | 0.5827 | 0.5657 | 0.6974 | 0.6680 | 0.5211 |
| MeSHProbeNet-P3 | 0.6730 | 0.6977 | 0.6598 | 0.6614 | 0.6616 | 0.5549 | 0.5404 | 0.6943 | 0.6530 | 0.5106 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| MTI First Line Index | 0.5021 | 0.7851 | 0.7076 | 0.7257 | 0.5460 | 0.4899 |
| Default MTI | 0.5190 | 0.7578 | 0.7567 | 0.7407 | 0.5322 | 0.5322 |
| iria-mix | 0.4406 | 0.6151 | 0.7302 | 0.6538 | 0.4182 | 0.4882 |
| DeepMeSH1 | 0.5533 | 0.8192 | 0.7579 | 0.7738 | 0.5900 | 0.5412 |
| DeepMeSH2 | 0.5525 | 0.8199 | 0.7552 | 0.7727 | 0.5912 | 0.5387 |
| DeepMeSH3 | 0.5522 | 0.8203 | 0.7537 | 0.7721 | 0.5912 | 0.5380 |
| DeepMeSH4 | 0.5424 | 0.8115 | 0.7458 | 0.7632 | 0.5805 | 0.5296 |
| DeepMeSH5 | 0.5825 | 0.8303 | 0.7748 | 0.7890 | 0.6161 | 0.5714 |
| MeSHProbeNet-P | 0.5511 | 0.8003 | 0.7742 | 0.7717 | 0.5730 | 0.5548 |
| NLM System 4 | 0.5169 | 0.8045 | 0.7127 | 0.7377 | 0.5626 | 0.5017 |
| MeSHProbeNet-P0 | 0.5496 | 0.8091 | 0.7642 | 0.7704 | 0.5791 | 0.5464 |
| MeSHProbeNet-P1 | 0.5498 | 0.8162 | 0.7589 | 0.7708 | 0.5837 | 0.5428 |
| MeSHProbeNet-P2 | 0.5464 | 0.8092 | 0.7602 | 0.7681 | 0.5776 | 0.5421 |
| MeSHProbeNet-P3 | 0.5393 | 0.8094 | 0.7481 | 0.7613 | 0.5754 | 0.5316 |
+ Test batch 2, week 2
Annotated articles:6621/6785
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| MTI First Line Index | 0.6355 | 0.6747 | 0.6213 | 0.6269 | 0.6375 | 0.5707 | 0.5450 | 0.6638 | 0.6094 | 0.4742 |
| Default MTI | 0.6456 | 0.6441 | 0.6690 | 0.6390 | 0.6064 | 0.6117 | 0.5669 | 0.6375 | 0.6539 | 0.4856 |
| iria-mix | 0.5365 | 0.4751 | 0.6227 | 0.5269 | 0.4713 | 0.5036 | 0.4524 | 0.4788 | 0.6099 | 0.3701 |
| DeepMeSH1 | 0.6963 | 0.7228 | 0.6859 | 0.6904 | 0.6898 | 0.6025 | 0.5921 | 0.7241 | 0.6706 | 0.5427 |
| DeepMeSH2 | 0.6961 | 0.7237 | 0.6839 | 0.6897 | 0.6906 | 0.5993 | 0.5901 | 0.7257 | 0.6688 | 0.5421 |
| DeepMeSH3 | 0.6957 | 0.7242 | 0.6834 | 0.6897 | 0.6892 | 0.5982 | 0.5885 | 0.7260 | 0.6678 | 0.5420 |
| DeepMeSH4 | 0.6850 | 0.7126 | 0.6711 | 0.6777 | 0.6755 | 0.5717 | 0.5635 | 0.7149 | 0.6575 | 0.5280 |
| DeepMeSH5 | 0.7288 | 0.7534 | 0.7180 | 0.7232 | 0.7309 | 0.6444 | 0.6356 | 0.7560 | 0.7036 | 0.5917 |
| MeSHProbeNet-P | 0.6916 | 0.7061 | 0.6931 | 0.6829 | 0.6703 | 0.5949 | 0.5759 | 0.7017 | 0.6818 | 0.5337 |
| NLM System 4 | 0.6581 | 0.6909 | 0.6405 | 0.6473 | 0.6192 | 0.5133 | 0.5072 | 0.6847 | 0.6335 | 0.4944 |
| MeSHProbeNet-P0 | 0.6922 | 0.7184 | 0.6829 | 0.6834 | 0.6826 | 0.5833 | 0.5706 | 0.7140 | 0.6716 | 0.5346 |
| MeSHProbeNet-P1 | 0.6928 | 0.7249 | 0.6778 | 0.6838 | 0.6906 | 0.5729 | 0.5647 | 0.7206 | 0.6671 | 0.5352 |
| MeSHProbeNet-P2 | 0.6933 | 0.7228 | 0.6801 | 0.6840 | 0.6907 | 0.5705 | 0.5629 | 0.7183 | 0.6700 | 0.5354 |
| MeSHProbeNet-P3 | 0.6794 | 0.7036 | 0.6720 | 0.6701 | 0.6669 | 0.5583 | 0.5433 | 0.6986 | 0.6613 | 0.5193 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| MTI First Line Index | 0.5078 | 0.7878 | 0.7230 | 0.7348 | 0.5472 | 0.5009 |
| Default MTI | 0.5215 | 0.7606 | 0.7653 | 0.7461 | 0.5322 | 0.5379 |
| iria-mix | 0.4520 | 0.6289 | 0.7495 | 0.6701 | 0.4271 | 0.5038 |
| DeepMeSH1 | 0.5593 | 0.8191 | 0.7681 | 0.7791 | 0.5907 | 0.5523 |
| DeepMeSH2 | 0.5581 | 0.8197 | 0.7666 | 0.7785 | 0.5908 | 0.5498 |
| DeepMeSH3 | 0.5576 | 0.8201 | 0.7659 | 0.7783 | 0.5904 | 0.5493 |
| DeepMeSH4 | 0.5474 | 0.8133 | 0.7573 | 0.7704 | 0.5806 | 0.5389 |
| DeepMeSH5 | 0.6071 | 0.8388 | 0.7920 | 0.8024 | 0.6361 | 0.6001 |
| MeSHProbeNet-P | 0.5555 | 0.8094 | 0.7760 | 0.7765 | 0.5796 | 0.5581 |
| NLM System 4 | 0.5245 | 0.8032 | 0.7268 | 0.7458 | 0.5632 | 0.5148 |
| MeSHProbeNet-P0 | 0.5546 | 0.8190 | 0.7662 | 0.7756 | 0.5868 | 0.5503 |
| MeSHProbeNet-P1 | 0.5552 | 0.8247 | 0.7609 | 0.7753 | 0.5911 | 0.5473 |
| MeSHProbeNet-P2 | 0.5556 | 0.8233 | 0.7623 | 0.7754 | 0.5904 | 0.5488 |
| MeSHProbeNet-P3 | 0.5452 | 0.8093 | 0.7592 | 0.7667 | 0.5768 | 0.5422 |
+ Test batch 2, week 3
Annotated articles:5927/6207
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| MTI First Line Index | 0.6244 | 0.6874 | 0.5905 | 0.6134 | 0.6358 | 0.5381 | 0.5236 | 0.6754 | 0.5805 | 0.4627 |
| Default MTI | 0.6378 | 0.6453 | 0.6491 | 0.6284 | 0.5969 | 0.5898 | 0.5526 | 0.6400 | 0.6357 | 0.4761 |
| iria-mix | 0.5324 | 0.4680 | 0.6214 | 0.5214 | 0.4553 | 0.4913 | 0.4419 | 0.4732 | 0.6087 | 0.3656 |
| DeepMeSH1 | 0.6909 | 0.7188 | 0.6755 | 0.6828 | 0.6812 | 0.5895 | 0.5821 | 0.7230 | 0.6614 | 0.5354 |
| DeepMeSH2 | 0.6895 | 0.7180 | 0.6736 | 0.6814 | 0.6806 | 0.5872 | 0.5801 | 0.7227 | 0.6593 | 0.5337 |
| DeepMeSH3 | 0.6898 | 0.7194 | 0.6730 | 0.6817 | 0.6804 | 0.5862 | 0.5797 | 0.7240 | 0.6587 | 0.5343 |
| DeepMeSH4 | 0.6791 | 0.7058 | 0.6636 | 0.6704 | 0.6619 | 0.5634 | 0.5551 | 0.7107 | 0.6503 | 0.5208 |
| DeepMeSH5 | 0.7186 | 0.7461 | 0.7036 | 0.7120 | 0.7212 | 0.6264 | 0.6214 | 0.7504 | 0.6894 | 0.5781 |
| MeSHProbeNet-P | 0.6865 | 0.7180 | 0.6684 | 0.6761 | 0.6785 | 0.5723 | 0.5630 | 0.7170 | 0.6585 | 0.5282 |
| NLM System 4 | 0.6541 | 0.6871 | 0.6341 | 0.6422 | 0.6142 | 0.5105 | 0.5048 | 0.6819 | 0.6286 | 0.4895 |
| MeSHProbeNet-P0 | 0.6871 | 0.7069 | 0.6795 | 0.6769 | 0.6668 | 0.5848 | 0.5696 | 0.7056 | 0.6696 | 0.5287 |
| MeSHProbeNet-P1 | 0.6869 | 0.6952 | 0.6901 | 0.6769 | 0.6546 | 0.5969 | 0.5756 | 0.6937 | 0.6803 | 0.5284 |
| MeSHProbeNet-P2 | 0.6880 | 0.7137 | 0.6746 | 0.6776 | 0.6746 | 0.5744 | 0.5639 | 0.7121 | 0.6655 | 0.5294 |
| MeSHProbeNet-P3 | 0.6880 | 0.7108 | 0.6765 | 0.6771 | 0.6741 | 0.5703 | 0.5609 | 0.7090 | 0.6683 | 0.5286 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| MTI First Line Index | 0.5019 | 0.8047 | 0.6925 | 0.7232 | 0.5603 | 0.4809 |
| Default MTI | 0.5175 | 0.7681 | 0.7473 | 0.7394 | 0.5380 | 0.5252 |
| iria-mix | 0.4535 | 0.6292 | 0.7496 | 0.6701 | 0.4290 | 0.5048 |
| DeepMeSH1 | 0.5537 | 0.8223 | 0.7605 | 0.7761 | 0.5896 | 0.5422 |
| DeepMeSH2 | 0.5527 | 0.8216 | 0.7590 | 0.7750 | 0.5893 | 0.5406 |
| DeepMeSH3 | 0.5522 | 0.8224 | 0.7576 | 0.7746 | 0.5889 | 0.5400 |
| DeepMeSH4 | 0.5451 | 0.8137 | 0.7528 | 0.7680 | 0.5811 | 0.5333 |
| DeepMeSH5 | 0.5955 | 0.8406 | 0.7813 | 0.7971 | 0.6291 | 0.5838 |
| MeSHProbeNet-P | 0.5520 | 0.8250 | 0.7551 | 0.7725 | 0.5910 | 0.5403 |
| NLM System 4 | 0.5235 | 0.8075 | 0.7224 | 0.7453 | 0.5646 | 0.5109 |
| MeSHProbeNet-P0 | 0.5536 | 0.8162 | 0.7650 | 0.7739 | 0.5849 | 0.5484 |
| MeSHProbeNet-P1 | 0.5549 | 0.8067 | 0.7751 | 0.7751 | 0.5783 | 0.5566 |
| MeSHProbeNet-P2 | 0.5537 | 0.8222 | 0.7604 | 0.7742 | 0.5894 | 0.5446 |
| MeSHProbeNet-P3 | 0.5540 | 0.8214 | 0.7616 | 0.7746 | 0.5894 | 0.5454 |
+ Test batch 2, week 4
Annotated articles:7079/7382
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| MTI First Line Index | 0.6383 | 0.6712 | 0.6308 | 0.6305 | 0.6282 | 0.5892 | 0.5581 | 0.6601 | 0.6179 | 0.4794 |
| Default MTI | 0.6485 | 0.6389 | 0.6826 | 0.6422 | 0.5951 | 0.6304 | 0.5780 | 0.6318 | 0.6660 | 0.4907 |
| iria-mix | 0.5358 | 0.4672 | 0.6335 | 0.5242 | 0.4583 | 0.5098 | 0.4526 | 0.4719 | 0.6197 | 0.3682 |
| DeepMeSH1 | 0.6925 | 0.7101 | 0.6929 | 0.6862 | 0.6715 | 0.6137 | 0.5969 | 0.7110 | 0.6749 | 0.5391 |
| DeepMeSH2 | 0.6918 | 0.7117 | 0.6902 | 0.6856 | 0.6769 | 0.6084 | 0.5941 | 0.7128 | 0.6720 | 0.5383 |
| DeepMeSH3 | 0.6913 | 0.7128 | 0.6881 | 0.6850 | 0.6765 | 0.6062 | 0.5928 | 0.7137 | 0.6701 | 0.5376 |
| DeepMeSH4 | 0.6783 | 0.7030 | 0.6692 | 0.6702 | 0.6644 | 0.5706 | 0.5613 | 0.7054 | 0.6531 | 0.5205 |
| DeepMeSH5 | 0.6954 | 0.7122 | 0.6955 | 0.6887 | 0.6745 | 0.6176 | 0.6007 | 0.7135 | 0.6782 | 0.5429 |
| MeSHProbeNet-P | 0.6917 | 0.7147 | 0.6869 | 0.6835 | 0.6709 | 0.5953 | 0.5785 | 0.7097 | 0.6746 | 0.5360 |
| NLM System 4 | 0.6573 | 0.6899 | 0.6418 | 0.6469 | 0.6137 | 0.5190 | 0.5099 | 0.6836 | 0.6329 | 0.4948 |
| MeSHProbeNet-P0 | 0.6915 | 0.7089 | 0.6916 | 0.6832 | 0.6650 | 0.6001 | 0.5805 | 0.7040 | 0.6794 | 0.5356 |
| MeSHProbeNet-P1 | 0.6913 | 0.7028 | 0.6970 | 0.6830 | 0.6596 | 0.6055 | 0.5828 | 0.6979 | 0.6848 | 0.5352 |
| MeSHProbeNet-P2 | 0.6919 | 0.7203 | 0.6815 | 0.6833 | 0.6791 | 0.5836 | 0.5721 | 0.7156 | 0.6696 | 0.5359 |
| MeSHProbeNet-P3 | 0.6803 | 0.7006 | 0.6780 | 0.6713 | 0.6538 | 0.5699 | 0.5515 | 0.6953 | 0.6659 | 0.5221 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| MTI First Line Index | 0.5123 | 0.7835 | 0.7289 | 0.7357 | 0.5474 | 0.5087 |
| Default MTI | 0.5256 | 0.7541 | 0.7730 | 0.7461 | 0.5319 | 0.5470 |
| iria-mix | 0.4506 | 0.6195 | 0.7544 | 0.6647 | 0.4222 | 0.5098 |
| DeepMeSH1 | 0.5563 | 0.8102 | 0.7724 | 0.7754 | 0.5839 | 0.5545 |
| DeepMeSH2 | 0.5556 | 0.8118 | 0.7700 | 0.7747 | 0.5850 | 0.5520 |
| DeepMeSH3 | 0.5545 | 0.8125 | 0.7682 | 0.7741 | 0.5854 | 0.5498 |
| DeepMeSH4 | 0.5427 | 0.8080 | 0.7532 | 0.7635 | 0.5773 | 0.5352 |
| DeepMeSH5 | 0.5599 | 0.8115 | 0.7745 | 0.7772 | 0.5870 | 0.5583 |
| MeSHProbeNet-P | 0.5568 | 0.8174 | 0.7682 | 0.7756 | 0.5881 | 0.5534 |
| NLM System 4 | 0.5260 | 0.8049 | 0.7257 | 0.7451 | 0.5661 | 0.5158 |
| MeSHProbeNet-P0 | 0.5568 | 0.8128 | 0.7728 | 0.7759 | 0.5846 | 0.5566 |
| MeSHProbeNet-P1 | 0.5571 | 0.8080 | 0.7775 | 0.7762 | 0.5811 | 0.5600 |
| MeSHProbeNet-P2 | 0.5560 | 0.8216 | 0.7632 | 0.7746 | 0.5916 | 0.5489 |
| MeSHProbeNet-P3 | 0.5479 | 0.8093 | 0.7625 | 0.7678 | 0.5791 | 0.5455 |
+ Test batch 2, week 5
Annotated articles:6756/7240
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| MTI First Line Index | 0.6406 | 0.6869 | 0.6260 | 0.6371 | 0.6425 | 0.5693 | 0.5483 | 0.6726 | 0.6115 | 0.4908 |
| Default MTI | 0.6450 | 0.6409 | 0.6739 | 0.6387 | 0.6014 | 0.6081 | 0.5644 | 0.6332 | 0.6571 | 0.4875 |
| iria-mix | 0.5179 | 0.4579 | 0.6037 | 0.5082 | 0.4298 | 0.4757 | 0.4243 | 0.4606 | 0.5915 | 0.3540 |
| DeepMeSH1 | 0.6895 | 0.7181 | 0.6807 | 0.6836 | 0.6855 | 0.5870 | 0.5786 | 0.7178 | 0.6634 | 0.5359 |
| DeepMeSH2 | 0.6880 | 0.7190 | 0.6769 | 0.6819 | 0.6879 | 0.5791 | 0.5731 | 0.7192 | 0.6594 | 0.5341 |
| DeepMeSH3 | 0.6883 | 0.7204 | 0.6758 | 0.6819 | 0.6895 | 0.5797 | 0.5750 | 0.7207 | 0.6586 | 0.5340 |
| DeepMeSH4 | 0.6756 | 0.7108 | 0.6578 | 0.6677 | 0.6748 | 0.5440 | 0.5418 | 0.7122 | 0.6427 | 0.5169 |
| DeepMeSH5 | 0.7120 | 0.7420 | 0.7041 | 0.7085 | 0.7142 | 0.6165 | 0.6095 | 0.7407 | 0.6854 | 0.5730 |
| MeSHProbeNet-P | 0.6839 | 0.6850 | 0.6986 | 0.6746 | 0.6480 | 0.5985 | 0.5741 | 0.6807 | 0.6872 | 0.5254 |
| NLM System 4 | 0.6514 | 0.6873 | 0.6329 | 0.6405 | 0.6175 | 0.5024 | 0.4989 | 0.6805 | 0.6246 | 0.4869 |
| MeSHProbeNet-P0 | 0.6852 | 0.7038 | 0.6828 | 0.6755 | 0.6669 | 0.5817 | 0.5666 | 0.6994 | 0.6716 | 0.5268 |
| MeSHProbeNet-P1 | 0.6848 | 0.7200 | 0.6673 | 0.6748 | 0.6847 | 0.5631 | 0.5561 | 0.7161 | 0.6562 | 0.5263 |
| MeSHProbeNet-P2 | 0.6857 | 0.7149 | 0.6727 | 0.6754 | 0.6811 | 0.5647 | 0.5567 | 0.7112 | 0.6620 | 0.5268 |
| MeSHProbeNet-P3 | 0.6832 | 0.6980 | 0.6848 | 0.6738 | 0.6591 | 0.5872 | 0.5685 | 0.6936 | 0.6731 | 0.5249 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| MTI First Line Index | 0.5260 | 0.7992 | 0.7244 | 0.7424 | 0.5691 | 0.5131 |
| Default MTI | 0.5246 | 0.7613 | 0.7656 | 0.7463 | 0.5367 | 0.5401 |
| iria-mix | 0.4401 | 0.6124 | 0.7339 | 0.6531 | 0.4161 | 0.4915 |
| DeepMeSH1 | 0.5535 | 0.8178 | 0.7613 | 0.7739 | 0.5891 | 0.5439 |
| DeepMeSH2 | 0.5513 | 0.8191 | 0.7574 | 0.7723 | 0.5885 | 0.5402 |
| DeepMeSH3 | 0.5515 | 0.8202 | 0.7558 | 0.7718 | 0.5897 | 0.5397 |
| DeepMeSH4 | 0.5366 | 0.8137 | 0.7409 | 0.7602 | 0.5779 | 0.5226 |
| DeepMeSH5 | 0.5903 | 0.8332 | 0.7785 | 0.7914 | 0.6242 | 0.5801 |
| MeSHProbeNet-P | 0.5520 | 0.7977 | 0.7793 | 0.7722 | 0.5708 | 0.5597 |
| NLM System 4 | 0.5180 | 0.8052 | 0.7167 | 0.7402 | 0.5607 | 0.5057 |
| MeSHProbeNet-P0 | 0.5507 | 0.8125 | 0.7646 | 0.7713 | 0.5812 | 0.5481 |
| MeSHProbeNet-P1 | 0.5476 | 0.8254 | 0.7496 | 0.7689 | 0.5888 | 0.5359 |
| MeSHProbeNet-P2 | 0.5494 | 0.8215 | 0.7540 | 0.7694 | 0.5873 | 0.5404 |
| MeSHProbeNet-P3 | 0.5496 | 0.8079 | 0.7670 | 0.7705 | 0.5770 | 0.5495 |
+ Test batch 3, week 1
Annotated articles:5835/6266
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| NLM System 4 | 0.0967 | 0.6878 | 0.5990 | 0.6183 | 0.0364 | 0.4866 | 0.1232 | 0.0527 | 0.5867 | 0.4653 |
| MTI First Line Index | 0.6499 | 0.6993 | 0.6377 | 0.6469 | 0.6475 | 0.5872 | 0.5646 | 0.6842 | 0.6189 | 0.5024 |
| Default MTI | 0.6547 | 0.6519 | 0.6869 | 0.6492 | 0.6037 | 0.6266 | 0.5804 | 0.6436 | 0.6661 | 0.4993 |
| iria-mix | 0.5221 | 0.4613 | 0.6129 | 0.5122 | 0.4250 | 0.4863 | 0.4335 | 0.4632 | 0.5983 | 0.3582 |
| DeepMeSH1 | 0.6985 | 0.7423 | 0.6818 | 0.6944 | 0.6962 | 0.5994 | 0.5922 | 0.7412 | 0.6604 | 0.5492 |
| DeepMeSH2 | 0.6977 | 0.7418 | 0.6808 | 0.6936 | 0.6986 | 0.5958 | 0.5894 | 0.7407 | 0.6594 | 0.5483 |
| DeepMeSH3 | 0.6992 | 0.7227 | 0.6990 | 0.6947 | 0.6814 | 0.6110 | 0.5971 | 0.7223 | 0.6776 | 0.5492 |
| DeepMeSH4 | 0.7277 | 0.7493 | 0.7261 | 0.7229 | 0.7148 | 0.6429 | 0.6317 | 0.7500 | 0.7068 | 0.5908 |
| DeepMeSH5 | 0.7279 | 0.7671 | 0.7107 | 0.7227 | 0.7292 | 0.6328 | 0.6285 | 0.7684 | 0.6914 | 0.5909 |
| MeSHProbeNet-P | 0.6912 | 0.7294 | 0.6790 | 0.6841 | 0.6851 | 0.5745 | 0.5647 | 0.7235 | 0.6616 | 0.5375 |
| MeSHProbeNet-P0 | 0.6921 | 0.7190 | 0.6897 | 0.6850 | 0.6735 | 0.5865 | 0.5717 | 0.7131 | 0.6723 | 0.5382 |
| MeSHProbeNet-P1 | 0.6926 | 0.7078 | 0.7007 | 0.6855 | 0.6610 | 0.5981 | 0.5780 | 0.7020 | 0.6834 | 0.5385 |
| MeSHProbeNet-P2 | 0.6925 | 0.7142 | 0.6940 | 0.6853 | 0.6705 | 0.5854 | 0.5707 | 0.7084 | 0.6772 | 0.5382 |
| MeSHProbeNet-P3 | 0.6913 | 0.7107 | 0.6943 | 0.6839 | 0.6704 | 0.5819 | 0.5683 | 0.7050 | 0.6781 | 0.5363 |
| lcy_test | 0.1748 | 0.3593 | 0.1338 | 0.1758 | 0.2689 | 0.1668 | 0.1424 | 0.3032 | 0.1228 | 0.1022 |
Hierarchical Measures
| System | LCA-F | HiP | HiR | HiF | LCA-P | LCA-R |
|---|---|---|---|---|---|---|
| NLM System 4 | 0.5007 | 0.8007 | 0.6974 | 0.7231 | 0.5559 | 0.4844 |
| MTI First Line Index | 0.5299 | 0.8030 | 0.7310 | 0.7455 | 0.5733 | 0.5201 |
| Default MTI | 0.5284 | 0.7619 | 0.7742 | 0.7494 | 0.5389 | 0.5476 |
| iria-mix | 0.4399 | 0.6113 | 0.7413 | 0.6541 | 0.4135 | 0.4966 |
| DeepMeSH1 | 0.5569 | 0.8311 | 0.7596 | 0.7777 | 0.5988 | 0.5438 |
| DeepMeSH2 | 0.5566 | 0.8318 | 0.7585 | 0.7774 | 0.5990 | 0.5433 |
| DeepMeSH3 | 0.5597 | 0.8156 | 0.7750 | 0.7791 | 0.5887 | 0.5573 |
| DeepMeSH4 | 0.6004 | 0.8327 | 0.7947 | 0.7988 | 0.6273 | 0.5979 |
| DeepMeSH5 | 0.5979 | 0.8469 | 0.7811 | 0.7978 | 0.6362 | 0.5859 |
| MeSHProbeNet-P | 0.5523 | 0.8259 | 0.7574 | 0.7718 | 0.5932 | 0.5429 |
| MeSHProbeNet-P0 | 0.5541 | 0.8177 | 0.7675 | 0.7736 | 0.5875 | 0.5511 |
| MeSHProbeNet-P1 | 0.5557 | 0.8087 | 0.7776 | 0.7750 | 0.5810 | 0.5596 |
| MeSHProbeNet-P2 | 0.5549 | 0.8146 | 0.7711 | 0.7744 | 0.5851 | 0.5544 |
| MeSHProbeNet-P3 | 0.5547 | 0.8125 | 0.7718 | 0.7740 | 0.5837 | 0.5549 |
| lcy_test | 0.2042 | 0.6267 | 0.2584 | 0.3375 | 0.3522 | 0.1596 |
+ Test batch 3, week 2
Annotated articles:10386/11455
Flat Measures
| System | MiF | EBP | EBR | EBF | MaP | MaR | MaF | MiP | MiR | Acc. |
|---|---|---|---|---|---|---|---|---|---|---|
| MTI First Line Index | 0.6413 | 0.6887 | 0.6304 | 0.6379 | 0.6446 | 0.5767 | 0.5523 | 0.6740 | 0.6116 | 0.4914 |
| Default MTI | 0.6454 | 0.6392 | 0.6810 | 0.6392 | 0.6015 | 0.6185 | 0.5695 | 0.6317 | 0.6598 | 0.4878 |
| iria-mix | 0.5139 | 0.4500 | 0.6094 | 0.5039 | 0.4261 | 0.4774 | 0.4231 | 0.4529 | 0.5939 | 0.3500 |
| DeepMeSH1 | 0.6956 | 0.7326 | 0.6832 | 0.6900 | 0.7005 | 0.6006 | 0.5945 | 0.7325 | 0.6622 | 0.5438 |
| DeepMeSH2 | 0.6950 | 0.7180 | 0.6938 | 0.6893 | 0.6945 | 0.6049 | 0.5950 | 0.7183 |