BioASQ Participants Area
Frequently Asked Questions
- Why participants should no longer submit synonyms for exact answers of list and factoid questions? Submission of synonyms by participants is redundant, since only one synonym is enough for an exact answer to be considered correct. Golden exact answers include synonyms, when appropriate, and, even if some system encounter and submit some valid synonym missing from the initial golden answer, this synonym will be included in the enriched golden answer, after the manual inspection of all system submissions by the biomedical experts.
Note: This change does not affect the format of the submissions.
Each unique exact answer is still represented by an array in JSON format, as done so far. Now, this array should only contain one element, instead of many synonyms. If the arrays in your submission contain more than one elements/synonyms, only the first of the synonyms will be taken into account during evaluation.
For more details, please see "JSON format of the datasets" section in Task 5B Guidelines.
- Why test articles are no longer selected from a list of selected journals? Lately, NLM indexers tend to work on a backlog of not indexed articles. Therefore, test-sets created only by recently added articles have big annotation periods. Under this conditions, the best way to have test sets with satisfyingly small annotation periods, is to use internal information from NLM, about the articles with high probability to be indexed soon. As a result, a list of selected journals is redundant and would only limit the size of test sets without any significant impact on expected annotation period.
- Which are the classes that are going to be used during classification? MeSH (Medical Subject Headings) is the vocabulary that is going to be used for classification.
- How many labels does each article usually receive? Usually articles receive around 10-15 labels.
- In what formats are the training and the test data going to be provided during the challenge? Training and test data are going to be provided both in raw text and in vectorized description. The vectorized description will be distributed as a Lucene Index. Detailed description of the fields of the index and the vectorization tools is provided analytically.