BioASQ Participants Area

The training dataset for this task is available for downloading. We provided a training dataset for each of the subtracks:

• [Sub-track 1] MESINESP - Scientific Literature: It contains all spanish records from LILACS and IBECS databases at the Virtual Health Library (VHL) with non-empty abstract written in Spanish. We have filtered out empty abstracts and non-Spanish abstracts. We have built the training dataset with the data crawled on 01/29/2021. This means that the data is a snapshot of that moment and that may change over time since LILACS and IBECS usually add or modify indexes after the first inclusion in the database. We distribute two different datasets:
  • Articles training set: This corpus contains the set of 237574 Spanish scientific papers in VHL that have at least one DeCS code assigned to them.
  • Full training set: This corpus contains the whole set of 249474 Spanish documents from VHL that have at leas one DeCS code assigned to them.
• [Sub-track 2] MESINESP - Clinical Trials: The training dataset contains records from Registro Español de Estudios Clínicos (REEC). REEC doesn't provide documents with the structure title/abstract needed in BioASQ, for that reason we have built artificial abstracts based on the content available in the data crawled using the REEC API. Clinical trials are not indexed with DeCS terminology, we have used as training data a set of 3592 clinical trials that were automatically annotated in the first edition of MESINESP and that were published as a Silver Standard outcome. Because the performance of the models used by the participants was variable, we have only selected predictions from runs with a MiF higher than 0.30, which corresponds with the submission of the best three teams. We have selected the intersection of all codes assigned by those teams.
• [Sub-track 3] MESINESP - Patents: It will be announced in some days.

DeCS is a trilingual and structured vocabulary created by BIREME to serve as a unique language in indexing, searching and retrieving material from biomedical scientific literature. It was developed from the MeSH aiming to serve as a common terminology for consistent searching in three languages, namely Spanish, Portuguese and English.

The MESINESP task dataset annotations are based on the DeCS 2020. There are two ways of description of a DeCS descriptor in the hierarchy: (i) the human name e.g. "trombólisis mecánica" and (ii) the index e.g. "D061185". In the submission of the results participants should use the index of the DeCS. The DeCS 2020 hierarchy is available in the OBO format here. The mapping between the names and the indexes is available here. All the nodes in the graph (and not only the leaf nodes) of DeCS are valid as classification answers for the BioASQ challenge.

Note: Thanks to our collaborators at BIREME, we have been able to include new COVID-related descriptors that will be used in future versions of DeCS. Training articles do not use these terms, but they will appear in a future version of the development set that will enable systems to properly classify this type of content.

• Accuracy (Acc.)
• Example Based Precision (EBP)
• Example Based Recall (EBR)
• Example Based F-Measure (EBF)
• Macro Precision (MaP)
• Macro Recall (MaR)
• Macro F-Measure (MaF)
• Micro Precision (MiP)
• Micro Recall (MiR)
• Micro F-Measure (MiF)

There will be a test set for each of the subtracks. Each test set will consist of non-annotated documents in spanish that has been uploaded to IBECS/LILACS, REEC and Google Patents, as well as other background biomedical texts in spanish. However, only the labels of a manually indexed documents by experts will be used for the official evaluation, not the background data.

The data of the test set will be served as JSON (Java Script Object Notation) strings. JSONs are light and can be easily parsed from programming languages. Each programming language offers modules for the interaction with JSON strings.

{"articles": [
  {"id":"biblio-1000005", "title":"Title",  "abstract":"Abstact..", "journal":"Journal.."},
  {"id":"biblio-1000072", "title":"Title",  "abstract":"Abstact..", "journal":"Journal.."},
           .
           .
  {"id":"lil-708981", "title":"Title",  "abstract":"Abstact..", "journal":"Journal.."}]}

Only registered users can download the test sets. Using the web interface. In the section Submitting/Task MESINESP you can find the available test sets. By clicking in the test, you can download it as a JSON text file.

In the section Submitting/Task MESINESP you can find a form with a "Browse" field and a system dropdown menu. After selecting the file in your computer that contains the JSON string and selecting the name of the system that corresponds to these results you can submit them.

{"documents": [{"labels":["label1","label2",...,"labelN"], "id": "biblio-1000005"},
                      {"labels":["label1", "label2",...,"labelM"],"id": "biblio-1000072"},
                                                .
                                                .
                      {"labels":["label1", "label2",...,"labelK"], "id":"lil-708981'}]}

• Users must upload DeCS labels for every article in the test set.
• The format of the JSON string is case sensitive. Thus, trying to upload a JSON with different values (i.e "ID" instead of "id") will result in a 500 error.
• Users must upload their results before the expiration of the test
• Users can upload results multiple times for the same system before the expiration of the test set. Each time that a user uploads new results the old ones are erased.
• The system before saving the results checks that:
  • The system in the JSON string belongs to the user,
  • The IDs in the provided JSON belong to the active test set,
  • There are DeCS indexes for every article of the test set,
  • The DeCS indexes exist, and
  • The test set is still active.
• The system responds with an OK message or an error message depending on the progress of the user request.
• After uploading results, participants can see some information about their uploads in the "Submit your results" section

Each user will have the opportunity to participate in Task MESINESP with a maximum of 5 systems. To register the systems, after logging in you have to visit "Edit Profile Settings" and follow the available instructions there.

ATTENTION: Trying to upload results without selecting a system will result in error while the results will not be saved.