BioASQ Participants Area
Task 8a begun on Monday, February 03, 2020!
What's new in BioASQ8, Task a
- This year the testsets will run from 10.00 GMT on Monday until 07.00 GMT on Tuesday, to comply with a time-shift observed in annotation release at PubMed. For more details please see FAQ
- In Task 8a a limit of 200 MeSH labels per article is introduced for participant responses. For more details please see FAQ
- Articles with "automated indexing" in PubMed will be ignored during the evaluation on testsets of task 8a. For more details please see FAQ
Guidelines for Task 8a
Task 8a will be based on the standard process followed by PubMed to index journal abstracts. The participants will be asked to classify new PubMed documents, written in English, as they become available on-line, before PubMed curators annotate them manually. As manual annotations from PubMed become available, they will be used to evaluate the classification performance of participating systems.The rest of the guidelines provide the essential information for participating in the Task 8a of the BioASQ challenge. They are organized in sections, by clicking on the titles you can find the relevant details.
+ Competition roll-out
- Three batches, of five test sets each (15 tests in total), will be released.
- New test sets will be available every Monday at 10.00 GMT. Participants that have checked "Receive Information" during registration will be sent an e-mail informing that a new test set is available.
- Participants will have a limited time window to upload their results, until 07.00 GMT of Tuesday.
- After the expiration of the test set, the evaluation measures will be calculated incrementally. The "Results" section will be initialized with the systems that participated in the test set.
- There will be four winning teams in each batch. Information about the prizes is available here. The winners will be decided from the rankings of the 4 best attempts of each team for each test batch. For example, if a team participates with 4 different systems, during the calculation of the rankings only the best performance of its systems will be used. In this way, a team can win a maximum of two prizes per batch.
+ Download Training Data
The training data sets for this task are available for downloading. They contain annotated articles from PubMed, where annotated means that MeSH terms have been assigned to the articles by the human curators in PubMed. Table 1 provides information about the provided datasets. Note that the main difference between those datasets among the different years, apart from the size, is the MeSH terms used. For example the 2015 training datasets contain articles where MeSH 2015 have been assigned. Also, for 2014, 2015 and 2016 there are two versions of the training data available. The small version (wrt size) consists of articles that belong to the pool of journals that the BioASQ team used to select the articles for the test data (this was a subset of the available journals). The bigger version consists of articles from every available journal. Since 2017 articles for the test data will be selected from all available journals, so only one corresponding training data set will be available. The evaluation of the results during each year of the challenge is performed using the corresponding version of the MeSH terms, thus their usage is highly recommended. The training datasets of previous years of the challenge are also available for reference reasons. Note that not every MeSH term is covered in the datasets. Participants, are allowed to use unlimited resources to train their systems.
The training set is served as a JSON string with the following format:{"articles": [{"abstractText":"text..", "journal":"journal..", "meshMajor":["mesh1",...,"meshN"], "pmid":"PMID", "title":"title..", "year":"YYYY"},..., {..}]}The JSON string contains the following fields for each article:
pmid : the unique identifier of each article,
title : the title of the article,
abstractText : the abstract of the article,
year : the year the article was published,
journal : the journal the article was published, and
meshMajor : a list with the major MeSH headings of the article.
Training v.2014 (txt/Lucene) | Training v.2014b (txt/Lucene) | Training v.2015 (txt/Lucene) | Training v.2015b (txt/Lucene) | Training v.2016 (txt/Lucene) | Training v.2016b (txt/Lucene) | Training v.2017 (txt/Lucene) | Training v.2018 (txt/Lucene) | Training v.2019 (txt/Lucene) | Training v.2020 (txt/Lucene) | |
Number of articles | 12,628,968 | 4,458,300 | 11,804,715 | 4,607,922 | 12,208,342 | 4,917,245 | 12,834,585 | 13,486,072 | 14,200,259 | 14,913,939 |
Avg. MeSH/article | 12.72 | 13.20 | 12.61 | 13.08 | 12.62 | 13.01 | 12.66 | 12.69 | 12.69 | 12.68 |
MeSH covered | 26,831 | 26,631 | 27,097 | 26,866 | 27,301 | 27,150 | 27,773 | 28,340 | 28,863 | 29,102 |
Size zip/unzip (txt) | 6.2Gb/20.31Gb | 1.9Gb/6.4Gb | 5.7Gb/19Gb | 2.2Gb/7.4Gb | 5.94Gb/19.4Gb | 2.4Gb/7.92Gb | 6.29Gb/20.5Gb | 6.68Gb/21.7Gb | 7.10Gb/23.1Gb | 7.51Gb/24.4Gb |
Size zip/unzip (Lucene) | 4.4Gb/6.2Gb | 1.3Gb/1.9Gb | 4.0Gb/5.6Gb | 1.6Gb/2.3Gb | 12.9Gb/15.8Gb | 5.25Gb/6.42Gb | 13.7Gb/16.7Gb | 14.5Gb/17.7Gb | 15.4Gb/18.8Gb | 16.3Gb/19.9Gb |
Attention: Note that only registered users can download the training set. You can register to the BioASQ challenge here. Not every MeSH term is covered to the provided datasets.
+ MeSH headings
MeSH is the National Library of Medicine's controlled vocabulary thesaurus. It is updated every year to include improvements or changes to the scientific domains it covers. It consists of sets of terms naming descriptors in a hierarchical structure. They are used from the human curators in NLM during the manual annotation of the biomedical articles that are uploaded.
There are 29657 descriptors in MeSH 2020 (available here) that will be used during this year of the challenge. The MeSH of 2019 , 2018 , 2017 , 2016 , 2015 , 2014 and 2013 are also available for reference. There are two ways of description of a MeSH in the hierarchy: (i) the humam name e.g. "Female" and (ii) the index e.g. "D005260". In the submission of the results participants should use the index of the MeSH. The mapping between the names and the indexes is available here. All the nodes in the graph (and not only the leaf nodes) of MeSH are valid as classification answers for the BioASQ challenge.
The hierarchy file, in parent-child format (available here - last update XX/XX/2020), contains a line
for each relation between the parent and a child node. For example, the line:
D001570 D000876
is to be read as node D001570|Benzodiazepinones is a parent of D000876|Anthramycin.
In addition to the MeSH Headings, other kinds of indexing information are available, e.g. Entry Terms and Supplementary Concept Records.
Entry Terms are synonyms of the MeSH. Entry Terms should not be provided in the evaluation, only MeSH Headings will be evaluated.
Supplementary Concept Records (SCRs) can be thought as subclasses or instances of MeSH headings. They are used to index chemicals, drugs, and other concepts such as rare diseases for MEDLINE. Each SCR is linked to one or more MeSH Headings and thus they can be helpful during classification. Since MeSH Headings are included in the index, SCRs are not, and they are not valid answers for the evaluation.
Some Headings have a special status as Check Tags, such as heading "Human", and are very frequent. Those are not removed, and will be valid answers during the evaluation.
+ Evaluation
For reasons of completeness, we are also going to provide the evaluation measures listed below:
- Accuracy (Acc.)
- Example Based Precision (EBP)
- Example Based Recall (EBR)
- Example Based F-Measure (EBF)
- Macro Precision (MaP)
- Macro Recall (MaR)
- Macro F-Measure (MaF)
- Micro Precision (MiP)
- Micro Recall (MiR)
- Micro F-Measure (MiF)
- Hierarchical Precision (HiP)
- Hierarchical Recall (HiR)
- Hierarchical F-Measure (HiF)
- Lowest Common Ancestor Precision (LCA-P)
- Lowest Common Ancestor Recall (LCA-R)
- Lowest Common Ancestor F-measure (LCA-F)
For more information regarding the LCA-F the interested reader is referred to "A. Kosmopoulos, I. Partalas, E. Gaussier, G. Paliouras and I. Androutsopoulos: Evaluation Measures for Hierarchical Classification: a unified view and novel approaches", available here.
+ Download test sets
Each test set will consist of non annotated abstracts of articles that have been uploaded in PubMed*.
The data of each test set will be served as JSON (Java Script Object Notation) strings. JSONs are light and can be easily parsed from programming languages. Each programming language offers modules for the interaction with JSON strings.
The format of the test set data in the JSON string will be the following:
{"documents": [ {"pmid":22511223, "title":"Title", "abstract":"Abstact..", "journal":"Journal.."}, {"pmid":22511224, "title":"Title", "abstract":"Abstact..", "journal":"Journal.."}, . . {"pmid":22511225, "title":"Title", "abstract":"Abstact..", "journal":"Journal.."}]}This JSON string represents an array with document objects. Each object has a pmid, an abstract, a title and a journal.
Only registered users can download the test sets. There are two ways that a user can download a test set:
- Using the web interface. In the section Submitting/Task 8a you can find the available test sets of the current edition of the BioASQ challenge. A repository with every BioASQ test is available in Oracle/Get data. By clicking in the test, you can download it either as a text file or as a lucene index.
- Using an API. By making a GET request in the URL http://participants-area.bioasq.org/tests/test_number/ along with authentication parameters. In the position of test_number a number should be placed indicating the test set that the user wants to download. For example, for downloading the test set of Week 2, you should make a GET request to http://participants-area.bioasq.org/tests/2/, while for downloading the 1st test set of the second batch you should make the GET request to http://participants-area.bioasq.org/tests/6/, provided that each test batch includes 5 test sets. To download the dry-run test set the URL should be http://participants-area.bioasq.org/tests/0/. In the following section "Code Snippets" you can find snippets in Python that perform this request.
+ Submit test results
There are two ways for a user to submit results of an active test set:
- Using the web interface. In the section Submitting/Task 8a you can find
a form with a "Browse" field and a system dropdown menu. After selecting the file in your computer that contains the JSON string
and selecting the name of the system that corresponds to these results you can submit them.
br>
The format of the JSON string in this case will be: br>
{"documents": [{"labels":["label1","label2",...,"labelN"], "pmid": 22511223}, {"labels":["label1", "label2",...,"labelM"],"pmid":22511224}, . . {"labels":["label1", "label2",...,"labelK"], "pmid":22511225}]}
where "label1",.."labelN" are the MeSH indicators e.g. "D005260" and not the human annotation i.e. "Female".
- Using an API. By making a POST request in the URi http://participants-area.bioasq.org/tests/uploadResults/test_number/ along with the JSON string.
In the position of test_number a number indicating the test set that the user wants to upload results for should be placed. For example, for uploading results for the Week 2 test set of the first test batch, you should POST your results in http://participants-area.bioasq.org/tests/uploadResults/2/. To submit results for the dry-run test set you should POST in http://participants-area.bioasq.org/tests/uploadResults/0/. The format of the JSON string in this case will be:
{"username":"your_username", "password":"your_password", "system":"your_system", "documents": [{"labels":["label1", "label2",...,"labelN"], "pmid": 22511223}, {"labels":["label1", "label2",...,"labelM"],"pmid":22511224}, . . {"labels":["label1", "label2",..., "labelK"], "pmid":22511225}]}
Again, "label1",.."labelN" are the MeSH indicators e.g. "D005260".
curl
that perform the POST and GET requests.
ATTENTION:
- Users must upload MeSH indexes for every article in the test set.
- The format of the JSON string is case sensitive. Thus, trying to upload a JSON with different values (i.e "PMID" instead of "pmid") will result in a 500 error.
- Users must upload their results before the expiration of the test
- Users can upload results multiple times for the same system before the expiration of the test set. Each time that a user uploads new results the old ones are erased.
- The system before saving the results checks that:
- The system in the JSON string belongs to the user,
- The PMIDs in the provided JSON belong to the active test set,
- There are MeSH indexes for every article of the test set,
- The MeSH indexes exist, and
- The test set is still active.
- The system responds with an OK message or an error message depending on the progress of the user request.
- After uploading results, participants can see some information about their uploads in the "Submit your results" section
+ Add a system
Each user will have the opportunity to participate in Task 8a with a maximum of 5 systems. To register the systems, after logging in you have to visit "Edit Profile Settings" and follow the available instructions there.
ATTENTION: Trying to upload results without selecting a system will result in error while the results will not be saved.
+ Code snippets and tools
Vectorization Tools
You can download tools for vectorizing the data from here. To use the tools, the lucene-core and lucene-analyzers jars must be included in the classpath. The tools produce a vectorized description as a Lucene Index.Python Snippets
In this section you can find some code snippets written in python. The library "Requests"
is used to perform the HTTP GET and POST requests.
Download Test Sets-Using API
Python Snippet
Assuming we want to download the articles of the test set of Week 1:import requests import json r=requests.get('http://participants-area.bioasq.org/tests/1/', auth=('your_username', 'your_password')) #Now the variable r contains the data (can been seen with r.text) data=json.loads(r.text) #Now the data are in the "data" variable in python native datatypes len(data) # This will give the number of the articles of the test set 1
Curl
The same could be done from the terminal using acurl
command:
curl -i --user your_username:your_password http://participants-area.bioasq.org/tests/1/
Upload Test Results-Using API
Python Snippet
Assuming we want to upload results for the test set of Week 1, we have created a JSON string i.e.{"username": "your_username", "password": "your_password", "documents": [{"pmid": 23476937, "labels": ["D000127", "D000128"]}], "system": "your_system"}In this string apart from your authentication credentials you mention that you have estimated D000127 and D000128 for the article with PMID 23476937. To upload this information using Python and
"Requests"
you execute the following snippet:
import requests import json d={"username": "your_username", "password": "your_password", "documents": [{"pmid": 23476937, "labels": ["D000127", "D000128"]}], "system": "your_system"} url='http://participants-area.bioasq.org/tests/uploadResults/1/' r=requests.post(url, data=json.dumps(d)) r.status_code #you can see 200 for success r.text #you can see the platform's answer
Curl
The same could be done from the terminal using curl
. From the directory you have saved the file with the test resuls, type the following command in a single line, where results.json
is the file that contains the JSON string with your results:curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST --data-binary @results.json http://participants-area.bioasq.org/tests/uploadResults/1/
+ Continuous Space Word Vectors
*Data from NLM are distributed based on the conditions described here. License Code: 8283NLM123.
If you used data obtained from the BioASQ challenges please support us by reporting BioASQ in your acknowledgements and citing our paper:
An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition: George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artiéres, Axel Ngonga, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos and Georgios Paliouras, in BMC bioinformatics, 2015 (bib).