BioASQ Participants Area

The benchmark datasets contain four types of questions:

1. Yes/no questions: These are questions that, strictly speaking, require "yes" or "no" answers, though of course in practice longer answers will often be desirable. For example, "Do CpG islands colocalise with transcription start sites?" is a yes/no question.
2. Factoid questions: These are questions that, strictly speaking, require a particular entity name (e.g., of a disease, drug, or gene), a number, or a similar short expression as an answer, though again a longer answer may be desirable in practice. For example, "Which virus is best known as the cause of infectious mononucleosis?" is a factoid question.
3. List questions: These are questions that, strictly speaking, require a list of entity names (e.g., a list of gene names), numbers, or similar short expressions as an answer; again, in practice additional information may be desirable. For example, "Which are the Raf kinase inhibitors?" is a list question.
4. Summary questions: These are questions that do not belong in any of the previous categories and can only be answered by producing a short text summarizing the most prominent relevant information. For example, "What is the treatment of infectious mononucleosis?" is a summary question.

In Phase A of Task 9b, the participants will be provided with English questions q₁, q₂,...,q_n. For each question q_i, each participating system will be required to return any (ideally all) of the following lists:

• A list of at most 10 relevant concepts c_i,1, c_i,2, c_i,3,... from the designated terminologies and ontologies. The list should be ordered by decreasing confidence, i.e., c_i,1 should be the concept that the system considers to be the most relevant to the question q₁, c_i,2 should be the concept that the system considers to be the second most relevant etc. A single concept list will be returned per question and participating system, and the list may contain concepts from multiple designated terminologies and ontologies. The returned concept list will actually contain unique concept identifiers (obtained from the terminologies and ontologies), rather than terms (words or phrases).
• A list of at most 10 relevant articles (documents) d_i,1, d_i,2, d_i,3,... from the designated article repositories. Again, the list should be ordered by decreasing confidence, i.e., d_i,1, should be the article that the system considers to be the most relevant to the question q₁,, d_i,2, should be the article that the system considers to be the second most relevant etc. A single article list will be returned per question and participating system, and the list may contain articles from multiple designated repositories. The returned article list will actually contain unique article identifiers (obtained from the repositories).
• A list of at most 10 relevant text snippets s_i,1, s_i,2, s_i,3,... from the returned articles. Again, the list should be ordered by decreasing confidence. A single snippet list will be returned per question and participating system, and the list may contain any number (or no) snippets from any of the returned articles d_i,1, d_i,2, d_i,3,... Each snippet will be represented by the unique identifier of the article it comes from, the identifier of the section the snippet starts in, the offset of the first character of the snippet in the section the snippet starts in, the identifier of the section the snippet ends in, and the offset of the last character of the snippet in the section the snippet ends in. The snippets themselves will also have to be returned (as strings).
• A list of at most 10 relevant RDF triples t_i,1, t_i,2, t_i,3,... from the designated ontologies. Again, the list should be ordered by decreasing confidence. A single triple list will be returned per question and participating system, and the list may contain any number (or no) triples from multiple designated ontologies.

In Phase B, the participants will be provided with the same questions q₁, q₂,...,q_n as in Phase A, but this time they will also be given gold (correct) lists of articles and snippets. The "gold" lists will contain articles and snippets identified by biomedical experts as relevant and providing enough information to answer the questions. For each question, each participating system may return an ideal answer, i.e., a paragraph-sized summary of relevant information. In the case of yes/no, factoid, and list questions, the systems may also return exact answers; for summary questions, no exact answers will be returned. The participants will be told the type of each question. A participating system may return only "exact" answers, or only "ideal" answers, or (ideally) both "exact" and "ideal" answers.

Exact Answers

• For each yes/no question, the exact answer of each participating system will have to be either "yes" or "no".
• For each factoid question, each participating system will have to return a list* of up to 5 entity names (e.g., up to 5 names of drugs), numbers, or similar short expressions, ordered by decreasing confidence.
• For each list question, each participating system will have to return a single list* of entity names, numbers, or similar short expressions, jointly taken to constitute a single answer (e.g., the most common symptoms of a disease). The returned list will have to contain no more than 100 entries of no more than 100 characters each.
• No exact answers will be returned for summary questions.

* Please consult section JSON format of the datasets for specific details regarding the needed submission format.

Ideal Answers

For each question (yes/no, factoid, list, summary), each participating system of Phase B may also return an ideal answer, i.e., a single paragraph-sized text ideally summarizing the most relevant information from concepts, articles, snippets, and triples retrieved in Phase A. Each returned "ideal" answer is intended to approximate a short text that a biomedical expert would write to answer the corresponding question (e.g., including prominent supportive information), whereas the "exact" answers are only "yes"/"no" responses, entity names or similar short expressions, or lists of entity names and similar short expressions; and there are no "exact" answers in the case of summary questions. The maximum allowed length of each "ideal" answer is 200 words.

Attention:

• In the exceptional case that strings submitted as ideal answers are not actual answers to the given question (e.g. submitting an empty string or a default message like “no answer”) this answer takes manual score zero for all four properties considered.
• Since BioASQ5, no synonyms should be submitted by participants for exact answers of factoid and list questions. This change does not affect the submissions format (see JSON example below).
• The gold lists that will be provided to the participants of Phase B will contain the relevant articles and snippets that the biomedical experts who prepared the questions managed to identify.
• There will be enough information in the provided gold lists (all two lists together, e.g., not necessarily in any single list) to find the "exact" answer and to formulate an "ideal" answer. However, there may be (and most probably, will be) additional correct (relevant) concepts, articles, snippets, and triples, which will not be included in the provided gold lists.
• When producing the "exact" and "ideal" answers in Phase B, the participating systems are allowed to use both the provided gold lists and any other resource (e.g., the concepts, articles, snippets, and triples they retrieved in Phase A, in addition to the gold provided lists).
• Before announcing the official results of Phase B, biomedical experts will examine the lists of concepts, articles, snippets, and triples returned by the participating systems, as well as the "exact" and "ideal" answers of the participating systems, in order to enhance the gold lists of concepts, articles, snippets, triples, and the gold "exact" and "ideal" answers that will be used as references for evaluation purposes, as discussed here .

A development ("dry-run") dataset for Phases A and B of Task 9b can be downloaded from here . The development dataset consists of 3742 questions, along with their gold concepts, articles, snippets, RDF triples, "exact" answers, and "ideal" answers in JSON format, explained below.
Attention:

• Only registered users can download the development dataset.
• The participants are allowed to use any resources they wish to train their systems.

The test dataset will be released in five batches, each containing approximately 100 questions. For each batch, first only the questions of the batch will be released, and the participants will have to submit their answers for Phase A (concepts, articles, snippets, RDF triples) within 24 hours; then the gold articles and snippets for the questions of the batch will also be provided, and the participants will again have 24 hours to submit their answers for Phase B ("exact" and "ideal" answers). The batches will be released according to the following schedule:

• Wednesday March 10, 2021, 10:00 GMT: Questions of batch 1 released. Phase A answers (concepts, articles, snippets, RDF triples) of batch 1 due within 24 hours.
• Thursday March 11, 2021, 11:00 GMT: Gold articles and snippets of batch 1 released. Phase B answers (“exact” and “ideal”) of batch 1 due within 24 hours.
• Wednesday March 24, 2021, 10:00 GMT: Questions of batch 2 released. Phase A answers of batch 2 due within 24 hours.
• Thursday March 25, 2021, 11:00 GMT: Gold articles and snippets of batch 2 released. Phase B answers of batch 2 due within 24 hours.
• Wednesday April 07, 2021, 10:00 GMT: Questions of batch 3 released. Phase A answers of batch 3 due within 24 hours.
• Thursday April 08, 2021, 11:00 GMT: Gold articles and snippets batch 3 released. Phase B answers of batch 3 due within 24 hours.
• Wednesday April 21, 2021, 10:00 GMT: Questions of batch 4 released. Phase A answers of batch 4 due within 24 hours.
• Thursday April 22, 2021, 11:00 GMT: Gold articles and snippets of batch 4 released. Phase B answers of batch 4 due within 24 hours.
• Wednesday May 05, 2021, 10:00 GMT: Questions of batch 5 released. Phase A answers of batch 5 due within 24 hours.
• Thursday May 06, 2021, 11:00 GMT: Gold articles and snippets of batch 5 released. Phase B answers of batch 5 due within 24 hours.

The evaluation measures that will be used in Phase A and Phase B are presented here .

In Phase A, relevant concepts and concept identifiers are to be retrieved from the following terminologies and ontologies:

• the Medical Subject Headings (MeSH)
• the Gene Ontology (GO)
• the Universal Protein Resource (UniProt, SwissProt subset)
• the Joint Chemical Dictionary (Jochem)
• the Disease Ontology (DO)

Relevant articles are to be retrieved from PubMed Annual Baseline Repository for 2021. Their unique identifiers are their URLs in PubMed (e.g., "http://www.ncbi.nlm.nih.gov/pubmed/23687640").

The relevant snippets will have to be parts of relevant articles.

The relevant RDF triples will have to come from the Linked Life Data platform, provided by the Linked Life Data project. The LinkedLifeData platform is a data warehouse that syndicates large volumes of heterogeneous biomedical knowledge in a common data model. The platform uses an extension of the RDF model that is able to track the provenance of each individual fact in the repository and thus update the information. It contains currently more than 8 billion statements, with almost 2 billion entities involved. The statements are extracted from 26 biomedical resources, such as PubMed, UMLS, DrugBank, Diseasome, and Gene Ontology.

Instructions on how to download the designated resources and/or how to use tools that the organizers provide to search the designated resources are available here.

The development and test datasets use the following JSON format:

{"questions":[
	{
		"type":"factoid",
		"body":"Is Rheumatoid Arthritis more common in men or women?",
		"id":"5118dd1305c10fae750000010",
		"ideal_answer": "Disease patterns in RA vary between the sexes; the condition is
			more commonly seen in women, who exhibit a more aggressive disease and a poorer
			long-term outcome.",
		"exact_answer": [
			["Women"]
		],
		"documents": [
			"http://www.ncbi.nlm.nih.gov/pubmed/12723987"
			, ...
		],
		"snippets":[
			{
				"document": "http://www.ncbi.nlm.nih.gov/pubmed/22853635",
				"text": "The expression and clinical course of RA are influenced by gender.
					In developed countries the prevalence of RA is 0,5 to 1.0%, with a
					male:female ratio of 1:3.",
				"offsetInBeginSection": 559,
				"offsetInEndSection": 718,
				"beginSection": "sections.0"
				"endSection": "sections.0"
			}, ...
		],
		"concepts":[
			"http://www.diseaseontology.org/api/metadata/DOID:7148"
			, ...
		],
		"triples": [
			{
				"s": "http://linkedlifedata.com/resource/umls/id/C2827401",
				"p": "http://www.w3.org/2008/05/skos-xl#prefLabel",
				"o": "http://linkedlifedata.com/resource/umls/label/A17680439"
			},...
		]
	}, ...
]}

In the case of factoid questions, the "exact_answer" field is a list of lists. Each of the inner list (up to 5 inner lists are allowed) should contain the name of the entity (or number, or other similar short expression) sought by the question; Since BioASQ5, no multiple names (synonyms) should be submitted for any entity, therefore each inner list should only contain one element. If the List contains more than one elements, only the first element will be taken into account for evaluation. Note that in the training data the field is a *simple* list containing the entity (and its synonyms if they were identified) that answers the question.

In the case of list questions, the "exact_answer" field is a list of lists. Each element of the outmost list is a list corresponding to one of the entities (or numbers, or other similar short expressions) seeked by the question. Since BioASQ5, no multiple names (synonyms) should be submitted for any entity, therefore each inner list should only contain one element. If the List contains more than one elements, only the first element will be taken into account for evaluation.
If any of the seeked entities has multiple names (synonyms), the corresponding inner list should only contain *one* of them. In the following example the exact golden answer to the list question contains three entities and the second entity has two names, i.e, "influenza" and "grippe":

"exact_answer": [["pneumonia"], ["influenza", "grippe"], ["bronchitis"]]

However, the submitted answer by the participants should be one of the following:

"exact_answer": [["pneumonia"], ["influenza"], ["bronchitis"]]

or "exact_answer": [["pneumonia"], ["grippe"], ["bronchitis"]]

Since, golden answer contains both synonyms, both answers are equivalent for evaluation.

Also note that "ideal_answer" in training data is a list, so that all golden ideal answers are available for training, if many. However, each system is expected to submit only one ideal answer per question as a string, as described in JSON format above.

Full text documents returned from PubMedCentral [Deprecated] As from BioASQ 3 only the abstracts and the titles of the articles are used. Thus, downloading the fulltext articles separately is not necessary. The required information concerning the articles can be obtained by the webservices provided by the BioASQ team. In those abstract-only documents that are returned from PubMed, where only the abstract and titles are available, the "beginSection" and "endSection" are both "abstract" (if the snippet comes from the abstract) or they are both "title" (if the snippet comes from the title). A snippet may not come from both the title and the abstract.

When submitting results, participants will use the same JSON format. The "text" field in the elements of the snippet list is also required.

Each participant (e.g., organization, research group) will be allowed to participate in Task b with a maximum of 5 systems. To declare the systems you are participating with, log in, go to "Edit Profile", and then click on the "Add System" button. The system name you will fill in the form will be the identifier of your system and it will be used in the "Results" section.
Attention: Trying to upload results without selecting a system will cause an error and the results will not be saved.

The word2vec tool (https://code.google.com/p/word2vec/) processes a large text corpus and maps the words of the corpus to vectors of a continuous space. The word vectors can then be used, for example, to estimate the relatedness of two words or to perform query expansion. We applied word2vec to a corpus of 10,876,004 English abstracts of biomedical articles from PubMed. The resulting vectors of 1,701,632 distinct words (types) are now publicly available here. File size: 1.3GB (compressed), 3.5GB (uncompressed). More information here.