Automating the process of critical appraisal and assessing the strength of evidence with information extraction technology

Jou Wei Lin, Chia Hsuin Chang, Ming Wei Lin, Mark H. Ebell, Jung Hsien Chiang

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Background Critical appraisal, one of the most crucial steps in the practice of evidence-based medicine, is expertise-dependent and time-consuming. The objective of this study was to develop and evaluate an automated text-mining system that could determine the evidence level provided by a medical article. Methods A text processor was designed and built to interpret the abstracts of medical literature. The system extracted information about: (1) the impact factor of the journal; (2) study design; (3) human subject involvement; (4) number of subjects; (5) P-value; and (6) confidence intervals. We used a classification tree algorithm (C4.5) to create a decision tree using supervised classification. Each article was categorized into evidence level A, B or C, and the output was compared to that determined by domain experts (the reference standard). Results We used a corpus of 3180 cardiovascular disease original research articles, of which 1108 were previously assigned evidence level A, 1705 level B and 367 level C by domain experts. The abstracts were analysed by our automated system and an evidence level was assigned. The algorithm accurately classified 85% of the articles. The agreement between computer and domain experts was substantial (κ-value: 0.78). Cross-validation showed consistent results across repeated tests. Conclusion The automated engine accurately classified the evidence level. Misclassification might have resulted from incomplete information retrieval and inaccurate data extraction. Further efforts will focus on assessing relevance and using additional study design features to refine evidence level classification.

Original languageEnglish (US)
Pages (from-to)832-838
Number of pages7
JournalJournal of Evaluation in Clinical Practice
Volume17
Issue number4
DOIs
StatePublished - Aug 1 2011
Externally publishedYes

Fingerprint

Information Storage and Retrieval
Technology
Journal Impact Factor
Decision Trees
Data Mining
Evidence-Based Medicine
Information Systems
Cardiovascular Diseases
Confidence Intervals
Research

Keywords

  • abstracting and indexing as topic
  • evaluation studies as topic
  • evidence-based medicine
  • information storage and retrieval

ASJC Scopus subject areas

  • Health Policy
  • Public Health, Environmental and Occupational Health

Cite this

Automating the process of critical appraisal and assessing the strength of evidence with information extraction technology. / Lin, Jou Wei; Chang, Chia Hsuin; Lin, Ming Wei; Ebell, Mark H.; Chiang, Jung Hsien.

In: Journal of Evaluation in Clinical Practice, Vol. 17, No. 4, 01.08.2011, p. 832-838.

Research output: Contribution to journalArticle

Lin, Jou Wei ; Chang, Chia Hsuin ; Lin, Ming Wei ; Ebell, Mark H. ; Chiang, Jung Hsien. / Automating the process of critical appraisal and assessing the strength of evidence with information extraction technology. In: Journal of Evaluation in Clinical Practice. 2011 ; Vol. 17, No. 4. pp. 832-838.
@article{7f5a1417b595462b8de9dc009bc387b6,
title = "Automating the process of critical appraisal and assessing the strength of evidence with information extraction technology",
abstract = "Background Critical appraisal, one of the most crucial steps in the practice of evidence-based medicine, is expertise-dependent and time-consuming. The objective of this study was to develop and evaluate an automated text-mining system that could determine the evidence level provided by a medical article. Methods A text processor was designed and built to interpret the abstracts of medical literature. The system extracted information about: (1) the impact factor of the journal; (2) study design; (3) human subject involvement; (4) number of subjects; (5) P-value; and (6) confidence intervals. We used a classification tree algorithm (C4.5) to create a decision tree using supervised classification. Each article was categorized into evidence level A, B or C, and the output was compared to that determined by domain experts (the reference standard). Results We used a corpus of 3180 cardiovascular disease original research articles, of which 1108 were previously assigned evidence level A, 1705 level B and 367 level C by domain experts. The abstracts were analysed by our automated system and an evidence level was assigned. The algorithm accurately classified 85{\%} of the articles. The agreement between computer and domain experts was substantial (κ-value: 0.78). Cross-validation showed consistent results across repeated tests. Conclusion The automated engine accurately classified the evidence level. Misclassification might have resulted from incomplete information retrieval and inaccurate data extraction. Further efforts will focus on assessing relevance and using additional study design features to refine evidence level classification.",
keywords = "abstracting and indexing as topic, evaluation studies as topic, evidence-based medicine, information storage and retrieval",
author = "Lin, {Jou Wei} and Chang, {Chia Hsuin} and Lin, {Ming Wei} and Ebell, {Mark H.} and Chiang, {Jung Hsien}",
year = "2011",
month = "8",
day = "1",
doi = "10.1111/j.1365-2753.2011.01712.x",
language = "English (US)",
volume = "17",
pages = "832--838",
journal = "Journal of Evaluation in Clinical Practice",
issn = "1356-1294",
publisher = "Wiley-Blackwell",
number = "4",

}

TY - JOUR

T1 - Automating the process of critical appraisal and assessing the strength of evidence with information extraction technology

AU - Lin, Jou Wei

AU - Chang, Chia Hsuin

AU - Lin, Ming Wei

AU - Ebell, Mark H.

AU - Chiang, Jung Hsien

PY - 2011/8/1

Y1 - 2011/8/1

N2 - Background Critical appraisal, one of the most crucial steps in the practice of evidence-based medicine, is expertise-dependent and time-consuming. The objective of this study was to develop and evaluate an automated text-mining system that could determine the evidence level provided by a medical article. Methods A text processor was designed and built to interpret the abstracts of medical literature. The system extracted information about: (1) the impact factor of the journal; (2) study design; (3) human subject involvement; (4) number of subjects; (5) P-value; and (6) confidence intervals. We used a classification tree algorithm (C4.5) to create a decision tree using supervised classification. Each article was categorized into evidence level A, B or C, and the output was compared to that determined by domain experts (the reference standard). Results We used a corpus of 3180 cardiovascular disease original research articles, of which 1108 were previously assigned evidence level A, 1705 level B and 367 level C by domain experts. The abstracts were analysed by our automated system and an evidence level was assigned. The algorithm accurately classified 85% of the articles. The agreement between computer and domain experts was substantial (κ-value: 0.78). Cross-validation showed consistent results across repeated tests. Conclusion The automated engine accurately classified the evidence level. Misclassification might have resulted from incomplete information retrieval and inaccurate data extraction. Further efforts will focus on assessing relevance and using additional study design features to refine evidence level classification.

AB - Background Critical appraisal, one of the most crucial steps in the practice of evidence-based medicine, is expertise-dependent and time-consuming. The objective of this study was to develop and evaluate an automated text-mining system that could determine the evidence level provided by a medical article. Methods A text processor was designed and built to interpret the abstracts of medical literature. The system extracted information about: (1) the impact factor of the journal; (2) study design; (3) human subject involvement; (4) number of subjects; (5) P-value; and (6) confidence intervals. We used a classification tree algorithm (C4.5) to create a decision tree using supervised classification. Each article was categorized into evidence level A, B or C, and the output was compared to that determined by domain experts (the reference standard). Results We used a corpus of 3180 cardiovascular disease original research articles, of which 1108 were previously assigned evidence level A, 1705 level B and 367 level C by domain experts. The abstracts were analysed by our automated system and an evidence level was assigned. The algorithm accurately classified 85% of the articles. The agreement between computer and domain experts was substantial (κ-value: 0.78). Cross-validation showed consistent results across repeated tests. Conclusion The automated engine accurately classified the evidence level. Misclassification might have resulted from incomplete information retrieval and inaccurate data extraction. Further efforts will focus on assessing relevance and using additional study design features to refine evidence level classification.

KW - abstracting and indexing as topic

KW - evaluation studies as topic

KW - evidence-based medicine

KW - information storage and retrieval

UR - http://www.scopus.com/inward/record.url?scp=79961023449&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79961023449&partnerID=8YFLogxK

U2 - 10.1111/j.1365-2753.2011.01712.x

DO - 10.1111/j.1365-2753.2011.01712.x

M3 - Article

VL - 17

SP - 832

EP - 838

JO - Journal of Evaluation in Clinical Practice

JF - Journal of Evaluation in Clinical Practice

SN - 1356-1294

IS - 4

ER -