In search of controlled evidence for health care quality improvement

E Andrew Balas, Marcia G. Stockham, Joyce A. Mitchell, Mary Ellen Sievert, Bernard G. Ewigman, Suzanne Austin Boren

Research output: Contribution to journalArticle

13 Citations (Scopus)

Abstract

The purpose of this study was to measure the efficiency of simple searches in retrieving controlled evidence about specific primary health care quality improvement interventions and their effects. Searches were conducted to retrieve evidence on seven interventions and seven effect variables. Specific words and the closest Medical Subject Headings (MeSH) recommended by professional librarians were used to search the MEDLINE database. Searches were restricted to the MeSH publication type 'randomized controlled trial.' Two reviewers independently judged retrieved citations for relevancy to the selected interventions and effects. In selecting MeSH terms, the average agreement among librarians was 64.3% (± 26.1) for interventions and 57.1% (± 19.9) for effects. Analysis of the 755 retrieved reports showed that MeSH term searches had an overall recall rate of 58% while the same rate for textword searches was significantly lower (11%o, p < .001). The difference in overall precision rates was nonsignificant (26% versus 33%, p = .15). In the group of MeSH searches, overall precision and recall was significantly lower for effects than for interventions (12% versus 52%, p < .001 and 41% versus 69%, p < .001). Two textwords appeared in more than 25% of the benchmark collection: reminder (25.7%) and cost (25.0%). The results of this study indicate that information needs for health care quality improvement cannot be met by simple literature searches. Certain MeSH terms and combinations of textwords yield moderately efficient recall and precision in literature searches for health care quality improvement. Clinicians and physician executives gaining direct access to bibliographic database could probably be better served by structured indexing of critical aspects of randomized controlled clinical trials: design, sample, interventions, and effects.

Original languageEnglish (US)
Pages (from-to)21-32
Number of pages12
JournalJournal of Medical Systems
Volume21
Issue number1
DOIs
StatePublished - Jun 12 1997
Externally publishedYes

Fingerprint

Medical Subject Headings
Quality Improvement
Health care
Delivery of Health Care
Librarians
Randomized Controlled Trials
Bibliographic Databases
Physician Executives
Benchmarking
MEDLINE
Publications
Costs
Primary Health Care
Databases
Costs and Cost Analysis

Keywords

  • health care
  • Health Services Research
  • MEDLINE
  • quality assurance
  • randomized controlled trials

ASJC Scopus subject areas

  • Medicine (miscellaneous)
  • Information Systems
  • Health Informatics
  • Health Information Management

Cite this

Balas, E. A., Stockham, M. G., Mitchell, J. A., Sievert, M. E., Ewigman, B. G., & Boren, S. A. (1997). In search of controlled evidence for health care quality improvement. Journal of Medical Systems, 21(1), 21-32. https://doi.org/10.1023/A:1022887224126

In search of controlled evidence for health care quality improvement. / Balas, E Andrew; Stockham, Marcia G.; Mitchell, Joyce A.; Sievert, Mary Ellen; Ewigman, Bernard G.; Boren, Suzanne Austin.

In: Journal of Medical Systems, Vol. 21, No. 1, 12.06.1997, p. 21-32.

Research output: Contribution to journalArticle

Balas, EA, Stockham, MG, Mitchell, JA, Sievert, ME, Ewigman, BG & Boren, SA 1997, 'In search of controlled evidence for health care quality improvement', Journal of Medical Systems, vol. 21, no. 1, pp. 21-32. https://doi.org/10.1023/A:1022887224126
Balas, E Andrew ; Stockham, Marcia G. ; Mitchell, Joyce A. ; Sievert, Mary Ellen ; Ewigman, Bernard G. ; Boren, Suzanne Austin. / In search of controlled evidence for health care quality improvement. In: Journal of Medical Systems. 1997 ; Vol. 21, No. 1. pp. 21-32.
@article{3cd6f02223ee4fa0afbc2b061926a7e6,
title = "In search of controlled evidence for health care quality improvement",
abstract = "The purpose of this study was to measure the efficiency of simple searches in retrieving controlled evidence about specific primary health care quality improvement interventions and their effects. Searches were conducted to retrieve evidence on seven interventions and seven effect variables. Specific words and the closest Medical Subject Headings (MeSH) recommended by professional librarians were used to search the MEDLINE database. Searches were restricted to the MeSH publication type 'randomized controlled trial.' Two reviewers independently judged retrieved citations for relevancy to the selected interventions and effects. In selecting MeSH terms, the average agreement among librarians was 64.3{\%} (± 26.1) for interventions and 57.1{\%} (± 19.9) for effects. Analysis of the 755 retrieved reports showed that MeSH term searches had an overall recall rate of 58{\%} while the same rate for textword searches was significantly lower (11{\%}o, p < .001). The difference in overall precision rates was nonsignificant (26{\%} versus 33{\%}, p = .15). In the group of MeSH searches, overall precision and recall was significantly lower for effects than for interventions (12{\%} versus 52{\%}, p < .001 and 41{\%} versus 69{\%}, p < .001). Two textwords appeared in more than 25{\%} of the benchmark collection: reminder (25.7{\%}) and cost (25.0{\%}). The results of this study indicate that information needs for health care quality improvement cannot be met by simple literature searches. Certain MeSH terms and combinations of textwords yield moderately efficient recall and precision in literature searches for health care quality improvement. Clinicians and physician executives gaining direct access to bibliographic database could probably be better served by structured indexing of critical aspects of randomized controlled clinical trials: design, sample, interventions, and effects.",
keywords = "health care, Health Services Research, MEDLINE, quality assurance, randomized controlled trials",
author = "Balas, {E Andrew} and Stockham, {Marcia G.} and Mitchell, {Joyce A.} and Sievert, {Mary Ellen} and Ewigman, {Bernard G.} and Boren, {Suzanne Austin}",
year = "1997",
month = "6",
day = "12",
doi = "10.1023/A:1022887224126",
language = "English (US)",
volume = "21",
pages = "21--32",
journal = "Journal of Medical Systems",
issn = "0148-5598",
publisher = "Springer New York",
number = "1",

}

TY - JOUR

T1 - In search of controlled evidence for health care quality improvement

AU - Balas, E Andrew

AU - Stockham, Marcia G.

AU - Mitchell, Joyce A.

AU - Sievert, Mary Ellen

AU - Ewigman, Bernard G.

AU - Boren, Suzanne Austin

PY - 1997/6/12

Y1 - 1997/6/12

N2 - The purpose of this study was to measure the efficiency of simple searches in retrieving controlled evidence about specific primary health care quality improvement interventions and their effects. Searches were conducted to retrieve evidence on seven interventions and seven effect variables. Specific words and the closest Medical Subject Headings (MeSH) recommended by professional librarians were used to search the MEDLINE database. Searches were restricted to the MeSH publication type 'randomized controlled trial.' Two reviewers independently judged retrieved citations for relevancy to the selected interventions and effects. In selecting MeSH terms, the average agreement among librarians was 64.3% (± 26.1) for interventions and 57.1% (± 19.9) for effects. Analysis of the 755 retrieved reports showed that MeSH term searches had an overall recall rate of 58% while the same rate for textword searches was significantly lower (11%o, p < .001). The difference in overall precision rates was nonsignificant (26% versus 33%, p = .15). In the group of MeSH searches, overall precision and recall was significantly lower for effects than for interventions (12% versus 52%, p < .001 and 41% versus 69%, p < .001). Two textwords appeared in more than 25% of the benchmark collection: reminder (25.7%) and cost (25.0%). The results of this study indicate that information needs for health care quality improvement cannot be met by simple literature searches. Certain MeSH terms and combinations of textwords yield moderately efficient recall and precision in literature searches for health care quality improvement. Clinicians and physician executives gaining direct access to bibliographic database could probably be better served by structured indexing of critical aspects of randomized controlled clinical trials: design, sample, interventions, and effects.

AB - The purpose of this study was to measure the efficiency of simple searches in retrieving controlled evidence about specific primary health care quality improvement interventions and their effects. Searches were conducted to retrieve evidence on seven interventions and seven effect variables. Specific words and the closest Medical Subject Headings (MeSH) recommended by professional librarians were used to search the MEDLINE database. Searches were restricted to the MeSH publication type 'randomized controlled trial.' Two reviewers independently judged retrieved citations for relevancy to the selected interventions and effects. In selecting MeSH terms, the average agreement among librarians was 64.3% (± 26.1) for interventions and 57.1% (± 19.9) for effects. Analysis of the 755 retrieved reports showed that MeSH term searches had an overall recall rate of 58% while the same rate for textword searches was significantly lower (11%o, p < .001). The difference in overall precision rates was nonsignificant (26% versus 33%, p = .15). In the group of MeSH searches, overall precision and recall was significantly lower for effects than for interventions (12% versus 52%, p < .001 and 41% versus 69%, p < .001). Two textwords appeared in more than 25% of the benchmark collection: reminder (25.7%) and cost (25.0%). The results of this study indicate that information needs for health care quality improvement cannot be met by simple literature searches. Certain MeSH terms and combinations of textwords yield moderately efficient recall and precision in literature searches for health care quality improvement. Clinicians and physician executives gaining direct access to bibliographic database could probably be better served by structured indexing of critical aspects of randomized controlled clinical trials: design, sample, interventions, and effects.

KW - health care

KW - Health Services Research

KW - MEDLINE

KW - quality assurance

KW - randomized controlled trials

UR - http://www.scopus.com/inward/record.url?scp=0030988838&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0030988838&partnerID=8YFLogxK

U2 - 10.1023/A:1022887224126

DO - 10.1023/A:1022887224126

M3 - Article

VL - 21

SP - 21

EP - 32

JO - Journal of Medical Systems

JF - Journal of Medical Systems

SN - 0148-5598

IS - 1

ER -