Priorities Among Effective Clinical Preventive Services Results of a Systematic Review and Analysis
BMJ. 2005 Apr ii; 330(7494): 765.
Improving clinical practice using clinical conclusion back up systems: a systematic review of trials to identify features disquisitional to success
Kensaku Kawamoto
1 Partitioning of Clinical Informatics, Section of Community and Family Medicine, Box 2914, Duke University Medical Center, Durham, NC 27710, USA
Caitlin A Houlihan
one Division of Clinical Informatics, Department of Community and Family Medicine, Box 2914, Duke Academy Medical Center, Durham, NC 27710, USA
E Andrew Balas
ii College of Health Sciences, Former Rule Academy, Norfolk, VA 23529, USA
David F Lobach
one Partition of Clinical Information science, Department of Community and Family Medicine, Box 2914, Duke University Medical Middle, Durham, NC 27710, USA
Abstruse
Objective To identify features of clinical determination back up systems critical for improving clinical practice.
Design Systematic review of randomised controlled trials.
Data sources Literature searches via Medline, CINAHL, and the Cochrane Controlled Trials Register up to 2003; and searches of reference lists of included studies and relevant reviews.
Written report option Studies had to evaluate the ability of decision back up systems to improve clinical practice.
Data extraction Studies were assessed for statistically and clinically pregnant improvement in clinical practice and for the presence of 15 determination support organization features whose importance had been repeatedly suggested in the literature.
Results Seventy studies were included. Decision support systems significantly improved clinical practice in 68% of trials. Univariate analyses revealed that, for five of the system features, interventions possessing the characteristic were significantly more probable to improve clinical practise than interventions defective the feature. Multiple logistic regression analysis identified four features as contained predictors of improved clinical practice: automatic provision of decision support as part of clinician workflow (P < 0.00001), provision of recommendations rather than just assessments (P = 0.0187), provision of conclusion support at the time and location of determination making (P = 0.0263), and reckoner based decision support (P = 0.0294). Of 32 systems possessing all four features, xxx (94%) significantly improved clinical exercise. Furthermore, directly experimental justification was establish for providing periodic operation feedback, sharing recommendations with patients, and requesting documentation of reasons for non following recommendations.
Conclusions Several features were closely correlated with determination support systems' power to improve patient intendance significantly. Clinicians and other stakeholders should implement clinical decision support systems that incorporate these features whenever viable and appropriate.
Introduction
Recent research has shown that wellness care delivered in industrialised nations often falls brusque of optimal, show based intendance. A nationwide audit assessing 439 quality indicators establish that The states adults receive but nearly half of recommended intendance,i and the US Institute of Medicine has estimated that up to 98 000 US residents die each twelvemonth as the outcome of preventable medical errors.2 Similarly a retrospective analysis at two London hospitals institute that 11% of admitted patients experienced adverse events, of which 48% were judged to be preventable and of which 8% led to death.3
To accost these deficiencies in care, healthcare organisations are increasingly turning to clinical determination support systems, which provide clinicians with patient-specific assessments or recommendations to help clinical decision making.four Examples include transmission or computer based systems that adhere care reminders to the charts of patients needing specific preventive care services and computerised physician society entry systems that provide patient-specific recommendations as part of the order entry process. Such systems accept been shown to improve prescribing practices,v - 7 reduce serious medication errors,8 ,9 enhance the delivery of preventive care services,10 ,xi and improve adherence to recommended care standards.4 ,12 Compared with other approaches to improve exercise, these systems have too generally been shown to be more effective and more likely to result in lasting improvements in clinical exercise.13 - 22
Clinical decision support systems do not always meliorate clinical practise, however. In a contempo systematic review of reckoner based systems, most (66%) significantly improved clinical practice, merely 34% did not.4 Relatively footling audio scientific prove is available to explain why systems succeed or fail.23 ,24 Although some investigators have tried to place the system features most important for improving clinical practise,12 ,25 - 34 they take typically relied on the opinion of a express number of experts, and none has combined a systematic literature search with quantitative meta-analysis. We therefore systematically reviewed the literature to place the specific features of clinical decision support systems most crucial for improving clinical exercise.
Methods
Data sources
We searched Medline (1966-2003), CINAHL (1982-2003), and the Cochrane Controlled Trials Register (2003) for relevant studies using combinations of the following search terms: decision back up systems, clinical; determination making, computer-assisted; reminder systems; feedback; guideline adherence; medical information science; advice; md's practice patterns; reminder$; feedback$; conclusion support$; and skillful system. Nosotros also systematically searched the reference lists of included studies and relevant reviews.
Inclusion and exclusion criteria
We divers a clinical determination support arrangement as any electronic or non-electronic arrangement designed to aid directly in clinical decision making, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are so presented to clinicians for consideration.4 Nosotros included both electronic and not-electronic systems because we felt the use of a figurer represented only one of many potentially important factors. Our inclusion criteria were any randomised controlled trial evaluating the ability of a clinical decision support system to improve an important clinical practice in a real clinical setting; employ of the arrangement by clinicians (physicians, doc assistants, or nurse practitioners) directly involved in patient care; and assessment of improvements in clinical do through patient outcomes or process measures. Our exclusion criteria were less than seven units of randomisation per study arm; study not in English language; mandatory compliance with decision back up system; lack of description of determination support content or of clinician interaction with system; and score of less than v points on a ten point scale assessing five potential sources of study bias.4
Study selection
Two authors independently reviewed the titles, index terms, and abstracts of the identified references and rated each paper as "potentially relevant" or "not relevant" using a screening algorithm based on written report type, report design, subjects, setting, and intervention. Ii authors then independently reviewed the full texts of the selected articles and again rated each newspaper as "potentially relevant" or "not relevant" using the screening algorithm. Finally, 2 authors independently applied the full set of inclusion and exclusion criteria to the potentially relevant studies to select the final fix of included studies. Disagreements betwixt reviewers were resolved by word, and we measured inter-rater understanding using Cohen's unweighted κ statistic.35
Information brainchild
A study may include several trial arms, and then that multiple comparisons may exist within the single report. For each relevant comparison, two reviewers independently assessed whether the clinical decision support system resulted in an improvement in clinical practise that was both statistically and clinically significant. In some cases changes in practice characterised as clinically significant by the report authors were deemed non-significant past the reviewers. Nosotros considered event size as an culling consequence measure but concluded that the use of result size would accept been misleading given the significant heterogeneity among the effect measures reported by the included studies. We also predictable that the use of effect size would accept led to the exclusion of many relevant trials, as many studies fail to report all of the statistical elements necessary to accurately reconstruct outcome sizes.
Adjacent, 2 reviewers independently determined the presence or absence of specific features of decision support systems that could potentially explain why a system succeeded or failed. To construct a set of potential explanatory features, nosotros systematically examined all relevant reviews and primary studies identified by our search strategy and recorded whatsoever factors suggested to be important for organisation effectiveness. Both technical and not-technical factors were eligible for inclusion. Also, if a cistron was designated as a barrier to effectiveness (such equally "the demand for clinician data entry limits arrangement effectiveness") nosotros treated the logically opposite concept every bit a potential success factor (such as "removing the demand for clinician information entry enhances system effectiveness"). Next, we limited our consideration to features that were identified every bit beingness potentially important by at to the lowest degree three sources, which left united states of america with 22 potential explanatory features, including general organization features, organization-clinician interaction features, communication content features, and auxiliary features (tables 1 and 2). Of these 22 features, xv could be included into our assay (table 1) considering their presence or absenteeism could be reliably abstracted from most studies, whereas the remaining 7 could not (table two).
Table 1
Descriptions of the 15 features of clinical determination back up systems (CDSS) included in statistical analyses
| Feature and sources * | Example |
|---|---|
| General system features | |
| Integration with charting or order entry system to support workflow integration25 ,26 ,36 ,37 w1 | Preventive care reminders attached to patient charts; clinician warned of raised creatinine concentration when using computerised dr. club entry arrangement to prescribe aminoglycoside for a hospitalised patient |
| Apply of a computer to generate the decision back up38 w2-w10 | Patients overdue for cervical cancer screening identified by querying a clinical database rather than by manual chart audits |
| Clinician-system interaction features | |
| Automatic provision of determination support as part of clinician workflow23 ,26 ,28 ,29 ,31 ,33 ,36 ,39 ,40 w11-w13 | Diabetes care recommendations printed on paper forms and attached to relevant patient charts by dispensary support staff, and then that clinicians practise non need to seek out the communication of the CDSS |
| No need for additional clinician information entry5 ,23 ,25 ,28 ,33 ,36 ,41-43 w12 | Electronic or manual chart audits are conducted to obtain all information necessary for determining whether a child needs immunisations |
| Request documentation of the reason for non following CDSS recommendations5 ,43 w12 w14 w15 | If a clinician does non provide influenza vaccine recommended by the CDSS, the clinician is asked to justify the decision with a reason such equally "The patient refused" or "I disagree with the recommendation" |
| Provision of conclusion support at time and location of decision making5 ,23 ,33 ,twoscore ,43-46 w1-w3 w5 w11-w13 w16-w19 | Preventive care recommendations provided as nautical chart reminders during an encounter, rather than as monthly reports listing all the patients in need of services |
| Recommendations executed past nothing agreementw3 w12 w14 w20 | Computerised physician order entry organisation recommends superlative and trough drug concentrations in response to an social club for aminoglycoside, and the clinician simply clicks "Okay" to lodge the recommended tests |
| Communication content features | |
| Provision of a recommendation, not but an assessment43 ,47 w21 | Arrangement recommends that the clinician prescribes antidepressants for a patient rather than simply identifying patient as being depressed |
| Promotion of action rather than inaction33 w11 w17 | System recommends an alternate view for an abdominal radiograph that is unlikely to be of diagnostic value, rather than recommending that the order for the radiograph be cancelled |
| Justification of decision support via provision of reasoning25 ,27 w14 w17 | Recommendation for diabetic foot examination justified by noting appointment of last test and recommended frequency of testing |
| Justification of decision back up via provision of inquiry evidence27 ,29 w17 w22 | Recommendation for diabetic foot test justified by providing information from randomised controlled trials that show benefits of conducting the exam |
| Auxiliary features | |
| Local user involvement in development procedure26 ,27 ,30 ,31 ,43-45 ,48 ,49 w17 w19 w23 | System design finalised after testing prototypes with representatives from targeted clinician user group |
| Provision of decision support results to patients likewise equally providerseleven ,fifty w4 w24-w26 | As well as providing chart reminders for clinicians, CDSS generates postcards that are sent to patients to inform them of overdue preventive intendance services |
| CDSS accompanied by periodic performance feedback13 ,29 ,49 w17 w27 w28 | Clinicians are sent emails every two weeks that summarise their compliance with CDSS recommendations for the care of patients with diabetes |
| CDSS accompanied by conventional educational activity51 w7 w17 w27 w29 | Deployment of a CDSS aimed at reducing unnecessary ordering of abdominal radiographs is accompanied by a "grand rounds" presentation on appropriate indications for ordering such radiographs |
Table 2
The vii potential explanatory features of clinical decision support systems (CDSS) that could not exist included in the statistical analyses
| Feature and sources * | Reason why characteristic could not exist abstracted and analysed |
|---|---|
| General arrangement features | |
| Organization is fastxxx ,31 ,33 ,45 | Nearly studies did not written report formal or informal assessments of system speed |
| Clinician-organisation interaction features | |
| Saves clinicians time or requires minimal time to use25 ,26 ,28 ,36 ,39-41 w3 | Nigh studies did not comport formal or informal evaluations of the time costs and savings associated with system utilise |
| Articulate and intuitive user interface5 ,23 ,25 ,26 ,30 ,31 ,33 ,42 ,45 ,52 w12 with prominent display of advice33 w1 w15 w30 | Nearly studies did non describe user interface with sufficient detail (such as via screenshots) to assess these aspects of user interface |
| Advice content features | |
| Assessments and recommendations are authentic26 ,30 ,31 ,43 w12 w17 | Most studies did non study the false positive or fake negative mistake rates associated with CDSS letters |
| Auxiliary features | |
| System developed through iterative refinement procedure30 ,31 ,33 ,43 ,45 ,53 | Well-nigh studies did not study the caste to which the system had undergone iterative refinement before evaluation |
| Alignment of decision back up objectives with organisational priorities30-32 ,43 ,49 w15 w31 and with the behavior23 ,25 ,27 ,54 w3 w12 w30 w32 and financial interests27 ,41 w6 w7 w17 w33 of individual clinicians | Nearly studies did not appraise whether CDSS supported organisational priorities (such as patient safety, toll containment) and were therefore amend positioned to receive institutional support, whether clinicians agreed with the practices encouraged by CDSS (such as increased use of β blockers for patients with congestive heart failure), or whether clinicians had any fiscal incentives to follow or reject CDSS communication |
| Active interest of local stance leaders30-32 ,43 | Unable to determine reliably, as many investigators were probably local opinion leaders themselves, but few identified themselves as such |
Data synthesis
We used 3 methods to identify clinical decision back up system features important for improving clinical do.
Univariate analyses—For each of the fifteen selected features we individually determined whether interventions possessing the feature were significantly more than likely to succeed (result in a statistically and clinically significant improvement in clinical practice) than interventions defective the feature. We used StatXact55 to summate 95% conviction intervals for individual success rates56 and for differences in success rates.57
Multiple logistic regression analyses—For these analyses, the presence or absenteeism of a statistically and clinically significant improvement in clinical practice constituted the binary outcome variable, and the presence or absence of specific determination support arrangement features constituted binary explanatory variables. Nosotros included merely cases in which the clinical decision back up arrangement was compared against a true control group. For the principal meta-regression analysis, we pooled the results from all included studies, so as to maximise the ability of the analysis while decreasing the risk of false positive findings from over-plumbing fixtures of the model.58 We also conducted separate secondary regression analyses for computer based systems and for non-electronic systems. For all analyses, nosotros included one indicator for the decision support subject matter (acute intendance v non-astute care) and ii indicators for the report setting (academic v not-academic, outpatient v inpatient) to appraise the role of potential misreckoning factors related to the study environment. With the 15 system features and the iii environmental factors constituting the potential explanatory variables, we conducted logistic regression analyses using LogXact-5.59 Independent predictor variables were included into the final models using forward choice and a significance level of 0.05.
Direct experimental evidence—Nosotros systematically identified studies in which the effectiveness of a given decision support arrangement was straight compared with the effectiveness of the aforementioned system with additional features. Nosotros considered a feature to have direct experimental evidence supporting its importance if its addition resulted in a statistically and clinically significant comeback in clinical practise.
Results
Description of studies
Of 10 688 potentially relevant articles screened, 88 papers describing lxx studies met all our inclusion and exclusion criteria (effigy).w1-w88 Inter-rater agreements for report option and data abstraction were satisfactory (table 3). The lxx studies contained 82 relevant comparisons, of which 71 compared a clinical decision support system with a control group (control-system comparisons) and 11 directly compared a system with the same system plus actress features (system-system comparisons). We used the control-organization comparisons to identify organisation features statistically associated with successful outcomes and the system-system comparisons to identify features with direct experimental evidence of their importance.
Selection procedure of trials of clinical decision support systems (CDSS) for review
Tabular array 3
Inter-rater understanding for study selection and information abstraction in review of trials of clinical decision support systems (CDSS)
| Conclusion evaluated | Raw agreement (%) | Agreement beyond gamble (Cohen's κ) (%) |
|---|---|---|
| Study selection | ||
| Study is potentially relevant based on examination of abstract and use of screening algorithm | 99.8 | 96.4 |
| Report is potentially relevant based on examination of total text and use of screening algorithm | 94.9 | 89.5 |
| Written report meets all inclusion and exclusion criteria based on examination of total text | 84.half dozen | 66.3 |
| Data abstraction | ||
| Employ of CDSS resulted in statistically and clinically significant improvement in clinical practice | 97.2 | 93.6 |
| CDSS intervention incorporated the potential success factor of interest (mean agreement level for the 15 features) | 97.viii | 90.five |
Table 4 describes the characteristics of the lxx included studies. Betwixt them, about 6000 clinicians acted every bit study subjects while caring for near 130 000 patients. The commonest types of decision back up system were computer based systems that provided patient-specific advice on printed run into forms or on printouts attached to charts (34%),w2 w4 w6 w8-w10 w12 w14 w19 w22-w28 w32 w34-w49 non-electronic systems that attached patient-specific advice to advisable charts (26%),w1 w29-w31 w50-w66 and systems that provided conclusion support within computerised physician guild entry systems (xvi%).w3 w7 w11 w15-w17 w20 w67-w71
Table 4
Characteristics of the lxx studies of clinical conclusion support systems (CDSS) included in review
| Characteristic | Frequency (%) |
|---|---|
| Setting: | |
| Academic setting | 59 |
| Outpatient setting | 77 |
| Multi-site trial | 43 |
| Clinician and patient subjects: | |
| Residents and fellows at least half of subjects | 57 |
| Mid-level clinicians (physician assistants, nurse practitioners) involved | 23 |
| Paediatric patients involved | 11 |
| System characterisation: | |
| Reminder or prompt organisation | 54 |
| Feedback system | 16 |
| Decision support system | xi |
| Skillful system | 0 |
| Clinical arena addressed by decision support: | |
| Management of chronic medical status or preventive care | 81 |
| Direction of acute medical condition | 23 |
| Direction of psychiatric condition | 14 |
| Pharmacotherapy | 53 |
| Laboratory test ordering | 46 |
| Non-surgical procedures | 41 |
| Radiology requisition | 31 |
| Specialist referral other than radiology | 24 |
| Diagnosis | 19 |
| Immunisation | 19 |
| Surgical procedures | three |
Univariate analyses of clinical conclusion support organization features
Table 5 summarises the success rates of clinical decision support systems with and without the 15 potentially important features. Overall, 48 of the 71 conclusion support systems (68% (95% confidence interval 56% to 78%)) significantly improved clinical practice. For 5 of the 15 features, the success rate of interventions possessing the feature was significantly greater than that of interventions defective the characteristic.
Tabular array 5
Success rates* of clinical determination support systems (CDSS) with and without 15 potentially important features. Results of 71 control-CDSS comparisons
| %success rate (95% CI) | ||||
|---|---|---|---|---|
| Feature | Feature prevalence (%) | With feature | Without feature | Charge per unit departure (95% CI) |
| General system features | ||||
| Integration with charting or order entry organisation | 85 | 73 (61 to 84) | 36 (xiv to 67) | 37 (6 to 61)† |
| Computer based generation of decision support | 69 | 76 (62 to 87) | fifty (28 to 72) | 26 (2 to 49)† |
| Local user interest in development process | 7 | 40 (eight to 81) | lxx (58 to 80) | −30 (−61 to 11) |
| Clinician-system interaction features | ||||
| Automated provision of decision support as office of clinician workflow | xc | 75 (63 to 85) | 0 (0 to 38) | 75 (37 to 84)† |
| Provision at time and location of determination making | 89 | 73 (61 to 83) | 25 (5 to 65) | 48 (0 to 70)‡ |
| Request documentation of reason for not following organisation recommendations | 21 | 100 (79 to 100) | 59 (45 to 72) | 41 (nineteen to 54)† |
| No need for additional clinician information entry | 89 | 71 (59 to 82) | 38 (11 to 71) | 34 (−2 to 61) |
| Recommendations executed by noting agreement | thirteen | 78 (44 to 96) | 66 (54 to 77) | 12 (−23 to 34) |
| Communication content features | ||||
| Provision of a recommendation, not just an cess | 76 | 76 (63 to 86) | 41 (xviii to 66) | 35 (viii to 58)† |
| Promotion of action rather than inaction | 92 | 68 (56 to 78) | 67 (27 to 94) | 1 (−27 to forty) |
| Justification via provision of research evidence | 7 | 100 (50 to 100) | 65 (53 to 76) | 35 (−13 to 48) |
| Justification via provision of reasoning | 39 | 75 (56 to 89) | 63 (47 to 76) | 12 (−11 to 34) |
| Auxiliary features | ||||
| Provision of decision support results to both clinicians and to patients | 10 | 86 (45 to 99) | 66 (54 to 77) | 20 (−23 to 39) |
| CDSS accompanied by periodic performance feedback | four | 67 (14 to 98) | 68 (55 to 78) | −1 (−fifty to 31) |
| CDSS accompanied by conventional teaching | 31 | 55 (33 to 74) | 73 (lx to 84) | −19 (−42 to 4) |
Well-nigh notably, 75% of interventions succeeded when the determination support was provided to clinicians automatically, whereas none succeeded when clinicians were required to seek out the communication of the decision support system (rate departure 75% (37% to 84%)). Similarly, systems that were provided equally an integrated component of charting or lodge entry systems were significantly more likely to succeed than stand up alone systems (rate difference 37% (6% to 61%)); systems that used a computer to generate the decision support were significantly more effective than systems that relied on manual processes (charge per unit departure 26% (2% to 49%)); systems that prompted clinicians to record a reason when non following the advised course of activeness were significantly more than likely to succeed than systems that allowed the system advice to be bypassed without recording a reason (rate difference 41% (xix% to 54%)); and systems that provided a recommendation (such as "Patient is at high run a risk of coronary avenue disease; recommend initiation of β blocker therapy") were significantly more likely to succeed than systems that provided merely an assessment of the patient (such as "Patient is at high chance of coronary artery disease") (rate deviation 35% (eight% to 58%)).
Finally, systems that provided determination support at the time and location of determination making were substantially more likely to succeed than systems that did non provide advice at the point of care, merely the difference in success rates fell just short of being meaning at the 0.05 level (rate deviation 48% (-0.46% to 70.01%)).
Meta-regression analysis
The univariate analyses evaluated each potential success factor in isolation from the other factors. We therefore conducted multivariate logistic regression analyses in gild to identify independent predictors of clinical conclusion support system effectiveness while taking into consideration the presence of other potentially important factors. Table half dozen shows the results of these analyses.
Table vi
Features of clinical decision back up systems (CDSS) associated with improved clinical exercise. Results of meta-regression analyses of 71 command-CDSS comparisons
| Feature * | Adapted odds ratio (95% CI) | P value |
|---|---|---|
| Chief analysis (all CDSS, n=71) | ||
| Automatic provision of decision support as part of clinician workflow | 112.1 (12.9 to ∞) | <0.00001 |
| Provision of decision back up at time and location of determination making | 15.4 (one.three to 300.6) | 0.0263 |
| Provision of recommendation rather than but an assessment | vii.ane (1.3 to 45.6) | 0.0187 |
| Computer based generation of decision support | six.iii (1.2 to 45.0) | 0.0294 |
| Secondary analysis (computer based CDSS, north=49) † ‡ | | |
| Automatic provision of decision support as part of clinician workflow | 105.0 (10.4 to ∞) | 0.00001 |
| Secondary assay (non-electronic CDSS, north=22) † § | | |
| Provision of recommendation rather than but an assessment | 19.4 (1.5 to 1263.0) | 0.0164 |
Of the half dozen features shown to be important past the univariate analyses, four were identified every bit independent predictors of system effectiveness by the master meta-regression analysis. Well-nigh notably, this analysis confirmed the disquisitional importance of automatically providing determination support as part of clinician workflow (P < 0.00001). The other three features were providing decision support at the fourth dimension and location of decision making (P = 0.0263), providing a recommendation rather than just an assessment (P = 0.0187), and using a computer to generate the determination support (P = 0.0294). Among the 32 clinical decision support systems incorporating all four features,w2-w6 w8-w10 w12 w16 w19 w20 w22 w24-w27 w32 w34-w49 w67 w69 w70 w88 30 (94% (fourscore% to 99%)) significantly improved clinical practice. In contrast, clinical decision support systems defective any of the four features improved clinical practice in simply 18 out of 39 cases (46% (xxx% to 62%)). The subset analyses for calculator based clinical decision support systems and for non-electronic clinical decision support systems yielded results consistent with the findings of the primary regression assay (tabular array 6).
Survey of straight experimental evidence
We identified xi randomised controlled trials in which a clinical decision support arrangement was evaluated straight against the same clinical decision support organisation with additional features (table 7).w14 w17 w19 w21 w22 w24-w26 w28 w38 w64 w86 In support of the regression results, one study found that system effectiveness was significantly enhanced when the decision back up was provided at the time and location of decision making.w19 Similarly, effectiveness was enhanced when clinicians were required to document the reason for non following organisation recommendationsw14 and when clinicians were provided with periodic feedback well-nigh their compliance with system recommendations.w28 Furthermore, two of four studies institute a meaning beneficial effect when conclusion back up results were provided to both clinicians and patients.w24-w26 w38 w86 In dissimilarity, clinical conclusion support system effectiveness remained largely unchanged when critiques were worded more strongly and the bear witness supporting the critiques was expanded to include institution-specific data,w17 when recommendations were made more than specific,w21 when local clinicians were recruited into the system development process,w64 and when bibliographic citations were provided to support the recommendations made by the organization.w22
Table vii
Details of 11 randomised controlled trials of clinical decision support systems (CDSS) that directly evaluated effectiveness of specific CDSS features
| Trial | No of clinicians *; No of patients *; duration of study | Control | Intervention | Consequence mensurate | Effect (intervention v command) |
|---|---|---|---|---|---|
| Tierney et al 1986w19 | 135; 6045; ten months | Computer generated reminders for 13 preventive care protocols, provided in a monthly study | Equally control, only protocols provided at the time of patient come across | % clinician compliance with protocols | Greater compliance for 3/13 protocols, P<0.05 |
| Litzelman et al 1993w14 | 176; 5407; 6 months | Computer generated reminders for faecal occult claret examination, mammography, and cervical smear test on encounter forms | As control, but users required to circumvolve 1 of 4 responses—"done/society today," "not applicable to this patient," "patient refused," or "side by side visit" | % clinician compliance with all reminders combined | 46 v 38, P=0.002 |
| Lobach 1996w28 | twenty; 205 encounters; 3 months | Computer generated diabetes guideline recommendations on special encounter forms | Equally control, plus biweekly email feedback summarising compliance with recommendations | Median level of % compliance with recommendations | 35.three v 6.one, P<0.01 |
| Becker et al 1989w25 | ane clinic; 371; 12 months | Calculator generated clinician reminders to provide 9 preventive intendance services | As control, plus mailed patient reminders | % overall compliance with preventive care guidelines | eighteen.5 v 12.ix, P=0.013 |
| McPhee et al 1989w24 | 21; 645; 9 months | Computer generated chart reminders for breast examination and mammography | As control, plus mailing of pamphlets and reminder letters to patients | % hateful compliance with mammography | 75 v 50, P=0.022 |
| Fordham et al 1990w26 | % mean compliance with breast examination | 80 v 82, NS | |||
| Gans et al 1994w86 | NA; 86; 18 months | Clinician notification of patients with previously undetected hypercholesterolaemia past mail service and provision of handling guidelines | Every bit control, plus mailing of reminder letters to patients | % of patients reporting follow-up visit to clinician | 57.5 v 53.ix, NS |
| % patient compliance with dietary recommendations | 74.5 five 61.5, NS | ||||
| % patient compliance with lifestyle recommendations | 36.2 v 35.9, NS | ||||
| Burack et al 1996w38 | 20; 758; 12 months | Figurer generated chart reminders for mammography referral | Every bit control, plus mailed patient reminders | % mammography completion among all eligible women | 31 v 32, NS |
| Harpole et al 1997w17 | 236; 491; 5 months | Computerised order entry system with real time critiques of appropriateness of abdominal radiograph orders | As control, only with critiques more than strongly worded and with supporting institutional evidence | % compliance with recommendations to abolish radiograph when unlikely to add diagnostic information | NS |
| % compliance with recommendations to order alternate views | NS | ||||
| Meyer et al 1991w21 | NA; 206; 12 months | Letter to clinicians identifying patients with ≥10 prescriptions and requesting a reduction in number of drugs | As control, followed past letter with specific recommendations for altering each drug regimen and gauge of each patient's compliance with drug regimen | Average number of drugs used at 4, half-dozen, and 12 months from intervention | NS |
| Sommers et al 1984w64 | 57; 145; 10 months | Manual nautical chart reminders for management of unexpected low haemoglobin levels | As command, plus baseline compliance feedback and interest of local clinicians in criteria evolution process | % compliance with management criteria | 61 v 77, P=NA |
| McDonald et al 1980w22 | 31; 3691 events; 3 months | Computer generated reminders for patient conditions requiring attention | As command, plus provision of bibliographic citations | % clinician response rate to detected events | forty.nine 5 35.9, P=0.154 |
Give-and-take
In this report, nosotros systematically reviewed the literature in order to determine why some clinical decision support systems succeed while others fail. In doing so, nosotros identified 22 technical and not-technical factors repeatedly suggested in the literature as important determinants of a system'southward ability to improve clinical exercise, and we evaluated 15 of these features in randomised controlled trials of clinical decision back up systems. We found five of the features were significantly correlated with system success, and 1 feature correlated with system success at but over the 0.05 significance level. Multiple logistic regression analysis identified four of these features equally independent predictors of a system'due south ability to improve clinical practice. Furthermore, we establish direct experimental prove to back up the importance of three additional features.
Strengths and limitations of our written report
This study has several important strengths. Firstly, our literature search was thorough, and we screened more than ten 000 manufactures to place potentially relevant studies. Secondly, nosotros generated the candidate fix of potentially important organization features by systematically reviewing the literature for relevant expert stance, rather than by relying on the views of a express set up of experts. Thirdly, we used two independent reviewers for study selection and data abstraction to increase the reliability of our findings. Fourthly, this study provides a quantitative gauge of the relative importance of specific clinical decision support system features. Finally, this study provides a comprehensive summary of randomised controlled trials that accept evaluated the importance of specific system features through direct experimentation.
I limitation of this study is that nosotros used a binary outcome measure out rather than a continuous measure such as effect size. We therefore could non adjust for variations in the size of outcomes. Another potential criticism is that nosotros pooled dissimilar types of clinical decision support systems in the regression assay. However, we believe that our methods were appropriate given that our objective was to determine the touch of heterogeneity among interventions rather than to estimate the effects of a homogeneous intervention, equally is usually the example for a meta-analysis.
Nosotros did not carry a subset analysis for studies in which patient upshot measures (as opposed to process measures) were evaluated—because the number of studies reporting patient issue measures was too small to allow for an fairly powered regression analysis. Moreover, because nosotros required an comeback in practice to be clinically meaning in lodge to be counted as a success, our methods precluded an improvement in a fiddling procedure measure from counting as a successful effect.
Our analyses were limited to published reports of randomised controlled trials. Thus, some of our findings may not exist extendable to clinical determination support organization categories for which we could non discover any studies meeting our inclusion criteria, such as clinical decision support systems provided on personal digital assistants. Also, publication bias against studies that failed to show an effect might have limited our power to identify features associated with ineffective systems.
The sample size for our regression assay was restricted past the size of the available literature. Thus, despite our best efforts to observe and include all relevant studies, our ratio of cases to explanatory variables was suboptimal, particularly for the subset regression analyses.58 ,60 Equally a outcome, we cannot rule out the importance of system features based on their absenteeism from the concluding regression models. As well, it is possible that one or more features were falsely included into the regression models because of over-fitting. Nonetheless, we do not believe this was the case, as our findings are consequent with our previous experiences of implementing clinical decision support systems in practice. An boosted limitation is that our analyses were restricted to features that could exist reliably bathetic. Every bit a consequence, we were unable to assess the significance of several potentially important features (table 2).
Implications
On a applied level, our findings imply that clinicians and other healthcare stakeholders should implement clinical decision support systems that (a) provide conclusion support automatically every bit part of clinician workflow, (b) deliver decision back up at the time and location of decision making, (c) provide actionable recommendations, and (d) use a figurer to generate the decision support. In detail, given the close correlation between automatic provision and successful outcomes (P < 0.00001), we believe that this feature should be implemented if at all possible. If a clinical decision support system must depend on clinician initiative for apply, we recommend that arrangement use be advisedly monitored and steps exist taken to ensure that clinicians access the resources every bit intended.
A mutual theme among all 4 features is that they make it easier for clinicians to utilize a clinical conclusion back up organisation. For example, automatically providing decision support eliminates the need for clinicians to seek out the advice of the system, and the employ of a calculator system improves the consistency and reliability of the clinical conclusion support organization by minimising labour intensive and fault prone processes such equally manual chart abstractions. As a general principle, then, our findings suggest that an constructive clinical conclusion support system must minimise the effort required by clinicians to receive and human activity on arrangement recommendations.
With regard to the three other system features shown to be important through direct experimentation, we think these features are of import and desirable but not as crucial as the four features identified by our regression analysis. Thus, when feasible and appropriate, clinical determination support systems should also provide periodic performance feedback, asking documentation of the reason for not following system recommendations, and share conclusion support results with patients. For the remaining clinical decision back up system features listed in table one, we consider them optional just still potentially benign, especially if they will get in easier for clinicians to use the clinical decision support arrangement or if the univariate analyses found that they were essentially more likely to be present in successful systems than in unsuccessful ones (tabular array v). Finally, with regard to the seven clinical decision support system features that could non be included in our regression analysis (table 2), nosotros recommend that they exist considered potentially important, especially if they reduce the fourth dimension, effort, or initiative required for clinicians to receive and deed on system recommendations.
Future directions
The promise of evidence based medicine will exist fulfilled simply when strategies for implementing all-time exercise are rigorously testify based themselves.61 ,62 In order to fulfil this goal in the context of clinical decision back up systems, two of import research needs must be addressed. Firstly, reports of clinical decision support arrangement evaluations should provide as much detail every bit possible when describing the systems and the mode in which clinicians interacted with them, so that others can acquire more effectively from previous successes and failures. Secondly, further straight experimentation is warranted to evaluate the importance of specific system features.
Supplementary Material
[actress: References w1 - w88]
Notes
References w1-w88, the studies reviewed in this commodity, are on bmj.com
We thank Vic Hasselblad for his assistance with the statistical analyses.
Contributors and guarantor: KK, DFL, and EAB contributed to the written report blueprint. KK, CAH, and DFL contributed to the data abstraction. All authors contributed to the information analysis. KK managed the project and wrote the manuscript, and all authors contributed to the critical revision and final blessing of the manuscript. DFL is guarantor.
Funding: This study was supported by inquiry grants T32-GM07171 and F37-LM008161-01 from the National Institutes of Health, Bethesda, Maryland, USA; and by enquiry grants R01-HS10472 and R03-HS10814 from the Agency for Healthcare Research and Quality, Rockville, Maryland, United states. These funders did not play a role in the design, execution, analysis, or publication of this written report.
Competing interests: None declared.
Upstanding approving: Non required.
References
ane. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health intendance delivered to adults in the United States. N Engl J Med 2003;348: 2635-45. [PubMed] [Google Scholar]
2. Kohn LT, Corrigan JM, Donaldson MS, eds. To err is human: building a safer health arrangement. Washington, DC: National Academy Press, 1999.
iii. Vincent C, Neale Thou, Woloshynowych M. Agin events in British hospitals: preliminary retrospective tape review. BMJ 2001;322: 517-9. [PMC free article] [PubMed] [Google Scholar]
4. Chase DL, Haynes RB, Hanna SE, Smith K. Furnishings of computer-based clinical decision support systems on doc functioning and patient outcomes: a systematic review. JAMA 1998;280: 1339-46. [PubMed] [Google Scholar]
5. Bennett JW, Glasziou PP. Computerised reminders and feedback in medication direction: a systematic review of randomised controlled trials. Med J Aust 2003;178: 217-22. [PubMed] [Google Scholar]
6. Walton RT, Harvey E, Dovey Southward, Freemantle Northward. Computerised communication on drug dosage to improve prescribing do. Cochrane Database Syst Rev 2001;1: CD002894. [PubMed] [Google Scholar]
7. Walton R, Dovey Due south, Harvey Eastward, Freemantle N. Figurer support for determining drug dose: systematic review and meta-analysis. BMJ 1999;318: 984-ninety. [PMC free commodity] [PubMed] [Google Scholar]
eight. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Curvation Intern Med 2003;163: 1409-16. [PubMed] [Google Scholar]
9. Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'Luf N, et al. The touch on of computerized physician club entry on medication error prevention. J Am Med Inform Assoc 1999;6: 313-21. [PMC free article] [PubMed] [Google Scholar]
x. Shea S, DuMouchel West, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate figurer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996;three: 399-409. [PMC free article] [PubMed] [Google Scholar]
xi. Balas EA, Weingarten S, Barb CT, Blumenthal D, Boren SA, Brownish GD. Improving preventive care by prompting physicians. Arch Intern Med 2000;160: 301-8. [PubMed] [Google Scholar]
12. Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc 1999;half-dozen: 104-14. [PMC free article] [PubMed] [Google Scholar]
thirteen. Thomson O'Brien MA, Oxman AD, Davis DA, Haynes RB, Freemantle N, Harvey EL. Inspect and feedback versus culling strategies: furnishings on professional practise and health care outcomes. Cochrane Database Syst Rev 2000;2: CD000260. [PubMed] [Google Scholar]
fourteen. Hulscher ME, Wensing M, van der Weijden T, Grol R. Interventions to implement prevention in primary care. Cochrane Database Syst Rev 2001;1: CD000362. [PubMed] [Google Scholar]
15. Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional exercise. CMAJ 1995;153: 1423-31. [PMC gratis article] [PubMed] [Google Scholar]
xvi. Kupets R, Covens A. Strategies for the implementation of cervical and breast cancer screening of women by primary care physicians. Gynecol Oncol 2001;83: 186-97. [PubMed] [Google Scholar]
17. Bero LA, Grilli R, Grimshaw JM, Harvey Eastward, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of enquiry findings. The Cochrane Constructive Practice and Organization of Intendance Review Group. BMJ 1998;317: 465-viii. [PMC gratis commodity] [PubMed] [Google Scholar]
18. Mandelblatt J, Kanetsky PA. Effectiveness of interventions to enhance physician screening for breast cancer. J Fam Pract 1995;40: 162-71. [PubMed] [Google Scholar]
19. Wensing M, Grol R. Single and combined strategies for implementing changes in primary care: a literature review. Int J Qual Health Care 1994;6: 115-32. [PubMed] [Google Scholar]
20. Mandelblatt JS, Yabroff KR. Effectiveness of interventions designed to increase mammography use: a meta-analysis of provider-targeted strategies. Cancer Epidemiol Biomarkers Prev 1999;8: 759-67. [PubMed] [Google Scholar]
21. Stone EG, Morton SC, Hulscher ME, Maglione MA, Roth EA, Grimshaw JM, et al. Interventions that increase use of adult immunization and cancer screening services: a meta-assay. Ann Intern Med 2002;136: 641-51. [PubMed] [Google Scholar]
22. Weingarten SR, Henning JM, Badamgarav Eastward, Knight One thousand, Hasselblad V, Gano A Jr, et al. Interventions used in illness management programmes for patients with chronic illness—which ones work? Meta-analysis of published reports. BMJ 2002;325: 925-32. [PMC free article] [PubMed] [Google Scholar]
23. Kaplan B. Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inf 2001;64: 39-56. [PubMed] [Google Scholar]
24. Kanouse DE, Kallich JD, Kahan JP. Dissemination of effectiveness and outcomes enquiry. Health Policy 1995;34: 167-92. [PubMed] [Google Scholar]
25. Wendt T, Knaup-Gregori P, Winter A. Decision support in medicine: a survey of problems of user acceptance. Stud Wellness Technol Inform 2000;77: 852-half dozen. [PubMed] [Google Scholar]
26. Wetter T. Lessons learnt from bringing knowledge-based decision support into routine use. Artif Intell Med 2002;24: 195-203. [PubMed] [Google Scholar]
27. Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H, et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001;8: 527-34. [PMC free article] [PubMed] [Google Scholar]
28. Payne TH. Figurer decision back up systems. Chest 2000;118: 47-52S. [PubMed] [Google Scholar]
29. Shiffman RN, Brandt CA, Liaw Y, Corb GJ. A design model for computer-based guideline implementation based on data management services. J Am Med Inform Assoc 1999;half-dozen: 99-103. [PMC free commodity] [PubMed] [Google Scholar]
30. Ash JS, Stavri PZ, Kuperman GJ. A consensus statement on considerations for a successful CPOE implementation. J Am Med Inform Assoc 2003;ten: 229-34. [PMC free article] [PubMed] [Google Scholar]
31. Trivedi MH, Kern JK, Marcee A, Grannemann B, Kleiber B, Bettinger T, et al. Development and implementation of computerized clinical guidelines: barriers and solutions. Methods Inf Med 2002;41: 435-42. [PubMed] [Google Scholar]
32. Solberg LI, Brekke ML, Fazio CJ, Fowles J, Jacobsen DN, Kottke TE, et al. Lessons from experienced guideline implementers: attend to many factors and use multiple strategies. Jt Comm J Qual Improv 2000;26: 171-88. [PubMed] [Google Scholar]
33. Bates DW, Kuperman GJ, Wang Southward, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003;10: 523-30. [PMC costless article] [PubMed] [Google Scholar]
34. Centre for Wellness Computer science, University of New Due south Wales. Appendix A: electronic decision support activities in different healthcare settings in Australia. In: National Electronic Decision Support Taskforce. Electronic decision support for Commonwealth of australia'southward health sector. Canberra: Commonwealth of Australia, 2003. www.ahic.org.au/downloads/nedsrept.pdf (accessed 28 Jan 2005).
35. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20: 37-46. [Google Scholar]
36. Aronsky D, Chan KJ, Haug PJ. Evaluation of a computerized diagnostic determination support organization for patients with pneumonia: study design considerations. J Am Med Inform Assoc 2001;eight: 473-85. [PMC gratis article] [PubMed] [Google Scholar]
37. Ramnarayan P, Britto J. Paediatric clinical decision support systems. Arch Dis Child 2002;87: 361-2. [PMC free article] [PubMed] [Google Scholar]
38. Ryff-de Leche A, Engler H, Nutzi Due east, Berger M, Berger Westward. Clinical application of 2 computerized diabetes management systems: comparison with the log-book method. Diabetes Res 1992;19: 97-105. [PubMed] [Google Scholar]
39. Hersh WR. Medical information science: improving wellness care through information. JAMA 2002;288: 1955-viii. [PubMed] [Google Scholar]
twoscore. Bodenheimer T, Grumbach K. Electronic technology: a spark to revitalize primary care? JAMA 2003;290: 259-64. [PubMed] [Google Scholar]
41. Lowensteyn I, Joseph L, Levinton C, Abrahamowicz Thousand, Steinert Y, Grover S. Can computerized risk profiles help patients better their coronary chance? The results of the coronary wellness assessment written report (CHAS). Prev Med 1998;27: 730-7. [PubMed] [Google Scholar]
42. Miller RA. Medical diagnostic decision back up systems—past, present, and future: a threaded bibliography and cursory commentary. J Am Med Inform Assoc 1994;ane: eight-27. [PMC gratuitous article] [PubMed] [Google Scholar]
43. Morris AH. Academia and dispensary. Developing and implementing computerized protocols for standardization of clinical decisions. Ann Intern Med 2000;132: 373-83. [PubMed] [Google Scholar]
44. Tierney WM. Improving clinical decisions and outcomes with information: a review. Int J Med Inf 2001;62: i-9. [PubMed] [Google Scholar]
45. Heathfield HA, Wyatt J. Philosophies for the design and evolution of clinical decision-back up systems. Methods Inf Med 1993;32: 1-8. [PubMed] [Google Scholar]
46. Wyatt JR. Lessons learnt from the field trial of ACORN, an expert system to advise on chest pain. Proceedings of the 6th World Conference on Medical Information science, Singapore 1989: 111-5.
47. Stock JL, Waud CE, Coderre JA, Overdorf JH, Janikas JS, Heiniluoma KM, et al. Clinical reporting to main care physicians leads to increased use and understanding of bone densitometry and affects the direction of osteoporosis. A randomized trial. Ann Intern Med 1998;128: 996-9. [PubMed] [Google Scholar]
48. Frances CD, Alperin P, Adler JS, Grady D. Does a fixed md reminder system amend the care of patients with coronary artery disease? A randomized controlled trial. West J Med 2001;175: 165-half dozen. [PMC free article] [PubMed] [Google Scholar]
49. Belcher DW, Berg AO, Inui TS. Practical approaches to providing better preventive care: are physicians a problem or a solution? Am J Prev Med 1988;4: 27-48. [PubMed] [Google Scholar]
50. McPhee SJ, Detmer WM. Role-based interventions to amend delivery of cancer prevention services by primary care physicians. Cancer 1993;72: 1100-12. [PubMed] [Google Scholar]
51. Strecher VJ, O'Malley MS, Villagra VG, Campbell EE, Gonzalez JJ, Irons TG, et al. Can residents be trained to counsel patients about quitting smoking? Results from a randomized trial. J Gen Intern Med 1991;half-dozen: ix-17. [PubMed] [Google Scholar]
52. Shannon KC, Sinacore JM, Bennett SG, Joshi AM, Sherin KM, Deitrich A. Improving commitment of preventive health care with the comprehensive annotated reminder tool (CART). J Fam Pract 2001;50: 767-71. [PubMed] [Google Scholar]
53. Delaney BC, Fitzmaurice DA, Riaz A, Hobbs FD. Can computerised conclusion support systems evangelize improved quality in primary intendance? BMJ 1999;319: 1281-3. [PMC free article] [PubMed] [Google Scholar]
54. Weir CJ, Lees KR, MacWalter RS, Muir KW, Wallesch CW, McLelland EV, et al. Cluster-randomized, controlled trial of computer-based conclusion support for selecting long-term anti-thrombotic therapy later acute ischaemic stroke. QJM 2003;96: 143-53. [PubMed] [Google Scholar]
55. StatXact [computer program]. Version 6.2.0. Cambridge, MA: Cytel Software, 2004.
56. Casella Thou. Refining binomial confidence intervals. CanJStat 1986;14: 113-29. [Google Scholar]
57. Agresti A, Min Y. On small-sample confidence intervals for parameters in discrete distributions. Biometrics 2001;57: 963-71. [PubMed] [Google Scholar]
58. Green SB. How many subjects does information technology take to do a regression analysis? Multivariate Behav Res 1991;26: 499-510. [PubMed] [Google Scholar]
59. LogXact [calculator program]. Version five.0. Cambridge, MA: Cytel Software, 2002.
60. Harrell FE Jr, Lee KL, Marker DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and capability, and measuring and reducing errors. Stat Med 1996;15: 361-87. [PubMed] [Google Scholar]
61. Grol R. Personal paper: behavior and show in irresolute clinical exercise. BMJ 1997;315: 418-21. [PMC gratuitous article] [PubMed] [Google Scholar]
62. Freemantle N, Grilli R, Grimshaw J, Oxman A. Implementing findings of medical research: the Cochrane Collaboration on Constructive Professional person Exercise. Qual Health Intendance 1995;4: 45-vii. [PMC free article] [PubMed] [Google Scholar]
Articles from The BMJ are provided here courtesy of BMJ Publishing Grouping
Source: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC555881/
0 Response to "Priorities Among Effective Clinical Preventive Services Results of a Systematic Review and Analysis"
Post a Comment