Article Text
PDF
Abstract
Background Emergency physicians frequently assess risk of acute cardiac events (ACEs) in patients with undifferentiated chest pain. Such estimates have been shown to have moderate to high sensitivity for ACE but are conservative. Little is known about the factors implicitly used by physicians to determine the pretest probability of risk. This study sought to identify the accuracy of physician risk estimates for ACE in patients presenting to the ED with chest pain and to identify the demographic and clinical information emergency physicians use in their determination of patient risk.
Methods This study used data from two prospective studies of consenting adult patients presenting to the ED with symptoms of possible acute coronary syndrome. ED physicians estimated the pretest probability of ACE. Multiple linear regression analysis was used to identify predictors of physician risk estimates. Logistic regression was used to determine whether there was a correlation between physicians’ estimated risk and ACE.
Results Increasing age, male sex, abnormal ECG features, heavy/crushing chest pain and risk factors were correlated with physician risk estimates. Physician risk estimates were consistently found to be higher than the expected proportion of ACE from the sampled population.
Conclusion Physicians systematically overestimate ACE risk. A range of factors are associated with physician risk estimates. These include factors strongly predictive of ACE, such as age and ECG characteristics. They also include other factors that have been shown to be unreliable predictors of ACE in an ED setting, such as typicality of pain and risk factors.
- acute coronary syndrome
- clinical assessment
- cardiac care, diagnosis
Statistics from Altmetric.com
Key messages
What is already known on this subject
Physicians tend to overestimate risk when dichotomising patients with chest pain into low-risk or high-risk categories.
The utility of specific elements in assessment, such as symptoms and traditional risk factors for cardiovascular disease, has been questioned in individual risk prediction.
What this study adds
In two prospective studies, we asked physicians to estimate patient risk of acute cardiac event (ACE) and found that physicians systematically overestimate risk. For example, the proportion of patients with ACE was 2%, 4%, 9% and 25% at physician risk estimates of 5%–10%, 25%–30%, 50%–55% and 70%–75%, respectively.
Increased age and male sex were associated with increased physician risk assessment for ACE, with estimates for women consistently being lower by comparison with men.
Physician risk estimates also are associated with factors that have been shown to be unreliable predictors of ACE in an ED setting, such as typicality of pain and risk factors.
Background
Patients presenting to the ED with undifferentiated chest pain represent a diagnostic challenge, with less than 20% of such patients being ultimately diagnosed with an acute cardiac event (ACE).1 The evaluation of patients with a potential ACE remains difficult as patients present with diverse symptoms and may not have clear diagnostic findings. For this reason, distinguishing the pretest probability of disease for patients in whom the physician is investigating for a potential ACE is an important component of assessment that facilitates tailoring of additional investigations and disposition planning.
One method for identifying patient risk is unstructured physician risk estimates (or gestalt). Gestalt is an appealing method of risk stratification due to its flexibility and inherent availability.2 Existing studies have shown that physician estimates of risk have moderate to high sensitivity for ruling out ACEs but tend to be conservative and overestimate risk.3–5 Such studies have dichotomised risk estimates into low-risk and high-risk when examining accuracy. Limited research has examined the accuracy of physicians’ risk estimates when retained as a continuous variable. This means that the accuracy of physician risk estimates across the complete spectrum of risk is unknown.
Another aspect of physician risk that remains underexplored is the factors used by physicians for the pretest probability assessment of risk. One study found that ED physicians used typicality of symptoms as the most important tool in determining suspicion for an ACE, with ECG and troponin results only having a minor role.6 This finding is in contrast to the research showing that symptoms are poor predictors of ACEs.7 8 It also is unclear whether the same pattern of results would emerge if examining predicted risk in patients whom physicians have chosen to investigate for an ACE.
This study will explore ED physician risk estimates. The major aim was to identify the accuracy of ED physician risk estimates in patients investigated for possible cardiac chest pain. The secondary aim was to identify the factors used by emergency physicians in determining their risk estimates for ACEs.
Methods
Study design
Data from two studies of patients presenting to the ED with suspected ACS were analysed. The first study was a prospective observational study of patients (n=983) presenting to the ED of an Australian tertiary care hospital between November 2007 and January 2011.9 The second was a non-randomised interventional trial (n=1366) conducted at the same site between February 2011 and March 2014.10 A detailed description of the studies has been previously published.9 10 An examination of physician risk estimates was included as a planned analysis for both studies. Informed consent was obtained from all participants.
Participants
Both studies had identical patient recruitment processes with the same inclusion and exclusion criteria. Eligible patients were those aged 18 years or older who presented to the ED with at least 5 min of chest pain suggestive of ACS and were being investigated for suspected ACS. Patients were excluded if they had a clear non-ACS cause for their symptoms, were unwilling or unable to provide informed consent (eg, language barrier), were considered by the staff as inappropriate for recruitment (eg, terminal illness), were transferred from another hospital, were pregnant, were recruited to the study within the previous 45 days, or were unable or unwilling to be contacted after discharge. At the study site, patients with a clear ST-segment myocardial infarction on presentation are referred directly to interventional cardiology on recognition. As such, these patients would not be included in the primary cohort as they would not have been managed by the ED clinicians to provide an estimate of risk.
All patients in the observational study were managed according to standard care, which included ECG and cardiac troponin I (cTnI) measurement at presentation, followed by cTnI measurements 6 hours later.9 The intervention study included an accelerated diagnostic protocol (the Improved Assessment of Chest Pain Trial [IMPACT] protocol) in which a subgroup of ‘low’-risk and ‘intermediate’-risk patients underwent 2-hour rather than 6-hour troponin testing.10 In the cohort of patients used in the current analyses, there were 369 IMPACT patients who underwent 2-hour testing with no additional testing at 6 hours. All were followed up at 30 days to ensure no events were missed. This intervention did not change patient care during the first 2 hours in the ED. The IMPACT protocol was invoked only after the physician risk estimates were obtained.10
Data collection
For both studies, research staff initially identified patients using the ED admissions database, an electronic record showing the patient’s presenting complaint. Nurses then screened any patient who presented with symptoms suspicious for ACS, including chest discomfort; shoulder, arm, jaw or upper abdominal pain; shortness of breath; nausea; vomiting; and sweating to determine eligibility for the study. During working hours (08:00–17:00), consecutive, eligible patients were recruited. Demographic, presenting details and medical history were collected directly from each patient. If a patient was unsure of an answer, a response of ‘no’ was recorded. The same staff also supervised ECG testing and drew blood samples for cTnI measurement. Patients were followed up for 30 days from initial presentation using hospital records and telephone follow-up. Only events during the index admission were considered for the primary outcome of this study, and all were considered to be index events.
The ED physician estimates were obtained for a convenience sample from the entire cohort. This convenience sample comprised all eligible patients where (1) research staff were available; (2) the treating clinician was available to provide a risk estimate; and (3) risk estimates were obtained before cTnI results were available. Physicians with advanced training in emergency medicine (either senior registrars or consultants) were asked to estimate the probability (using a 100 mm visual analogue scale) of each patient experiencing an ACE during the index admission. This was recorded after the presentation ECG results were known but before presentation cTnI results were available. Risk estimates were not obtained if either the cTnI result was known before research staff were able to collect the data, or the physician or research staff were unavailable.
Outcomes
The primary endpoint was ACEs, defined as death from a cardiac cause, diagnosis of acute myocardial infarction (AMI) (type I or II), emergency revascularisation, unplanned revascularisation or unstable angina pectoris during the index admission (online supplementary table 1).11 The endpoints were adjudicated in line with the third universal definition of myocardial infarction (MI),12 with the exception that the 369 IMPACT patients without 6-hour samples were adjudicated based on 2-hour rather than 3-hour to 6-hour samples. One cardiologist assigned endpoints using the clinical record, ECGs, troponin results and investigations from standard care. A second cardiologist conducted a blind review of all patients who received a cardiovascular endpoint and 10% of cases with a non-cardiovascular endpoint. This adjudication process was undertaken by cardiologists who were part of the research team. In cases of disagreement, endpoints were agreed by consensus between the two cardiologists and an emergency physician. The secondary endpoint was physician estimate of risk. This was used as an endpoint in identifying patient predictors of physicians’ risk estimates.
Supplemental material
Statistical analysis
Statistical analyses were performed in R V.3.5.1. The study cohort comprised all patients where a risk estimate was made (figure 1). Physician risk estimates were not available in 1223 cases.
Participant flow.
Baseline characteristics and outcomes of the study cohort were reported for the primary cohort. These data also were reported for those patients without data on physician risk scores to identify whether there were systematic differences between the cohort of patients with and without risk estimates. Logistic regression was conducted to examine the relationship between the physician’s estimate of ACEs and index ACEs. Physician risk estimates were modelled using restricted cubic splines to allow for the relationship between index ACEs and physician risk estimates to be non-linear. To identify best fit, models were fit with three, four and five knots, with knot location based on the recommendations of Harrell.13 The final number of knots were chosen based on Akaike’s information criterion (
).13
Linear regression analyses were conducted to identify predictors of physician risk estimates. This model incorporated patient age, sex, risk factors, cardiovascular history, presentation ECG and pain characteristics.11 These variables were chosen a priori as they reflect the information available to clinicians when making risk estimates. They were also variables that plausibly would be used by clinicians based on previous research.
A sensitivity analysis also was conducted wherein the primary logistic regression analysis was repeated, but multiple imputation was used for missing data. This analysis was conducted to identify whether the relationships reported in the primary analyses were obscured by the use of a convenience sample. Missing physician estimates were imputed in 50 samples using age, sex, risk factors, cardiovascular history, ECG, pain characteristics and index ACEs. We also conducted a sensitivity analysis examining the relationship between physician risk estimates and ACE up to 30 days (including index ACEs). This analysis sought to identify the accuracy of risk estimates for both index and short-term events.
Results
Baseline demographics and outcomes of the patients with and without risk estimates were similar (table 1). A total of 130 (12%) patients were diagnosed with at least one ACE during their index presentation. For patients with an ACE, the median physician risk estimate was 75 (IQR 50–70). Median physician risk estimates (figure 2) for ACE were similar for patients diagnosed with type I MI (75, IQR 60–90), type II MI (75, IQR 50–90) and unstable anginga pectoris (UAP) (76.5, IQR 40–90). However, the spread of estimates for type II MI and UAP was slightly higher than that for type I MI (figure 2). Compared with those with ACE, physician risk scores were lower for those with other cardiac conditions (50, IQR 20–65) or a non-cardiac diagnosis (26, IQR 15–50).
Baseline characteristics of the cohort
Physician risk estimates by patient outcome. T1MI, type 1 myocardial infarction; T2MI, type 2 myocardial infarction; UAP, unstable angina pectoris.
For the analysis regressing ACEs on risk estimates, the optimal model incorporated cubic splines with three knots. These knots were located at physician risk estimates of 10, 40 and 80. Increasing physician risk estimates were associated with increased probability of an index ACE. Figure 3A shows the proportion of patients who had an index ACE across the range of physician risk estimates (grouped into five-point increments). Figure 3B provides the same data across individual risk estimates smoothed using logistic regression. Physician estimates of risk were consistently higher than the proportion of patients with an ACE. For example, the proportion of patients with an ACE was 2%, 4%, 9% and 25% at physician risk estimates of 5%–10%, 25%–30%, 50%–55% and 70%–75%, respectively. Sensitivity and specificity of various physician risk estimates for index ACEs are provided in online supplementary tables 2 and 3. To further explore the data, we performed a second (unplanned) regression analysis incorporating risk estimates, sex and their interaction. This provided data on physician risk estimates of male and female participants (online supplementary figure 1). Physician risk estimates for male and female participants were of similar accuracy; both curves showed a relative overestimation by physicians with respect to ACEs.
Predicted probability of an ACE. (A) Provides a calibration plot comparing predicted and observed ACEs. Patients were grouped into 20 groups of predicted risk (categories include five risk points). Dots reflect the proportion of patients with an ACE and horizontal lines are 95% confidence intervals. If dots fall on the 45 degree line, the proportion of patients with an ACE equals the risk predicted by clinicians. Dots below the line indicate that the actual proportion of ACEs is lower than that predicted by clinicians. (B) Provides the expected proportion of ACEs from regression of ACEs on physician risk estimates. ACE, acute cardiac event.
A number of sensitivity analyses also were performed. First, the regression analyses were repeated, but multiple imputation was used to impute physician risk estimates for those patients without missing data (n=1223). The findings from this analysis yielded the same pattern of results to those of the main analysis. A comparison of results with and without multiple imputations is provided in online supplementary figure 2. Second, the regression analysis was repeated, but the endpoint was ACEs up to 30 days (including index events). The same overestimation of risk is seen for this endpoint as that seen using index ACEs as the endpoint (online supplementary figure 3).
Results from multiple linear regression of physician risk estimates are provided in figure 4. Physician risk estimates increased with increasing age until age 60 but started to plateau above 60 years. Risk estimates for women were less than those for men. Patients having any abnormality on ECG had higher physician risk estimates compared with those with a normal ECG; the increase in risk estimates ranged from 4 to 30 points higher, depending on the ECG categorisation. Abnormalities in order of increasing risk estimates included (1) non-specific ST-T wave changes, (2) abnormal ECG but not diagnostic of ischaemia, (3) ischaemia or infarction known to be old, (4) infarction or ischaemia not known to be old and (5) ECG changes consistent with AMI. The presence of chest pain was associated with a higher risk estimate (4–15 points higher), while being female was associated with a lower risk estimate (2–8 points lower). Pain characteristics, such as dull, burning or stabbing, were not associated with physician estimates of risk, but pain that worsened on exertion was associated with increased estimates of risk. Risk factors and prior AMI were associated with a small increase in risk estimates.
Predictors of physician risk estimates. (A) Provides the association between patient age and unstructured physician risk estimates. (B) Provides the coefficients from regression of physician risk estimates. The coefficients represent the average increase in risk score assigned to patients with each feature. AMI, acute myocardial infarction.
Discussion
This study found that physician risk estimates for ACEs were conservative and overestimated risk across the entire spectrum. A number of factors were identified as being associated with higher ratings of risk. These included factors predictive of an ACE, such as male sex, increasing age and ECG findings. They also included other factors that have been shown to be unreliable predictors of an ACE in an ED setting, such as typicality of pain and risk factors.
Overall, unstructured risk estimates were associated with overestimation of ACE risk. This finding is in line with previous research examining whether clinical gestalt is accurate for ruling out AMI.2–5 Such studies have reported moderate to high sensitivity for ruling out patients, but specificity is very low, typically around 20%–30%.3 5 This is not surprising as the significant consequences of missing a diagnosis of AMI means that ED clinicians are necessarily conservative and risk averse. When combined with objective tests, such as troponin, physician gestalt is likely to be a safe method for risk stratifying ACEs. However, this can lead to unnecessary investigations and admissions, raising the potential for patient harms and overcrowding within the healthcare system.
With physician gestalt overestimating risk, clinicians may be encouraged to instead use a validated structured risk estimate. Several tools have been used for the assessment of ACE risk in ED patients, including the History, EKG, Age, Risk factors, Troponin (HEART) score,14 the Thrombolysis In Myocardial Infarction (TIMI) score,15 the Emergency Department Assessment of Chest Pain Score (EDACS),16 Global Registry of Acute Coronary Events (GRACE) score17 and the Manchester ACS (MACS) rules (MACS and T-MACS18 19). Of these, GRACE and MACS/T-MACS provide the greatest detail in terms of a predicted probability of cardiac events. However, there is limited research to date comparing the accuracy of clinician gestalt to such clinical decision rules. The literature that does exist has not equivocally supported the benefit of validated scores over physician estimates.20 Further, there has been limited research focussing on whether structured risk estimates are well calibrated for predicting risk. Notable exceptions include a multicentre study showing that the T-MACS rule was accurate for identifying risk.19 Another study found overestimation of risk when using the T-MACS score in a very low-risk cohort, but this study retrospectively validated T-MACS using several proxy measures.21 Further research is required to identify the utility of physician gestalt in comparison to clinical decision rules.
Male sex is known to be predictive of an ACE in emergency patients with acute undifferentiated chest pain,22 and our study found sex to be influential in unstructured risk prediction for ACEs. In clinical care, caution is needed about the weighting of sex in ACE-risk determination, as the bias against recognition of ACE risk in women with an ACE has previously been reported,23 and may lead to undertreatment. A strong association between age and risk prediction for ACEs was also found in this study, with older patients having increased risk. Age is a strong predictor for underlying coronary artery disease,24 and hence this is an unsurprising finding, but the effect of age on physician risk estimates has not been described previously.
Cardiac pain characteristics are not reliable predictors of an ACE7 8 but have been incorporated as a component of some risk assessment tools. Pain traditionally seen as typical, including chest pain that is heavy or crushing in nature, has not been found to have a strong association with ACEs in undifferentiated chest pain cohorts.22 Despite this, we found that the presence of chest pain and pain that is heavy or crushing in character was a predictor of risk estimates. This indicates that physicians may still implicitly use pain characteristics despite their lack of utility in risk prediction.
Traditional predictors of risk for coronary artery disease had a weak association with unstructured physician estimates. These risk factors are useful on a population basis to identify cohorts at higher risk of coronary artery disease, but their utility for individual patient risk prediction is limited, except potentially in younger patients.7 22 24 Over-reliance on such risk factors may reduce accuracy in the identification of patients with an ACE.
Of the historical examination and investigation findings evaluated, ECG changes, particularly those changes consistent with AMI or ischaemic changes, were most influential. This is unsurprising as ECG changes are an objective measure that are associated with higher risk of 30-day death, AMI and revascularisation.25 This finding is counter to previous research showing that ECG findings only played a minor role in determining suspicion for an ACE.6
Study limitations
Our study has several limitations. For the primary cohort, only a subset of the original patients was included because of missing risk estimates. However, there were no differences found in baseline characteristics in the subset not included, and imputation of missing data did not alter the pattern of results. The study is from a single centre and so may not be generalisable across sites. Risk estimates were widely variable, indicating high uncertainty in physician estimates. This study used previously collected data. As such, not all possible predictors of physician risk indicators were assessed in the study. Our study evaluated unstructured risk assessment without access to supportive investigations, including troponin that has been shown to improve accuracy of risk stratification by physicians and may enable a reduction in admissions following a suspected ACE.3
Conclusion
Our study of unstructured physician risk estimates for ACEs in emergency patients shows that ED physicians continue to overestimate risk of ACEs. A range of factors are associated with physician risk estimates. These include factors strongly predictive of an ACE, such as age, history of AMI and ECG characteristics. They also include other factors that have been shown to be unreliable predictors of an ACE in an ED setting, such as typicality of pain and risk factors.
Acknowledgments
We are indebted to the patients involved in our studies and to the research and clinical staff involved in our trials.
References
Footnotes
Twitter @kiwiskiNZ
Contributors All authors met the requirements of authorship. JHG, NS, TK, LR, JWP, MT, WAP and LC conceived and designed the study. JHG, WAP, MT, TH and LC were involved in acquisition of data. JHG analysed the data. WAP, NS, TK, LR, JWP, MT and LC assisted with the interpretation of the data. JHG, NS and LC drafted the work, and all authors revised the manuscript critically for important intellectual content.
Funding Data collection was funded by research grants from the Emergency Medicine Foundation (QEMRF-PROJ-2008-002; QEMRF-PROG-2010-004).
Competing interests WAP and LC report funding from Beckmans and Abbott point of care in the form of a research grant paid to the institution. MT and JWP report funding from Abbott point of care and Roche in the form of a research grant paid to the institution. They also report consultancy from Abbott diagnostics.
Patient consent for publication Not required.
Ethics approval The study protocols were approved by the institutions’ human research and ethics committees and complied with the Declaration of Helsinki. The study was approved by the Royal Brisbane Women’s Hospital. The ethics/project approval numbers are HREC/2008/101 and HREC/10/QRBW/403.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement No data are available.