We love a bit of meta-cognition here at St.Emlyn’s. As time passes and we all become a little longer in the tooth it becomes increasingly apparent that it’s not just ‘what’ we know it’s how we use it. In diagnostic testing in the ED (something that we are supposed to be awesome at) we must understand how we use tests and how our clinical judgement affects the use and subsequent performance of those tests.
We are always on the look out for papers that examine how we think and how we make decision and so I was intrigued to look at this paper on the ability of clinicians to determine the probability of underlying MI/IHD in patients presenting to the emergency department. If you’re a regular reader you will know that Rick Body has a great deal of experience in this area and has published widely on clinician ability to identify patient risk. You might want to look at these blog posts where we have explored judgement and metacognition in the past.
The paper is from Bristol UK and the abstract is below. At the moment it is open access on the website so please read the abstract, but as always, if you can find the time please read the full paper before going further.
What kind of paper is this?
This study was part of a larger study at the use of chest pain pathways and high sensitivity troponins. This is a pre-planned study within that larger study and it’s essentially an observational cohort.
Tell me about the patients
They have looked at 912 patients of whom 12.5% had an AMI. This is a fairly low event rate but not dissimilar to other studies in this area as they have only looked at patients with a non diagnostic ECG. Interestingly they excluded patients who had no diagnostic testing which may have missed a small number of patients. The study is looking at judgement and part of that judgement is whether or not to test a chest pain patient for AMI. It’s tricky as the alternative would be to test everyone with chest pain according to predetermined criteria. This way is more pragmatic but it may have missed some patients. This is a single centre UK medium sized hospital, so limited because single site, but not untypical of many UK departments.
What did they do?
In the larger study they are following patients through to a definitive diagnosis of AMI/not AMI and IHD/not IHD using troponin testing and angiography. Since that will give a pretty robust gold standard outcome they have collected data on physician impression of pain as patients presented in the emergency department. They have asked clinicians to state whether the pain is typical or not then followed the patients through to see if they are right. They have also compared experienced vs inexperienced clinicians to see if seniority improves performance.
AMI was determined using the third universal definition of MI and IHD determined in some patients undergoing angiography although that was quite a small percentage (17.2%).
What did they find?
The key findings from the authors are that fewer than half of patients were characterised as having typical chest pain and that experienced clinicians. There was a small difference in classifying patients as ‘typical’ between experienced and non-experienced clinicians (35.2 v 45.8%).
The ROC curves are interesting and reminiscent of work we did on risk factors in cardiac disease. In other words the performance of doctors in assessing risk is rubbish! Hang on though. This is a little bit unusual as chest pain typicality is a dichotomous variable (the chest pain can either be typical or atypical – there are only two possibilities). You usually use an ROC curve to plot sensitivity versus specificity for something that could have several possible cut-offs (an ordinal or continuous variable). However, they have plotted a single value on these curves and then drawn lines to the corners. If you only have one data point then it’s really just a dot and therefore you shouldn’t really use an AUC (Area under the curve) analysis. Have a look at the graphs below for interest, but bear in mind that the most sensible way to look at this data is to consider the specificity of the opinion of the clinician.
It’s also worth asking what we mean by sensitivity in this study. Arguably it’s irrelevant as all patients are at suspected disease (that was one of the inclusion criteria) but in order to plot a ROC curve you need a sensitivity and a specificity so it appears that they have plotted the sensitivity at the point of describing the pain as ‘typical’.
So, the graphs are interesting but I would not focus on them. The key finding is the specificity.
The authors have looked at experience and in the paper discuss the fact that experienced clinicians have slightly better specificity than non-experienced. In terms of specificity experienced clinicians had a specificity of 65.8% vs non-experienced at 55.4%. However, I’m not sure how much use that is. This data tells me that they are both rubbish at it. Sorry folks, but I suspect that you are just as poor at determining this as other clinicians around the world (including me). Although there is a difference it’s clinically unimportant and I’m not sure that we can use experience as a determinant of probability in clinical practice.
It’s also interesting to see that 685/912 patients in this study were seen and assessed by doctors with less than 2 years of emergency medicine experience. International readers may find this surprising but it’s typical of many UK EDs and I would argue that in many hospitals the proportion of patients seen by juniors is even higher. On a marginally political note and with no wish to offend any hard working junior docs this study does show that UK EDs have a seniority staffing issue even when assessing chest pain patients who are clearly a high risk group.
So in summary?
The bottom line from me is close to that of the authors. You’re a bit rubbish at determining whether your patient has AMI or IHD on the basis of gestalt, clinical judgement or gut feeling. Don’t guess – test.
Questions for FRCEMers
1. On the ROC curve looking at the diagnosis of Acute Myocardial Infarction place a cross at the point where the test performs best (in statistical terms).
2. If 1000 patients were assessed by senior emergency physicians on the basis of this study and 300 of those patients were selected as having typical pain then how many of the 300 identified as having typical pain would have a final diagnosis of AMI assuming a specificity of 65.8%.
Before you go please don’t forget to…