BETs and CATs

The Best Evidence Topic (BET) Report

A modified CAT for summarising the available evidence in emergency medicine

• K Mackway-Jones
• S D Carley
• R J Morton
• S Donnan

Address:	Department of Emergency Medicine, Manchester Royal Infirmary, Oxford Road, MANCHESTER, M13 9WL
Correspondence:	Kevin Mackway-Jones, Consultant
Email:	kevin.mackway-jones@man.ac.uk

Introduction

In order to achieve the optimal care for patients in the Emergency department it is essential that their management is based upon the best available evidence. While the concept of evidence based medicine is widely accepted across many hospital and community specialties, emergency medicine has lagged behind in formalising the approach to evidence based practice.

One commonly used approach is that of the critically appraised topic (CAT) ^{1 2}. This was designed to be a one page summary of the evidence related to a particular clinical question. This established method would appear to be a reasonable starting point for developing a clinically based approach to examining the evidence in emergency medicine. Unfortunately problems have been identified. CATs work best when based on papers that stand up to rigorous critical appraisal. In emergency medicine the evidence that does exist is frequently not of high quality, and consequently the critical appraisal process would tend to discard many papers because of either methodological flaws or poor design. The experience of the critical appraisal journal club ¹³ was that an absolute requirement for high quality evidence meant that large numbers of clinical questions in emergency medicine could not be answered at all. It became apparent during the searches that evidence of lower quality was available, and that this could be assimilated to give the best available answer at the present time.

This realisation that it was necessary to use the best available evidence led to the development of the Best Evidence Topic (BET) report discussed below.

Design

The Best Evidence Topic reports (BETs) are constructed in four stages, based on the principles underlying all evidence based medicine. These stages are summarised in Box 1 below:

Asking the right questions

If a BET report is to have real meaning to practising clinicians it is important that each topic selected is rooted in the practice of emergency medicine. In order to ensure that this is the case a clinical scenario is developed to illustrate the topic of interest. This scenario will be familiar to the person preparing the report, and will represent a real clinical problem in emergency medicine.

In order to ensure that the question is well defined and answerable a three part question is prepared from the clinical scenario. This process is well described as a tool for evidence based medicine ^{2 4}. The three part question has the structure shown in Box 2.

It is vitally important to define the three part question well as this helps ensure that an appropriate search strategy can be formulated. An example of a general question of interest and a derived three part question are shown in Box 3.

Searching for evidence

In order to ensure that as much evidence as possible is included in the Best Evidence Topic report, it is essential that search strategies are sound and explicit. Two sources are explored in the construction of a BET. First the medline database is searched using a strategy constructed as outlined below. Secondly the Best Evidence CD-ROM ⁵ is searched to see if there are any relevant systematic reviews.

Medline

There are three parts to the Medline search. First it is necessary to use search terms to produce a subset of the medline database that contains all the papers relevant to the subject defined by the three part question. Second evidence based or methodological filters are applied if appropriate Finally the titles and abstracts of the "hits" are scanned to see which papers are actually relevant to the question posed. Techniques for searching the Medline database have been reviewed recently ⁶.

The subject search

In order to achieve the aims of the BET report (finding and summarising the best evidence) it is important that the subject search has a high sensitivity (that is it has a high likelihood of retrieving all relevant items) so that important evidence is not overlooked. It may be necessary to use a number of different search strategies (that is use different search terms and combinations of terms) as well as using search terms that cover a slightly broader question. Greater sensitivity is achieved by using a combination of free text and text words together with medical subject headings (MeSH). A particularly useful feature enabling increased search sensitivity is the explosion command; this maps individual MeSH term on to other related MeSH terms. Since some 50% of the articles on medline are misclassified by subject heading, this ability to search for related terms is invaluable.

If the highly sensitive search yields too many "hits" to be manageable, then it may be necessary to increase the specificity (the proportion of documents which are relevant) by refining the search. In general techniques for increasing specificity are the opposite of those for increasing sensitivity. Thus the question needs to be better defined and more specific terms need to be used in free text. Other techniques for increasing specificity include combining terms using the boolean operator AND to combine different aspects of the question, and use of the Medline LIMIT command to set various general limits of the search such as language, species or publication type.

As has been mentioned above the proper construction of a three part question holds the key to the search. A properly constructed question will allow a sensitive search to be carried out in a particular area of interest, so that all relevant papers relating to that area can be gathered and appraised.

Applying filters

If a properly sensitive search carried out in a relevant area still yields a large number of hits then filters can be used. These are search strategies that have been designed to perform particular tasks such as finding randomised controlled trials or research overviews. A number of these are available in the public domain and have been published ⁶ or can be found at appropriate web sites ⁷.

The vast majority of questions of interest to Emergency Physicians at the present time (and therefore those most likely to appear as BET reports) involve either treatments (interventions) or diagnosis. An appropriate filter for intervention studies in the context of BET reports is a maximally sensitive randomised control trial filter - one such is shown in Box 4.

1	Randomized controlled trial.pt.
2	Controlled clinical trial.pt.
3	Randomized controlled trials.sh.
4	Random allocation.sh.
5	Double-blind method.sh.
6	Single-blind method.sh.
7	or/1-6
8	Animals.sh not Human.sh.
9	7 not 8
10	Clinical trial.pt.
11	Exp Clinical trials
12	(clin$ adj25 trial$).ti,ab.
13	((single or double or treble or triple) adj25 (blind$ or mas$)).ti,ab.
14	Placebos.sh.
15	placebo$.ti,ab
16	random$.ti,ab.
17	Research design.sh.
18	or/10-17
19	18 not 8
20	19 not 9
21	Comparative study.sh.
22	exp Evaluation studies/
23	Follow up studies.sh.
24	Prospective studies.sh.
25	(control$ or prospective$ or volunteer$).ti,ab.
26	or/21-25
27	26 not 8
28	26 not (9 or 20)
29	9 or 20 or 28

Diagnostic questions require a different approach since randomised controlled trials are not in general an appropriate method of investigation. In such cases an appropriate diagnostic evidence based medicine filter can be used. Diagnostic evidence filters and other evidence based filters appropriate to different types of questions (such as epidemiological or aetiological ones ) are available from the Centre for Evidence Based Medicine and other appropriate sources ⁷.

Making the searches explicit meant that they can easily be repeated to update the BET at a later date.

Scanning titles and abstracts

Once the subject search has been completed and any necessary filters have been applied a number of papers will fulfil the search criteria. There is no guarantee that these will all be relevant to the question posed and a final scan of titles and, if there is any doubt, abstracts will be necessary. Ideally a maximum of fifty papers will be left for scanning in this way, but on some occasions more than this number will remain despite increasing the specicivity of the search. It is better to have a higher number of papers to scan than to miss relevant papers by over refining the search.

At the end of the scanning process a number of papers will be discarded as "irrelevant to the question". All the other papers remaining will need to be critically appraised. This process is described below.

The overall search strategy used in the construction of best evidence topic report is summarised in Box 5.

Appraising the evidence

Evidence based medicine is not about dismissing all evidence that has not been gained by randomised trials and meta-analyses ⁸. Some types of question are not best answered by these approaches and appropriate evidence needs to be found for these. Furthermore for many questions in emergency medicine there is no appropriate research that stands up to rigorous critical appraisal. Best evidence topic reports are designed to find the best evidence that exists to answer a particular question. If high level evidence does exist then the question posed can be answered with some certainty. If high level evidence does not exist then best evidence available can be summarised, and the failure to find good evidence can be used as a spring board for appropriate research.

The first critical appraisal filter to be applied to papers that are found by the search is methodological. The question underlying the methodological filter is: "what is the appropriate design of research to answer this question?", and, if more than one approach is possible the second question is: "what is the relative merit of each of the possible methods?" In the case of questions concerning interventions the hierarchy of evidence shown in Box 6 is generally accepted ⁷ and is used in constructing BETs. Although other hierarchies have been proposed ⁹ the concepts underlying them all are the same.

The highest available level of evidence is used to construct the BET - thus level I evidence will be used if it exists; if there is no level I evidence then level II will be sought and so on.

Once papers containing the highest level of evidence available have been identified, they are critically appraised. The critical appraisal methods used are standard and have been published in a number of places ^{2 10 11}. Key questions include methodological and analytical soundness.

I	Strong evidence from at least one published systematic review of multiple well-designed randomised controlled trials
II	Strong evidence from at least one published properly designed randomised controlled trial of appropriate size and in an appropriate clinical setting
III	Evidence from published well designed trials without randomisation, single group pre-post, cohort, time series or matched case-control studies
IV	Evidence from well designed nonexperimental studies from more than one centre or research group
V	Opinions of respected authorities, based on clinical evidence, descriptive studies or reports of expert consensus committees

After critical appraisal a paper may be found to be so flawed as to be unusable and will thus be discarded as of "insufficient quality for inclusion". If a paper is not fatally flawed or if, despite being flawed it represents the best evidence that is available despite flaws it will be included in the BET report. Comments about study weaknesses will be included in the last column of the table summarising the evidence.

The overall strategy for critical appraisal used in the construction of best evidence topic report is summarised in Box 7.

Summarising the evidence

In order to keep a record of appraised topics a summary of the evidence is made in the form of a best evidence topic report (BET). The first of these BETs appear in this month's Journal. The reports follow a standardised format. This includes the title, the clinical scenario, the derived three part question, the detailed search strategy and the results of the search. The number of "hits" is followed by the number of papers which were not relevant to the question and the number which were of insufficient quality for inclusion (following critical appraisal). Those papers which are included in the best evidence topic report are reported in table format. The table includes the author date and country of the research, the patients groups the study type (and the level of evidence), the outcomes that were investigated, the key results for each outcome and the study weaknesses that were identified. A further analysis of the results of the best papers may be given if the results allow this. Free text comment about the search and its outcome is then given, followed by the clinical bottom line (the answer to the original question). The references of the relevant papers are given in full.

The template for the best evidence topic report is shown in Box 9.

Best Evidence Topic Report
Title
Report by
Search checked by
Clinical scenario
Three part question
Search strategy
Search outcome	x papers found of which y irrelevant and z of insufficient quality for inclusion
Relevant papers

Author, date and country	Patient group	Study type (level of evidence)	Outcomes	Key results	Study weaknesses

Comment
Clinical bottom line
References

Box 9. Template for a BET report

Each BET is attributed to a main author and also a subsidiary author who is responsible for checking the search strategy and outcome.

Negative BETs

It is foreseen that a significant number of topics will fail to produce any evidence to answer the clinical question posed. These "negative" BETs are indicative of areas for further research, and a list of the questions for which no answer at all can be found should be circulated alongside "positive" BETs.

Discussion

BETs are an attempt to formalise evidence based practice in the emergency department. Their very name indicates the uncertain nature of much of the "evidence" in this area. The hope is that by basing BETs on real problems in emergency medicine, real questions will be answered with the best evidence available. This process is essential as real questions about patient care arise on a daily basis in the emergency department, yet there is rarely time to search for evidence at the time of a patient presentation. It is therefore essential to be pro-active in the approach to evidence based practice. Emergency medicine is still a relatively young specialty and there is a paucity of high quality evidence for many aspects of practice, BET reports will therefore not be confined to questions regarding new or novel practices and will examine the evidence for well established practices as well.

The BET report does have its weaknesses. While it reports the best available evidence obtained through literature searching on medline and other information resources, it does not examine unpublished literature in the way that systematic reviews do. It is unlikely that systematic reviews will become available for many of the more minor complaints that present to the emergency department for some time. Even if they did, as the BETs will show, it is unlikely that much of the evidence would pass the methodological filters used in systematic reviews.

Best Evidence Topic reports contain the best evidence that can be practically obtained by busy practising clinicians and should be used to inform best practice. They can be disseminated around members of journal clubs, training schemes and can be published in peer reviewed journals or web sites. It is hoped that BETs will demonstrate both the strengths and the weaknesses of the evidence base on which the practice of emergency medicine is based.

References

1. Sauve S, Lee HN, Meade MD et al. The critically appraised topic: A practical approach to learning critical appraisal. Ann Roy Coll Phys Surg Canada 1995; 28 396-398.

2. Sackett DL, Richardson WS, Rosenberg W et al. Evidenced-based medicine: How to practice and teach EBM. New York. Churchill Livingstone 1997

3. Carley SD, Mackway-Jones K, Jones A et al. Moving towards evidence based emergency medicine: use of a structured critical appraisal journal club. J Acc Emerg Med 1998;15:220-222.

4. Richardson WS, Wilson MC, Nishikawa J et al. The well built clinical question: A key to evidence-based decisions (editorial). ACP J Club 1995; 123 A12-13.

5. Best Evidence CD-ROM. London. BMJ Publishing. 1997.

6. Greenhalgh T. The Medline database. Br Med J 1997; 315 180-183

7. Centre for Evidence Based Medicine http://cebm.jr2.ox.ac.uk/

8. Sackett DL, Rosenberg WMC, Gray JAM et al. Evidence based medicine: what it is and what it isn't. Br Med J 1996; 312 71-72.

9. US task Force on Preventative Health Care in Stevens A, Raftery J (Ed) Health Care Needs Assessment: The epidemiologically based needs assessment reviews. Second Series. Radcliffe Medical Press. 1997.

10. Crombie IK. The pocket guide to Critical Appraisal. London. BMJ Publishing. 1996.

11. Greenhalgh T. How to read a paper: The basics of evidence based medicine. London. BMJ Publishing. 1997.