Randomised control trial

Rosenberg WMC, et al.
Improving searching skills and evidence retrieval.
Journal of the Royal College of Physicians of London
November/December 1998; 557-563

Submitted by: -
Institution:
Date submitted:17th April 2009

Before CA, i rated this paper: 5/10

1 Objectives and hypotheses

1.1 Are the objectives of the study clearly stated?

Yes. The study evaluated the effect of a three-hour training session in formulating questions and searching databases.

2 Design

2.1 Is the study design suitable for the objectives

The primary study design was a randomised controlled trial. There was also a "before and after" element to the design. Both of these are suitable to answer the stated objective.

2.2 Who / what was studied?

The study was performed on 108 first year clinical (medical) students. Prior to the study most students had had experience of searching without formal training.

2.3 Was this the right sample to answer the objectives?

In the study's narrow focus of improving the use of EBM in medical students yes it was the right sample. It does not, however, evaluate this training session in other groups who may also require these skills.

2.4 Is the study large enough to achieve its objectives? Have sample size estimates been performed?

There was no sample size calculation performed. Instead all students were enrolled into the programme. Some of the results are presented with confidence intervals which do not cross which imlies that the study was large enough.

2.5 Were all subjects accounted for?

Yes.

2.6 Were all appropriate outcomes considered?

The outcomes were i) searching performance ii)quality of evidence retrieved and iii) student satisfaction. These would appear to be all the appropriate outcomes in view of the study question.

2.7 Has ethical approval been obtained if appropriate?

N/A

2.8 Were the patients randomised between treatments?

Yes. The two groups were randomised to receive EITHER i) a three-hour training session in question formulation and searching OR ii) nothing.

2.9 How was randomisation carried out?

A random numbers chart was used. This was blocked to ensure that equal numbers of students were allocated.

2.10 Are the outcomes clinically relevant?

Not so "clinically" but "educationally" relevant, yes.

3 Measurement and observation

3.1 Is it clear what was measured, how it was measured and what the outcomes were?

The students were asked to return computer printouts of thier sessions. The main outcomes were i) search score, ii) yield score and iii) student satisfaction. The search score was evaluated against a structured score sheet marked between 0-18. The yield score was a generated using the hierarchy of evidence scale for the single highest quality article between 0-4. Student satisfaction was measured on a 1-6 point score. In all cases higher score indicated "better" results.

3.2 Are the measurements valid?

Probably.

3.3 Are the measurements reliable?

Probably.

3.4 Are the measurements reproducible?

Ther was no mention of reproducibilty in this study. The search and yield scores would be reproducible elsewhere. The satisfaction score is not clearly described and it is unusual to have a discrete 6 point score of satisfaction.

3.5 Were the patients and the investigators blinded?

The investigators who generated the search and yield scores were blinded to the allocations.

4 Presentation of results

4.1 Are the basic data adequately described?

For this study probably.

4.2 Were groups comparable at baseline?

Again probably. There is explicit mention that "most" volunteers had previously had "weekly" experience of searching "without having received formal training". Later there is mention that one student in each group had recieved formal trainig before the study.

4.3 Are the results presented clearly, objectively and in sufficient detail to enable readers to make their own judgement?

Yes.

4.4 Are the results internally consistent, i.e. do the numbers add up properly?

Yes.

4.5 Were side effects reported?

N/A

5 Analysis

5.1 Are the data suitable for analysis?

Yes.

5.2 Are the methods appropriate to the data?

Yes. Both normal and non-normal data was analysed using the appropriate tests.

5.3 Are any statistics correctly performed and interpreted?

I have not repeated the test but the interpretation seems correct as presented.

6 Discussion

6.1 Are the results discussed in relation to existing knowledge on the subject and study objectives?

Yes. Comparison is made to other studies in this area and the results are in line with others.

6.2 Is the discussion biased?

7 Interpretation

7.1 Are the authors' conclusions justified by the data?

The conclusions were that the teaching session was an effective way of improving searching skills in terms of search results and quality. These conclusions are backed up by the data presented.

7.2 What level of evidence has this paper presented? (using CEBM levels)

7.3 Does this paper help me answer my problem?

This paper tells us that a formal course in question formulation and searching is a way of improving medical student skills in this area.

8 Implementation

8.1 Can any necessary change be implemented in practice?

Yes.

8.2 What aids to implementation exist?

There are other courses available that provide this kind of education although they have not been assessed in this way.

8.3 What barriers to implementation exist?

Barriers are time and financial constraints upon medical curricula.