GUEST POST by Sarah Seaton, research adviser from the RDS East Midlands

Prognostic modelling: what is it and how can it be used?

by Sarah Seaton, RDS East Midlands

Within the RDS, different members of team have varying expertise and interests. Mine is largely in observational research, and prognostic modelling, and this is generally what I advise on. Recently, I gave some staff training on this topic within the East Midlands and the below is a transcribed version of that session.

What is prognosis?

Prognostic modelling aims to estimate the risk of a future outcome in an individual. The outcomes are generally very specific. Examples include: mortality; disease progression and disease recurrence. Different factors (variables) which help predict the risk of this event occurring are used. This is known as multivariable analysis. There are two kinds of studies: development and validation. Here I will focus on development, but the validation studies are as important, if not more so. So, what we are doing is taking factors about the patient (e.g. age, weight, smoking status) and using it to predict an outcome (e.g. development of lung cancer).

Selecting factors that predict the outcome

The factors which are used need to be objective (easy to measure). These factors need to have data available at the time the prediction is made. For example, think of a situation where a GP might want to predict a patient’s risk of developing diabetes in the next 12 months. They have a fifteen minute consultation period where they will have very basic information available to them, for example: blood pressure, weight, height and ethnic group. These simple, objective factors are appropriate for the setting (GP practice) and are then used to predict the risk of diabetes.

Statistical parsimony

Keeping in mind our example of a GP appointment brings us to the idea of statistical parsimony. This is basically the idea that everything should be as simple as it can be (for medical people this is very much like Occam’s Razor). Choosing which factors predict an outcome can be a statistical exercise, but (and bearing in mind I am a statistician!) it is probably far more sensible to consider this as a clinical question, which needs literature reviewing and discussion amongst colleagues. The key issue here is to have as few variables as you can, for statistical reasons and for simplicities sake.

Statistical analysis and sample size calculations

This is not the time or place to discuss the methods behind this, but I will recommend a few excellent articles and books for anyone interested in further reading. Suffice to say, the work is not always trivial, and definitely needs a statistician. Sample size calculations for these types of studies are perhaps a little “unorthodox” and won’t be what you are used to, so make sure the statistician involved has done one before, or has read about them.

Presentation of results

The main issue of risk scores is presentation and ease of their use. There are some very good examples of risk score presentation and use. For example, take a look at the CRASH-2 “calculator” for predicting mortality after head injury. Although not beautiful, it is very simple to get a result out.

However, this remains a difficult area, and important consideration needs to be given to the fact that the model needs to be possible to easily implement. I think, possibly, there is great i-phone app potential here, if a sensible model is developed.

Getting funding

It is very hard to get funding for prognostic studies, and they are often “bolt ons” to other studies. In my opinion there is a good reason for this. It is not enough to tell a patient they are “at risk” of a disease. Something has to change in light of this new found risk. If nothing changes, then it was perhaps unethical to identify the patient when nothing could be offered, and a feeling of “so what?” is left behind. This, I think, is particularly true of the NIHR funders. So, as you write that application, or develop that model, think to yourself: what is going to change with this knowledge?

This is a whistle stop article, and much more detail is needed for anyone wanting to do one of these studies. A good starting point is:

Moons et al. Prognosis and prognostic research: what, why and how? BMJ 2009; 338.

Steyerberg, E. Clinical prediction models: a practical approach to development, validation and updating.

GUEST POST: Report from the COMET IV meeting Rome 19-21st Nov 2014 by Duncan Barron

My colleague, Duncan Barron, recently attended the COMET IV meeting in Rome and has written a guest post about the conference, the reasons for developing Core Outcome Sets (COS) and some of the presentations.

COMET IV meeting Rome 19-21st Nov 2014

Guest post by Duncan Barron, PPI Lead, Research Design Service South East

“Clinical trials are only as credible as their outcomes” (Tugwell, 1993).

The COMET (Core Outcome Measures in Effectiveness Trials) 4th meeting in Rome was held recently on the 19th-21st November in Rome and I was lucky enough to attend.

The COMET Initiative brings together people interested in the development and application of agreed standardised sets of outcomes, known as a ‘core outcome set.’ These sets should represent the minimum that should be measured and reported in all clinical trials, audits of practice or other forms of research for a specific condition.

There were speakers attending from all over Europe (with good representation from the UK) including patient representatives, some of whom presented on their experiences of being involved in developing outcome measures relevant to patients and their conditions.

The Core Outcome Sets (COS) are an “agreed standardised set of outcomes measures and reported as a minimum”. The aim of the COS is to increase consistency of reporting across trials and improve the quality of research.

The outcomes need to be appropriate, including being so to patients and the public. COMET provides guidance on what should be measured when developing a COS. This includes considering thing such as which domains to measure (eg. QoL; adverse effects) and considering the different ways of measuring outcomes.

A summary of some of the sessions

Roberto D’Amico spoke about his research in relapse remitting MS. He found that different reporting of data in trials meant it was impossible to analyse in a Systematic Review and therefore a waste of patient data. Therefore, there is a need for Core Outcome Scales.

Silvio Gavattini spoke on surrogate and composite end-points, highlighting that end-points of relevance to the patients (e.g. better QoL; decreasing mortality) are important. He gave an example of cholesterol as a surrogate for lowering MI, but that cholesterol is not a good surrogate for all drugs. In cancer, research decreasing the size of volume of tumour is usually viewed as good, but this is not always equivalent to a therapeutic end point. Survival is often the patients’ focus. Reducing tumour size can happen, but there can be considerable side effects which are not positive for the patient. Composite end points are a combination of individual end points in to one single measure (this can benefit in reducing the number of patients needed in a trial). However, the new measure can be influenced by the contribution of just one of the components which may be less meaningful overall. Therefore, each component should be equally meaningful.

Paula Williamson, from the Uni of Liverpool, spoke about the remit of COMET to encourage evidence based COS development and uptake. She highlighted a Systematic Review (Gargon et al (2014) of 198 COS (from 250 papers) that has been used to populate the COMET COS database – see here for more details. The systematic review revealed PPI in COS development in only 16% of the published COS. However, ongoing current PPI in COS is 90%. The Delphi technique is a regularly used component of COS (85% in ongoing COS work).

Trial funders including the NIHR Health Technology Assessment programme endorse COMET and recommend consulting the COMET database. “Where established Core Outcomes exist they should be included amongst the list of outcomes unless there is good reason to do otherwise. Please see The COMET Initiative website to identify whether Core Outcomes have been established.”
See here for more details.

Christian Apfelbacher (Uni of Regensburg) spoke about the methods used to develop COS in the field of atopic eczema in the HOME study.
1) Define the scope of the condition eg. setting (eg. trials); geo scope; stakeholders
2) Define the domains: what to measure
– used a Delphi to decide on domains
– 4 diff domains decided (eg. QoL; L/T flares).
3) Define core set of outcome measurement instruments
– beginning with a systematic review of existing tools (which are good enough instruments)
4) Dissemination and implementation:
– roadmap completed for one domain (“clinical signs”)
See here for more details.

Finn Gottrup, Prof of Surgery, Uni of Southern Denmark spoke about the development of Wound COS. Pervious COS had focused mainly on healing. His team undertook a Delphi study to identify a consensus on core outcomes for wound research. The work is still underway, but is linked with the European Wound Management (EWMA) group.

Holgar Schunemann (McMaster Uni, Canada) gave an interesting talk on Summary of Findings (SOF) Tables in published journals (as part of the The Grading of Recommendations Assessment, Development and Evaluation (short GRADE) Working Group). SOF tables help improves understanding of findings for journal readers. He is working on Interactive SOF tables (electronic) where the reader can manipulate way info is presented and Diagnostic SOF tables where the reader can test the accuracy of outcomes.

Peter Tugwell (Uni of Ottawa) & David Tovey (Cochrane Collaboration) spoke about core outcomes in pain research. Different Cochrane working groups all have an interest in chronic pain. These groups are beginning to develop alliances in developing outcomes across groups. The focus should be on the patient perspective for example pain interference with function versus Pain intensity.

Rosemary Humphreys spoke about a patient perspective in COS in the HOME (Eczema) study at the Uni of Notts. She highlighted the following:
– Patients can help define outcomes and new ones researchers haven’t considered
– Clinicians know about the condition but
– Patients know about the impact on their lives incl family, relationships.
– Involve patients early help design the COS
– Challenges to patients: language; jargon – produce a glossary for the condition of interest

Iain Bruce (Royal Manchester Children’s Hospital) MOMENT (cleft palate) study highlighted some lessons learned in involving patients and carers:
– Benefit from patients with direct experience of condition
– Involved the CEO of CLAPA, a patient support group in the cleft palate field
– Ensure patient voice is heard (lived experience)
– Ensure outcomes of most importance to patients are considered
– PPI is not an add on. It’s a fundamental theme!
– Challenges: engaging patients and use of language; practical meeting dates/ costs; scientific studies can be daunting
– Researchers need to explain why patients’ opinions are important.
– Use the same PES for clinicians + patients for Delphi survey
– Use SMOG calculator for PES and readability
– GRADE system 1-9 (Guyatt et al)
– Traffic light system for children
– All views are given equal value
– Advice to researchers developing COS: use lay language and variety of diff views; sell the benefits, tell them why it’s important for them to be involved; engage early at the design stage