Face to Face

My tour of the RDSs – which commenced with my trip to Leicester to observe RDS EM’s NIHR Fellowship Day – continued recently with a visit to RDS London. I was invited by Lauren Bell to observe their Proposal Feedback Forum and to meet Lauren and her RDS London colleagues.

It has always been one of my intentions, with this blog, to reach out to other RDS staff and advisers across England. Although we all offer the same type of support and guidance to researchers applying for funding for health and social care research, our different areas are quite distinct – each with its own needs and requirements when it comes to how this support is delivered. RDS London serves, as you might expect, a tremendously research-active part of the country, and I was excited by the opportunity to observe them in action.

I’ve written before about what I view as the main ‘USPs’ of the RDS. Peer review is, in my mind, one at the very top of that list: the fact that researchers, regardless of stage or experience, can access free help and advice from methodological experts in a wide variety of areas. Offering advice on a one-to-one basis is a core part of the way all RDSs do business: a researcher who approaches us asking for advice will typically liaise with a single adviser, who will be their main point of contact with the wider RDS. However, in addition to this lead adviser, researchers can access a breadth of methodological expertise, as required. One of the ways that many of the RDSs choose to expand upon this is to hold meetings where advisers act as a funding panel, reading, assessing and giving feedback to researchers before they submit their applications to be faced with the real deal. This trial run can be tremendously valuable to researchers – allowing them to fine-tune their applications and, on occasion, make the sometimes difficult decision that more work is required before they’re ready to submit for real.

One of the main difficulties faced by researchers who have had applications rejected by funders is the paucity of information about how to improve their applications prior to resubmission. I’m a big believer in re-designing and re-submitting, as I’ve written about before. However, to many clinical researchers who are relative novices when it comes to research, this type of rejection can be devastating and mean that they turn away from research, leaving their important, clinically-driven questions unanswered. This is unfortunate as these are the very researchers that the NIHR is keen to engage and to encourage to conduct research.

In the South East, we hold a regular Pre-submission Panel, where we ask researchers to submit draft funding applications which are reviewed by a large group of our advisers. Our collective advice is then fed back to the researcher by the lead adviser. In essence, we mimic a funding panel but with the added advantage of offering detailed advice on how to improve applications. RDS London do something similar, as do many other RDSs.

In the past couple of years, in addition to this type of review, RDS London have offered the opportunity for researchers to come and present to their panel in person. They still submit their near-to-completion funding application for review, but they also prepare a 10-minute presentation, including any specific questions they want answered, to present to RDS panel prior to a larger discussion of the proposal. This allows for immediate feedback to the researcher about their plans and their questions, and also for a discussion to take place with them about any concerns identified by the panel.

The particular panel I observed had 2 researchers presenting – one an experienced academic researcher with a clinical background and the other a practicing clinician who was a more novice researcher. After each of the of their presentations, they would get comments and questions from a range of methodologists, covering statistics, health economics, Patient & Public Involvement (PPI), qualitative research and health psychology. A discussion would ensue during which various aspects of the question, application and design would be clarified or altered. It was a fascinating process to observe, not least because I could see the benefit of the process to both the experienced researcher and the less experienced. The entire process was positive and encouraging, with an emphasis on making the research application stronger and more competitive. There was no doubt that both researchers left having received valuable feedback which will hopefully translate into successful applications, better research and ultimately benefit to patients.

There is definitely something to be said for having the researcher present to face a panel and answer questions directly. Issues can often be resolved far more easily in this kind of environment when compared to via email or even face-to-face meetings with just a single adviser. It can also help with what is one of my biggest frustrations when working with a researcher: what to do when your advice is ignored. There is something about advisers coming together in this way that makes it harder to brush our advice aside.

It is also a wonderful opportunity of personal development, and RDS London recognise and take advantage of this. It is reassuring for more junior or newly-appointed advisers to observe more experienced colleagues and learn from the type of advice they give.

For my part, there are three take home messages. First, that RDS advice can make a real difference to the competitiveness of research applications regardless of how experienced the researcher may be. Secondly, that the format used by RDS London is something worth exploring in my own patch down in the South East. And, finally, that despite our geographical barriers, we are one RDS, offering the same high-quality advice and support to researchers across England.

If you’re a researcher in London working in health or social care, do get in touch with RDS London. If you can, do ask to present at their Proposal Feedback Forum.

If you’re elsewhere in England – there is an RDS for you! Find out where here.

If you’re in the South East – Surrey, Sussex or Kent – come talk to me and my colleagues!

Time to Write?

Finding time to focus on research can be challenging. Research is, for many of the clinicians with whom I work, something that is done in their ‘free’ time. They recognise the value of research and believe passionately about delivering evidence-based care to their patients, yet it is not something they can necessarily dedicate time to during their working week. They have heavy clinical responsibilities and having the time to think about their research plans can be difficult; more difficult still is finding the time to actually sit down and tackle one of the NIHR research programme’s detailed application forms.

And, yet, these are the very people the NIHR is trying to reach. As I talked about here, the NIHR is a funder of research for, and from, the NHS. It is these clinical researchers – those who know intimately the problems faced in clinical practice – who we want to encourage to put forward questions to address through research to find evidence-based solutions that they can use in their practice to the benefit of their patients.

There are a number of ways to tackle the problem of time. I believe that working with an RDS adviser, like myself, can help make the process of designing a research study and submitting a funding application easier. As I’ve said before, there are many advantages to making use of the services the NIHR Research Design Service offer to health and social care researchers free of charge. I won’t go into detail of these here, but our collective experience with funding applications and funders, our individual expertise as researchers, and our ability to offer peer and lay review are just a few ways we can support busy clinicians who wish to pursue research.

Another way of making time for research is to actually allocate a block of it specifically for writing. Bits and pieces, written and read here and there in evenings and weekends can make for a difficult to follow funding application. It is surprisingly easy to spot when applications have been written by different people and no one has had the time to go though the entire document and pull it all together. But, often, there are few alternatives.

Last March, I was fortunate enough to be invited to speak and advise at a Research Retreat run by the Research Design and Conduct Service (the Welsh equivalent of the RDS). I wrote about my experience of the event specifically, and about the usefulness of Research Retreats in general, on my return. I found the event positive, productive and rewarding. It was a real pleasure to have an extended amount of time to dedicate to the development of specific research proposals and to observe just how far the research teams had progressed in the development of these over the course of just 3 days. With dedicated time to write, and experienced advisers to offer support, real progress was evident.

RDS SW have run a similar type of Retreat annually since 2009. Like my experience in Wales, they have received very positive feedback. Participants, in particular, have welcomed the dedicated time to concentrate solely on their research with RDS staff available throughout to offer expert support and guidance. It is always hard to come up with reliable measures of success when it comes to funding applications (as I’ve commented on here). It is also difficult to know for sure what leads to success. That said, RDS SW are aware of 20 applications, worked on during their Retreat, which were successfully submitted to funders. Of these, 9 were funded. This, if you’re at all familiar with the typical success rates of the various NIHR funding programmes (RfPB, for example, has a success rate around 20%), is pretty impressive.

Over the past couple of months, I’ve been involved in a group made up of RDS staff from across the country with the aim of offering something similar to health and social care researchers in England. This is the first time such an event has been offered nationally, but it is our hope that it will give busy clinicians the time, space and support to act on their research aspirations and produce an almost submission-ready funding application.

For this first venture, teams of 2 or 3 researchers can apply to take part in a 5-day RDS Residential Grant Writing Retreat to be held from 13th – 17th June 2016 in Bath. More details of the entry requirements, application and selection process, cost, and venue are available here.

If you think this is something that might interest you and your fellow researchers, do have a look. I can only speak from my own experiences of a similar type of event, but I do believe there is much to be gained by participating. At the very least, you are guaranteed dedicated time to write, with the help and guidance of experienced RDS advisers. And working in a supportive environment with other committed researchers and RDS advisers is likely to bring about even more.

NIHR Fellowships & ‘One NIHR’

I was lucky enough recently to visit my colleagues at the Research Design Service in the East Midlands (RDS EM), in Leicester to be specific. This was a first for me – to actually meet in person other RDS colleagues in their own patch. As I’ve written about before, we have a so-called ‘National Training Day’ the is held roughly every 2 years, but attending that has been, until now, been my only experience of meeting other RDS colleagues.

Social media can be a great enabler of actual face-to-face contact. My very first ‘follower’ on twitter was Sarah Seaton, an RDS adviser and NIHR doctoral fellow. Through twitter we’ve had numerous conversations on a wide range of health-research and RDS-related topics. As I’ve commented on before, the health research community on twitter is lively and varied. This interaction then led up to a hugely enjoyable Google hangout via the NIHR Hub (where there may or may not have been virtual hats involved at various points in proceedings), and finally to us deciding that it would be a good idea to arrange to meet up in person. Although we are badged as ‘one RDS’ and ‘one NIHR’, the fact still remains that many of us on-the-ground advisers don’t really have all that much contact with the wider, national RDS or, indeed, NIHR.

One of the things that Sarah does in RDS EM is to arrange a 2-hour NIHR Fellowship event every year. As she is herself an NIHR fellow, as well as being a RDS adviser, she is in the perfect position to do so. As we in the RDS SE were due to hold our fellowship event, for the first time, in a couple of weeks, it seemed like the perfect opportunity for me to visit, observe the event, and meet Sarah and her RDS EM colleagues.

Now, I can’t really write this particular post without sharing what I learned from the event. NIHR Fellowships, to quote Dawn Biram from the Trainees Coordinating Centre (TCC) who spoke, fund research and training to develop the research leaders of the future. With a variety of pathways open – for medics, clinically-trained health professionals, and non-clinical health researchers – at a variety of levels – Masters right the way up to Senior Lectureships – they are certainly a something worth considering when planning your research. And, as with all health and social care research applications, they are something on which your local RDS can provide advice, support and guidance.

Sarah had arranged for a range of speakers, all of whom play a different role in the fellowships. There was Dawn from the TCC, the secretariat which manage the fellowship scheme for the NIHR, Matt Bown, a current panel member for the doctoral fellowship pathway, Rhiannon Owen, a current NIHR fellow, Emma Watson, a doctoral fellow with Kidney Research UK, and finally Clare Gilles, an RDS EM adviser.

I was tweeting pretty much continuously during the event as there were so many tips and hints from each of the speakers. I use #NIHRtips, if anyone is interested, as this is a hashtag that I, and others, use for all sorts of NIHR and funding-related advice.

The overwhelming message from all the speakers was just how long it takes to put together a competitive application. These are huge endeavours and require input not just from yourself, but from a range of people both within and outside your NHS Trust or HEI. It is not unexpected for a fellowship application to take 6 months to a year to get right. Make sure you look at the guidance from last year’s competition and use that as a starting place – don’t wait until the new competition opens to put pen to paper (or fingers to keyboard). Try to get your hands on a successful fellow’s application form, so you can see what a really good application looks like (and just how long and complex it is!).

Applications are judged on 3 main things: you as an applicant, your research project & training plan and your institution (both NHS and HEI). Each one of these – person, plans, place – will require careful thought and preparation. So, update your CV, get that publication in, and apply for that small pot of funding to kick-start things. Talk to your local RDS about your plans and carry out a PPI consultation on your research ideas and design. Think about what other organisations you should involve – a clinical trials unit, the clinical research network, a charity or patient stakeholder group. Look at your options for HEIs, supervisors, and mentors. Approach the top people in your field and get advice. Get an idea of what training is out there and plan to attend the best there is available. Get your research networks started now.

There were also lots of tips about filling in the actual form. Panel members are only given a couple of weeks to shortlist 15 or so applications, so do whatever you can to make yours stand out. Be neat and careful with your spelling and grammar. Be consistent in your use of numbering and make sure your references are correct. Use bold, italics and underline to make your application clear and easy to read – don’t just have unbroken lines of dense text.

If you’re invited to interview, this is your chance to really demonstrate that you live up to your application in person – candidates are only interviewed if they are potentially fundable on paper. Make sure your presentation is second-perfect and set up as many mock interviews as you possibly can – your local RDS can help you with this. Google your panel and make sure you’re familiar with them, their research interests and their institutions. You can probably work out fairly easily who will be leading your interview, so anticipate what you might be asked. Be ready to defend your research plan, but don’t ignore input from the panel either. These people are experts in their fields, so give in to their greater experience if they suggest things to you. While you’re speaking, be confident and come out and move around the room. Remember, the panel is looking for future research leaders, so show your passion for your topic. Finally, your last slide will be left up during the Q&A portion of you interview, so use it to leave the panel with the message you want to convey.

Such events are invaluable and getting to talk to people directly can be really helpful if you’re planning to apply for an NIHR fellowship. Contact your local RDS and see if they’re doing something. Even if they’re not, they’ll have the experience of working with many NIHR fellows at various levels. They can help you with your application and may also be able to put you in touch with others who’ve been on the same journey.

From my point of view, it’s this collective experience that is so very valuable. When we share our experiences across the NIHR, we become stronger as a research community. We can help and support each other, offer advice and critical evaluation, and provide a network and a support group for researchers at all levels of the NIHR research pathway. I gained a lot from my visit, both in terms of knowing more about the NIHR fellowships, but also about what it means to be part of the larger NIHR and how, as a research community, we all have our parts to play in ensuring that patients and the public benefit from the very best evidence-based health and social care.

Adaptive Trials

We recently had a speaker come talk about adaptive clinical trials. It was a good seminar with lots of clear, real-life examples and it was well-attended by RDS advisers and researchers based both in universities and in the NHS. I won’t go into detail about the content of the seminar – people far better qualified than me have already done so in abundance – but it did get me thinking about how such designs could be used by the researchers with whom I and my colleagues work.

A large proportion of the health research funded by the NIHR, through such programmes as HTA, EME and RfPB, are clinical trials. And there are many issues to consider when designing an RCT, some of which I’ve discussed before. Most researchers engaged in such projects have input from one of the many Clinical Trials Units (CTUs) and have experienced statisticians, data managers and trial managers on their research teams. And, of course, many RDS advisers who support such studies ourselves have this experience.

What is particularly relevant about adaptive trials, not least in the context of providing advice for health researchers, is the opportunity for flexibility they offer. The goal of an adaptive design is to enable researchers to learn from the accumulating data and make key design changes accordingly as things progress. Usually, while a trial is still in progress, we do not look at any of the data. However, the accumulating material may be more informative than that available before the trial had started. And it is this previously available data – things like effect sizes, recruitment strategies and randomisation, dosages – on which the design of the trial was based. The idea behind adaptive designs is that a trial can be improved by making use of interim data to refine certain aspects of it.

Improvement can mean a number of things, usually to do with making a clinical trial more efficient. It can, for example, lead to doing away with a treatment arm if a particular dosage or intervention is demonstrated to be inappropriate. It can also mean identifying with greater accuracy the number of patients needed or lead to refinements in the recruitment strategy, target population or treatment randomisation. It can even lead to decision-making about, amongst others, key objectives, end-points, test statistics, or subsequent phases of the research.

The trick to it all is in the pre-specification: making it very clear in the trial protocol what purpose will be served by carrying out an interim analysis, when it will be done, on what measures, by whom, and to what end. You need to put procedures in place to ensure that the blind holds and limit the people who see the data at this pre-specified interim stage.

As an RDS adviser, I can see merit in this approach. The ability to carry out a sample size re-estimation, for example, is something that I think could benefit many projects. So too is the ability to drop inferior treatment groups or use preferential randomisation. There are, of course, many other options for adaptive designs, and these are just two examples that I can think of which could be applied to many projects.

The drive for efficiency in health research is nothing new. I’ve written about it before and there are examples of it in the NIHR funding programmes themselves – a recent example being the call the HTA programme issued in the summer of 2014 for ‘efficient study designs’. And, as always, an integral part of any NIHR funding application is the demonstration of the value for money of the research itself. This is, of course, not to say that adaptive trials are the answer or even appropriate. Such designs come with their own risks – errors due to their greater complexity, more time needed in the planning stages, uncertainties around the ethical implications, and the need for greater regulatory review, to name just a few.

As always, the design of a study needs to fit its research question. Adaptive trials offer an intriguing option when uncertainties mean refinements are required during the trial itself in order to optimise its design.

RDS USPs?

I’ve done a few short talks recently about the NIHR Research Design Service and the services we offer. It has been an interesting exercise – to try to distill in a short space of time the hours of concentrated effort that RDS advisers put into the applications we support and the variety of guises our advice can take. In many ways, this has been a marketing exercise – detailing the ‘unique selling points’ of the RDSs in a way that would appeal to NHS clinicians who are either already involved in research or are interested in so being.

I’ve talked numerous times before about the support RDS advisers can give, but it has been interesting to really focus in on what is unique about the RDSs. I think it comes down to 3 things: (1) collective experience, (2) peer review, and (3) lay review.

First of all, collectively, we have experience with literally hundreds of funding applications from a wide range of applied health and social care research funders. We’ve seen what works as well as what doesn’t. We’ve worked on bids that have been funded first time around and ones that have been funded fifth time around. We’ve worked on a huge variety of research topics, we know our remit as advisers and our strengths as researchers and methodologists in our own right. And, perhaps most importantly of all, we occupy a unique position of being committed to an applications success and yet not being part of the research team and, as such, not too close to the research idea. Of course, this is not to say that we can therefore guarantee a particular application’s success – but making use of our expertise certainly can’t hurt.

Peer review is, in my opinion, another big USP. As far as I am aware, all of the 10 RDSs in England offer some form of formal peer review. Like many, the RDS SE holds a regular meeting, which we call a ‘pre-submission panel’. Advisers from across our region get together and review funding applications in detail in a way that mirrors as closely as possible the assessment process of the NIHR research programmes. This trial run gives researchers an invaluable opportunity to address any potential weaknesses identified in their application prior to submission. It also allows them to make the, sometimes vital, decision not to submit just yet.

Thirdly, lay review is something that many RDSs offer. At the RDS SE, we offer researchers the opportunity to have their applications reviewed by 2 lay reviewers. This is also linked to the peer review process – where 2 lay reviewers sit on our pre-submission panel. Again, this reflects to some extent the NIHR’s assessment procedure and also allows researchers to hear directly from lay representatives their views on the research question and the research team’s plans to address it.

Now, I don’t in any way believe that consultation with an NIHR RDS will necessarily mean a successful application for funding. Certainly this would be impossible given that, in the South East at any rate, our aim is to provide advice and methodological support for as many projects as possible.

However, this does bring us to a fourth USP: we provide our service to researchers free of charge.

So, my take home message is this: if you are preparing an application for health or social care research funding, come and talk to us.

From frustration to funding

I attended the R&D meeting of one of our local NHS Trusts a few months ago where the main agenda item was the organization of the Trust’s annual research conference. During the meeting, two of the members started a fascinating discussion about the research process, which stayed with me long after the meeting had finished.

They were talking about the very beginning of the research process – about that spark of inspiration that can start an entire programme of research and from where such a spark can come. Interestingly, rather than talking about a fascinating theory or a novel approach, the fantastic force of inspiration they were discussing was that of simple frustration. The sources of this frustration encompassed a wide range of issues – from frustrations with clinical practice, patient progress and lack thereof, right through to frustrations with service organisation and delivery.

I found this a fascinating idea and it is one that is mirrored by the provision of research funding by the National Institute for Health Research (NIHR). The NIHR is largely a funder of NHS-based research – the funds are held by NHS Trusts rather than HEIs, the principle investigators are usually NHS clinicians, and the research topics prioritised are those which will bring about direct benefit to patients. This focus makes the NIHR unique amongst the variety of other funders of research that exist, including the research councils and numerous charities. The NIHR is all about research for, and from, the NHS. Research topics are prioritised by various panels precisely to ensure that the topics believed to be of greatest import to patients and NHS staff are those that receive funding.

When discussing funding applications with researchers, my colleagues and I often talk about the ‘story’ behind the research. This story is a vital one to tell when making an application for funding. It straddles various parts of the application form – the background and rationale sections, the PPI sections, the outcomes sections. Identifying the source of your spark of inspiration is an important part of telling this story. And, often, this spark is one of frustration with how things currently stand and the desire to change things for the better.

I wrote a blog post a while ago about turning the frustrations associated with poor feedback following funding application rejection into fuel to redesign and reapply. It seems only fitting to now attend to the other end of the spectrum and say that the frustrations felt about any issues of clinical practice can equally be used to inspire research questions. And, as with any research question, your local RDS can help refine and direct these frustrations into a research question appropriate for NIHR funding.

Don’t let frustrations get you down; use them to fuel your research questions and apply for funding to identify solutions. By so doing, you may uncover solutions that will improve care for patients as well as easing the frustrations of staff.

***

ETA: As Nikki points out below, the NIHR does fund non-NHS research – the Public Health Research programme being a prime example of this.

Research Retreats – do they work?

I was fortunate recently to be invited to speak at a Research Retreat run by the Research Design and Conduct Service (RDCS – another acronym for you!). The RDCS operates pretty much the same as the Research Design Service (RDS) does, but in Wales rather than in England. There are some differences between us – the acronyms change (rather NISCHR than NIHR; RfPPB rather than RfPB) and there are slight differences in the NHS set-up (Health Boards rather than NHS Trusts; differences in how support costs are provided)– but nothing too different overall. Most importantly, however, the research and funding landscape remain pretty familiar.

The concept of a research retreat is not a new one. I know that certain RDSs in England, RDS SW for example, have been running them for years. But it was not something I had been a part of before and I was keen to see one in action. How productive would researchers be when provided with a few days dedicated research time, with the support of a range of methodologists and advisers, away from the pressures of their clinical responsibilities?

The RDCS team had done their homework. They’d booked out a small country hotel that had lots of room where the teams could work, wonderful food and, perhaps most importantly, a continual supply of tea and coffee. The timing was also good; there was a funding deadline that many of the teams were working toward. And the group was good – 9 teams of researchers in total, making it a small enough group to get some really interesting discussions going yet large enough to allow groups to work independently on their own projects.

There were a couple of talks each day – my own on mixed-methods research, an invaluable one on project management and costings, a fascinating talk by a PPI representative, and a thorough run-down of the funding remit and requirements. These gave the teams direction, tips, and allowed us to get together for group discussions.

The rest of the time was dedicated to writing, with advisers circulating to help out where needed. I spent time with almost all of the teams, as did the other advisers. It was an illuminating process: to give the advice as I normally do, but then, instead of waiting a month or so for the next meeting to see how the team had progressed, to meet with them again later on that day to work on the next part of the project. It was like putting in several months worth of advising work into a couple of days. It was also extremely fulfilling to see the teams grow in confidence and enthusiasm as they had the time together to work on their projects with expertise on tap to really move things forward.

Of course, the proof of it all will be in the outcomes of the projects. How will the 3 applications to RfPPB fare? Will the research teams at the start of their research journey make it all the way through to submission and ultimate project funding?

I am optimistic.

When given the time and space to connect with the others in their team and really focus on their research question, study design and funding application, their dedication to and enthusiasm for their projects was tangible. These were people who had seen problems in their clinical practice and were driven to explore solutions by research in order to make things better for their patients.

And that, in a nutshell, is what fundable health research is really all about.

***~~~***

I have to say a big thank you to Mark Kelson, Kerry Hood and everyone else at the RDCS for inviting me!

GUEST POST by Sarah Seaton, research adviser from the RDS East Midlands

Prognostic modelling: what is it and how can it be used?

by Sarah Seaton, RDS East Midlands

Within the RDS, different members of team have varying expertise and interests. Mine is largely in observational research, and prognostic modelling, and this is generally what I advise on. Recently, I gave some staff training on this topic within the East Midlands and the below is a transcribed version of that session.

What is prognosis?

Prognostic modelling aims to estimate the risk of a future outcome in an individual. The outcomes are generally very specific. Examples include: mortality; disease progression and disease recurrence. Different factors (variables) which help predict the risk of this event occurring are used. This is known as multivariable analysis. There are two kinds of studies: development and validation. Here I will focus on development, but the validation studies are as important, if not more so. So, what we are doing is taking factors about the patient (e.g. age, weight, smoking status) and using it to predict an outcome (e.g. development of lung cancer).

Selecting factors that predict the outcome

The factors which are used need to be objective (easy to measure). These factors need to have data available at the time the prediction is made. For example, think of a situation where a GP might want to predict a patient’s risk of developing diabetes in the next 12 months. They have a fifteen minute consultation period where they will have very basic information available to them, for example: blood pressure, weight, height and ethnic group. These simple, objective factors are appropriate for the setting (GP practice) and are then used to predict the risk of diabetes.

Statistical parsimony

Keeping in mind our example of a GP appointment brings us to the idea of statistical parsimony. This is basically the idea that everything should be as simple as it can be (for medical people this is very much like Occam’s Razor). Choosing which factors predict an outcome can be a statistical exercise, but (and bearing in mind I am a statistician!) it is probably far more sensible to consider this as a clinical question, which needs literature reviewing and discussion amongst colleagues. The key issue here is to have as few variables as you can, for statistical reasons and for simplicities sake.

Statistical analysis and sample size calculations

This is not the time or place to discuss the methods behind this, but I will recommend a few excellent articles and books for anyone interested in further reading. Suffice to say, the work is not always trivial, and definitely needs a statistician. Sample size calculations for these types of studies are perhaps a little “unorthodox” and won’t be what you are used to, so make sure the statistician involved has done one before, or has read about them.

Presentation of results

The main issue of risk scores is presentation and ease of their use. There are some very good examples of risk score presentation and use. For example, take a look at the CRASH-2 “calculator” for predicting mortality after head injury. Although not beautiful, it is very simple to get a result out.

However, this remains a difficult area, and important consideration needs to be given to the fact that the model needs to be possible to easily implement. I think, possibly, there is great i-phone app potential here, if a sensible model is developed.

Getting funding

It is very hard to get funding for prognostic studies, and they are often “bolt ons” to other studies. In my opinion there is a good reason for this. It is not enough to tell a patient they are “at risk” of a disease. Something has to change in light of this new found risk. If nothing changes, then it was perhaps unethical to identify the patient when nothing could be offered, and a feeling of “so what?” is left behind. This, I think, is particularly true of the NIHR funders. So, as you write that application, or develop that model, think to yourself: what is going to change with this knowledge?

This is a whistle stop article, and much more detail is needed for anyone wanting to do one of these studies. A good starting point is:

Moons et al. Prognosis and prognostic research: what, why and how? BMJ 2009; 338.

Steyerberg, E. Clinical prediction models: a practical approach to development, validation and updating.

What is success?

I was part of a lively twitter discussion some time ago where the topic of ‘success’ was raised. What is success? From a researchers perspective, that’s a simple question to answer – success is getting the funding to do your study – but from an RDS perspective, the answer is much more complicated.

The discussion was started by Sarah Seaton, a statistician & RDS adviser for RDS East Midlands, who tweeted that she’d had the good news that a study she’d been advising on had been funded by the HTA programme. A group of us offered our congrats and then the Research Development & Innovation at University Hospitals Coventry & Warwickshire NHS Trust tweeted to say they were also waiting to hear about some funding outcomes for projects on which they’d advised. This comments sparked a new conversation about how services which support researchers making funding applications often do not hear about the outcome of these applications.

This is something that we at the RDS SE have discussed time and time again and, fortunately, it is something that is slowly changing. We will now hear officially on a regular basis about which projects have been funding by NIHR. This information can only be a good thing, especially in terms of evaluating and improving the advice we offer.

Many of the researchers I work with are ‘regulars’. Over time, we’ve build up a good relationship and whenever they are working on a project they come to me for advice. They’re great about letting me know the outcomes, something about which I love to hear from both a professional and a personal point of view. When you’ve been involved with a project from the early stages, you’re invested in the outcome even if you’re not actually going to be involved in the project itself. I put a lot of work and great number of hours into the projects on which I advise and it gives me a great sense of satisfaction to hear that one of them has been funded. This is, I am certain, true of every RDS adviser.

That said, it’s equally important to me to hear about the projects that have been rejected, especially when a researcher sends me the reviewers’ reports and the comments from the panel. Going through these helps me learn how to improve the advice I give. It also enables the researcher and I together to work out where to go next. As I’ve said before, many projects are funded upon resubmission, undoubtedly stronger for the extra work the initial rejected prompted.

However, I do sometimes work on projects which come in at the last minute and/or with researchers with whom I have not worked before. In these cases, I often don’t hear about the outcome of their applications. Even though I email to ask about the outcome once I know that the funding decisions have been made, I sometimes don’t hear back. It’s tempting in these scenarios to assume that their application has not been successful, but you can’t know for certain.

Hearing officially about the funding outcomes will have a number of benefits. For a start, it would make evaluating ourselves and improving our services much easier and more effective. This would be especially true if we could see the details of the panels and/or reviewers comments on the applications we supported. It will also help with the cohesiveness of the NIHR as a whole and be a great motivator for advisers who put so much time and effort into proposals they may never be involved with again.

It would also help us with our definition of ‘success’. To link this post back to where I originally started, for an RDS ‘success’ can be tricky to define. It isn’t always a case of getting a researcher to submit an application – this is a crude metric that misses out on the subtleties of what we do.

Often, for us, success is the exact opposite. Rather than getting an application submitted, success is getting a researcher to realize that submitting the application is the wrong decision. For many projects it would be far better to keep working on improving the research plan and wait until the next deadline. With many NIHR programmes, that’s only a 4 month delay. If you rush and submit a less than perfect project, you’ve then got a wait of usually 6 months to hear the outcome. And, if it’s a rejection, with many NIHR programmes there’s a 12-month waiting period before you can resubmit.

So, to sum this all up, I’d like to leave you with a couple of points. First, when you hear about a funding outcome, please do let your RDS (and whoever else helped you with the application) know and, if possible, let them see the feedback the submission received. This way, they can celebrate with you if the application was successful. And, if it wasn’t, then they can learn from the feedback, just as you can, and help you with the decision of where to go from here.

Secondly, do keep in mind that rushing to submit for that funding deadline isn’t always the best thing to do. Sometimes success is waiting in the short-term to enable you to get to the longer-term success of actually running your project.

Frustrating Funding Feedback – is there any other kind?

If you’re a researcher with even the smallest amount of experience, then you’re no stranger to rejection. It goes with the territory and the ups and downs of the application cycle is something to which we all have to adjust.

When it comes to rejection, perhaps the most frustrating thing about it all is the paucity of feedback. Many hundreds of hours collectively go into a single funding application. They are, of course, the work of the PI and collaborators primarily, but also finance people, R&D staff, possibly input from a CTU and, often, research advisers like myself. It is disheartening to learn that, after all this work, a project has been rejected. And it is beyond frustrating to see that the rejection of all this effort boils down to a few short bullet points on the rejection letter.

Of course I understand the sheer volume of applications that funders get and the pressure of the time limits under which they work. And, of course, different funders do offer different levels of feedback. Many programmes, RfPB for example, will send the PI the outcome letter with the ‘bullet points of doom’ and also include the reviews the application received.

The reviewers’ comments can be a bit of a double-edged sword. On the one hand, it can be gratifying to read the positive comments made by the reviewers in the face of an overall rejection by the funder. However, it can also be incredibly frustrating to realize that these positive comments were seemingly over looked by the panel and that the discussion obviously took a very different path. And when the bullet points don’t seem to match up with any of the reviewers comments, then you can be left rather bemused by the whole process.

At this point, I can understand the temptation to throw in the towel. And, for some projects, this probably is the end of the road.

However, this isn’t always the right move. If this was the project’s first rejection, then it is worth looking through all the feedback and identifying where changes could and should be made. I have seen many projects funded on their second or even third attempts. Indeed, many research teams benefit from getting the feedback about their project, however sparse it may seem, and being forced to rethink their original design. Although RfPB do not implement a time limit on resubmissions, other NIHR programmes do. However, this too is to the benefit of the project – in reality, you will need those 12 months to properly re-design the project and re-draft the application into something that is fundable.

Consult with your co-applicants and do get in contact with your local RDS. Even if you didn’t use an RDS in your first application, we would be happy to help with another attempt to secure funding. It is likely that this consultation alone will allow you to address some of the issues raised by the funding panel.

Research that comes from the right place – that is patient-centred, appropriately designed and proposed by a strong research team – is rarely entirely un-fundable.

Don’t let rejection get you down; just regroup, redesign and resubmit.