Introduction

This information sets out how we developed the evaluation criteria that were used to assess and score the two possible options for the future Principal Treatment Centre for children’s cancer. These are the options that we are consulting on.

It explains some of the areas where there have been different views and provides detail on the weighting of the criteria.

1. Overview

After we identified two options for the future Principal Treatment Centre for children with cancer living in Brighton and Hove, East Sussex, Kent, Medway, south London and most of Surrey, it was vital to have a clear set of criteria for a careful and rigorous assessment of both options as part of our pre-consultation process. This was to enable us to have a clear understanding of the options we were taking forward for consultation and be able to present information about them for this pre-consultation business case.

This ensured the NHS organisations putting in proposals for the future centre – Guy’s and St Thomas’ NHS Foundation Trust, which runs Evelina London Children’s Hospital, and St George’s University Hospitals NHS Foundation Trust, which runs St George’s Hospital – knew what evidence they had to provide and what specific aspects would be assessed and scored for the pre-consultation options appraisal.

The criteria also provided the basis for the assessment and scoring by expert panels of both options’ proposals for the pre-consultation options appraisal. (Some criteria were assessed and scored using quantitative methods, rather than by the panels.)  

The process we followed drew on the expert knowledge and experience of clinicians, managers, parents, charities, staff and research leads, and the views of children and families – see the next section for detail about the experts involved.

Working with the programme, they supported us to develop criteria which identified:

• the key areas (domains) that are most important for the future service. Four key areas were identified, reflecting the complex range of factors to be considered in making a decision on the future site of this service.
• the specific aspects of the domains we should focus on as ‘sub-criteria’ to help us assess the two options in detail.

They also played an important role in deciding how the sub-criteria should be “weighted” in the scoring depending on their importance. There is more about this in section 5. ‘Weighting’ recognises that, although different factors need to be considered, some may be more important than others.

The content of the national service specification for Principal Treatment Centres underpins much of the criteria. The national service specification had already gone through public consultation and, before that, significant stakeholder engagement to identify what is important for delivering children’s cancer care.

Some specific elements of the evaluation criteria were also included as the result of feedback through our pre-consultation engagement. For instance, engagement work with children and young people in London and the south east, as well as the Stakeholder Group, resulted in a fifth sub-criterion being added to the patient and carer experience domain – ‘support at times of crisis’

2. Who was involved in developing the criteria?

These criteria were developed by the programme through work with:

• a clinical advisory group – who advised on the criteria for assessing and scoring clinical services
• a parent stakeholder group – who advised on engagement work to develop the criteria for assessing and scoring patient and carer experience
• an options appraisal working group – who advised on the criteria for assessing and scoring enabling factors (workforce, capacity, resilience).

A parallel process focused on proposed changes to children’s cancer research. It involved lead researchers and clinicians – who advised on the criteria for assessing and scoring research.

The Clinical Advisory Panel included clinical leaders from Evelina London Children’s Hospital, Great Ormond Street Hospital, King’s College Hospital, St George’s Hospital, The Royal Marsden, NHS England London and NHS England South East.  The group had an independent chair, Professor Sir Terence Stephenson, who is former President of the Royal College of Paediatrics and Child Health, former chair of the Academy of Medical Royal Colleges, former chair of the General Medical Council, and chair of the Health Research Authority.

The Stakeholder Group consisted primarily of parents.  It was chaired by Michelle McLoughlin, CBE, an independent consultant who was previously NHS England’s national specialty advisor and former chief nurse of Birmingham Women’s and Children’s Hospital.

The Options Appraisal Working Group included senior managers and clinical leaders from Evelina London Children’s Hospital, St George’s Hospital, The Royal Marsden, NHS England London and NHS England South East; NHS England Women and Children and Cancer Programme of Care leads; and public health leads.  It was chaired by the medical director for specialised commissioning for London. It was advised by a workforce sub-group.

Sir Terence Stephenson led work with research leaders to find out what they thought were the most important aspects of research to test with the Trusts.

NHS England London drew in additional external support from people experienced in complex decision making, including additional public health support to assist in structuring the evaluation process. It was recognised that the evaluation criteria needed to be able to differentiate (identify meaningful differences) between options which met all the hurdle requirements and therefore had to be measurable. 

3. Review of the criteria 

Before the evaluation criteria were finalised, Guy’s and St Thomas’ and St George’s provided feedback on them. Guy’s and St Thomas’ raised some questions. St George’s asked NHS England to take into account the facts that:

• St George’s has 25 years experience of providing important elements of care for children with cancer (St George’s and The Royal Marsden provide the current Principal Treatment Centre in partnership)
• St George’s has complex, multidisciplinary teams working across a range of services including paediatric surgery, pathology and intensive care, complementing The Royal Marsden’s children’s cancer (oncology) services. These staff are all experienced in managing cancer complications. St George’s provides children’s intensive care for the current Principal Treatment Centre.
• St George’s provides neurosurgery on site (brain, nervous system and spinal surgery). Although this is not on the list of services that a Principal Treatment Centre must have, St George’s argued that having neurosurgery on site should matter more than having other services that the national service specifications requires to be ‘readily available’, because so many cancer patients need such surgery.

To help finalise the criteria, in July 2022 a new independent Clinical Review Group was set up. It was chaired by Professor Sir Terence Stephenson and included clinical directors from children’s cancer and strategic paediatric networks, clinicians and medical directors from inside and outside London, and nursing specialists.

The group was asked for their expert view on the criteria, including points raised on the criteria by St George’s and Evelina London.

They gave serious consideration to these points. They amended two of the sub-criteria for the clinical services key area to reinforce the opportunity for scoring high marks by giving answers drawing on experience of delivering care for children with cancer (the sub-criteria on network effectiveness and transition to teenage and young adult services).   

However, they also considered that, although St George’s Hospital’s experience in children’s cancer care is important, it is not more important than experience in delivering very complex non-cancer children’s services and the organisational and clinical skills that this requires. This was particularly so as neither St George’s Hospital nor Evelina London has experience in the very specialist cancer treatment services for children that The Royal Marsden currently provides and in which the 170+ staff who are eligible to move to the future Principal Treatment Centre are expert.

The Clinical Review Group felt that the changes they made and the consideration they gave provided the right balance. It was their collective view that the most important thing is that the specialist children’s cancer services currently at The Royal Marsden go to a hospital expert in children’s care that can welcome and work with them to provide a Principal Treatment Centre for children  with cancer that will be able to build on the strengths of the existing service and provide best quality care and achieve world class outcomes that are maintained for decades to come.

They recognised the importance of neurosurgery but also that neurosurgery is one of the services that the national service specification says must be readily available if they are not on site. It is not a service that must be on site. As this was the view taken by the national service specification for the whole of England after working with patients, parents and professionals, the Clinical Review Group did not feel there was a reason to take a different view for London. Neurosurgery will stay at King’s College Hospital and St George’s whichever site is chosen for the future Principal Treatment Centre. In its overall decision making, NHS England must consider the impact on other services. St George’s contends the evaluation should also have taken into account the potential impact on its other services if it does not become the future Principal Treatment Centre.

St George’s is specifically concerned about:

• its ability to attract and keep children’s specialist surgical staff and trainee surgeons if children’s cancer surgery moved away, and the impact that could potentially have on the non-cancer surgery it could provide for children.
• the potential impact on its children’s pathology services (which help to diagnose and guide treatment for cancer, as well as many other diseases). Children’s cancer makes up a lot of the pathology team’s work and helps to attract and keep staff. 
• its ability to deliver wider improvements and other benefits for non-cancer patients, including its interventional radiology service, children’s research, and improvements for specific conditions such as sickle cell disease.

St George’s is also concerned about ongoing costs which the Trust might continue to have to pay for a while if the future Principal Treatment Centre were to be at Evelina London and therefore cancer services that St George’s provides for the Principal Treatment Centre moved to Evelina London.

Again, this was given serious thought by the Clinical Review Group. The group felt that, while the impacts would need careful consideration at implementation, they did not need to be included in the evaluation criteria. This was because analysis had found that the scale of changes would be unlikely to have a significant impact on other services such that they should be included in the evaluation criteria at this stage.  Importantly, alongside this, the group also advised it would also be difficult (at that stage) to accurately measure the impact of any changes, including how to reduce them. We followed this advice.

The members of the Clinical Review Group discussed each criterion and made recommendations on revisions; the evaluation criteria were updated to reflect them.  The Clinical Review Group confirmed that, with these revisions, the criteria were suitable, comprehensive and would allow us to differentiate between the options.

4. The criteria

Four domains (key areas) were agreed by the Programme Board for the evaluation, each with their own sub-criteria.  This reflects the various areas stakeholders said were particularly important to consider in evaluating any option:

• clinical
• patient and carer experience
• enabling (non-clinical) factors
• research.

The table below has a description of the sub-criteria for each of the four domains.

Domain	Sub-criteria
Clinical	Interdependencies: ensuring children have access to as many other specialties as possible on the same site   In addition to the mandatory services listed in the Principal Treatment Centre service specification, 13 other services are listed that are not required to be delivered on site but must be ‘readily available.’ With the aim of having as many of these services on site as possible, this sub-criterion therefore factored in these services to ensure that children gain from the greatest density of specialty skill through the co-location of services.
Clinical	Transfers: reducing avoidable transfers of patients to other hospitals for care, particularly where a transfer would have an impact on patient experience and safety  There was particular concern within Professor Sir Mike Richards’ review about the ‘shuttling’ of children between sites. There is a small group of treatment transfers which are potentially avoidable, and where a transfer would adversely impact on patient experience.
Clinical	Network effectiveness: experience of providing leadership for, and working with a network of other hospitals (to provide care as close to home as possible) The Clinical Advisory Panel (CAP) felt it was important for shortlisted providers to demonstrate their experience in leading networks given the centrality of this role for the Principal Treatment Centre in the service specification, and the support this will provide to children’s cancer shared care units at district general hospitals, enabling more children to have care closer to home.
Clinical	Transition: supporting children to make the move to teenage and young adult cancer services when they are ready While the scope of the service reconfiguration relates to children up to the age of 16, transition to teenage and young adult cancer services is an important consideration and is specifically referenced in the service specification.

Domain	Sub-criteria
Patient and carer experience	Quality of facilities: patient environments are an important contributor to overall experience of care, with age-appropriate environments, play facilitation, patient privacy and dignity, space for parents/carers to remain with the child, and an education model for children and young people. [1]    Patient environments are an important contributor to the holistic experience of care. This is set out specifically in the service specification and was also a key issue reported by parents and young people in the Association for Young People’s Health survey report.  [1] Parents along with other members of the ‘Patient and carer experience’ domain panel determined the weighting of the sub-criteria. Members of the panel gave the highest weighting to ‘Quality of facilities’ than to other sub-criteria such as ‘Service accessibility.
Patient and carer experience	Patient navigation to services, including offsite care: patients and families want positive and connected experience of being guided through their treatments in a joined-up way, enabled by technology
Patient and carer experience	Family support during periods of extreme difficulty: the need for support and wrap around care, particularly during periods of difficulty   This sub-criterion was developed by parents on the Stakeholder Group, and work with children and young people.
Patient and carer experience	Engagement: organisations that successfully engage patients and carers are most likely to be successful in delivering a service that meets the needs of users We expect the Principal Treatment Centre to work with local and national charities which support children with cancer, talk to patient groups and panels with direct experiences of services, and involve service users and parents and carers in key decisions.
Patient and carer experience	Service accessibility: minimising the impact of relocation on accessibility by car and public transport, with a focus on those who are less able to choose flexible arrangements   The impact of the move based on travel times looking at car and public transport travel, and the impact on more socially deprived areas. The impact of travel to either Evelina London Children’s Hospital or St George’s Hospital was measured using a standard methodology for car and public transport by levels of deprivation. This was then converted into a ‘score’ for each component and a 50/50 weighting given to car travel and public transport to create a final overall score. After feedback from parents, this weighting was converted to 70/30 in favour of car travel as one of the sensitivity tests on the scores – see [how travel times were assessed and scored].

Domain	Sub-criteria
Enabling	Capacity: sufficient capacity to treat children from a wide geography for a condition that requires speedy access, including for bone marrow transplant. Data on activity delivered by the current Principal Treatment Centre was shared with providers as a guide to the capacity required to accommodate the required level of activity.
Enabling	Resilience: patients who use the services must be able to access care when required, including surgery within reasonable timescales; this was evaluated by assessing Trusts’ business continuity plans against NHS England Emergency Preparedness, Resilience and Response Core Standards.
Enabling	Organisational support for staff: Staff must be supported through this period of change   This sub-criterion looked at current organisational performance based on published workforce statistics – staff survey results, vacancy rates, staff stability and sickness.  Some of the risks involved in staff leaving the service during transition to the new site should be mitigated by moving to an organisation that staff rate highly.
Enabling	Impact on staff: the service change must not have an unnecessary or significantly negative impact on the workforce who deliver the service   This sub-criterion looked at ‘non-pay’ benefits that would be offered to staff compared to those received currently at The Royal Marsden (such as nursery provision, education benefits, staff wellbeing offer), impact on training programmes (professional programmes and continuous professional development (CPD)) and changes to staff travel times. Equivalence or improvements on existing experience was deemed important for attracting and retaining current and future staff. Royal Marsden staff, as stakeholders had asked for this criteria to be included, they wanted to be confident that a future employer would create the same positive environment that exists at the Royal Marsden.

Domain	Sub-criteria
Research	All patients within the Principal Treatment Centre have the same access to clinical trials and research is supported through:   Performance and capability: assessed current research performance and capability, providers’ ambition and future vision for research and innovation   Proven research capability should help mitigate against the risks involved in moving to a new delivery model during the transition period, as well as setting aspirations for the future.
Research	People: research workforce; staff development programmes; income supporting research staffing; research networks and collaboration; previous impact on collaborating to advance international health policy
Research	Place: current capacity and excellence – physical space for research, including infrastructure to support and enhance transferring research teams, capacity for (phase I, II, and III research) trials and tissue studies, ability to link with industry; plans to improve existing provision, and capacity to scale.

The structure of the evaluation criteria (the key domains and sub-criteria) allowed for a review of each option against a combination of different factors which have all been identified as important in thinking about the best site for the future children’s cancer principal treatment centre.

The final evaluation criteria can be found here (PDF).

5. Weighting of the criteria

All the work for this proposed reconfiguration is overseen by a Programme Board which is made up of leaders from the hospitals involved, senior doctors from within NHS England London and NHS England South East, and external experts.

Reflecting the complexity of the decision, the Programme Board agreed separate processes for weighting the criteria and scoring the options’ proposals. A two-level system was accepted by the Programme Board for weighting, giving weights firstly to the domains, and then weighting the sub-criteria within each domain. This meant the more important the domains (and the sub-criteria within the domains) were for the future Principal Treatment Centre, the bigger percentage of the available scores they could get. These weights were applied to scores post-panel evaluation. 

Domain weights

The domain weights were determined by the Programme Board. Board members first individually assigned the weights they felt were most appropriate and shared their rationale with one another, and then assigned weights again in the light of what they had heard from other members of the Board. This two-stage process resulted in the weighting for the four high level domains (clinical services, patient and carer experience, enabling factors, and research) shown below. 

Domain	Clinical services	Patient and carer Experience	Enabling factors	Research
Weight	36%	26%	19%	19%

Sub domain weights

Four expert panels were established to weight and score the criteria within each domain. Each panel comprised 10 people, except for the research panel which had seven independent researchers for the weighting stage of the process (and two more members later). 

Overall, 32 different people were involved in the panels in October and November 2022 (seven people were members of two panels, but no one was on more than two panels). The expert panels who undertook the weighting comprised:

• independent cancer specialists and other leading nurses and doctors from London, the south east and further afield with no involvement in the current Principal Treatment Centre or either of the options
• parents and representatives of children’s cancer charities
• independent researchers
• senior managers and experts in specific fields including emergency preparedness and human resources
• staff from NHS England London and NHS England South East.

The panels were provided with training on the weighting process, and what to consider in weighting proposals. Each panel member individually weighted the different sub-criteria for their domain. Panel members asked for a ‘decision tool’ to be included with their weighting materials and this was provided.  

Weighting the sub-criteria was a two-stage process – first, panel members provided initial weights on each sub criteria with justification for the chosen weight. NHS England London fed back the initial weights anonymously to the panel. The panel members were given an opportunity to update their weight, or confirm their initial weight, before the final score for each sub-criterion was calculated by NHS England London, using the mean (average) of all the scores. The weighting process was undertaken virtually and weights were not discussed between panel members.  

Weighting determined the percentage of the total score each sub-criterion was given when calculating the overall score for the domain. For instance, the panel of parents and charities involved in developing the sub-criteria for the patient and carer experience domain gave highest weighting to the sub-criterion ‘quality of facilities’ for the future Principal Treatment Centre. This meant ‘quality of facilities’ (which contains five different aspects each scored separately) got a bigger percentage of the score available than the other sub-criteria in the patient and carer experience domain. ‘Service accessibility’ (travel) was clearly seen as important but as one of a number of aspects of patient and carer experience that had to be taken into account in the evaluation of the two proposals.

The experts who had weighted the criteria were also invited to assess the proposals. You can read more about this in How the options were assessed and scored. The full set of sub-domain weightings were agreed by the Programme Board and is shown in the diagram below.

Sub-criteria	Interdependent services	Treatment transfers	Network effectiveness and system benefits	Transition to teenage and young adult services
Final weight	35%	27.5%	19.5%	18%

Sub-criteria	Quality of facilities	Patient navigation	Family support during periods of extreme difficulty	Engagement and collaboration	Service accessibility
Final weight	25%	23%	21%	16%	15%

Sub-Criteria	Capacity	Resilience	Organisational support for staff	Impact on staff
Final weight	30.5%	25%	23.5%	21%

Sub-criteria	Performance and capability	People	Place
Final weight	39%	32%	29%