National Citizen Service (NCS) 2013 – Evaluation – Main Report – Part 2 of 2
In a civilsociety.co.uk blog post on 14th August 2014 I read about an evaluation of the NCS 2013, which concluded the service had provided good value for money. It was the comment about good value for money that caught my attention. It’s abstract, meaningless and generally used when people can’t or won’t be specific
These extracts explain what the NCS 2013 is, give some details about it and provide an overview of the evaluation report.
National Citizen Service (NCS) is a Government-backed initiative that brings together young people aged 15 to 17 from different backgrounds to help them develop greater confidence, self-awareness and responsibility, with a view to creating a more cohesive, responsible and engaged society.
In total, 31,738 young people took part in NCS programmes in summer 2013, and 7,828 in autumn 2013. The Cabinet Office appointed Ipsos MORI to evaluate the impact and value for money of these programmes.
This report summarises the first stage of the 2013 evaluation. Baseline and follow-up surveys, conducted three months after NCS took place, were undertaken with NCS participants and control groups. Further follow-ups are planned to identify any longer-term impacts there may be.
The report under the heading “Aims of the evaluation” explains that,
The key objectives of this evaluation were:
- To assess the impact of the summer and autumn programmes on four outcome areas: social mixing; transition to adulthood; teamwork, communication and leadership; and community involvement
- To understand whether NCS represents good value for money and improves on the value for money of the 2012 pilots
Here I’m concentrating on the second of these two objectives that is, “whether NCS represents good value for money”. I’ve written about the first in a separate blog post.
Good value for money
What a daft thing to determine. Good relative to what? In this context good is meaningless drivel because there’s no indication of what constitutes good. Value follows a similar path, because to know if you achieve value you first have to know what it is. I consider value to be ….
The subjective judgment beneficiaries make about how well the impact of the benefits they expect, perceive and actually experience satisfy their needs.
This requires a definition of beneficiaries needs. The best we have are some outcomes for the NCS 2013, which require each participant that finishes to be:
- More confident
- More self-aware
- More responsible
I’ve assumed ‘more’ means more than before the NCS 2013. A further outcome, a need that isn’t specific to the participants, is that the collective impact of the participants should result in:
- A more cohesive, responsible and engaged society
In my version of the world I’d have outcomes that are more specific and definite, such that I’d know if the service was successful. That is did the service the beneficiaries experienced satisfy their needs?
This evaluation can’t do that because we don’t know what more means. So it concentrates on monestising the benefits it claims the NCS 2013 has delivered. It converts the subjective value of the benefits delivered by the NCS 2013 into its money equivalent and compares this to:
- The monetised benefits from the NCS 2012
- The £62m cost of the NCS 2013
At the risk of sounding churlish if good is simply better than what happened before or the monetised benefits being more than the cost of achieving them then that’s a cop out. However, for me, it’s even worse than that. I reviewed the contents of Section 7: Value for money and this led to me being especially interested in:
- Misleading presentation
- Calculation mistakes
- Monetising benefits
- 95% confidence level
The paragraph below, from “Key findings”, summarises the monetised benefits from the NCS 2013 and, in my view, misleads readers by maximising the ‘apparent’ success of the NCS 2013.
Overall it is estimated that NCS 2013 delivered benefits (excluding health impacts) of between £68m and £236m in summer, and between £14m and £60m in autumn. Including health impacts, the estimated benefits rise to between £84m and £300m in summer, and between £16m and £78m in autumn.
It misleads because:
- ‘Delivered’ benefits are estimates and ranges, but if they’ve been ‘delivered’ then why aren’t they more certain than estimates and ranges?
- ‘Delivered’ is what happens to pizzas, achieved is what happens to benefits
- 10 days and 60 hours of interaction with 37,266 15 to 17 year olds cannot have already ‘delivered’ between £100m and £378m of benefit (inc. health benefits)
- The monetised benefits are dependent on the lifetime earnings of participants and their life expectancy
- The monetised benefits can’t be ‘delivered’ until participants retire and die, perhaps 60 years hence
The ‘delivered’ aspect and ‘apparent certainty of estimates’ are summarily contradicted by the caveats below.
….. there is some uncertainty driven by the short-term nature of results obtained to date, alongside the long-term nature of some of the impacts involved.
….. these results are based on projections of how individuals might behave in the future
The “short-term nature of results” is the impact measured by the difference in answers from a baseline survey, before the NCS 2013, and a follow up survey around four months after the finish of the NCS. It is this short term impact that is used to calculate results (monetised benefits) that predominately depend on how “individuals might behave in the future”. Future refers to their lifetime earnings and how long they live for, perhaps another 60 years. And then there’s the “highly uncertain” nature of health impacts, “as they are contingent on behaviour change being sustained for the long term.”
What also annoys me is the apparent overly optimistic confidence, seemingly accurate estimates and the huge £278m difference between the most pessimistic and the most optimistic estimates. Is this credible?
The writer lessens attention on this credibility issue and the magnitude of the amounts involved by presenting monetised benefits as an amount for each £1 of expenditure. To bring this home what is your reaction to the ‘as reported’ and ‘corrected’ monetised benefits for each £1 of expenditure for the most pessimistic scenarios.
* I explain what the ‘corrected’ amounts are in ‘Calculation mistakes’ below
Now contrast your reaction to the above with your reaction to the same ‘as reported’ and ‘corrected’ monetised benefits (below) for the most pessimistic scenario, but now shown as totals.
When you consider the amount of monetised benefits for each £1 of expenditure there aren’t any reference points you can rely on to determine if it’s good, bad or indifferent. But the totals are different they clearly show the huge amounts of money involved and suddenly you have reference points which you can use to make relevant comparisons.
I found two mistakes. Correcting the first mistake has a significant negative impact on the ‘as reported’ monetised benefits ascribed to “Leadership skills”. Here are the calculations with the ‘as reported’ and ‘corrected’ totals.
There is nothing wrong with the calculations it’s just whoever did the maths got it wrong. I’ve shown the impact of the corrections in more detail in the tables below.
Unfortunately correcting this mistake renders this statement invalid, as the most pessimistic scenarios now have monetised benefits of £39.9m (exc. health) and £57.9m (inc. health) but still cost £62m.
Nonetheless, it should also be noted that even under the most pessimistic scenarios, the scheme is estimated to have delivered greater social benefits than the costs involved.
The second mistake is in the calculation of the monetised benefits from “Health impacts” (smoking) for the autumn programme. I’ve highlighted it in bold below.
Summer: 30,045 × (0.014 or 0.061) × 1.29 × £20,000 = £10.9m to £47.3m
Autumn: 7,221 × (0.010 or 0.071) × 0.32 × £20,000 = £1.9m to £13.2m
This appears to be a copy and paste accident. The previous calculation for “Health impacts” (alcohol) is the same except that it uses a life expectancy increase of 0.32 years not the 1.29 years that should be used for “Health impacts” (smoking). Whilst careless, this doesn’t change the totals from the calculation. However, it does diminish the credibility of the overall evaluation. It caused me to suspect the worst and search for other mistakes.
95% confidence level
A 95% level of confidence means the impacts and differences, calculated for the 4,401 participants who completed the follow up survey, could be wrong by plus or minus 5%.
Throughout this report, only impacts and differences that are statistically significant at the 95% level of confidence are commented on.
This table shows the monetised benefits ascribed to the most pessimistic and optimistic scenarios with the 95% level of confidence applied. However, keep in mind that the impacts used to calculate the monetised benefits came from 4,401 participants but the monetised benefits are calculated as if they apply to all 37,266 participants that finished the NCS 2013.
The most pessimistic scenarios in the above table do not achieve sufficient monetised benefits to cover the cost of the NCS 2013. The next table (below) shows this is a slightly different way, as the amount of monetised benefit for each participant. To compare the monetised benefits against something I’ve included the cost for each of the 39,566 participants that started the NCS 2013, although only 37,266 finished.
Why can’t a simple description of the benefits suffice? Why do they need the embellishment of being translated into an amount of money?
One answer is that those who need to continually top up their political and managerial moats like to trumpet the success of an initiative by showing that the benefits clearly outweigh the cost. They appear to think that the best way to do this is to have costs and benefits in the same currency because:
- It makes it easy for them to trumpet success
- The translation of subjective benefits into pounds can be easily manipulated beyond the bounds of reasonableness whilst appearing to retain a respectful veneer of reasonableness
- They don’t think others are capable of making judgments about services when costs and benefits are in different currencies
The paragraph below shows we have H M Treasury to blame for the method used by Ipsos MORI to monetise benefits and the impetus to use it.
This section summarises the assessment of value for money associated with NCS 2013. This analysis has been undertaken in line with the principles of the H M Treasury Green Book, and seeks to monetise (as far as possible) the resource costs and benefits associated with the scheme. Full details of the analysis are set out in the separately-published Technical Report.
Monetising benefits appears to ignite the competitive juices to:
- Maximise the amount of money ascribed to the impact of a benefit
- Maximise the number of benefits to which an amount of money can be ascribed
In this evaluation report, and many others, monetising benefits also appears to engender the use of a multitude of overly precise and apparently accurate assumptions and estimates, such as:
- Not drinking 6 units of alcohol on a single occasion over a month (cause) will increase the life expectancy of participants of 0.32 years (effect)
- Not smoking cigarettes for one week (cause) increases participants’ quality-adjusted life expectancy by 1.29 years (effect)
The overriding assumption of the NCS 2013 is the 10 days and 60 hours of activities in the summer programme (30 hours in the autumn) will have a positive impact on (all 37,266 who finished not just the 4,401 that completed a follow up survey) participants:
- Earnings over their lifetime
- Life expectancy
Perhaps the main aim of monetising benefits is to create a veneer of certainty and accuracy from assumptions that are far from certain as alluded to in the extract below.
….. there is some uncertainty driven by the short-term nature of results obtained to date, alongside the long-term nature of some of the impacts involved.
….. these results are based on projections of how individuals might behave in the future ….
Other issues for this specific instance include:
- Assumptions being made in isolation about the next 50 or 60 years are far from certain, for example, living longer also means you’re able to volunteer more than someone who doesn’t live as long, vice versa
- The amount of monetised benefits is largely irrelevant as no one will track whether or not they are realised over the next 60 years
- If someone does monitor the realisation of benefits can they do so with sufficient accuracy to validate the original estimates?
- There is a credibility issue with a massive £306,300,000 (exc. health) and £375,000,000 (inc. health) difference between the most pessimistic and optimistic scenarios
- The heavy use of statistics infers an undeserved confidence, accuracy and legitimacy on the monetary amounts ascribed to the impact of the benefits
Did the evaluation of the NCS 2013 achieve its second objective? To understand:
- Whether the NCS 2013 represents good value for money
- If NCS 2013 improves on the value for money of the 2012 pilots
The answer Ipsos MORI gave, to the first of the above bullet points, is …..
… it is not possible to provide a conclusive assessment of the value for money associated with NCS
It appears the answer to the second bullet point is a likely yes, although there is the fundamental underlying difficulty of the value for money concept.
The aforementioned results are not directly comparable to 2012. Reusing the 2012 methodology, it is estimated that NCS delivered between £2.39 and £4.46 per £1 of expenditure in 2013, compared to between £1.50 and £2.80 in 2012.
This evaluation was never going to be able to “understand whether the NCS 2013 represents good value for money” because:
- Good is a relative term, so you should consider and compare what else you could achieve for the £62m cost of the NCS 2013, for example improvements to hospitals
- The method used to determine ‘value’ is too precise and smothered in statistics and has assumptions and estimates spanning participants lifetimes, possibly more than 60 years hence
- The impact determined from 4,401 participants follow up surveys should not be extrapolated across all 37,266 participants that finished the NCS 2013
- It is the wrong question to ask
I’ve discussed at reasonable length the difficulties I think there are with the method used to evaluate the NCS 2013. That leaves what I think should have been evaluated.
The NCS came about because someone somewhere (politicians perhaps) decided there were beneficiaries with needs to satisfy. This requires accurate descriptions, in definite and specific terms, of the outcomes that would satisfy beneficiaries’ needs. The best I could find was this extract.
…… to help them develop greater confidence, self-awareness and responsibility, with a view to creating a more cohesive, responsible and engaged society.
I commented on these desired outcomes at the beginning, saying they were not as definite or specific as they should be, for example, what is greater confidence and how much more would be considered a success. If this isn’t known then how can anyone know if the NCS 2013 has been successful?
Relying so heavily on results that won’t be fully realised for up to 60 years dampens, considerably, the usefulness and credibility of the evaluation.It appears that even the most pessimistic scenarios for the NCS 2013 have not delivered greater social benefits than the costs involved.
Here are three more points that I’m uncomfortable with.
- The scheme has not delivered (pizzas are delivered not benefits, which are achieved) anything to date, other than a good time for participants and the benefit from some volunteering that was an integral part of the programmes
- The translation of potential long term subjective social benefits into an amount of money to compare against the cost of the scheme is, at best, a guess or perhaps an informed estimate, it depends on your point of view
- Whether the scheme will actually achieve more in benefits than the costs involved will only be known for certain in 60 years, who’ll be watching then?
For me it would have been eminently more sensible to stick with subjective descriptions of likely benefits. For example, participants are more likely to continue with further education. We know this is a good thing for most and we can quite reasonably assume that if they do continue it’s likely they will earn more over their lifetime than if they don’t. I’m happier using that explanation of benefits rather than benefits from higher educational attainment could be £17.7m to £138.7m. It’s easier and significantly more practical to track whether or not participants continue with further education, more so than the complex matter of tracking what they earn over their lifetime.
And very finally, before publishing this evaluation report I would have asked someone, not intimately involved, to critique it, to actively work at finding fault. This would have made it infinitely stronger.