Balanced Scorecard for Public Food Procurement – Part 3 of 3
This is the third of my blog posts commenting on Defra’s “Balanced scorecard for public food procurement”, published in July 2014. The main point to learn; write what is literally true, if not metaphorically. I mention this because the balanced scorecard declares that most of its award questions have been framed to specify desired outcomes (see below). The intention might have been genuine, but what is said to be a desired outcome isn’t. A genuine outcome is about what you desire to achieve, not how.
“The award questions relate to these criteria. In most cases they have been framed to specify a desired outcome, rather than requiring a particular solution. This is to allow for flexibility and innovation by bidders. Procurers use the evaluation guidance to help them rate each response.”
I like using outcomes, any mention of them attracts my attention. That’s why, “framed to specify a desired outcome” piqued my interest, because it’s rare to find outcomes that are genuine, rather than being inputs or outputs labeled as outcomes. I decided to find out if this ‘balanced scorecard’ has genuine outcomes.
What is an outcome?
The Chambers dictionary describes an outcome as a consequence or a result. It’s what the beneficiaries of a contract should experience to satisfy their needs. An outcome should be specific, not abstract. I often use conditions with outcomes to increase specificity and relevance. The more specific an outcome the more certainty potential suppliers have about what the purchaser needs. If you can’t formulate genuine outcomes then you’re better off using specific inputs or outputs.
Why use outcomes?
Outcomes help specify what beneficiaries should experience from the purchase and use of goods or services. They should not include how potential suppliers might achieve them.
Outcomes encourage potential suppliers to use their experience and expertise to propose how they will achieve them. The main assumption being potential suppliers have experience and expertise that purchasers don’t. Therefore, they are more able to propose how the desired outcomes could be achieved. Whereas purchasers should be eminently better equipped to specify what will satisfy the needs of their beneficiaries.
There is a disconnect between the purchaser’s ability to distinguish between, and to define, what and how. Purchasers’ often say they use outcomes to specify when they’re really describing how (inputs) but giving it an outcome label. Why? I’m not sure. But it could be because they don’t understand what a genuine outcome is.
Here are two examples that illustrate what I mean, one from the balanced scorecard and another from household waste collection.
- This is an extract from my third example below, “all food supplied to the UK public sector meets food hygiene requirements”. This is about as close as the balanced scorecard gets to an outcome. Unfortunately it isn’t an outcome because it describes how and not what. In this example what could be, ‘food served did not cause illness because of poor hygiene’, although including ‘poor hygiene’ limits it.
- An example of an outcome (what) from household waste collection is ‘Each resident’s bin is empty once every fortnight’, how, because the supplier empties each resident’s bin once every fortnight.
Are the award questions framed to specify desired outcomes?
Here are three award questions from the balanced scorecard along with my comments about them being framed, or not, to specify desired outcomes. My comments on the characteristics of these three award questions apply to, most, if not all, the rest.
The first award question is about supply chain management.
“Please describe the systems you will have in place for approving your food and drink suppliers, for managing compliance with technical specifications for food and drink, and managing the continuing fulfilment of contract performance clauses.” The typo (fulfilment rather than fulfillment) is the original writer’s.
To recap the balanced scorecard frames most award questions to specify desired outcomes. We’ve just discussed what an outcome is, so we’ll use this to determine if this question is framed to specify a desired outcome. So what is Defra’s desired outcome or outcomes? Could they be descriptions of systems, approval of suppliers, compliance with technical specifications and fulfillment of performance clauses? Are any of these actually outcomes?
There are no reasons I can think of why you would consider any of these to be outcomes. Systems are inputs that help to achieve outcomes, as is the approval of suppliers. However, perhaps we could interpret compliance with technical specifications as an outcome? The technical specification is the mandatory criteria and in this instance Defra have said it isn’t applicable (N/A), this means there isn’t a technical specification. This means the technical specification can’t be an outcome; maybe there’s one in the award criteria? The award criteria that represents Excellent is ….
“The applicant has provided detailed and comprehensive evidence that it has systems in place for the approval of its food and drink suppliers, and for monitoring of metrics required to satisfy technical specifications, and for the continuing fulfilment of contract performance clauses. This will include full details of documented management systems.”
Again there are inputs, but no outcomes. As a bit of a side issue, it is interesting to note the need to “include full details of documented management systems” suggests that the purchaser knows what a good management system is. It is more likely, than not, that most, if not all those evaluating answers, will lack experience and expertise of such systems. Even if some have relevant experience and expertise it’s unlikely to span multiple systems so they can evaluate accurately, consistently and fairly. This could call into question the relevance and authenticity of their evaluation.
And finally there is “fulfilment of contract performance clauses”, could this specify a desired outcome. If “fulfilment of contract performance clauses” is a desired outcome it’s abstract and its status as an outcome relies on performance clauses being outcomes. We can’t say whether they are or aren’t because there’s no distinction, yet, between a technical specification and a contract performance condition (clause).
Now I’ve done the first example to death let’s move on to the second example award question about animal welfare.
“Please state the % of total monetary value of animal derived foods that will be supplied from farm assured sources, where the associated schemes provide assurance that UK legislative, or equivalent standards for animal welfare, have been met or exceeded.
The writer isn’t making it easy to spot desired outcomes. Perhaps “assurance that UK legislative, or equivalent standards for animal welfare, have been met or exceeded” could be an outcome? Especially if, at the same time, you consider Defra’s Mandatory Criteria associated with this award question.
“All food served must be produced in a way that meets UK legislative standards for animal welfare, or equivalent standards.”
It’s still difficult to determine a desired outcome. At this point I was a bit fed up at the distinct lack of outcomes, so I had a go at writing a desired outcome for animal welfare.
When food includes animal products, the animals these products originate from have been well looked after (outcome), to at least UK legislation standards (condition).
Here ‘s another couple of interesting side issues, the clarity of the award question and the use of the term “monetary value”. The award question could be clearer, much clearer, my suggestion is ….
How much will you spend, as a % of the total you propose to spend to supply [the purchaser], on foods made with animal products that originate from animals raised on farms in schemes that assure animal welfare standards meet or exceed UK legislation?
But making the award question clearer doesn’t give us a desired outcome. Finally there’s the term “monetary value”. What’s wrong with using spend?
At last, this is my third example award question and it’s about food safety and hygiene.
“Supply Chain (applicable to caterers and to the direct supply of products) Please describe the systems that your organisation will use to assess risks and manage food safety and food hygiene throughout the supply chain, including how mitigating actions are linked to the outcomes of a systematic risk assessment, and details of any independently audited food safety schemes you will use.”
There are no desired outcomes but, as in my first example, there’s the request for descriptions of systems, which presupposes those evaluating know about systems to assess risks and manage food safety and hygiene.
This question is mainly about inputs. The only hint of an outcome is implied; food must be safe and hygienic. I looked to the Aim (see below) associated with this question for more clarity. There is the hint of an outcome, in bold, although it does need rewording to improve its status as an outcome.
- “To improve food safety, and ensure that consumers and public health is protected, by ensuring that all food supplied to the UK public sector meets food hygiene requirements.
- To encourage a risk-based, proportionate approach to food hygiene and safety throughout the food chain, including transport and storage.
- To incentivise the appropriate adoption of independently audited food safety assurance schemes in the supply chain.”
To change the words in bold to an outcome we should first interpret the intention behind them. Perhaps best to say that the purchaser doesn’t want anyone to become ill because of the food they eat. Looking a little further in the mandatory criteria associated with this example award question I found evidence to support this interpretation, “Food must be safe” that is “it must not be injurious to health or unfit for human consumption”.
This is as close to an outcome as we get. It’s a bit cumbersome and I’m not sure why they thought it wise to include “food must not be injurious to health” and “food must not be unfit for human consumption”. If food is unfit for human consumption then it is likely to be injurious to health, if it’s injurious to health then it’s likely to be unfit for human consumption. Why use both? Could food be fit for human consumption and injurious to health? Possible, I suppose, if some of those who eat the food could suffer an allergic reaction. However, the writer could have been more positive than negative and written ‘food is fit for human consumption’ rather than “food must not be unfit for human consumption”.
Here’s another quick side issue. This award question is a leading question, as it leads potential suppliers to assume their answers must include what the purchaser mentions in the question. This part of the question shows why, “how mitigating actions are linked to the outcomes of a systematic risk assessment”.
The purchaser wants potential suppliers to write in their answers about mitigating actions, systematic risk assessment and the link between outcomes of such an assessment and mitigating actions. As a potential supplier I’d be daft not to mention all this in my answer, even if I don’t actually do it or I use another more effective method.
If you ask a leading question you run the risk of receiving answers that include everything you think relevant and valuable. For some this is disadvantageous (the better potential suppliers), for others it is advantageous and for the rest it has no impact. However, it could homogenise potential suppliers answers. This puts purchasers in an awkward position because they need to know how potential suppliers differ, if they are to evaluate accurately, responsibly, fairly and reliably.
My suggestion for a desired outcome (the what) is …
The food served did not cause illness
Now I’m on a roll we could eliminate the leading element of the award question and link it to this desired outcome by rewording it. The new award question could be ….
How will you make sure the food you serve will not cause those who eat it to be ill?
And if you want to be a bit more positive ….
How will you make sure the food you serve is beneficial to the health of those who eat it?
What potential suppliers can learn
There’s not much to add beyond what I’ve already said in the previous two Blog posts. Here I’ll include some of the more important points.
- Always test what you are told
- Especially for vague seductive expressions that appear positive
- Start as a skeptic until proven otherwise
- Read carefully, very carefully
- Understand what you read, ask if you don’t
- The purchaser is responsible for your understanding
- Ask until you do understand
- Don’t take anything for granted
What purchasers can learn
Just as I mentioned above there’s not much to add beyond what I’ve already said in the previous two Blog posts. Here I’ll include some of the more important points.
- If you contradict promises, it will cause amnesia (of the good stuff)
- It makes potential suppliers more cautious not more optimistic
- They’ll place greater emphasis on uncertainty in your documents
- Uncertainty increases prices, risk premiums are expensive
- Write for potential suppliers
- You are responsible for their understanding
- Be specific and definite
- Be clear about cause and effect
- Potential suppliers aren’t daft, don’t treat them as though they are
- Be realistic and straightforward
- Use your common sense, not management speak
Other comments about the balanced scorecard
There’s one aspect of the balanced scorecard that deserves a positive mention. It’s the use of evaluation scores that aren’t numbers, this moves away from the perceived authority and legitimacy that numbers give to an evaluation score. Unfortunately it does hamstring evaluators’ ability to recognise smaller differences in quality.
The one word I’d use to sum up the balanced scorecard is ‘disappointing’, although I’d temper this with obvious ‘good intentions’. Unfortunately it appears to have been written by purchasers’ for purchasers’, rather than for potential suppliers. It’s as though the purchaser wrote it to garner kudos from fellow purchasers rather than to attract competitive tenders. There’s also the unmistakable whiff of a committee and the need to please all involved, which manifests itself through wordiness, complexity and much unnecessary padding.
It’s a shame because good food bought well will maximise a multitude of positive influences on those who produce, transport, prepare and eat it.