The Evaluation Scorecard
That Holds Up

An evaluation scorecard has two jobs. The obvious one is picking the right vendor. The less obvious one — and the reason scorecards are built the way they are — is producing a documented, defensible reason for the decision that holds up when the losing bidder asks why, or when someone reopens the question in eighteen months.

Most scorecards fail at the second job, and quite a few fail at the first. Here is how to build one that does both.

Build the scorecard before the proposals arrive

This is the rule that matters most and the one broken most often. Criteria and weights must be fixed and published before the submission deadline — ideally in the RFP itself.

The reason is not only fairness, though it is that. It is that human judgement runs backwards without a fixed frame. A panel that reads the proposals first will, entirely sincerely, build criteria that describe the proposal they liked. Everyone involved will experience this as objective evaluation. It is not, and it is exactly what a challenge will surface.

Fix the weights, publish them, and do not change them after the deadline. If a criterion turns out to be badly framed, note it, apply it consistently anyway, and fix it next time. Adjusting weights mid-evaluation is the single most challengeable thing a panel can do.

Weighting: fewer criteria, more meaning

The most common structural mistake is too many criteria. A scorecard with twenty-five weighted criteria produces totals within two points of each other, because averaging over many dimensions crushes variance. You will have spent forty hours to produce a tie.

Five to eight top-level criteria is the workable range, each with a meaningful weight — no criterion below 5%, because anything smaller cannot change an outcome and only adds scoring labour. Put the detail in sub-questions beneath each criterion rather than in more top-level rows.

The scoring scale must be described, not numbered

"Score each criterion 1–5" produces noise. Without described levels, one evaluator's 4 is another's 2, and the consensus meeting becomes a negotiation between scoring styles rather than a discussion of the proposals.

Describe every level in terms of evidence, so scorers are judging the same thing.

Notice that the difference between 4 and 3 is evidence quality, and between 3 and 2 is a material gap. That framing pushes the panel toward "what did they actually show us" rather than "how did that make me feel", which is both a better decision process and a far more defensible record.

A worked example: scoring the same answer three ways

The scale only works if a panel applies it consistently, and the place consistency breaks is the 3/4 boundary. Here is one requirement, three vendor answers, and the score each earns.

The gap between the second and third answers is the one panels most often collapse, and collapsing it is expensive. Both promise a 30-minute Severity 1 response. Only one has shown it can deliver, disclosed where it failed, and attached a consequence. Score them the same and you have told every bidder that evidence is optional — and you have thrown away the only signal that distinguishes a real capability from a confident sentence.

This is also the most useful thing to run your calibration exercise on. Give the panel these three answers cold, before real scoring, and any disagreement about the 3/4 boundary surfaces in twenty minutes rather than in the consensus meeting three weeks later.

Handling price without letting it dominate

Price is where most scorecards get quietly distorted. Two approaches work, and one common approach does not.

The formula approach. The lowest compliant price scores full marks, and others score proportionally: score = (lowest price ÷ this price) × available marks. Transparent and challenge-resistant. Its weakness is that one unrealistically low bid compresses everyone else's score — which is why you should exclude bids you assess as abnormally low, on documented grounds, rather than letting them warp the scale.

The banded approach. Define price bands in advance against a budget estimate, and score by band. Less sensitive to outliers, but requires you to publish or at least record the bands beforehand.

What does not work is scoring price by judgement — "this feels expensive". It is unrepeatable, it is where preference for a favoured vendor enters unnoticed, and it is the first thing a challenge will probe.

Whichever you use, score on total cost of ownership over the contract term, not on the headline year-one figure. The bid that wins on year one and loses over five years is a common and expensive outcome, and it is entirely a scorecard design failure.

Calibrating the panel

Even with a described scale, scorers vary. Some mark generously, some harshly, some avoid the extremes entirely. Two hours of calibration removes most of that variance.

Run a calibration exercise before real scoring. Take one criterion and one proposal — ideally a mid-field one — and have every panel member score it independently. Compare and discuss the outliers. What you are looking for is not agreement on the score but shared understanding of what evidence a 4 requires. Twenty minutes of this recalibrates a panel for the whole evaluation.

Score independently before meeting. Panel members must not see each other's scores first. Discussion before independent scoring means the most senior or most confident voice sets the anchor and everyone converges on it, producing false consensus.

Require written justification for every score. One or two sentences citing the specific evidence: "Scored 4 — provided three references in comparable organisations with contactable referees and stated overrun on one, section 6.2." This is the single most valuable habit. It disciplines the scoring, it makes the consensus meeting productive, and it is exactly the record you need if the decision is questioned.

Investigate divergence rather than averaging it. When two scorers differ by two or more points, that is information — usually one of them found something the other missed, or they interpreted the requirement differently. Discuss it and reach a reasoned position. Silently averaging a 2 and a 5 into 3.5 throws away the most useful signal the process generated.

The hard case: the numbers pick a vendor nobody wants

It happens, and it is the moment that defines whether your process is real. The scores are in, and the winner is a vendor the panel has reservations about. Meanwhile the vendor everyone liked came second by four points.

The wrong response — common, and dangerous — is to revisit scores until the preferred vendor wins. It is indefensible if challenged, it destroys the panel's trust in the process for every future procurement, and everyone in the room knows what happened.

The legitimate paths are narrower, and all of them are about examining the reasoning rather than adjusting the outcome.

• Check for a scoring error. Recheck the arithmetic and confirm each score matches its written justification. Genuine mistakes happen and correcting them is proper.
• Ask what the reservation actually is. Frequently it is a real risk that your criteria failed to capture — cultural fit, delivery track record with organisations like yours, a concerning reference call. If it is real and material, it belongs in the evaluation, and if the RFP allowed for reference checks or clarifications, use that stage properly rather than as a backdoor.
• Distinguish risk from familiarity. Often the discomfort is simply that the winner is unfamiliar and the runner-up is known. That is not a finding, and acting on it is how organisations end up never changing supplier.
• If the criteria were genuinely wrong, say so. The honest remedy is to document the flaw and, if it is serious enough, cancel and re-run with corrected criteria. That is painful and occasionally correct. It is still far better than an unexplainable award.

The discipline to internalise: if the scorecard produces an answer you dislike, the time to fix the scorecard was before publication. Afterwards, your options are to follow it or to restart it — not to quietly reshape it.

What to document

Assume the decision will be questioned by a losing bidder, an auditor, or a successor in your role. Keep, as a matter of routine:

• The criteria and weights, with the date they were fixed and the version published to bidders.
• Individual scorer sheets with written justifications, retained unaltered.
• The consensus record, including where scores changed and the reasoning.
• Conflict-of-interest declarations from every panel member.
• All clarification questions and the answers issued to all bidders.
• The final ranking and the award rationale in plain language.

The most useful of these is the written justifications. A panel that can point to a specific sentence in a specific proposal for every score has a decision that stands. A panel with only numbers has a decision that rests on the memory of a meeting.

Giving a useful debrief

Worth a paragraph, because it costs little and pays back. Unsuccessful bidders who receive a specific, evidence-based debrief are markedly less likely to challenge — and markedly more likely to bid, seriously, next time. A vague "another bidder offered better value" reads as concealment even when it is not, and it degrades your future field.

Within whatever your rules permit, tell them which criteria they scored below the field average on and what evidence was missing. You are not obliged to reveal other bidders' content, and you do not need to in order to be useful.

The scorecard is a commitment, not a calculation

A scorecard does not make the decision objective — human judgement is doing the work at every scoring row. What it does is make the judgement visible, consistent, and recorded, so the reasoning can be examined rather than assumed.

That is why building it before the proposals arrive matters so much. You are committing to what will count while you still have no idea who benefits. Everything else in the process depends on having made that commitment honestly.