Why we publish our accessibility defects

Read enough accessibility conformance reports and you notice the genre conventions. “Substantially conforms.” “Minor exceptions apply.” “Remediation is ongoing.” Nobody is lying, exactly, and nobody is telling you what is broken either.

We publish the defect list. Every open issue from the annual audit, with its WCAG success criterion, a severity, the date it was found, and — where it is old — a sentence explaining why it is still there.

What this costs

It costs tenders. Twice in the last three years we have been marked down against a competitor whose report said “substantially conforms” while ours listed nineteen open items, several of them minor. A procurement scoring matrix cannot easily distinguish between fewer defects and fewer disclosed defects, and there is no box on the form for that.

We have decided to keep losing those. Here is the reasoning.

The obligation outlives the contract

An accessibility statement is a legal instrument in most of the jurisdictions our customers operate in. It has to name the parts of the service that are not accessible, and it has to be accurate. If we hand an institution a summary, we have handed them the job of discovering the specifics themselves — usually after a student has already hit the problem.

An institution that can read our open list can write an accurate statement on day one. An institution that cannot has to either take our word for it or run its own audit, and running your own audit of someone else’s software is a strange thing to have to budget for.

Severity, honestly applied

The temptation with a public list is to grade everything as minor. We use three bands and we try to be strict:

• Blocking — a task cannot be completed by a user relying on this assistive technology. There is a workaround only if we name one.
• Serious — the task is completable but significantly harder, slower, or requires knowledge a first-time user would not have.
• Minor — non-conformance with no practical impact we can demonstrate.

We currently have zero blocking items, four serious ones, and the rest minor. The four serious ones are all in the same place: the question-authoring interface, which is keyboard-navigable but has a focus-management problem in nested item editing that we have now failed to fix twice.

That is embarrassing to write and it is the most useful sentence on the page.

What we ask for in return

If you are evaluating any LMS, including this one, ask for the defect list rather than the report. Ask when the audit was done and by whom. Ask what the severity definitions are, because a vendor grading everything as minor has told you something even if the list looks short.

And if a vendor cannot give you a list at all, that is not evidence that there is nothing on it.