A vendor benchmark is evidence about a benchmark. Evaluation on your own data, your own population and your own thresholds is the only kind that answers a question anybody will ask you.
Evaluation goes wrong when the test is designed around what is easy to measure rather than around what was promised.
Written down, in terms that can pass or fail. A great many claims dissolve at this stage, which is itself a finding.
Performance on a public benchmark and performance on your customers are different numbers, and the gap is usually widest where it matters most.
A model is not good or bad in the abstract. It is above or below a threshold somebody was willing to put their name against.
Independent evaluation sits in the testing row below, and it is the row most often empty. Choose an obligation to see what it is meant to hold.
Six artefacts. Every obligation on this list draws on the same six, which is why preparing for one of them prepares you for most of the others.
Vendor benchmarks are evidence about a benchmark. Evaluation on your own data, your own population and your own thresholds is evidence about your system.
Four stages, every engagement. Hover a stage to see what happens in it.
Three moves. Two of them are ours, and the one in the middle deliberately is not.
We evaluate the system against what it was claimed to do and against the obligations that apply to you, and we record what the test showed — including where the answer was that no record exists.
Remediation is performed by whoever built or runs the system. We do not take the build work, because taking it would mean the next evaluation is us examining our own hands.
The system is re-examined against the same tests on a schedule, because an evaluation is a statement about a configuration on a date, and configurations move.
Three engagements inside this pillar. Start with the question you can name, or take the whole estate at once.
Governance that produces evidence, not a document describing an intention.
Testing a model or a system against what it was claimed to do.
The record, assembled before somebody asks for it rather than after.
Thirty minutes. Bring the sentence from the vendor’s proposal you would least like to see put to a test.
Book a 30-minute scoping callORVIX · INDEPENDENT AI & TECHNOLOGY ASSURANCE · WE DISCLOSE EVERY COMMERCIAL RELATIONSHIP ON THE PAGE FOR THE SERVICE IT BELONGS TO. WHERE LICENSING IS REQUIRED, DELIVERY IS PERFORMED BY NAMED PARTNERS UNDER THEIR OWN LICENCE.