Vendor evaluation that actually filters

Three ways to see it

1. 1

   A serious vendor evaluation rests on five legs. Technical scoring: does the system work, on our data, in our network, with our latency, in our languages. Financial position: is the vendor solvent for the contract term, can we see audited accounts. References: who has used this product at our scale, what did they really say off the record. Pilots: can we run a paid four to six week pilot on a non-production slice before committing. Inclusion mandates: does the vendor meet the female participation, PWD accessibility, and local capacity requirements we wrote into the RFP.

2. 2

   Way one to score technical capability: insist on a benchmark on your data. Generic benchmarks like MMLU, GLUE, or vendor white papers tell you almost nothing about how a system will behave on a Pakistani complaint corpus mixed with Urdu and Punjabi. Build a 200-sample evaluation set drawn from your own records. Ask each vendor to run their system on it and produce results in a comparable format. Score them on accuracy, latency, and total cost.

3. 3

   Way two to score local capacity and inclusion: write quantitative thresholds, not aspirations. 30 percent female engineers on the delivery team is a threshold. 'We support diversity' is an aspiration. PWD accessibility WCAG 2.2 AA conformance is a threshold. 'Accessible UI' is an aspiration. Pakistani vendors will rise to clear thresholds. Foreign vendors will negotiate around vague ones. Always write the threshold.

The why-tree

Try this with Claude

AI-edge prompt to try: 'Acting as an off-record reference checker, give me ten sharp questions to ask a customer of a major AI vendor that will surface honest signal beyond marketing. Order them from softest to hardest.' Use the list in your next reference call.