The Reference Call That Reveals Everything: What to Ask a Vendor’s Existing Clients

The Call That Tells You More Than the RFP

Vendor RFP responses are polished marketing documents. Every vendor describes their AI as explainable, their methodology as two-way, and their quality as best-in-class. The reference call is where those claims get tested against operational reality, but only if you ask the right questions.

Most reference calls follow a predictable script: “How was the implementation? Are you satisfied with the relationship? Would you recommend them?” These questions generate polite, positive responses that don’t differentiate vendors. The questions that produce useful intelligence are specific, metric-driven, and focused on the operational realities that RFP responses don’t cover.

The Questions That Reveal the Truth

Ask the reference: “What is your internal error rate on codes this vendor produced, based on your own MEAT validation audits?” If the reference hasn’t independently validated vendor output against MEAT criteria, they don’t know whether the codes are defensible. If they have validated and the rate is above 20%, the vendor’s quality claims don’t match the operational output.

Ask: “What is the deletion rate the vendor produces across your chart review volume?” If the reference reports zero or near-zero deletions, the vendor operates add-only regardless of what the RFP response claimed. If the reference doesn’t track this metric, the program isn’t measuring what the OIG identified as a critical compliance indicator.

Ask: “When you needed to produce an evidence trail for a specific submitted code during an audit response, how long did it take and what did the package look like?” This tests whether the vendor’s output is genuinely audit-ready or requires rework. If the reference needed weeks to assemble evidence from vendor deliverables, the output isn’t audit-ready despite what the vendor claims.

Ask: “Has the vendor’s AI recommendation ever caused a problem you didn’t expect?” This open-ended question surfaces issues the reference wouldn’t volunteer unprompted: categories where the AI consistently overrecommends, documentation types the system handles poorly, or edge cases where explainability breaks down.

Reading Between the Answers

The most informative reference calls are the ones where the reference pauses before answering. A reference who immediately praises the vendor is delivering a rehearsed endorsement. A reference who thinks carefully about specific metrics is telling you what they actually know from operational experience.

Pay attention to what the reference doesn’t say. If they describe the vendor’s coding volume enthusiastically but can’t speak to defensibility metrics, the relationship is measured on volume. If they describe smooth operations but haven’t tested audit readiness, the relationship hasn’t been stress-tested. What’s absent from the conversation is as informative as what’s present.

The Call That Protects the Contract

Plans evaluating risk adjustment vendors should conduct at least three reference calls per finalist, using the metric-specific questions above rather than generic satisfaction questions. The 30 minutes invested in each call produces more reliable intelligence about the vendor’s actual operational quality than the 30 pages of their RFP response. The vendor controls the RFP. The reference controls the truth.