AI Vendor Due Diligence

Do You Know Which Model Your AI Vendor Is Running?

Your SaaS vendor probably knows, but your contract probably does not. The gap matters more than most small and midsize businesses realize. AI features now slide into ordinary software the way tracking pixels used to: quietly, everywhere. A help desk adds “summarize this ticket.” Your CRM offers email drafting, and your document platform launches semantic search. Employees are suddenly feeding customer conversations, support history, contracts, and sales notes into an AI system whose underlying model, hosting location, retention rules, and subcontractors may be a total mystery. That is AI vendor due diligence.

A recent Federal Register incident illustrates the point. On September 16, the site’s advanced search interface showed options tied to Qwen3-0.6B, a compact open-weight model from Alibaba’s Qwen team. The model can be downloaded and deployed without a procurement process, and Qwen’s published documentation supports local deployment through common serving tools. Its base-model license is Apache 2.0. In other words: a developer can treat it like a useful building block, not a strategic supplier decision.

Meanwhile, on September 8, the FBI, NSA, and CISA issued a joint advisory alleging that Alibaba and five other China-based AI companies had conducted large-scale knowledge-distillation campaigns against U.S. frontier models. The advisory is an allegation, not a judicial ruling, but the timing made the Federal Register discovery a useful wake-up call about how quickly model provenance can become relevant.

Set the geopolitics aside for a minute. The business lesson is simpler and more uncomfortable:
The AI model touching your data may have changed months after you signed the vendor contract.

The Model Is Not on the Order Form

Most companies don’t buy “a model.” They buy a ticketing system, sales tool, accounting package, HR platform, or search product. Then the vendor bolts on AI because apparently no software roadmap may now survive without a sparkle emoji and a “copilot” tab. The model sits several layers below the purchase decision.

You bought a help desk platform. The platform added a ticket-summary feature. That feature might call:

• The vendor’s own fine-tuned model
• A frontier-model API
• A cloud provider’s managed AI service
• An open-weight model on the vendor’s infrastructure
• A third-party AI routing layer that selects models dynamically

Each path carries different implications for data handling, geography, subprocessors, retention, and contractual recourse. The awkward part is that model selection often behaves more like an engineering decision than a vendor-management decision. Engineers compare latency, benchmark scores, cost per million tokens, tool-use reliability, and whether the thing breaks when someone types an em dash. Then they ship, which is rational from an engineering perspective. It’s less appealing when the AI feature ingests your customer data.

For SMBs, risk multiplies because AI adoption often arrives through tools you already own. Nobody launches a formal evaluation – the feature just appears. Employees use  it because it’s convenient. Finance renews the software twelve months later. Congrats: your data has entered a black box with a pretty UI.

The Cheap-Model Switcheroo

The bigger risks are what model your vendor uses today and what they use after the next cloud bill arrives. AI inference costs hit vendor margins hard. A company may launch an AI feature on a premium API to ship it, then swap in a cheaper model after usage grows. The button and marketing pages stay the same. Your team keeps pasting customer information into the same text box. But underneath, the data path may have changed:

The Federal Register example is not proof that every open model creates a security problem. In fact, a locally hosted model used for retrieval can have a very different risk profile from a hosted chatbot sending prompts to an external API. That distinction is exactly why hand-wavy answers are not good enough. You cannot assess the risk if nobody will tell you what is really running.

Three Questions for Every Renewal

Get answers to these questions in writing for every vendor whose AI features touch customer, employee, financial, health, legal, or proprietary business data. Do it at renewal when vendors suddenly rediscover the classic art of replying to emails.

1. Which model powers this feature, and who publishes it?

Ask for the model family and version (not “our proprietary AI).” “Proprietary AI” often means a wrapper around another company’s model, possibly with a retrieval layer, prompts, fine-tuning, and a bold landing page. That architecture may be perfectly reasonable. You still deserve to know what is inside. A useful answer looks like:

“Ticket summarization uses Model X, version Y, hosted by Provider Z. Our retrieval layer uses Model A for embeddings.”

A useless answer looks like:

“We use best-in-class AI to enhance customer outcomes.”

2. Where does inference run?

Ask whether processing happens:

• On the vendor’s own infrastructure
• In a named cloud-provider environment
• Through a model-provider API
• Through an AI gateway, aggregator, or subcontractor
• In a particular region or set of regions

You want a location and architecture (not “globally distributed infrastructure).” A real answer names regions, providers, and the path your data takes.

Jurisdiction, contractual protections, incident response, privacy obligations, and government-access rules are not identical everywhere. “Somewhere in North America” is not a data-governance strategy.

3. What happens to our inputs and outputs?

Get definitive answers on:

• Whether prompts or outputs are retained
• The retention period
• Whether data is used for training, evaluation, or quality review
• Whether the policy differs by plan or feature
• The current subprocessor list
• Whether admins can disable AI features or block sensitive fields

You want plain language: your data is not used to train shared models, retention is limited & documented. There is no surprise opt-out buried in a settings page created by a scavenger hunt enthusiast.

A Non-Answer Is Indeed an Answer

If a vendor cannot identify the model, hosting arrangement, retention policy, and subprocessors behind an AI feature, record that as a risk. There are only a few likely explanations:

• They do not know, which means their AI governance is weak.
• They know but will not say, which is no better.
• Their stack changes so frequently that the answer is unstable, which means you need notice requirements.

The practical move is to ask for written notification before the vendor makes a material change to the model, inference provider, hosting geography, or data-use terms for an AI feature that processes your data. It doesn’t need to be a 19-page appendix written in legal fog. Start with a narrow clause:

Vendor will provide written notice before materially changing the AI model provider, inference location, or data-processing practices for AI features that process Customer Data.

That clause triggers when the change happens, not nine months later when a security questionnaire gets dusted off for renewal. If the vendor will not agree, reduce the blast radius. Decide which categories of data never go into their AI feature: customer support transcripts, credentials, payroll information, health data, confidential deal terms, source code, or regulated records. Document the rule internally, train the users, and review at every renewal.

The Contract Cycle Is Too Slow

Open-weight models evolve quickly, API pricing changes, and new providers appear. Vendors have every incentive to optimize cost and performance beneath a stable product interface. That translates to AI governance cannot be a one-time checkbox at onboarding. Make it a lightweight operating habit:

• List the three SaaS tools that handle your most sensitive data.
• Identify every AI feature your team uses, not just the ones you remember buying.
• Ask the three questions above.
• Save the responses with the contract and security documentation.
• Ask for notice before material model or data-path changes.
• Restrict sensitive inputs where answers are vague.

We are not suggesting you ban AI, expect a perfect supply chain, or turn your SMB into a tiny compliance department with a complicated spreadsheet. We recommend not confusing a polished interface with transparency. Your vendor may be running a model you would happily approve. Great. Get that in writing. It may be running something nobody on the account team can explain. Also useful to know: before your customers, regulator, board, or largest client asks the same question with less patience.