AI vendor red flags

The short answer

None of these prove a vendor is bad. Each one is a place where a good vendor has a specific answer and a weak one changes the subject. That difference is the actual signal.

Watch for the shape of the response, not the content. Teams who have run systems in production answer these quickly and with detail, because they have lived them.

The demo uses their data

Every demo is impressive on data chosen to make it impressive. If they will not run it on a sample of yours, even a small anonymised one, that is a statement about how well it generalises.

The follow up worth asking: what kind of input makes it perform worst. A vendor with production experience answers immediately, because they know. One who does not will tell you it handles everything, which is not a property any system has.

No real answer on evaluation

Ask how they will know it is working and listen for whether the answer contains a method or a reassurance. Monitoring uptime is not evaluating correctness, and the two get conflated deliberately.

What good sounds like: a reference set, a rubric, who wrote it, how often it runs, what happens when it drops. That vocabulary is in LLM evals. Its absence means quality will be assessed by whoever complains loudest.

Accuracy quoted as a single number

Ninety five percent accurate is not a claim, it is a shape of a claim. On what data, measured how, on which task, compared to what baseline, and what does the remaining five percent look like.

The last part matters most. Five percent that is politely unhelpful is fine. Five percent that is confidently wrong on credit decisions is not the same product, and a single number hides which one you are buying.

Vague about integration

Integration is where most of the schedule and most of the risk actually live. A vendor who has done it asks you pointed questions early: which systems, do they have APIs, is there a staging environment, who approves changes, how long does approval take.

A vendor who waves at integration as a later detail is either inexperienced or planning to reprice it once you are committed. Both end the same way.

You cannot meet the engineers

If the people in the room are all commercial, ask when you meet the delivery team. Resistance here predicts the bait and switch that follows: senior team sells, junior team delivers.

You do not need to interview them. You need to confirm they exist, that they will be on your project, and that they were involved in something comparable.

Pricing that does not match the work

Per seat pricing for a system that makes decisions rather than serving users usually means it was built for a different problem. Hourly pricing on a build puts you and them on opposite sides of every delay.

And be careful when discovery is the largest number in the proposal without ending in a production price. A discovery that does not price production is a paid exploration. We broke down what the real cost buckets are in what an AI agent costs.

No path to you owning it

Ask what you keep if you leave in month three. If the answer involves their platform, their environment, or a licence, you are renting a capability rather than building one.

That can be a fine choice, deliberately made. It is a bad choice made accidentally, which is what happens when nobody asks. The layer by layer version of this decision is build vs buy.

Green flags worth paying for

They ask what decision you are trying to improve before describing what they build. They volunteer a case that went badly and what changed. They tell you when your problem does not need AI, which is the strongest signal on this list.

They put a date on when something will be live in front of a real user. They price by outcome. And they are specific about guardrails and evaluation without being asked, because those are the parts you only care about after you have been burned. That is the standard we hold ourselves to when we build production AI, end to end, and you can talk to sales to test it.