How to choose an AI partner

The short answer

Almost every evaluation tests how well a vendor presents. What you actually need to know is whether they have taken something like your problem into production and stayed there while it broke.

The questions below are designed to be uncomfortable for anyone who has not. They are not technical questions, which is the point. You do not need to evaluate their architecture. You need to evaluate whether the thing will be live, owned by your team, and still working in a year.

The three kinds of vendor you will meet

The consultancy. Strong on framing, discovery and stakeholder management. Delivers strategy, a roadmap and a recommendation. Frequently does not build, or builds through a separate delivery arm you have not met. We compared that shape directly in a consultancy.

The software house. Builds what you specify, competently and on contract. The gap is that specifying an AI system correctly requires knowing which decision is worth automating, and that is the part you were hoping to buy.

The product vendor. Sells a platform. Fast to start, and the fit question is whether your problem is genuinely the shape their product assumes. Often it is, and then buying is the right call.

None of these is wrong. They are wrong when you need one and hire another, which is what most bad AI engagements actually are.

Ask about production, not about the pitch

Ask what they have running in production right now, with real users, that they did not build for themselves. Ask how long it has been live. Ask what broke in the first month, because everything breaks in the first month and a vendor who says nothing did is either new or not telling you the truth.

Then ask what they do when a model provider ships an update that changes behaviour. This one is diagnostic. A team that has operated a system through that has a specific, slightly weary answer. A team that has not will talk about how they would approach it.

The portfolio question that reveals everything

Ask for a case where the result was worse than expected, and what they did. The answer separates people who have shipped from people who have sold.

What you want to hear is a specific failure, an actual diagnosis, and a change they made. What you do not want is a smooth transition into a success story, or an answer implying it has never happened. Everyone who builds AI in production has been surprised by it. Claiming otherwise is a stronger negative signal than the failure would have been.

Who actually does the work

Ask to meet the engineers who will be on your project. Not the practice lead, not the solutions architect who wrote the proposal, the people writing the code.

The pattern to watch for is a senior team in the sales process and a different, cheaper team in delivery. It is common, it is rarely disclosed, and it is the single biggest predictor of a disappointing engagement.

A follow up that works well: ask which of these people will still be on the project in month four.

Code and ownership

The code should be yours, in your repositories, from the first commit. Not delivered at the end, not held in their environment, not licensed back to you.

Ask specifically what happens if you end the engagement in month three. If the honest answer is that you keep a working system your team can operate, good. If it is that you keep an artifact nobody internally understands, that is a dependency, not a deliverable.

And ask what they use that is proprietary. Using their own tooling is fine. Building your core decision logic inside something only they can maintain is not.

How they handle being wrong

Ask how they will know the system is working, and expect a specific answer about evaluation rather than a sentence about monitoring. If nobody can describe how correctness gets measured, nobody is measuring it. The shape of a real answer is in LLM evals and observability.

Ask what happens when the system produces a wrong answer for a customer. Who finds out, how fast, and what stops it repeating. A partner who has been through this describes a process. One who has not describes an intention. The engineering behind that process is what we called guardrails.

Pricing structure is a signal

Hourly billing rewards slowness and puts you and the vendor on opposite sides of every estimate. Per use case pricing puts you on the same side, because they only finish when it is live.

Be careful with anything priced per seat for a system that makes decisions rather than serves users, and with proposals where the largest number is discovery. A discovery that does not end with a production price and a plan is a paid exploration. The full breakdown is in what an AI agent costs.

A one page scorecard

Score each vendor from one to five on: something comparable live in production today, you met the actual engineers, code is yours from day one, they can describe how correctness is measured, they answered the failure question specifically, pricing is per outcome rather than per hour, and there is a date when something is live in front of a real user.

Anything below a four on the first, third or seventh is worth stopping over. Those three predict the outcome more than anything in the proposal. This is also how we structure our own engagements, and why our pricing is public rather than discovered in negotiation. You can talk to sales or read how we build production AI, end to end.