How to vet an AI development agency before you hire one

To vet an AI development agency, ask for proof of each specific capability your project needs rather than a general portfolio: a code sample you or an adviser can read, and a customer reference for that capability you can contact yourself. Then check how they plan to test the AI's answers, how they price the work, and who will own what they build.

1. Ask for proof of each capability, not a portfolio

An agency that built a polished app has not necessarily built a search system over documents, an agent that takes actions, or an evaluation of a model. List the capabilities your project needs and ask for evidence of each one. A short code sample shows more than a case-study page.

2. Contact a reference for each claim

Ask for a customer reference for each capability, with a company email address, website and phone number. Email them, and call if they do not reply. Ask what was delivered, whether it is still in use, and what went wrong along the way.

3. Ask how they will know the AI works

Good agencies raise evaluation early: a set of real examples, how answers are scored, and what level of quality is acceptable. If the plan covers screens and integrations but not testing the AI's output, expect surprises after launch.

4. Notice whether they asked about your data

The first questions should be about your data: what exists, how clean it is, where it lives and what you are allowed to do with it. A quote given before those questions is a guess.

5. Get the price phase by phase

Ask for a quote broken into phases, for example discovery, a first working version, hardening and launch. It shows where the cost and the risk sit, and lets you compare agencies against the same requirements.

6. Tie payment to accepted work

Agree acceptance criteria for each milestone before its work starts, and pay milestone by milestone. Escrow, where the money is held until you accept the work, protects both sides: the agency knows the money is there, and you pay only for work that meets the criteria.

7. Put ownership and data handling in writing

The statement of work should say who owns the code, the prompts, the configuration and anything trained or tuned on your data, how your data is stored and deleted, and which outside AI services will see it.

8. Meet the people who will do the work

Ask who will be on the project and what each person has built before. The team on the sales call is not always the team on the project.

Red flags

• References that cannot be reached, or only for unrelated work.
• Confident promises about accuracy before anyone has seen your data.
• Large up-front payments not tied to a deliverable.
• No plan for maintenance, although models, outside services and data all change after launch.

How abracodabra vets agencies

Every capability an agency claims needs a code sample abracodabra checks is real, plus a customer reference with a company email address, website and phone number. abracodabra emails the reference and a person calls if there is no reply. Only confirmed claims count as verified when projects are matched. Clients see which capabilities each matched agency proved before they ask up to three of them for quotes. Read where to find people to build your AI idea, or start a project.