How to Choose an AI Brand Monitoring Tool

The AI visibility category filled up fast, and the products vary enormously in rigour behind similar-looking dashboards. Two tools can report very different numbers for the same brand in the same week, and both can be defensible — or one can be close to meaningless.

These are the questions that separate them. We build in this category, so read this with that in mind; the questions below are ones we think any buyer should ask of any vendor, including us.

TL;DR

Ask every vendor:

How many models, and are they the ones my buyers use?
How many times is each question run?
Are prompts blind, or is my brand named in them?
Can I see the raw responses behind the number?
Is the question set mine, or generic to my industry?
How is "mentioned" defined?
Do you capture cited sources?
Do you distinguish positive from negative mentions?
How volatile is the metric run to run?
What happens after the diagnosis?
Can I export my data?

The questions that matter most

1. Model coverage

Ask: which models, and how often is the roster reviewed?

Single-model tracking is a partial picture, because buyer usage is split. Coverage should include the assistants your buyers actually use — which you should verify against your own referral data rather than accepting a vendor's list.

Beware coverage claims that count variants of the same underlying model as separate systems to inflate a number.

2. Sample size per question

Ask: how many times do you run each question?

This is the question that most cleanly separates rigorous tools from superficial ones.

Responses are non-deterministic. Asking once produces one draw from a distribution. A tool that runs each question once and reports a precise percentage is reporting noise with false precision. Repeated runs are what make a number stable enough to trend.

If a vendor cannot answer this, that is informative.

3. Prompt bias

Ask: do your prompts contain my brand name?

If they do, your brand appears in the responses because it was in the question. That inflates the score and measures nothing about discovery.

A serious tool separates unaided measurement (blind prompts, real discovery) from aided measurement (brand named, reputation and characterisation). If a vendor reports one blended number, ask which it is. If the answer is unclear, the number is not trustworthy.

4. Transparency of raw data

Ask: can I read the actual responses behind my score?

You should be able to click a number and see the responses that produced it. Without that, you cannot verify the metric, debug a surprising result, or learn anything about why you were omitted.

Every vendor's exact weighting will be proprietary — ours included, and that is reasonable. But proprietary weighting is different from an unauditable black box. You should always be able to see the underlying responses even if you cannot see the formula.

5. Question set construction

Ask: is my question set specific to me, or a generic industry template?

Generic sets produce generic results. Your buyers ask questions shaped by your segment, geography, and price point. The set should be built for you, editable by you, and stable over time so your trend stays valid.

6. Mention definition

Ask: what counts as a mention?

Being listed sixth in a list of ten is not equivalent to being the single recommendation. A tool that scores both identically is throwing away the most important distinction in the data. Ask how prominence is handled.

7. Source capture

Ask: do you record which sources the model cited?

This is where the actionable intelligence lives. Knowing you appear in 20% of answers is a status report. Knowing which three properties drive inclusion in your category is a plan.

8. Sentiment and accuracy

Ask: do you distinguish how I am mentioned?

Being named as a cautionary example counts as a mention on a naive counter. Being described inaccurately — wrong category, wrong pricing, confused with another company — is actively harmful. Both should be visible.

9. Volatility

Ask: what is the typical run-to-run variation for a stable brand?

An honest vendor will give you a figure. A tool reporting scores to a decimal place with variance wider than the decimal is selling false precision. You need to know what movement is real.

10. What happens after the diagnosis

Ask: does this tell me what to do, and can it help me do it?

Many tools stop at the dashboard. That leaves the hard part — deciding what to change and getting it shipped — entirely with you. Ask whether the tool produces specific recommendations, and whether it can carry them into your CMS.

11. Data portability

Ask: can I export everything?

Your historical trend is the most valuable thing you accumulate. If you cannot export it, switching costs are artificially high. Ask specifically about raw responses, not just aggregates.

Warning signs

No answer on sample size. The most reliable indicator of a superficial product.
Blended aided and unaided scores. Flattering by construction.
Industry benchmarks presented as authoritative. Category concentration makes cross-industry comparison close to meaningless.
No access to raw responses. Unverifiable numbers.
Coverage claims that count model variants separately. Inflation.
Guaranteed results. Nobody controls model outputs. Anyone promising a specific score is overselling.

Running a fair evaluation

Give every vendor the same brand, the same competitors, and the same questions. Run for at least three weeks — one reading tells you nothing about stability.

Then compare: do the numbers move together? Where they disagree, can each vendor explain why from the raw data? Which one taught you something you did not already know?

That last question is usually the deciding one.

Frequently asked questions

Can I do this in-house? Yes for a baseline audit. It becomes impractical as a continuous discipline once you multiply questions by assistants by repetitions.

Why do two tools report different numbers? Different question sets, sample sizes, model coverage, and mention definitions. Compare trends, not absolute values across vendors.

Is a single-model tool worth it? Only if your buyers overwhelmingly use that one assistant. Verify with your own referral data before assuming.

How much should this cost? It varies widely. Judge on rigour and on what happens after the diagnosis, not on dashboard polish.

Published by the SIQA Editorial Team.