[Original research]
One AI answer is a sample, not a score.
We asked the same buyer questions several times through one search-and-answer pipeline and recorded whether each brand was named, and where. Here is what moved, with the method and every limit stated.
- Domains measured
- 3
- Question–domain pairs
- 6
- Answer runs
- 17
- Domains where position moved
- 33%
An early, small sample from real Spotlit scans. Read it as a direction, not a benchmark.
[01]Findings/ tied to the data
What we found
Three findings, each tied to the data behind it.
- 1
Same question, same pipeline, different placement.
The brand's position moved between runs for 2 of 6 question–domain pairs, and at 33% of the domains measured.
EvidenceSpotlit scan database — 17 runs across 3 domains, October 2026. Position spread computed from min/max position across 2–3-run repeat sets, same pipeline.
- 2
Being named varied between runs too.
1 of 6 pairs named the brand on some runs but not others. Average mention rate across all measured pairs: 42%.
EvidenceMean of mention_rate across 6 question–domain pairs.
- 3
One run is a sample, not a measurement.
Without repeat runs, you cannot distinguish signal from noise.
EvidenceStatistical inference — a sample of 1 has no variance estimate. This is not a finding; it is a mathematical fact.
[02]Data/ distributions
How the answers moved
How mention rates and position spreads fall across every measured question–domain pair.
How often the brand was named
Share of repeat runs that mentioned the brand, grouped.
- 0%3 pairs
- 1–20%0 pairs
- 21–40%0 pairs
- 41–60%1 pairs
- 61–80%0 pairs
- 81–100%2 pairs
n = 6 question–domain pairs
How far its position moved
Gap between best and worst position across repeat runs.
- No spread4 pairs
- 1 position0 pairs
- 2 positions0 pairs
- 3+ positions2 pairs
n = 6 question–domain pairs
[03]Takeaways/ recommendations, not findings
What to do with this
If you report AI visibility to clients, this is how we would use it. These are our recommendations, not findings.
Report a rate, not a yes or no.
“Named in 2 of 3 runs” tells a client far more than “named”. A single run can't show the difference.
Confirm a move before you report it.
A position change seen once may be noise. Re-run the same question before calling it a win or a loss.
Keep the questions frozen.
Compare the same wording over time. Change the question and you've changed the test.
[04]Method/ same questions, repeated
How we measured
Same questions, repeated through the same pipeline, recorded on every run.
- 1
Freeze the questions
Each domain gets a fixed prompt set: the same questions, worded identically, locked on first scan.
- 2
Repeat each one
Each question runs 3–5 times through the same pipeline: Tavily search, then a Gemini answer.
- 3
Record every run
Whether the brand was mentioned, and at what position in the answer.
- 4
Measure the spread
Mention rate and position spread, for every question.
- Mention rate
- What fraction of runs mentioned the brand. Mentioned 3 out of 5 times = 60%.
- Position spread
- The min–max range of where the brand appeared. Position 2 on one run and 5 on another = a spread of 3.
The data on this page is aggregated across all domains in our scan database that have repeat-run data.
[05]Limits/ read before quoting
Material limitations
What this data can and can't support. Read these before quoting any number above.
Sample size. This data comes from our scan database, not a controlled study. The domains are self-selected (they ran a Spotlit scan), not randomly sampled from the internet. The distribution may not represent all domains or all categories.
Pipeline-specific. Our pipeline uses Tavily web search, then an AI model answers from those results (the model provider can differ between runs). A different search provider or LLM would produce different variance characteristics. We are measuring variance in our pipeline, not in “AI” generically.
Repeat count. We run 3–5 repeats per question. Higher repeat counts would produce tighter confidence intervals. The variance we report is itself an estimate.
Temporal. All data is from scans run through the current version of the pipeline. AI model behaviour changes over time; this snapshot may not hold in six months.
No causal claims. We record what we measured. We do not claim that any specific factor causes the observed variance.
Data generated October 21, 2026 · Spotlit by ReachFlow
[→] Free scan
See how steady your own AI answers are.
Every free scan re-asks key buyer questions three times and shows whether the answer held, next to every page the answers were built from. One domain, about 2 minutes, free account.
- Who AI names for your buying questions
- Every page each answer cited
- The pages to get listed on first
Want to look first? See how it works