Summarize with ChatGPTClaudePerplexity
TL;DR
- AI engines do not reliably issue identical fan-out queries every time, and Profound reported query-string uniqueness of 91% for ChatGPT, 14% for Perplexity, and 47% for Copilot.[1]
- Bee LLM found recurring query phrasings in 20.1% of ChatGPT query occurrences and 84.1% of Gemini occurrences, but these figures do not measure semantic intent stability.[2]
- Changes in answers, citations, or recommendations do not directly prove changes in internal fan-out queries, so each measurement layer should be reported separately.[15]
- Approximately 100 runs per fixed prompt condition provides a practical planning target for roughly ±10-percentage-point precision under stated binomial assumptions; no run count guarantees a permanent fan-out pattern.[8]
- Honest fan-out reporting should publish raw conditions, search counts, normalization rules, per-engine results, uncertainty intervals, and the distinction between literal wording and semantic intent.[2][1]
No. Repeated prompts can produce different fan-out wording, different search counts, and different cited sources, but the amount of variation depends heavily on the engine, measurement layer, and test conditions. In Finseo’s own German-language (DACH) data, a prompt’s ChatGPT fan-out searches shared no identical string on 96.8% of consecutive days, and 92.3% of ChatGPT searches were unique strings. Profound, as an independent second sample, reports query-string uniqueness of 91% for ChatGPT, 14% for Perplexity, and 47% for Copilot; these figures describe unique query strings across executions, not the probability that an entire fan-out changes on every rerun.[1]
The evidence supports four separate questions:
- Does the same literal search wording recur?
- Does the engine issue the same number of searches?
- Does it pursue the same underlying intent?
- Does it cite the same domains or URLs?
Those outcomes should not be collapsed into one “AI search stability” score. A system may change its wording while pursuing the same intent, or preserve one highly recurring query while changing the rest of its retrieval plan.[2]
What does “fan-out query stability” mean?
A fan-out is the set of related searches an AI engine issues while answering a user’s prompt. OpenAI describes ChatGPT Search as rewriting a question into one or more targeted queries and potentially sending additional queries after reviewing initial results.[3]
Fan-out query stability is the degree to which those searches remain consistent when the same prompt is submitted repeatedly under the same conditions. Stability can be measured at several levels:
- Literal stability: whether the exact query strings recur.
- Semantic stability: whether different wordings represent the same intent.
- Count stability: whether each run issues a similar number of searches.
- Source stability: whether the same domains or URLs are cited.
- Answer stability: whether the resulting entities, recommendations, or conclusions recur.
These layers can diverge. A query may be lexically different but semantically equivalent, while identical searches may still return different pages as indexes, rankings, locations, or model backends change.[4]
What do published studies measure about AI visibility in ChatGPT, Perplexity, Gemini, Claude, and Google AI Mode?
The strongest direct evidence comes from studies that captured issued search queries rather than inferring them from visible answers. Other studies measure answer lists, citations, domains, or URLs instead. The distinction matters because a changed answer does not prove that the underlying fan-out changed.[4][2]
| Study | Engines | Sample size | What varied | Finding | Source |
|---|---|---|---|---|---|
| Finseo fan-out study (own data) | ChatGPT, Claude | About 520,000 German-language answers (DACH), 31 Oct to 14 Nov 2026 | Issued query strings | 92.3% of ChatGPT searches were unique strings; on 96.8% of consecutive days a prompt’s searches shared no identical string. ChatGPT ran 1.36 searches per answer, Claude 1.14 when it searched. | Finseo |
| Profound repeated-prompt study | ChatGPT, Perplexity, Copilot | 10,000 prompts over 14 days | Issued query strings | Query-string uniqueness was 91% for ChatGPT, 14% for Perplexity, and 47% for Copilot.[1] | |
| Bee LLM query-fan-out study | ChatGPT, Gemini | 1,595 query-bearing responses from 75 prompts, eight accounts, and 11 projects | Recurrence of normalized auxiliary-query phrasings | Recurring phrasings accounted for 20.1% of ChatGPT query occurrences and 84.1% of Gemini occurrences.[2] | [2] |
| SparkToro/Gumshoe recommendation study | ChatGPT, Claude, Google AI | 2,961 submissions from 600 volunteers; 12 prompts repeated 60–100 times | Brand-list membership, order, and length | ChatGPT and Google AI repeated the same brand list in fewer than 1 in 100 responses; the same ordered list occurred about 1 in 1,000 times across tools.[5] | [5] |
| SE Ranking AI Mode test | Google AI Mode | 5,000 keywords, 15 parses, three runs in five U.S. cities | Domain and URL overlap | Same-city general local queries showed about 35% domain overlap and 18–20% URL overlap; location-specific prompts were more stable.[6] | [5] |
| Self-Consistency preprint | Google deep-research-preview-04-2026 API | One cybersecurity prompt, 10 runs, 45 run pairs | Entity and citation-URL overlap | Median entity Jaccard overlap was 0.12; median citation-URL Jaccard overlap was just over 0.01.[7] |
The studies are not directly poolable. Profound measures query-string uniqueness, Bee LLM measures recurrence of normalized query occurrences, SparkToro measures recommendations, and the other studies measure sources or citations.[2]
How often do ChatGPT, Perplexity, and Gemini repeat searches?
Profound’s results show a large difference between engines: ChatGPT generated unique query strings at a much higher rate than Perplexity, while Copilot fell between them.[1] Perplexity typically repeated the same one or two searches on daily runs, whereas ChatGPT more often reformulated the search plan.[1]
Bee LLM found that 20.1% of ChatGPT query occurrences belonged to recurring phrasings, compared with 84.1% for Gemini.[2] The study recorded 4,739 ChatGPT auxiliary-query occurrences and 1,613 Gemini occurrences, with mean query counts of 4.79 and 2.67 respectively among included responses.[2]
These numbers describe different samples and denominators. Profound’s figures concern query-string uniqueness across executions, while Bee LLM’s recurrence percentages concern query occurrences in responses that contained recorded auxiliary searches.[2]
A complete published distribution of per-prompt search-count variance—such as the standard deviation, interquartile range, or full run-by-run count distribution—was not identified in these studies.[8] Average query counts should therefore not be treated as fixed fan-out sizes.
Do changing brand recommendations and citations mean the AI searches changed?
Not necessarily. A visible answer can change because of retrieval ranking, index freshness, generation randomness, personalization, or source selection even when the internal search queries remain similar.[4]
SparkToro/Gumshoe found that complete recommendation lists were highly unstable, but individual brands could still recur frequently. For one digital-marketing-consultant prompt, Google AI recommended Smartsites in 85 of 95 responses despite the broader lists changing.[5] This shows why list-level instability does not imply that every entity is equally unstable.
The Google AI Mode evidence has a similar limitation. ACME.BOT directly logged ChatGPT queries but reconstructed Google AI Mode and AI Overview searches from cited pages because those Google surfaces did not expose their internal queries.[9] Its results therefore cannot establish Google’s literal fan-out stability.
The citation preprint found a median of 79 distinct citation URLs per report, with a range of 66–101 across repeated runs of one prompt.[7] Different URLs can support similar claims, so citation turnover is evidence of source instability, not a direct measurement of changed fan-out wording.[10][7]
Why can the same prompt produce different searches?
Several mechanisms can alter the effective request or the execution path.
- Query rewriting: ChatGPT may create one or more targeted searches and issue additional searches after reviewing initial results.[3]
- Personalization: OpenAI says saved memories may inform query rewriting, while Google says AI Mode can use previous conversations and places searched or tapped in Search and Maps.[11]
- Location and device: OpenAI documents approximate location from IP, and Google identifies location, language, and device as factors affecting relevance.[3]
- Index freshness: Google continuously revisits known pages and discovers new content, so the available retrieval landscape changes over time.[4]
- Model and backend changes: OpenAI documents backend configuration signals such as
system_fingerprintand describes deterministic sampling as best effort rather than guaranteed.[12] - Sampling and numerical execution: Temperature affects output randomness, while floating-point arithmetic, batching, hardware, and parallel execution can introduce differences even under greedy decoding.[8]
- Conversation state: A follow-up conversation may supply context that is absent from a fresh session, changing the searches generated from the same visible wording.[1][6]
These drivers should not be assigned precise percentages without controlled evidence. In particular, mixture-of-experts routing is a possible architectural consideration, but the reviewed evidence does not establish what proportion of consumer-search variance it causes.[13]
Does temperature zero make fan-out queries deterministic?
No. Lower temperature makes outputs more focused and generally more deterministic, but it does not guarantee identical results in hosted systems.[8]
A controlled Thinking Machines experiment produced 80 unique completions from 1,000 temperature-zero attempts using a Qwen model; the most frequent completion appeared 78 times. With batch-invariant kernels enabled, all 1,000 completions were identical.[8][14] The result demonstrates that serving configuration can matter, but it does not measure the cause of variance in a consumer AI search product.[8][5]
A NeurIPS study found that changes in hardware, batch size, runtime configuration, and floating-point reduction order can affect greedy-decoding results when competing token probabilities are close.[8] Separate hosted-environment research also observed nondeterminism under fixed temperature, top-p, and seed settings, while noting that the infrastructure was uncontrolled.[8]
The practical conclusion is narrower than “AI engines are random.” Reproducibility depends on the model, sampling settings, backend, serving configuration, context, and retrieval system.
What methodology should AI search visibility tracking use for prompts, citations, and conditions?
A meaningful run-to-run comparison should hold as many variables constant as possible and record the remainder.
- Define the unit of analysis. Decide whether the experiment measures exact query strings, semantic query clusters, query counts, domains, URLs, entities, recommendations, or answer text.
- Freeze the prompt. Preserve exact wording, spelling, language, punctuation, and any system or project instructions.
- Use a fixed context. Record account, memory state, conversation history, location, device, browser, and personalization settings.
- Identify the surface. Do not combine ChatGPT Search, Gemini, Google AI Mode, Google AI Overviews, and ordinary web search into one result. Finseo’s own analysis found that Google AI Mode and AI Overviews share only 13.7% of cited URLs on the same query.
- Record model and backend signals. Capture model identifiers, dates, available version information, and API fields such as a system fingerprint where provided.
- Capture the entire search trace. Record every observable query, not only the first query or the final cited pages.
- Separate zero-search runs. A run with no exposed auxiliary search is different from a run with an empty query set; report the no-search rate separately.
- Normalize transparently. Document treatment of capitalization, accents, punctuation, whitespace, spelling variants, stop words, and URL parameters.
- Set the precision target in advance. Decide how wide an uncertainty interval is acceptable before selecting the number of runs.
- Report distributions. Provide counts, ranges, medians, intervals, and per-engine results rather than one pooled average.
IAB recommends repeated observations, documented variation ranges, trend reporting, and uncertainty around AI visibility measurements.[15] It also distinguishes prompt-library breadth from repeated runs: a well-repeated but narrow prompt set can produce precise results about the wrong subject.[15]
How many times should you run a prompt?
There is no published universal threshold at which a fan-out pattern becomes “stable.” Bee LLM required at least five query-bearing responses per prompt-engine combination for part of its analysis, but that was an inclusion rule rather than a validated stability guarantee.[2]
A more defensible approach is to choose a precision target for a clearly defined occurrence rate. If a candidate query appears in x of n eligible runs, estimate its occurrence rate as p=x/n. Under the usual independent-run and constant-probability assumptions, the approximate planning formula is:
[ n=\left\lceil\frac{z^2p(1-p)}{e^2}\right\rceil ]
Using a 95% confidence level, the conservative value p(1-p)=0.25, and the corresponding margin of error produces these calculated planning values:[16][17]
| Nominal margin at 95% confidence | Approximate runs per prompt and condition |
|---|---|
| ±20 percentage points | 25 |
| ±15 percentage points | 43 |
| ±10 percentage points | 97 |
| ±5 percentage points | 385 |
These are planning calculations, not AI-search benchmarks.[17][18] They estimate one proportion, such as whether a particular query appears at least once, rather than the stability of an entire evolving fan-out vocabulary.
For practical research:
- Use 10–20 runs to identify obvious variation and capture problems.
- Use about 100 runs when a roughly ±10-point estimate is sufficient under the stated assumptions.
- Use about 385 runs when a roughly ±5-point estimate is required.
- Repeat the design separately for each engine, surface, location, account condition, and time window.
- Treat smaller samples as exploratory and avoid calling a pattern stable without an interval.
An observation such as “the query appeared in 8 of 10 runs” estimates an occurrence rate of 80%, but its approximate 95% Wilson interval is 49.0%–94.3%. The same 80 of 100 runs gives an interval of approximately 71.1%–86.7%.[5] The point estimate is identical; the larger sample is more precise.
How do you calculate overlap between two fan-out runs?
For two runs, convert the normalized query strings into sets A and B. The Jaccard overlap is:
[ J(A,B)=\frac{|A\cap B|}{|A\cup B|} ]
Illustrative arithmetic only—not a published study result:
- Run A contains:
best hiking boots,waterproof hiking boots, andhiking boot reviews. - Run B contains:
best hiking boots,waterproof trail shoes, andhiking boot reviews. - The intersection contains two queries.
- The union contains four distinct queries.
- Therefore, J(A,B)=2/4=0.50, or 50% overlap.
Jaccard treats each set element as present or absent. If a query occurs twice in one run, ordinary set Jaccard does not give it double weight; use occurrence rates or a multiset measure when frequency matters.[2][5]
Do not calculate Jaccard on unnormalized strings and then describe the result as semantic stability. Exact query strings, intent clusters, domains, URLs, entities, and recommendations are different objects and require separate definitions.[7]
What is Jaccard overlap, and why is it useful for fan-out stability?
Jaccard overlap compares the size of the intersection between two sets with the size of their union.[19] It is useful because it expresses shared fan-out elements relative to all distinct elements observed across both runs, rather than counting matches against only one run.
For fan-out tracking, define the elements before calculating the score. Whole query strings measure literal recurrence; semantic clusters measure recurring intent; domains measure publisher overlap; and URLs measure page-level source overlap. A score calculated on one object cannot be transferred to another.
If both runs contain no elements, the denominator is zero and the score is undefined. Report that comparison as N/A and provide the no-search rate separately instead of silently converting it to zero.[7][6]
How should variability be reported honestly?
A useful report should state exactly what was observed and under which conditions. For example:
“Under the documented account, location, model, and time-window conditions, the query appeared in 68 of 100 runs; the 95% interval was reported alongside the estimate.”
Avoid claims such as “ChatGPT always runs this query,” “AI Mode changes every time,” or “the engine searches five times per prompt” unless the experiment actually supports those statements. Profound’s 91% uniqueness figure does not mean that 91% of ChatGPT’s searches change on each rerun, and Bee LLM’s recurrence measure cannot be treated as a direct measure of semantic-plan stability.[2]
At minimum, publish:
- Prompt text and prompt-library scope.
- Engine and surface.
- Run count and eligible-run count.
- Dates, timestamps, and time window.
- Account, memory, conversation, location, and device conditions.
- Model or backend identifiers when available.
- Capture method and known blind spots.
- Search count per run, including zero-search runs.
- Literal-query normalization rules.
- Semantic-clustering rules, if used.
- URL-level and domain-level source overlap.
- Confidence intervals or other uncertainty estimates.
- Failures, missing traces, and exclusions.
Keep answer variability, citation variability, and fan-out variability in separate columns. The strongest conclusion supported by current evidence is that AI engines often vary their fan-out wording, with major engine-level differences, while complete per-prompt search-count distributions and repeated Google AI Mode internal-query measurements remain underreported.[4]
Frequently asked questions
Are ChatGPT search queries consistent?
Not reliably. Profound reported 91% query-string uniqueness for ChatGPT in its repeated-prompt dataset, while Bee LLM found that 20.1% of ChatGPT query occurrences belonged to recurring phrasings.[2] These are different metrics, so neither should be interpreted as a complete probability that a full fan-out changes.
Does ChatGPT search the same way twice?
It may not. OpenAI says ChatGPT can rewrite a question into one or more targeted queries and issue additional searches after reviewing results.[3] Personalization, location, conversation context, index freshness, sampling, and backend execution can all affect the effective search process.[8]
Why do AI search results change every time?
Results can change because the engine rewrites the prompt differently, retrieves from a changing index, applies personalization or location signals, uses different model or backend configurations, or generates a different answer from similar evidence.[8] A changed answer alone does not prove that the internal fan-out changed.
How many times should I run a prompt?
There is no universally validated number. For a defined occurrence rate, approximately 100 runs supports a nominal ±10-percentage-point planning margin at 95% confidence under stated binomial assumptions, while approximately 385 supports ±5 points.[10] Use fewer runs for exploratory work and report the resulting uncertainty instead of calling the pattern stable.
What is a good stability metric for fan-out queries?
Use Jaccard overlap for clearly defined sets, such as normalized query strings, semantic clusters, domains, or URLs.[7] Pair it with per-query occurrence rates and search-count distributions, because Jaccard alone does not show which elements recur most often or whether one run performed no searches.
Are AI citations more stable than AI fan-out queries?
The available evidence does not support a universal answer because studies measure different layers. Repeated-answer and citation studies show substantial variation, but citation turnover does not directly reveal whether the underlying search queries changed.[8] Report fan-out, citations, domains, URLs, and answer entities as separate outcomes.
Can brand visibility be tracked in Claude answers?
Yes, and Claude has appeared in published measurements. The SparkToro/Gumshoe recommendation study covered ChatGPT, Claude, and Google AI, and it found that the same ordered brand list occurred about 1 in 1,000 times across tools.[5] In Finseo's own German-language (DACH) data, ChatGPT ran 1.36 searches per answer, while Claude ran 1.14 when it searched.
How should Google AI Mode and AI Overviews visibility be tracked?
Track them as separate surfaces. Finseo's own analysis found that Google AI Mode and AI Overviews share only 13.7% of cited URLs on the same query. SE Ranking found that same-city general local queries in AI Mode showed about 35% domain overlap and 18 to 20% URL overlap.[6] Because those Google surfaces do not expose their internal queries, ACME.BOT had to reconstruct them from cited pages.[6][9]
How stable are brand recommendations in AI answers?
Full recommendation lists are highly unstable. ChatGPT and Google AI repeated the same brand list in fewer than 1 in 100 responses.[5][2] Individual brands can still recur often, though: Google AI recommended Smartsites in 85 of 95 responses for one prompt.[5] Report brand recurrence rates rather than relying on a single list snapshot.
Sources
- 1What AI engines actually search for and why ChatGPT never searches the same way twice | Profound — tryprofound.com
- 2Query Fan-Out in ChatGPT and Gemini Across 6,352 Searches | Bee LLM — beellm.ai
- 3Searching the web with ChatGPT | OpenAI Help Center — help.openai.com
- 4Characterizing Web Search in the Age of Generative AI — aclanthology.org
- 5NEW Research: AIs are highly inconsistent when recommending brands or products; marketers should take care when tracking AI visibility - SparkToro — sparktoro.com
- 6How Volatile Are AI Mode Results? 2025 Local Search Test — seranking.com
- 7Self-Consistency of Entity Sets and Citations in a Deep Research System under Repeated Prompting — arxiv.org
- 8Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference — proceedings.neurips.cc
- 9Fan-out query analysis: ChatGPT, AI Overview and AI Mode - ACME.BOT — acme.bot
- 10Citation Drift: Measuring Reference Stability in Multi-Turn LLM Conversations — aclanthology.org
- 11AI Mode in Google Search adds personalization, agentic features — blog.google
- 12How to make your completions outputs reproducible with the new seed parameter — developers.openai.com
- 13Mixture-of-Experts with Expert Choice Routing — papers.neurips.cc
- 14Defeating Nondeterminism in LLM Inference - Thinking Machines Lab — thinkingmachines.ai
- 15Executive Summary — iab.com
- 163.3.3.3. Selecting Sample Sizes — itl.nist.gov
- 177.2.4.1. Confidence intervals — itl.nist.gov
- 18Web search — developers.openai.com
- 19jaccardscore — scikit-learn 1.9.1 documentation — scikit-learn.org

