Answer Volatility

Three mechanisms drive volatility. First, generation is probabilistic: models sample tokens under a temperature setting, so two identical requests can legitimately produce different brand lists. Second, providers ship model updates and silent revisions that shift behavior overnight. Third, engines using retrieval-augmented generation depend on live search results that change as indexes refresh, so the source material itself moves between runs.

Volatility varies by prompt type. Questions with one dominant answer are stable; open recommendation prompts ("best tools for X") are the most volatile, because many brands plausibly fit and sampling decides who makes the cut on any given run, precisely the prompts marketers care about most.

Asking ChatGPT once whether it mentions your brand is a coin flip presented as a measurement. A founder who checks one morning and sees the brand listed may conclude all is well, while the brand actually appears in a minority of runs; the reverse panic is equally common. Single observations cannot distinguish a real visibility change from sampling noise, and decisions made on them, celebrating, escalating, reallocating budget, are decisions made on noise.

The statistical fix is the same as any noisy measurement: repeated sampling. Running the same prompt set daily and aggregating over a week turns a coin flip into a mention rate with a confidence you can act on, and makes genuine shifts, like a model update dropping you from a category, distinguishable from variance.

Treat volatility as a property to measure, not just noise to remove. A brand mentioned in 90% of runs has durable presence; one at 40% is on the bubble and most exposed to model updates. Track stability per prompt and per provider, report weekly aggregates rather than daily points, and annotate known model releases so real regime changes are not mistaken for drift. This sampling discipline is why dedicated platforms run prompts on an automated daily schedule: Geonimo samples every tracked prompt across providers each day through prompt tracking, so its visibility scores reflect distributions rather than lucky or unlucky single runs.