Published September 30, 2026
Why does ChatGPT give a different answer every time I ask who to hire?
Because it's built to. ChatGPT, Claude, and Google's AI write each answer fresh, so the businesses they name, and the order they name them in, shift from one ask to the next. In a study published in January 2026, the chance of getting the same list of brand recommendations twice was under 1 in 100 (Fishkin, 2026). The useful question isn't "where do I rank in ChatGPT." It's "how often am I named when people ask."
What this means
If you asked an AI assistant who to hire, saw your business named, then asked again and watched it disappear, nothing necessarily broke. Large language models build answers by predicting likely next words, with some randomness built in. Many assistants also run live web searches before answering. Different searches pull different pages, and different pages lead to different names.
Google says this plainly about its own products: AI Overviews and AI Mode "may use different models and techniques, so the set of responses and links they show will vary" (Google Search Central, 2025). Both can also use what Google calls query fan-out, running several related searches behind the scenes to build one answer.
So a single screenshot of a single answer is one roll of the dice. It's real, but it isn't a verdict.
Photo: "Dice" by Mak Ting Him, CC BY 2.0
Who this applies to
Any business owner who has checked their AI visibility by typing a question once, and anyone who has been shown a "rank" in ChatGPT by a vendor. It matters most in crowded categories. The more credible options an AI has to choose from, the more its answers scatter.
How we'd evaluate it
We don't treat one answer as a result. We run the same customer-style question several times, on more than one engine, using a few phrasings a real person might type, and record every business named each time. What we look at is the share of answers that include you (named in 7 of 10 runs is a very different position from 2 of 10), who else keeps showing up, and whether the details the AI gives about you are accurate. Position within the list gets little weight, for reasons the research below makes clear.
Available options
- Re-run your own test. Free. Ask the same question 5 to 10 times in fresh chats and tally who's named. Crude, but far better than one try.
- Structured, dated testing. A fixed set of prompts, run repeatedly across engines on a schedule, so a change from one month to the next means something.
- Work on the inputs, not the output. You can't control the roll, but you can influence what the AI has to choose from: consistent business details across your website, Google Business Profile, and directories; specific, recent reviews; and clear pages about what you do and where.
Benefits, limitations, and tradeoffs
Measuring how often you're named, instead of where, gives a more honest picture and is harder to misread. The tradeoff is effort. Repeated testing takes time, and even a well-built sample is an estimate. No amount of testing makes AI answers stable, and results can differ by account, location, and chat history. Frequency tells you roughly how likely you are to be named. It doesn't tell you why, and it doesn't promise the next answer.
What we know
The clearest public data comes from Rand Fishkin of SparkToro and Patrick O'Donnell of Gumshoe.ai. In November and December 2025, about 600 volunteers ran 12 prompts through ChatGPT, Claude, and Google's AI a combined 2,961 times (Fishkin, 2026). What they found:
- Under 1 in 100: the chance that ChatGPT or Google's AI returned the same list of brands in any two responses.
- About 1 in 1,000: the chance of the same list in the same order.
- 2 to 10+: even the number of names in an answer varied.
Search Engine Land's summary put it bluntly: tracking a position in AI answers is mostly noise, while how often a brand appears across many runs held up better (Goodwin, 2026). In tight categories with few options, such as Los Angeles Volvo dealerships in the study, answers clustered around a small group of names. In broad categories, they scattered.
Two caveats. Gumshoe sells AI tracking software, a possible conflict of interest the author disclosed himself. And the study covered three tools over a specific window, so newer models may behave differently. The direction of the finding matches Google's own documentation, though: answers vary by design.
Next steps
Pick the two or three questions your customers are most likely to ask an AI assistant. Run each one 5 to 10 times in fresh sessions on ChatGPT and Google, and write down every business named each time. If you show up in most answers, focus on keeping your information accurate. If you show up in few or none, that's worth diagnosing. Our 5-minute AI visibility test and our guide to tracking AI visibility over time walk through both.
Orlando considerations
Orlando has a lot of businesses competing in the same categories, from law firms and contractors to dental practices and home services, often within a few miles of each other. By the study's logic, more credible options means more variation in who gets named on any given ask. That makes a single test less reliable here, and repeated testing more useful, than in a small town with two choices.
Frequently asked questions
If ChatGPT named my business once, does that mean I'm in ChatGPT?
It means you're in the pool it draws from for that question, which is a good sign. It doesn't mean you'll be named the next time someone asks. How often you're named across repeated asks is the better measure.
Can I make ChatGPT give the same answer every time?
No. Nobody can, including AnswerFoundry. These systems are built to generate a fresh answer each time. What you can influence is how likely you are to be named, by making accurate, consistent, well-corroborated information about your business easy to find.
Is an AI ranking report worth paying for?
Be careful with any report that gives you a single rank or position inside an AI answer. Published research shows the order changes almost every time. Ask how many runs, prompts, and engines sit behind any number you're shown.
How many times should I run my own test?
There's no settled number. The SparkToro researchers suggested 60 to 100 runs for a reliable read on a single prompt. For a quick owner check, 5 to 10 runs per question on two engines will show whether you're usually named, sometimes named, or rarely named.
References
Fishkin, R. (2026, January 27). NEW research: AIs are highly inconsistent when recommending brands or products; marketers should take care when tracking AI visibility. SparkToro. https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/
Goodwin, D. (2026, January 28). AI recommendation lists repeat less than 1% of the time: Study. Search Engine Land. https://searchengineland.com/ai-recommendation-lists-rarely-repeat-study-468076
Google Search Central. (2025, December 10). AI features and your website. Google for Developers. https://developers.google.com/search/docs/appearance/ai-features
Last updated: September 30, 2026
If you ran the test a few times and your name came up rarely, or not at all, that's the gap an AI visibility audit is built to map, with a prioritized list of what to fix first.