Anyone trying to measure a brand’s visibility inside ChatGPT without opening ChatGPT now has a number to reckon with. A preprint posted on October 11 by researchers at the National University of Singapore and Xi’an Jiaotong University tested whether two stand-ins that can be queried at scale — Google’s organic search results and OpenAI’s own Responses API — reproduce the citations ChatGPT itself gives. Together, they found just 26.2% of the pages ChatGPT cited in tests of its GPT-5.5 Instant setting, and 48.7% of the sites behind those citations.
Search Engine Journal reported the paper on October 18. The figures matter because a growing visibility-tracking industry is built on exactly these stand-ins — and a tracker measuring them may be measuring a neighbouring system rather than the one users see.
How The Researchers Watched ChatGPT Cite Sources
The team recorded the data flow ChatGPT’s web application received during its own writing process, using their own signed-in sessions. That stream showed the searches ChatGPT performed, the pages those searches returned, and the pages ChatGPT finally selected as sources. A page counted as cited if ChatGPT selected it as a source, whether it appeared as its own link or inside a grouped selection.
The test set was 80 questions taken from Search Arena, a public database of real user interactions with search-enabled chatbots — 40 in English and 40 in Chinese, spanning recommendations, fact lookups, information synthesis and analysis. Each question was run 20 times on GPT-5.3 Instant and GPT-5.5 Instant. GPT-5.2 Instant and GPT-5.4 Thinking covered fewer questions because both were decommissioned before data collection finished; most collection ran from March 29 through July 29, 2026.
For the stand-ins, the researchers pulled up to five pages of Google organic results per question through SerpAPI — about 43 results per question on average — and sent 20 queries per question to gpt-5.3-chat-latest and gpt-5.5 through OpenAI’s Responses API with web search enabled, all within three days of the matching ChatGPT runs. The paper is an arXiv preprint, and arXiv does not peer-review the papers it hosts.
The Stand-Ins Matched Sites Far Better Than Pages
Combined, Google’s results and the API found 28.3% of the pages GPT-5.3 Instant cited and 26.2% of the pages GPT-5.5 Instant cited. At site level the same combination did roughly twice as well, finding 50.3% and 48.7% of cited sites respectively, with subdomains treated as separate sites.
Google alone accounted for only 9.4% and 6.4% of cited pages, and 26.1% and 22.8% of cited sites. The API’s full source list covered more — 22.9% and 22.5% of cited pages — and the pages the API cited in its own answers were a closer match still: GPT-5.5 Instant cited 23.3% of the pages the API cited, up from 16.4% for GPT-5.3 Instant. The API is a better proxy than Google rankings, but neither substitutes for the app.
ChatGPT Rewrote Every Question And Changed Sources Between Runs
OpenAI’s help page explains that when ChatGPT search uses an external provider, it usually transforms a question into one or more specific search terms — and in the GPT-5.3 Instant and GPT-5.5 Instant runs where searches were visible, ChatGPT never once searched using the exact wording of the original question. GPT-5.2 Instant did so in just 5.5% of runs.
Language added another layer of drift. For Chinese-language questions, GPT-5.3 Instant searched only in English in 67.0% of visible runs and GPT-5.5 Instant in 69.6% — consistent with Peec AI results reported in February, which found 43% of fan-out queries for non-English prompts ran in English. And the answers were unstable run to run: 44.5% of the distinct pages returned for a given question appeared in only one of the 20 GPT-5.5 Instant runs, and 40.8% for GPT-5.3 Instant.
Why It Matters
ChatGPT citations are the currency of the AI-visibility tracking business, and this paper says the two cheapest ways of auditing them — an API call or a Google ranking check — capture the right neighbourhood but miss most of the exact pages. That is the core problem of generative engine optimisation: the entity being measured, ChatGPT’s citation list, is rewritten, rerun and re-selected on every ask, so a single measurement is a sample, not a census.
The authors’ own monitoring advice follows from that: track visibility at both page and site level, run the same prompt multiple times, account for the fact that ChatGPT rewrites prompts into its own searches, and check whether a tool’s page-level numbers come from the API or from the ChatGPT app itself. The paper tested no commercial tracking tool, so it does not grade any vendor — but it does set the ceiling on what an API-based tool can claim, which is the measurement question at the heart of the AEO versus SEO shift.
Background
ChatGPT’s search citations became a marketing metric after OpenAI added web search to the chatbot, spawning a wave of GEO and AEO tools that promise brands a view of how often AI answers cite them. Every figure in this paper predates GPT-5.6 and GPT-6 — Sistrix data cited by Search Engine Journal showed day-to-day variation in German-language ChatGPT citations jumping to 47%, against a normal 1% to 2%, around the GPT-5.5 change in May — and the authors say monitoring strategies should be re-evaluated whenever the model behind ChatGPT changes.
Source: Search Engine Journal, reporting the researchers’ arXiv preprint posted October 11, 2026
0 Comments